Very cool, thanks Mike. Kevin
From: Mike Thomsen <[email protected]> Sent: Monday, April 30, 2018 6:25 AM To: [email protected] Subject: KSQL + NiFi KSQL is pretty new, so a lot of Kafka users probably haven't heard about it yet. TL;DR it is "SQL for Kafka." Confluent has put out a lot of blog posts and videos showing off the basic capabilities, and here's a quickstart guide: https://docs.confluent.io/current/ksql/docs/tutorials/basics-docker.html#ksql-quickstart-docker You can do this to create a general stream of user events: CREATE STREAM messages (username varchar, msg varchar) WITH (kafka_topic='input_messages', value_format='JSON'); and then if you want to spy on bad actor John Smith, you could tell KSQL to inspect input_messages and route his messages to a new topic like this: create stream from_john_smith with (kafka_topic='from_john_smith', value_format='JSON') as select username, msg from messages where username = 'john.smith'; That's just the example I played around with over the weekend. There's a lot more available from KSQL like joins, using topics as lookup tables, etc. IMO there's a lot of potential there to have KSQL work w/ Kafka to do some serious heavy lifting to sort things out before they even hit NiFi. Nothing needed to be added to NiFi to make it work out of the box for me since Kafka Streams are just a high level abstraction on top of Kafka. Mike
