Hey Mike, we are interested a lot in KSQL as well. I wonder how do you see KSQL playing role in NiFi setup? From data provenance perspective I imaging data flow administrator would like to see a processor somewhere on the flow that says 'run this KSQL script' that is not necessarily connected with specific Kafka topic (through readkafka processor) but rather disconnected from anything else and represented as a standalone processor box on the flow. The objective of having such a processor is again to make sure that entire flow is managed in a single place - nifi flow.
What do you think? On 30 April 2018 at 18:26, Kevin Verhoeven <[email protected]> wrote: > Very cool, thanks Mike. > > > > Kevin > > > > *From:* Mike Thomsen <[email protected]> > *Sent:* Monday, April 30, 2018 6:25 AM > *To:* [email protected] > *Subject:* KSQL + NiFi > > > > KSQL is pretty new, so a lot of Kafka users probably haven't heard about > it yet. TL;DR it is "SQL for Kafka." Confluent has put out a lot of blog > posts and videos showing off the basic capabilities, and here's a > quickstart guide: > > > > https://docs.confluent.io/current/ksql/docs/tutorials/ > basics-docker.html#ksql-quickstart-docker > > > > You can do this to create a general stream of user events: > > > > CREATE STREAM messages (username varchar, msg varchar) WITH > (kafka_topic='input_messages', value_format='JSON'); > > > > and then if you want to spy on bad actor John Smith, you could tell KSQL > to inspect input_messages and route his messages to a new topic like this: > > > > create stream from_john_smith with (kafka_topic='from_john_smith', > value_format='JSON') as select username, msg from messages where username = > 'john.smith'; > > > > That's just the example I played around with over the weekend. There's a > lot more available from KSQL like joins, using topics as lookup tables, > etc. IMO there's a lot of potential there to have KSQL work w/ Kafka to do > some serious heavy lifting to sort things out before they even hit NiFi. > > > > Nothing needed to be added to NiFi to make it work out of the box for me > since Kafka Streams are just a high level abstraction on top of Kafka. > > > > Mike > -- Uladzimir Palkhouski Senior Solution Architect Skype: uladzimir.palkhouski Email: [email protected] <[email protected]> Mobile: +41 79 6553559 LinkedIn: https://de.linkedin.com/in/uladzimirpalkhouski
