I don't think you would even need that because it appears that KSQL is controllable through REST. With that said, I don't think KSQL should be seen as something that NiFi controls in most cases, but as something that controls Kafka on behalf of NiFi.
On Mon, Apr 30, 2018 at 5:34 PM Uladzimir Palkhouski < [email protected]> wrote: > Hey Mike, > > we are interested a lot in KSQL as well. I wonder how do you see KSQL > playing role in NiFi setup? From data provenance perspective I imaging data > flow administrator would like to see a processor somewhere on the flow that > says 'run this KSQL script' that is not necessarily connected with specific > Kafka topic (through readkafka processor) but rather disconnected from > anything else and represented as a standalone processor box on the flow. > The objective of having such a processor is again to make sure that entire > flow is managed in a single place - nifi flow. > > What do you think? > > On 30 April 2018 at 18:26, Kevin Verhoeven <[email protected]> > wrote: > >> Very cool, thanks Mike. >> >> >> >> Kevin >> >> >> >> *From:* Mike Thomsen <[email protected]> >> *Sent:* Monday, April 30, 2018 6:25 AM >> *To:* [email protected] >> *Subject:* KSQL + NiFi >> >> >> >> KSQL is pretty new, so a lot of Kafka users probably haven't heard about >> it yet. TL;DR it is "SQL for Kafka." Confluent has put out a lot of blog >> posts and videos showing off the basic capabilities, and here's a >> quickstart guide: >> >> >> >> >> https://docs.confluent.io/current/ksql/docs/tutorials/basics-docker.html#ksql-quickstart-docker >> >> >> >> You can do this to create a general stream of user events: >> >> >> >> CREATE STREAM messages (username varchar, msg varchar) WITH >> (kafka_topic='input_messages', value_format='JSON'); >> >> >> >> and then if you want to spy on bad actor John Smith, you could tell KSQL >> to inspect input_messages and route his messages to a new topic like this: >> >> >> >> create stream from_john_smith with (kafka_topic='from_john_smith', >> value_format='JSON') as select username, msg from messages where username = >> 'john.smith'; >> >> >> >> That's just the example I played around with over the weekend. There's a >> lot more available from KSQL like joins, using topics as lookup tables, >> etc. IMO there's a lot of potential there to have KSQL work w/ Kafka to do >> some serious heavy lifting to sort things out before they even hit NiFi. >> >> >> >> Nothing needed to be added to NiFi to make it work out of the box for me >> since Kafka Streams are just a high level abstraction on top of Kafka. >> >> >> >> Mike >> > > > > -- > > Uladzimir Palkhouski > > Senior Solution Architect > > Skype: uladzimir.palkhouski > > Email: [email protected] <[email protected]> > > Mobile: +41 79 6553559 <+41%2079%20655%2035%2059> > > LinkedIn: https://de.linkedin.com/in/uladzimirpalkhouski >
