Re: Creating a Kafka Stream connector for PLC4X?
Hi Niclas, Thanks for your explanation ... helps a lot :-) I wasn't suggesting that we immediately jump on this. Much more I thought this would be a great thing for someone involved with Kafka or at least a lot of Kafka knowledge to jump in. After all we're trying to get as many new people on board as possible. Giving as many suggestions for options to get involved should help ... Perhaps it would be great to have a page on the website with areas we think could be interesting ... sort of a project-local "help-wanted". So if someone has a look and sees that there is some non-byte-counting way to get involved. Maybe he is encouraged to do so. Chris Am 24.03.18, 02:32 schrieb "hedh...@gmail.com im Auftrag von Niclas Hedhman" : Kafka Connect is the ability to run a "source" and/or "sink" of streaming data on the Kafka instance itself. It defines the APIs so people can include the component (and its configuration) from the command-line. Part of the Kafka Connect component are Key and Value serialization, and in Kafka Connect itself, one can apply additional transformations of those messages. For PLC4X, I would imagine that the Source Connector would be configurable with a long list of data points to be scanned, possibly with some conditions, priorities and other algorithms to decide what/when to be scanned, at what interval and which topic is the destination. Of course PLC4X configuration of protocols would also be needed. The Sink Connector would read from N topics, and the message would contain what values to be written to where. It should be enough to do this on-demand. I have not figured out how the Source Connector will distributed the work load in the cluster. One would assume that there is some mechanism in place, otherwise the ETL (Extract Load Transform) use case would not work, and ETL seems to be a major case for Kafka Connect. My conclusion; I think this is a great idea, and that we should aim for implementing. However, I think it is more important to have more protocols in place first, so that the PLC4X abstraction is tuned for disparate types of systems, before attempting to apply another abstraction onto PLC4X API. And for that, I think we need to increase the number of hands (my own included) to help out, which requires that the people with the vision and understanding of the details, need to work more on documentation and less on exotic integration. I think it is the most urgent issue in this project. Cheers Niclas On Thu, Mar 22, 2018 at 5:58 PM, Niclas Hedhman wrote: > > Chris, > I came across a presentation by coincidence that mentioned "Kafka > Connect", so I will research this over the weekend and see what it is, and > how it fits with PLC4X. The presenter only mentioned that a "Kafka Connect" > component is enable by configuration (in Kafka was implied but not sure) > rather than programming. > > More next week. > Niclas > > On Wed, Mar 21, 2018 at 5:19 PM, Niclas Hedhman > wrote: > >> >> Alright... So to repeat (and I think you get this much)... Kafka is in >> principle very simple; Put any byte array into a topic in one end, and >> readers can get that byte array from that topic somewhere else. Just a >> simple message queue, at an abstract level. And the Kafka client libraries >> for doing this is dead easy to use, producer.send() and consumer.poll() >> respectively. >> >> So, I suspect that you are talking about some higher level than base >> Kafka, possibly masquerading the Kafka transport as some other transport >> abstraction or a component implementing some protocol, such as MQTT [1], >> that behind the end point feeds Kafka. But I could be completely off-track >> and some really clever ideas are at play here, that I don't realize yet. >> So, yes, please invite people to discuss further. >> >> Niclas >> >> [1] MQTT feeding Kafka is something I will implement myself in the not >> too distant future >> >> On Wed, Mar 21, 2018 at 4:50 PM, Christofer Dutz < >> christofer.d...@c-ware.de> wrote: >> >>> Hi Niclas, >>> >>> have to admit I don't know in detail either ... it was part of a talk a >>> colleague of mine held at a Kafka meetup in our office. Also I'm not into >>> Kafka half as deep as you ... from a user point of view it looked as if >>> such a Kafka Connect adapter would be a good idea. I'm trying to convince >>> that colleague to contribute ... perhaps he could register here and shed >>> some light on the topic ... I'll continue the convincing (even if another >>> codecentric participant wouldn't increase company diversity) >>> >>> Chris >>> >>> >>> Am 13.03.18, 16:41 schrieb "hedh
Re: Creating a Kafka Stream connector for PLC4X?
Kafka Connect is the ability to run a "source" and/or "sink" of streaming data on the Kafka instance itself. It defines the APIs so people can include the component (and its configuration) from the command-line. Part of the Kafka Connect component are Key and Value serialization, and in Kafka Connect itself, one can apply additional transformations of those messages. For PLC4X, I would imagine that the Source Connector would be configurable with a long list of data points to be scanned, possibly with some conditions, priorities and other algorithms to decide what/when to be scanned, at what interval and which topic is the destination. Of course PLC4X configuration of protocols would also be needed. The Sink Connector would read from N topics, and the message would contain what values to be written to where. It should be enough to do this on-demand. I have not figured out how the Source Connector will distributed the work load in the cluster. One would assume that there is some mechanism in place, otherwise the ETL (Extract Load Transform) use case would not work, and ETL seems to be a major case for Kafka Connect. My conclusion; I think this is a great idea, and that we should aim for implementing. However, I think it is more important to have more protocols in place first, so that the PLC4X abstraction is tuned for disparate types of systems, before attempting to apply another abstraction onto PLC4X API. And for that, I think we need to increase the number of hands (my own included) to help out, which requires that the people with the vision and understanding of the details, need to work more on documentation and less on exotic integration. I think it is the most urgent issue in this project. Cheers Niclas On Thu, Mar 22, 2018 at 5:58 PM, Niclas Hedhman wrote: > > Chris, > I came across a presentation by coincidence that mentioned "Kafka > Connect", so I will research this over the weekend and see what it is, and > how it fits with PLC4X. The presenter only mentioned that a "Kafka Connect" > component is enable by configuration (in Kafka was implied but not sure) > rather than programming. > > More next week. > Niclas > > On Wed, Mar 21, 2018 at 5:19 PM, Niclas Hedhman > wrote: > >> >> Alright... So to repeat (and I think you get this much)... Kafka is in >> principle very simple; Put any byte array into a topic in one end, and >> readers can get that byte array from that topic somewhere else. Just a >> simple message queue, at an abstract level. And the Kafka client libraries >> for doing this is dead easy to use, producer.send() and consumer.poll() >> respectively. >> >> So, I suspect that you are talking about some higher level than base >> Kafka, possibly masquerading the Kafka transport as some other transport >> abstraction or a component implementing some protocol, such as MQTT [1], >> that behind the end point feeds Kafka. But I could be completely off-track >> and some really clever ideas are at play here, that I don't realize yet. >> So, yes, please invite people to discuss further. >> >> Niclas >> >> [1] MQTT feeding Kafka is something I will implement myself in the not >> too distant future >> >> On Wed, Mar 21, 2018 at 4:50 PM, Christofer Dutz < >> christofer.d...@c-ware.de> wrote: >> >>> Hi Niclas, >>> >>> have to admit I don't know in detail either ... it was part of a talk a >>> colleague of mine held at a Kafka meetup in our office. Also I'm not into >>> Kafka half as deep as you ... from a user point of view it looked as if >>> such a Kafka Connect adapter would be a good idea. I'm trying to convince >>> that colleague to contribute ... perhaps he could register here and shed >>> some light on the topic ... I'll continue the convincing (even if another >>> codecentric participant wouldn't increase company diversity) >>> >>> Chris >>> >>> >>> Am 13.03.18, 16:41 schrieb "hedh...@gmail.com im Auftrag von Niclas >>> Hedhman" : >>> >>> Sorry, don't know what that is or what it really is, compared to >>> what I >>> wrote... >>> >>> On Tue, Mar 13, 2018 at 8:50 PM, Christofer Dutz < >>> christofer.d...@c-ware.de> >>> wrote: >>> >>> > Argh ... I wanted to propose a Kafka Connect and not a Kafka Steam >>> adapter >>> > ( ... >>> > >>> > By the way my currently most used setup is actually using PLC4X >>> for PLC >>> > communication and using Edgents Kafka Connector to publish to >>> Kafka. >>> > >>> > A Kafka Connect adapter would allow to directly connect it to >>> Kafka. >>> > >>> > Chris >>> > >>> > >>> > Am 13.03.18, 00:02 schrieb "hedh...@gmail.com im Auftrag von >>> Niclas >>> > Hedhman" : >>> > >>> > I have some Kafka experience, but not used Kafka Streams. But >>> I think >>> > the >>> > most straight forward approach would still be to have a Kafka >>> Producer >>> > on >>> > the PLC4X side simply writing to one or more Kafka topics, >>> which is >>> > relatively si
Re: Creating a Kafka Stream connector for PLC4X?
Chris, I came across a presentation by coincidence that mentioned "Kafka Connect", so I will research this over the weekend and see what it is, and how it fits with PLC4X. The presenter only mentioned that a "Kafka Connect" component is enable by configuration (in Kafka was implied but not sure) rather than programming. More next week. Niclas On Wed, Mar 21, 2018 at 5:19 PM, Niclas Hedhman wrote: > > Alright... So to repeat (and I think you get this much)... Kafka is in > principle very simple; Put any byte array into a topic in one end, and > readers can get that byte array from that topic somewhere else. Just a > simple message queue, at an abstract level. And the Kafka client libraries > for doing this is dead easy to use, producer.send() and consumer.poll() > respectively. > > So, I suspect that you are talking about some higher level than base > Kafka, possibly masquerading the Kafka transport as some other transport > abstraction or a component implementing some protocol, such as MQTT [1], > that behind the end point feeds Kafka. But I could be completely off-track > and some really clever ideas are at play here, that I don't realize yet. > So, yes, please invite people to discuss further. > > Niclas > > [1] MQTT feeding Kafka is something I will implement myself in the not too > distant future > > On Wed, Mar 21, 2018 at 4:50 PM, Christofer Dutz < > christofer.d...@c-ware.de> wrote: > >> Hi Niclas, >> >> have to admit I don't know in detail either ... it was part of a talk a >> colleague of mine held at a Kafka meetup in our office. Also I'm not into >> Kafka half as deep as you ... from a user point of view it looked as if >> such a Kafka Connect adapter would be a good idea. I'm trying to convince >> that colleague to contribute ... perhaps he could register here and shed >> some light on the topic ... I'll continue the convincing (even if another >> codecentric participant wouldn't increase company diversity) >> >> Chris >> >> >> Am 13.03.18, 16:41 schrieb "hedh...@gmail.com im Auftrag von Niclas >> Hedhman" : >> >> Sorry, don't know what that is or what it really is, compared to what >> I >> wrote... >> >> On Tue, Mar 13, 2018 at 8:50 PM, Christofer Dutz < >> christofer.d...@c-ware.de> >> wrote: >> >> > Argh ... I wanted to propose a Kafka Connect and not a Kafka Steam >> adapter >> > ( ... >> > >> > By the way my currently most used setup is actually using PLC4X for >> PLC >> > communication and using Edgents Kafka Connector to publish to Kafka. >> > >> > A Kafka Connect adapter would allow to directly connect it to Kafka. >> > >> > Chris >> > >> > >> > Am 13.03.18, 00:02 schrieb "hedh...@gmail.com im Auftrag von Niclas >> > Hedhman" : >> > >> > I have some Kafka experience, but not used Kafka Streams. But I >> think >> > the >> > most straight forward approach would still be to have a Kafka >> Producer >> > on >> > the PLC4X side simply writing to one or more Kafka topics, >> which is >> > relatively simple. >> > >> > You need to 'configure' during the instance creation; >> > >> > Properties properties = new Properties(); >> > properties.put( ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, >> cli.kafka ); >> > properties.put( ProducerConfig.LINGER_MS_CONFIG, 1 ); >> > properties.put( ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, >> > StringSerializer.class.getName() ); >> > properties.put( ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, >> > StringSerializer.class.getName() ); >> > KafkaProducer producer = new KafkaProducer<>( >> > properties ); >> > >> > And once you have the producer, you can push a message to the >> topic >> > with or >> > without a key, like this; >> > >> > producer.send( new ProducerRecord<>( "your-topic", value ) ); >> > >> > >> > >> > I am pretty sure Kafka Streams can take over from the topic >> queue. >> > >> > >> > Cheers >> > Niclas >> > >> > On Mon, Mar 12, 2018 at 10:38 PM, Christofer Dutz < >> > christofer.d...@c-ware.de >> > > wrote: >> > >> > > Hi, >> > > >> > > last week I attended the Kafka meetup of a colleague of mine >> and in >> > his >> > > talk he introduced Kafka Streams and to me it sounded like a >> > „Clustered >> > > Edgent“. >> > > >> > > My second thought was: would it not be cool to implement such >> a Kafka >> > > Stream Connector? >> > > >> > > Anyone here got the knowhow to do that? Shouldn’t be that >> difficult. >> > > >> > > Chris >> > > >> > >> > >> > >> > -- >> > Niclas Hedhman, Software Developer >> > http://polygene.apache.org - New Energy for Java >> > >> > >> > >> >> >> -- >>
Re: Creating a Kafka Stream connector for PLC4X?
Alright... So to repeat (and I think you get this much)... Kafka is in principle very simple; Put any byte array into a topic in one end, and readers can get that byte array from that topic somewhere else. Just a simple message queue, at an abstract level. And the Kafka client libraries for doing this is dead easy to use, producer.send() and consumer.poll() respectively. So, I suspect that you are talking about some higher level than base Kafka, possibly masquerading the Kafka transport as some other transport abstraction or a component implementing some protocol, such as MQTT [1], that behind the end point feeds Kafka. But I could be completely off-track and some really clever ideas are at play here, that I don't realize yet. So, yes, please invite people to discuss further. Niclas [1] MQTT feeding Kafka is something I will implement myself in the not too distant future On Wed, Mar 21, 2018 at 4:50 PM, Christofer Dutz wrote: > Hi Niclas, > > have to admit I don't know in detail either ... it was part of a talk a > colleague of mine held at a Kafka meetup in our office. Also I'm not into > Kafka half as deep as you ... from a user point of view it looked as if > such a Kafka Connect adapter would be a good idea. I'm trying to convince > that colleague to contribute ... perhaps he could register here and shed > some light on the topic ... I'll continue the convincing (even if another > codecentric participant wouldn't increase company diversity) > > Chris > > > Am 13.03.18, 16:41 schrieb "hedh...@gmail.com im Auftrag von Niclas > Hedhman" : > > Sorry, don't know what that is or what it really is, compared to what I > wrote... > > On Tue, Mar 13, 2018 at 8:50 PM, Christofer Dutz < > christofer.d...@c-ware.de> > wrote: > > > Argh ... I wanted to propose a Kafka Connect and not a Kafka Steam > adapter > > ( ... > > > > By the way my currently most used setup is actually using PLC4X for > PLC > > communication and using Edgents Kafka Connector to publish to Kafka. > > > > A Kafka Connect adapter would allow to directly connect it to Kafka. > > > > Chris > > > > > > Am 13.03.18, 00:02 schrieb "hedh...@gmail.com im Auftrag von Niclas > > Hedhman" : > > > > I have some Kafka experience, but not used Kafka Streams. But I > think > > the > > most straight forward approach would still be to have a Kafka > Producer > > on > > the PLC4X side simply writing to one or more Kafka topics, which > is > > relatively simple. > > > > You need to 'configure' during the instance creation; > > > > Properties properties = new Properties(); > > properties.put( ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, > cli.kafka ); > > properties.put( ProducerConfig.LINGER_MS_CONFIG, 1 ); > > properties.put( ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, > > StringSerializer.class.getName() ); > > properties.put( ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, > > StringSerializer.class.getName() ); > > KafkaProducer producer = new KafkaProducer<>( > > properties ); > > > > And once you have the producer, you can push a message to the > topic > > with or > > without a key, like this; > > > > producer.send( new ProducerRecord<>( "your-topic", value ) ); > > > > > > > > I am pretty sure Kafka Streams can take over from the topic > queue. > > > > > > Cheers > > Niclas > > > > On Mon, Mar 12, 2018 at 10:38 PM, Christofer Dutz < > > christofer.d...@c-ware.de > > > wrote: > > > > > Hi, > > > > > > last week I attended the Kafka meetup of a colleague of mine > and in > > his > > > talk he introduced Kafka Streams and to me it sounded like a > > „Clustered > > > Edgent“. > > > > > > My second thought was: would it not be cool to implement such > a Kafka > > > Stream Connector? > > > > > > Anyone here got the knowhow to do that? Shouldn’t be that > difficult. > > > > > > Chris > > > > > > > > > > > -- > > Niclas Hedhman, Software Developer > > http://polygene.apache.org - New Energy for Java > > > > > > > > > -- > Niclas Hedhman, Software Developer > http://polygene.apache.org - New Energy for Java > > > -- Niclas Hedhman, Software Developer http://polygene.apache.org - New Energy for Java
Re: Creating a Kafka Stream connector for PLC4X?
Hi Niclas, have to admit I don't know in detail either ... it was part of a talk a colleague of mine held at a Kafka meetup in our office. Also I'm not into Kafka half as deep as you ... from a user point of view it looked as if such a Kafka Connect adapter would be a good idea. I'm trying to convince that colleague to contribute ... perhaps he could register here and shed some light on the topic ... I'll continue the convincing (even if another codecentric participant wouldn't increase company diversity) Chris Am 13.03.18, 16:41 schrieb "hedh...@gmail.com im Auftrag von Niclas Hedhman" : Sorry, don't know what that is or what it really is, compared to what I wrote... On Tue, Mar 13, 2018 at 8:50 PM, Christofer Dutz wrote: > Argh ... I wanted to propose a Kafka Connect and not a Kafka Steam adapter > ( ... > > By the way my currently most used setup is actually using PLC4X for PLC > communication and using Edgents Kafka Connector to publish to Kafka. > > A Kafka Connect adapter would allow to directly connect it to Kafka. > > Chris > > > Am 13.03.18, 00:02 schrieb "hedh...@gmail.com im Auftrag von Niclas > Hedhman" : > > I have some Kafka experience, but not used Kafka Streams. But I think > the > most straight forward approach would still be to have a Kafka Producer > on > the PLC4X side simply writing to one or more Kafka topics, which is > relatively simple. > > You need to 'configure' during the instance creation; > > Properties properties = new Properties(); > properties.put( ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, cli.kafka ); > properties.put( ProducerConfig.LINGER_MS_CONFIG, 1 ); > properties.put( ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, > StringSerializer.class.getName() ); > properties.put( ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, > StringSerializer.class.getName() ); > KafkaProducer producer = new KafkaProducer<>( > properties ); > > And once you have the producer, you can push a message to the topic > with or > without a key, like this; > > producer.send( new ProducerRecord<>( "your-topic", value ) ); > > > > I am pretty sure Kafka Streams can take over from the topic queue. > > > Cheers > Niclas > > On Mon, Mar 12, 2018 at 10:38 PM, Christofer Dutz < > christofer.d...@c-ware.de > > wrote: > > > Hi, > > > > last week I attended the Kafka meetup of a colleague of mine and in > his > > talk he introduced Kafka Streams and to me it sounded like a > „Clustered > > Edgent“. > > > > My second thought was: would it not be cool to implement such a Kafka > > Stream Connector? > > > > Anyone here got the knowhow to do that? Shouldn’t be that difficult. > > > > Chris > > > > > > -- > Niclas Hedhman, Software Developer > http://polygene.apache.org - New Energy for Java > > > -- Niclas Hedhman, Software Developer http://polygene.apache.org - New Energy for Java
Re: Creating a Kafka Stream connector for PLC4X?
Sorry, don't know what that is or what it really is, compared to what I wrote... On Tue, Mar 13, 2018 at 8:50 PM, Christofer Dutz wrote: > Argh ... I wanted to propose a Kafka Connect and not a Kafka Steam adapter > ( ... > > By the way my currently most used setup is actually using PLC4X for PLC > communication and using Edgents Kafka Connector to publish to Kafka. > > A Kafka Connect adapter would allow to directly connect it to Kafka. > > Chris > > > Am 13.03.18, 00:02 schrieb "hedh...@gmail.com im Auftrag von Niclas > Hedhman" : > > I have some Kafka experience, but not used Kafka Streams. But I think > the > most straight forward approach would still be to have a Kafka Producer > on > the PLC4X side simply writing to one or more Kafka topics, which is > relatively simple. > > You need to 'configure' during the instance creation; > > Properties properties = new Properties(); > properties.put( ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, cli.kafka ); > properties.put( ProducerConfig.LINGER_MS_CONFIG, 1 ); > properties.put( ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, > StringSerializer.class.getName() ); > properties.put( ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, > StringSerializer.class.getName() ); > KafkaProducer producer = new KafkaProducer<>( > properties ); > > And once you have the producer, you can push a message to the topic > with or > without a key, like this; > > producer.send( new ProducerRecord<>( "your-topic", value ) ); > > > > I am pretty sure Kafka Streams can take over from the topic queue. > > > Cheers > Niclas > > On Mon, Mar 12, 2018 at 10:38 PM, Christofer Dutz < > christofer.d...@c-ware.de > > wrote: > > > Hi, > > > > last week I attended the Kafka meetup of a colleague of mine and in > his > > talk he introduced Kafka Streams and to me it sounded like a > „Clustered > > Edgent“. > > > > My second thought was: would it not be cool to implement such a Kafka > > Stream Connector? > > > > Anyone here got the knowhow to do that? Shouldn’t be that difficult. > > > > Chris > > > > > > -- > Niclas Hedhman, Software Developer > http://polygene.apache.org - New Energy for Java > > > -- Niclas Hedhman, Software Developer http://polygene.apache.org - New Energy for Java
Re: Creating a Kafka Stream connector for PLC4X?
Argh ... I wanted to propose a Kafka Connect and not a Kafka Steam adapter ( ... By the way my currently most used setup is actually using PLC4X for PLC communication and using Edgents Kafka Connector to publish to Kafka. A Kafka Connect adapter would allow to directly connect it to Kafka. Chris Am 13.03.18, 00:02 schrieb "hedh...@gmail.com im Auftrag von Niclas Hedhman" : I have some Kafka experience, but not used Kafka Streams. But I think the most straight forward approach would still be to have a Kafka Producer on the PLC4X side simply writing to one or more Kafka topics, which is relatively simple. You need to 'configure' during the instance creation; Properties properties = new Properties(); properties.put( ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, cli.kafka ); properties.put( ProducerConfig.LINGER_MS_CONFIG, 1 ); properties.put( ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName() ); properties.put( ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName() ); KafkaProducer producer = new KafkaProducer<>( properties ); And once you have the producer, you can push a message to the topic with or without a key, like this; producer.send( new ProducerRecord<>( "your-topic", value ) ); I am pretty sure Kafka Streams can take over from the topic queue. Cheers Niclas On Mon, Mar 12, 2018 at 10:38 PM, Christofer Dutz wrote: > Hi, > > last week I attended the Kafka meetup of a colleague of mine and in his > talk he introduced Kafka Streams and to me it sounded like a „Clustered > Edgent“. > > My second thought was: would it not be cool to implement such a Kafka > Stream Connector? > > Anyone here got the knowhow to do that? Shouldn’t be that difficult. > > Chris > -- Niclas Hedhman, Software Developer http://polygene.apache.org - New Energy for Java
Re: Creating a Kafka Stream connector for PLC4X?
I have some Kafka experience, but not used Kafka Streams. But I think the most straight forward approach would still be to have a Kafka Producer on the PLC4X side simply writing to one or more Kafka topics, which is relatively simple. You need to 'configure' during the instance creation; Properties properties = new Properties(); properties.put( ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, cli.kafka ); properties.put( ProducerConfig.LINGER_MS_CONFIG, 1 ); properties.put( ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName() ); properties.put( ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName() ); KafkaProducer producer = new KafkaProducer<>( properties ); And once you have the producer, you can push a message to the topic with or without a key, like this; producer.send( new ProducerRecord<>( "your-topic", value ) ); I am pretty sure Kafka Streams can take over from the topic queue. Cheers Niclas On Mon, Mar 12, 2018 at 10:38 PM, Christofer Dutz wrote: > Hi, > > last week I attended the Kafka meetup of a colleague of mine and in his > talk he introduced Kafka Streams and to me it sounded like a „Clustered > Edgent“. > > My second thought was: would it not be cool to implement such a Kafka > Stream Connector? > > Anyone here got the knowhow to do that? Shouldn’t be that difficult. > > Chris > -- Niclas Hedhman, Software Developer http://polygene.apache.org - New Energy for Java
Creating a Kafka Stream connector for PLC4X?
Hi, last week I attended the Kafka meetup of a colleague of mine and in his talk he introduced Kafka Streams and to me it sounded like a „Clustered Edgent“. My second thought was: would it not be cool to implement such a Kafka Stream Connector? Anyone here got the knowhow to do that? Shouldn’t be that difficult. Chris
