Hi Chris, All data processing framework could think it as a *pipeline . *The Edgent's point of view, there could be two endpoints :
- source : means data injection; - sink : means data export; There are many frameworks use this conventional naming rule, such as Apache Flume, Apache Flink, Apache Spark(structured streaming) . I think "KafkaConsumer" could be replaced with "KafkaSource" and "KafkaProducer" could be named "KafkaSink". And middle of the pipeline is the transformation of the data, there are many operators to transform data ,such as map, flatmap, filter, reduce... and so on. Vino yang. Thanks. 2018-03-20 20:51 GMT+08:00 Christofer Dutz <christofer.d...@c-ware.de>: > Hi, > > have been using the Kafka integration quite often in the past and one > thing I always have to explain when demonstrating code and which seems to > confuse everyone seeing the code: > > I would expect a KafkaConsumer to consume Edgent messages and publish them > to Kafka and would expect a KafkaProducer to produce Edgent events. > > Unfortunately it seems to be the other way around. This seems a little > unintuitive. Judging from the continued confusion when demonstrating code > eventually it’s worth considering to rename these (swap their names). > Eventually even rename them to “KafkaSource” (Edgent Source that consumes > Kafka messages and produces Edgent events) and “KafkaConsumer” (Consumes > Edgent Events and produces Kafka messages). After all the Classes are in > the Edgent namespace and come from the Edgent libs, so the fixed point when > inspecting these should be clear. Also I bet no one would be confused if we > called something that produces Kafka messages a consumer as there should > never be code that handles this from a Kafka point of view AND uses Edgent > at the same time. > > Chris > > >
