Re: Anyone else mis-interpret the "KafkaConsumer" and "KafkaProducer" all the time?

[vino yang]

Hi Chris,

All data processing framework could think it as a *pipeline . *The Edgent's
point of view, there could be two endpoints :


   - source : means data injection;
   - sink : means data export;

There are many frameworks use this conventional naming rule, such as Apache
Flume, Apache Flink, Apache Spark(structured streaming) .

I think "KafkaConsumer" could be replaced with "KafkaSource" and
"KafkaProducer" could be named "KafkaSink".

And middle of the pipeline is the transformation of the data, there are
many operators to transform data ,such as map, flatmap, filter, reduce...
and so on.

Vino yang.
Thanks.

2018-03-20 20:51 GMT+08:00 Christofer Dutz :

> Hi,
>
> have been using the Kafka integration quite often in the past and one
> thing I always have to explain when demonstrating code and which seems to
> confuse everyone seeing the code:
>
> I would expect a KafkaConsumer to consume Edgent messages and publish them
> to Kafka and would expect a KafkaProducer to produce Edgent events.
>
> Unfortunately it seems to be the other way around. This seems a little
> unintuitive. Judging from the continued confusion when demonstrating code
> eventually it’s worth considering to rename these (swap their names).
> Eventually even rename them to “KafkaSource” (Edgent Source that consumes
> Kafka messages and produces Edgent events) and “KafkaConsumer” (Consumes
> Edgent Events and produces Kafka messages). After all the Classes are in
> the Edgent namespace and come from the Edgent libs, so the fixed point when
> inspecting these should be clear. Also I bet no one would be confused if we
> called something that produces Kafka messages a consumer as there should
> never be code that handles this from a Kafka point of view AND uses Edgent
> at the same time.
>
> Chris
>
>
>

Re: [discuss] What about splitting the kafka Connector into kafka 0.8 and 0.9?

[vino yang]

Hi Chris,

I will try to do this, if I have any question, keep in touch!

Vino yang
Thanks.

2018-03-20 17:32 GMT+08:00 Christofer Dutz :

> Ok,
>
> So I just added a new Annotation type to the Kafka module.
>
> org.apache.edgent.connectors.kafka.annotations.KafkaVersion
>
> It has a fromVersion and a toVersion attribute. Both should be optional so
> just adding the annotation would have no effect (besides a few additional
> CPU operations). The annotation can be applied to methods or classes (every
> method then inherits this). I hope that's ok, because implementing this on
> a parameter Level would make things extremely difficult.
>
> @vino yang With this you should be able to provide Kafka version
> constraints to your code changes. Just tell me if something's missing or
> needs to be done differently
>
> For now this annotation will have no effect as I haven't implemented the
> Aspect for doing the checks, but I'll start working on that as soon as you
> have annotated something.
>
> Chris
>
> Am 20.03.18, 10:11 schrieb "Christofer Dutz" :
>
> Ok ... maybe I should add the Annotation prior to continuing my work
> on the AWS connector ...
>
>
> Chris
>
> Am 04.03.18, 08:10 schrieb "vino yang" :
>
> The reason is that Kafka 0.9+ provided a new consumer API which
> has more
> features and better performance.
>
> Just like Flink's implementation :
> https://github.com/apache/flink/tree/master/flink-connectors.
>
> vinoyang
> Thanks.
>
>
>
>
>

Re: [disscuss] make TStream support groupBy operator?

[vino yang]

Hi Chris,

I think the Edgent community seems not very active. Now industry IoT and
Edge Computing (fog) are more and more popular. It's the Edgent's
opportunity now. In China, many big IT company are force on the IoT and
edge computing.

The old committers do not pay close attention to the project now?

Vino yang
Thanks

2018-03-20 17:28 GMT+08:00 vino yang :

> Hi Chris,
>
> My background is BigData (MapReduce, Spark, Flink, Kafka Stream) those
> data processing frameworks all provide the groupBy / keyBy operation and
> aggregation operator. It comes from traditional RDBMS. Edgent like a single
> JVM's Flink / Kafka Stream works on edges(gateway or IoT).
> They are similar with each other in some ways.
>
> Vino yang.
> Thanks!
>
> 2018-03-20 17:05 GMT+08:00 Christofer Dutz :
>
>> Hi Vino,
>>
>> unfortunately I can't contribute any opinion on this as I don't yet
>> understand the consequences.
>> I know that in an asynchronous event processing system some operations
>> that might be useful have to be sacrificed for the sake of asynchonisity.
>>
>> For me Kafka Stream sort of feeling like the cloud-brother of Edgent, it
>> does seem to support groupBy.
>>
>> Would be really cool if some of the formerly active people could at least
>> leave some comments on questions like this. You don't have to actually work
>> on things, but giving us new guys some guidance would be awesome.
>>
>> I don't want to ruin thing you built over years, just because I'm not
>> that into the topic ... yet.
>>
>> Chris
>>
>>
>>
>>
>> Am 16.03.18, 13:02 schrieb "Christofer Dutz" > >:
>>
>> I'm currently at a conference, so I can't be as responsive as I used
>> to be ... All will be back to normal next Tuesday ;-)
>>
>> Chris
>>
>> Outlook for Android herunterladen
>>
>> 
>> From: vino yang 
>> Sent: Friday, March 16, 2018 2:26:10 AM
>> To: dev@edgent.apache.org
>> Subject: Re: [disscuss] make TStream support groupBy operator?
>>
>> Hi all,
>>
>> Anyone can give some opinion? Chris ? I think we should support some
>> reduce
>> operation(aggregation function, such as max / avg / min sum) for both
>> stream and windowed stream, these features based on the keyBy or
>> groupBy
>> operation.
>>
>> Vino yang
>> Thanks!
>>
>> 2018-03-13 12:52 GMT+08:00 vino yang :
>>
>> > Hi guys,
>> >
>> > Does Edgent current support groupBy operator?
>> >
>> > Vino yang
>> > Thanks.
>> >
>>
>>
>>
>

Re: [discuss] What about splitting the kafka Connector into kafka 0.8 and 0.9?

[Christofer Dutz]

Ok,

So I just added a new Annotation type to the Kafka module. 

org.apache.edgent.connectors.kafka.annotations.KafkaVersion

It has a fromVersion and a toVersion attribute. Both should be optional so just 
adding the annotation would have no effect (besides a few additional CPU 
operations). The annotation can be applied to methods or classes (every method 
then inherits this). I hope that's ok, because implementing this on a parameter 
Level would make things extremely difficult.

@vino yang With this you should be able to provide Kafka version constraints to 
your code changes. Just tell me if something's missing or needs to be done 
differently

For now this annotation will have no effect as I haven't implemented the Aspect 
for doing the checks, but I'll start working on that as soon as you have 
annotated something.

Chris

Am 20.03.18, 10:11 schrieb "Christofer Dutz" :

Ok ... maybe I should add the Annotation prior to continuing my work on the 
AWS connector ...


Chris

Am 04.03.18, 08:10 schrieb "vino yang" :

The reason is that Kafka 0.9+ provided a new consumer API which has more
features and better performance.

Just like Flink's implementation :
https://github.com/apache/flink/tree/master/flink-connectors.

vinoyang
Thanks.

Re: [disscuss] make TStream support groupBy operator?

[vino yang]

Hi Chris,

My background is BigData (MapReduce, Spark, Flink, Kafka Stream) those data
processing frameworks all provide the groupBy / keyBy operation and
aggregation operator. It comes from traditional RDBMS. Edgent like a single
JVM's Flink / Kafka Stream works on edges(gateway or IoT).
They are similar with each other in some ways.

Vino yang.
Thanks!

2018-03-20 17:05 GMT+08:00 Christofer Dutz :

> Hi Vino,
>
> unfortunately I can't contribute any opinion on this as I don't yet
> understand the consequences.
> I know that in an asynchronous event processing system some operations
> that might be useful have to be sacrificed for the sake of asynchonisity.
>
> For me Kafka Stream sort of feeling like the cloud-brother of Edgent, it
> does seem to support groupBy.
>
> Would be really cool if some of the formerly active people could at least
> leave some comments on questions like this. You don't have to actually work
> on things, but giving us new guys some guidance would be awesome.
>
> I don't want to ruin thing you built over years, just because I'm not that
> into the topic ... yet.
>
> Chris
>
>
>
>
> Am 16.03.18, 13:02 schrieb "Christofer Dutz" :
>
> I'm currently at a conference, so I can't be as responsive as I used
> to be ... All will be back to normal next Tuesday ;-)
>
> Chris
>
> Outlook for Android herunterladen
>
> 
> From: vino yang 
> Sent: Friday, March 16, 2018 2:26:10 AM
> To: dev@edgent.apache.org
> Subject: Re: [disscuss] make TStream support groupBy operator?
>
> Hi all,
>
> Anyone can give some opinion? Chris ? I think we should support some
> reduce
> operation(aggregation function, such as max / avg / min sum) for both
> stream and windowed stream, these features based on the keyBy or
> groupBy
> operation.
>
> Vino yang
> Thanks!
>
> 2018-03-13 12:52 GMT+08:00 vino yang :
>
> > Hi guys,
> >
> > Does Edgent current support groupBy operator?
> >
> > Vino yang
> > Thanks.
> >
>
>
>

Re: [discuss] What about splitting the kafka Connector into kafka 0.8 and 0.9?

[Christofer Dutz]

Ok ... maybe I should add the Annotation prior to continuing my work on the AWS 
connector ...


Chris

Am 04.03.18, 08:10 schrieb "vino yang" :

The reason is that Kafka 0.9+ provided a new consumer API which has more
features and better performance.

Just like Flink's implementation :
https://github.com/apache/flink/tree/master/flink-connectors.

vinoyang
Thanks.

Re: Rename "iotp" to "iot-ibm" or similar?

[Christofer Dutz]

Hi,

Well on our company in-house conference last Thursday to Sunday I had one 
session on "implementing IoT Platform adapters for Apache Edgent" and I got 
some people together and we started to work on an AWS IoT Connector. Seems that 
the concepts of the iot module can be implemented 1-to-1 with AWS. 

So this will be the thing I'll be working on actively. I think supporting as 
many IoT Platforms as possible will help Edgents adoption. Probably Mindsphere 
which is probably very interesting for PLC4X+Edgent solutions will stay a 
fantasy as I'm not willing to pay the 10k€/developer/year just to be able to 
access their libs (Not even mentioning that this totally disqualifies it for 
being added to an Apache project).

So probably Googles platform will follow 

But I still want to rename the "iotp" module to something like "iot-watson" or 
"iot-ibm-watson"


Chris



Am 12.03.18, 03:37 schrieb "vino yang" :

+1 , I think we also could support to EdgexFoundry in the future.

2018-03-11 20:00 GMT+08:00 Christofer Dutz :

> Hi all,
>
> I am currently thinking of adding modules to support other IoT Platforms:
>
>   *   Google IoT
>   *   AWS IoT
>   *   Siemens MindSphere
>
> For that the “iotp” sort of doesn’t quite fit as it’s not just “one” IoT
> platform.
> So would you be ok with renaming that to something like: “iot-ibm” or
> similar?
>
> Then we’d have:
>
>   *   Iot-ibm
>   *   Iot-aws
>   *   Iot-google
>   *   Iot-mindsphere/siemens
>
> Chris
>

Re: [disscuss] make TStream support groupBy operator?

[Christofer Dutz]

Hi Vino,

unfortunately I can't contribute any opinion on this as I don't yet understand 
the consequences. 
I know that in an asynchronous event processing system some operations that 
might be useful have to be sacrificed for the sake of asynchonisity.

For me Kafka Stream sort of feeling like the cloud-brother of Edgent, it does 
seem to support groupBy.

Would be really cool if some of the formerly active people could at least leave 
some comments on questions like this. You don't have to actually work on 
things, but giving us new guys some guidance would be awesome. 

I don't want to ruin thing you built over years, just because I'm not that into 
the topic ... yet.

Chris




Am 16.03.18, 13:02 schrieb "Christofer Dutz" :

I'm currently at a conference, so I can't be as responsive as I used to be 
... All will be back to normal next Tuesday ;-)

Chris

Outlook for Android herunterladen


From: vino yang 
Sent: Friday, March 16, 2018 2:26:10 AM
To: dev@edgent.apache.org
Subject: Re: [disscuss] make TStream support groupBy operator?

Hi all,

Anyone can give some opinion? Chris ? I think we should support some reduce
operation(aggregation function, such as max / avg / min sum) for both
stream and windowed stream, these features based on the keyBy or groupBy
operation.

Vino yang
Thanks!

2018-03-13 12:52 GMT+08:00 vino yang :

> Hi guys,
>
> Does Edgent current support groupBy operator?
>
> Vino yang
> Thanks.
>