Re: Kafka Streams: from a KStream, aggregating records with the same key and updated metrics ?

2017-01-17 Thread Michael Noll
Thanks for sharing back your findings/code, Nicolas! -Michael On Mon, Jan 16, 2017 at 11:15 PM, Nicolas Fouché wrote: > If anyone is interested, here is my custom timestamp extractor: > https://gist.github.com/nfo/54d5830720e163d2e7e848b6e4baac20 . > > 2017-01-16 15:52 GMT+01:00 Nicolas Fouché

Re: Kafka Streams: from a KStream, aggregating records with the same key and updated metrics ?

2017-01-16 Thread Nicolas Fouché
If anyone is interested, here is my custom timestamp extractor: https://gist.github.com/nfo/54d5830720e163d2e7e848b6e4baac20 . 2017-01-16 15:52 GMT+01:00 Nicolas Fouché : > Hi Michael, > > got it. I understand that it would be less error-prone to generate the > final "altered" timestamp on the Pr

Re: Kafka Streams: from a KStream, aggregating records with the same key and updated metrics ?

2017-01-16 Thread Nicolas Fouché
Hi Michael, got it. I understand that it would be less error-prone to generate the final "altered" timestamp on the Producer side, instead of trying to compute it each time the record is consumed. Thanks. Nicolas. 2017-01-16 10:03 GMT+01:00 Michael Noll : > Nicolas, > > quick feedback on timest

Re: Kafka Streams: from a KStream, aggregating records with the same key and updated metrics ?

2017-01-16 Thread Michael Noll
Nicolas, quick feedback on timestamps: > In our system, clients send data to an HTTP API. This API produces the > records in Kafka. I can't rely on the clock of the clients sending the > original data, (so the records' timestamps are set by the servers ingesting > the records in Kafka), but I can

Re: Kafka Streams: from a KStream, aggregating records with the same key and updated metrics ?

2017-01-15 Thread Nicolas Fouché
Hi Eno, 2. Well, records could arrive out of order. But it should happen rarely, and it's no big deal anyway. So let's forget about the version number if it makes things easier ! 3. I completely missed out on KTable aggregations. Thanks a lot for the pointer, that opens new perspectives. ... a f

Re: Kafka Streams: from a KStream, aggregating records with the same key and updated metrics ?

2017-01-13 Thread Eno Thereska
Hi Nicolas, There is a lot here, so let's try to split the concerns around some themes: 1. The Processor API is flexible and can definitely do what you want, but as you mentioned, at the cost of you having to manually craft the code. 2. Why are the versions used? I sense there is concern about r