Re: Kafka Streams vs Spark Streaming : reduce by window

2017-06-20 Thread Guozhang Wang
amp; IoT* > *Microsoft Azure Advisor* > > Twitter : @ppatierno <http://twitter.com/ppatierno> > Linkedin : paolopatierno <http://it.linkedin.com/in/paolopatierno> > Blog : DevExperience <http://paolopatierno.wordpress.com/> > > > --

Re: Kafka Streams vs Spark Streaming : reduce by window

2017-06-18 Thread Paolo Patierno
p://it.linkedin.com/in/paolopatierno> Blog : DevExperience<http://paolopatierno.wordpress.com/> From: Michal Borowiecki Sent: Sunday, June 18, 2017 9:34 AM To: d...@kafka.apache.org; Jay Kreps Cc: users@kafka.apache.org; Matthias J. Sax Subject: Re: Kafka Streams vs Spark

Re: Kafka Streams vs Spark Streaming : reduce by window

2017-06-18 Thread Michal Borowiecki
DevExperience<http://paolopatierno.wordpress.com/> <http://paolopatierno.wordpress.com/> ____________ From: Eno Thereska <mailto:eno.there...@gmail.com> Sent: Thursday, June 15, 2017 3:57 PM To:users@kafka.apache.org <mailto:user

Re: Kafka Streams vs Spark Streaming : reduce by window

2017-06-16 Thread Matthias J. Sax
; your article ! Thanks ! >>> >>> >>> Paolo >>> >>> >>> Paolo Patierno >>> Senior Software Engineer (IoT) @ Red Hat >>> Microsoft MVP on Windows Embedded & IoT >>> Microsoft Azure Advisor >>> >>> Twitter

Re: Kafka Streams vs Spark Streaming : reduce by window

2017-06-16 Thread Jay Kreps
: @ppatierno<http://twitter.com/ppatierno> > <http://twitter.com/ppatierno> > Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno> > <http://it.linkedin.com/in/paolopatierno> > Blog : DevExperience<http://paolopatierno.wordpress.com/> >

Re: Kafka Streams vs Spark Streaming : reduce by window

2017-06-16 Thread Jay Kreps
: @ppatierno<http://twitter.com/ppatierno> > <http://twitter.com/ppatierno> > Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno> > <http://it.linkedin.com/in/paolopatierno> > Blog : DevExperience<http://paolopatierno.wordpress.com/> >

Re: Kafka Streams vs Spark Streaming : reduce by window

2017-06-16 Thread Michal Borowiecki
no<http://it.linkedin.com/in/paolopatierno> Blog : DevExperience<http://paolopatierno.wordpress.com/> From: Eno Thereska Sent: Thursday, June 15, 2017 1:45 PM To: users@kafka.apache.org Subject: Re: Kafka Streams vs Spark Streaming : reduce by window Hi Pa

Re: Kafka Streams vs Spark Streaming : reduce by window

2017-06-15 Thread Matthias J. Sax
erno<http://twitter.com/ppatierno> > Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno> > Blog : DevExperience<http://paolopatierno.wordpress.com/> > > > > From: Eno Thereska > Sent: Thursday, June 15, 2017 3:57 PM > To: users@ka

Re: Kafka Streams vs Spark Streaming : reduce by window

2017-06-15 Thread Paolo Patierno
ierno<http://it.linkedin.com/in/paolopatierno> > Blog : DevExperience<http://paolopatierno.wordpress.com/> > > > > From: Eno Thereska > Sent: Thursday, June 15, 2017 1:45 PM > To: users@kafka.apache.org > Subject: Re: Kafka Stream

Re: Kafka Streams vs Spark Streaming : reduce by window

2017-06-15 Thread Eno Thereska
o> > Blog : DevExperience<http://paolopatierno.wordpress.com/> > > > > From: Eno Thereska > Sent: Thursday, June 15, 2017 1:45 PM > To: users@kafka.apache.org > Subject: Re: Kafka Streams vs Spark Streaming : reduce by window >

Re: Kafka Streams vs Spark Streaming : reduce by window

2017-06-15 Thread Paolo Patierno
Blog : DevExperience<http://paolopatierno.wordpress.com/> From: Eno Thereska Sent: Thursday, June 15, 2017 1:45 PM To: users@kafka.apache.org Subject: Re: Kafka Streams vs Spark Streaming : reduce by window Hi Paolo, That is indeed correct. We don’t believe in cl

Re: Kafka Streams vs Spark Streaming : reduce by window

2017-06-15 Thread Eno Thereska
Hi Paolo, That is indeed correct. We don’t believe in closing windows in Kafka Streams. You could reduce the number of downstream records by using record caches: http://docs.confluent.io/current/streams/developer-guide.html#record-caches-in-the-dsl

Re: Kafka Streams vs Spark Streaming : reduce by window

2017-06-15 Thread Tom Bentley
It sounds like you want a tumbling time window, rather than a sliding window https://kafka.apache.org/documentation/streams#streams_dsl_windowing On 15 June 2017 at 14:38, Paolo Patierno wrote: > Hi, > > > using the streams library I noticed a difference (or there is a lack of > knowledge on my

Kafka Streams vs Spark Streaming : reduce by window

2017-06-15 Thread Paolo Patierno
Hi, using the streams library I noticed a difference (or there is a lack of knowledge on my side)with Apache Spark. Imagine following scenario ... I have a source topic where numeric values come in and I want to check the maximum value in the latest 5 seconds but ... putting the max value in