Re: Kafka Streams Aggregate By Date

2016-10-19 Thread Matthias J. Sax
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 About auto-topic creation: If your broker configuration allows for this, yes it would work. However, keep in mind, that the topic will be created with default values (according to broker config) with regard to number of partitions and replication fac

Re: Kafka Streams Aggregate By Date

2016-10-19 Thread Furkan KAMACI
I could successfully get the total (not average). As far as I see, there is no need to create a topic manually before I run the app. Topic is created if there is data and topic name not exists. Here is my code: KStreamBuilder builder = new KStreamBuilder(); KStream longs = builder.stream(

Re: Kafka Streams Aggregate By Date

2016-10-18 Thread Matthias J. Sax
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 You should create input/intermediate and output topic manually before you start you Kafka Streams application. - -Matthias On 10/18/16 3:34 PM, Furkan KAMACI wrote: > Sorry about concurrent questions. Tried below code, didn't get any > error but c

Re: Kafka Streams Aggregate By Date

2016-10-18 Thread Furkan KAMACI
Sorry about concurrent questions. Tried below code, didn't get any error but couldn't get created output topic: Properties props = new Properties(); props.put("bootstrap.servers", "localhost:9092"); props.put("acks", "all"); props.put("retries", 0); props.pu

Re: Kafka Streams Aggregate By Date

2016-10-18 Thread Matthias J. Sax
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Two things: 1) you should not apply the window to the first count, but to the base stream to get correct results. 2) your windowed aggregation, doew not just return String type, but Window type. Thus, you need to either insert a .map() to transform

Re: Kafka Streams Aggregate By Date

2016-10-18 Thread Furkan KAMACI
Hi Matthias, I've tried this code: *final Properties streamsConfiguration = new Properties();* *streamsConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, "myapp");* *streamsConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");* *streamsCo

Re: Kafka Streams Aggregate By Date

2016-10-18 Thread Matthias J. Sax
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 I see. KGroupedStream will be part of 0.10.1.0 (should be release the next weeks). So, instead of > .groupByKey().count() you need to do > .countByKey() - -Matthias On 10/18/16 12:05 PM, Furkan KAMACI wrote: > Hi Matthias, > > Thanks for you

Re: Kafka Streams Aggregate By Date

2016-10-18 Thread Furkan KAMACI
Hi Matthias, Thanks for your detailed answer. By the way I couldn't find "KGroupedStream" at version of 0.10.0.1? Kind Regards, Furkan KAMACI On Tue, Oct 18, 2016 at 8:41 PM, Matthias J. Sax wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA512 > > Hi, > > You just need to read you stream

Re: Kafka Streams Aggregate By Date

2016-10-18 Thread Matthias J. Sax
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Hi, You just need to read you stream and apply an (windowed) aggregation on it. If you use non-windowed aggregation you will get "since the beginning". If you use windowed aggregation you can specify the window size as 1 hour and get those results.