Structured streaming performance issues

2019-02-21 Thread gvdongen
Hi everyone, I have the following pipeline: Ingest 2 streams from Kafka -> parse JSON -> join both streams -> aggregate on a key over the last second -> output to Kafka with Join: inner join in interval of one second, with watermarking 50 ms Aggregation: tumbling window of one second, with waterma

Benchmarking streaming frameworks

2017-04-03 Thread gvdongen
Dear users of Streaming Technologies, As a PhD student in big data analytics, I am currently in the process of compiling a list of benchmarks (to test multiple streaming frameworks) in order to create an expanded benchmarking suite. The benchmark suite is being developed as a part of my current wo

Benchmarking streaming frameworks

2017-04-03 Thread gvdongen
Dear users of Streaming Technologies, As a PhD student in big data analytics, I am currently in the process of compiling a list of benchmarks (to test multiple streaming frameworks) in order to create an expanded benchmarking suite. The benchmark suite is being developed as a part of my current wo