RE: Weird performance pattern of Spark Streaming (1.4.1) + direct Kafka

2015-10-07 Thread Goodall, Mark (UK)
I would like to say that I have also had this issue. In two situations, one using Accumulo to store information and also when running multiple streaming jobs within the same streaming context (e.g. multiple save to hdfs). In my case the situation worsens when one of the jobs, which has a long

DStream union with different slideDuration

2015-09-29 Thread Goodall, Mark (UK)
Hi, I was wondering if there is a reason for limiting union to only work on streams with the same slideDuration. Looking at UnionDStream.scala, if slideDuration was set to the minimum of the parents, and there was a require to enforce that all slideDuration were divisible wholly by the minimum,