What does exponential data means? Does this mean that the amount of the
data that is being received from the stream in a batchinterval is
increasing exponentially as the time progresses?
Does your process have enough memory to handle the data for a batch
interval?
You may want to share Spark task UI snapshots and logs.
On Thu, Aug 13, 2015 at 9:25 PM, UMESH CHAUDHARY
wrote:
> Hi,
> I was working with non-reliable receiver version of Spark-Kafka streaming
> i.e.
> KafkaUtils,createStream... where for testing purpose I was getting data at
> constant rate from kafka and it was acting as expected.
> But when there was exponential data in Kafka, my program started crashing
> saying
> "Cannot Compute split on input data..." also I found on console logs that
> it was adding data continuously in memory while receiving from Kafka.
>
> How Spark Streaming behaves towards exponential data.
>
