etienne created SPARK-17606:

             Summary: New batches are not created when there are 1000 created 
after restarting streaming from checkpoint.
                 Key: SPARK-17606
             Project: Spark
          Issue Type: Bug
          Components: Spark Core
    Affects Versions: 1.6.1
            Reporter: etienne

When spark restarts from a checkpoint after being down for a while.
It recreates missing batch since the down time.

When there are few missing batches, spark creates new incoming batch every 
batchTime, but when there is enough missing time to create 1000 batches no new 
batch is created.

So when all these batch are completed the stream is idle ...

I think there is a rigid limit set somewhere.

I was expecting that spark continue to recreate missed batches, maybe not all 
at once ( because it's look like it's cause driver memory problem ), and then 
recreate batches each batchTime.

Another solution would be to not create missing batches but still restart the 
direct input.

Right know for me the only solution to restart a stream after a long break it 
to remove the checkpoint to allow the creation of a new stream. But losing all 
my states.

ps : I'm speaking about direct Kafka input because it's the source I'm 
currently using, I don't know what happens with other sources.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to