[ https://issues.apache.org/jira/browse/SPARK-17606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15571870#comment-15571870 ]
etienne edited comment on SPARK-17606 at 10/13/16 1:17 PM: ----------------------------------------------------------- Theses string are log as DEBUG or in upper logs level ? ( because there is a lot of log to process ) edit : I found that log4j.logger.org.apache.spark.streaming.scheduler.JobGenerator=DEBUG I think this will help was (Author: crakjie): Theses string are log as DEBUG or in upper logs level ? ( because there is a lot of log to process ) > New batches are not created when there are 1000 created after restarting > streaming from checkpoint. > --------------------------------------------------------------------------------------------------- > > Key: SPARK-17606 > URL: https://issues.apache.org/jira/browse/SPARK-17606 > Project: Spark > Issue Type: Bug > Components: Streaming > Affects Versions: 1.6.1 > Reporter: etienne > > When spark restarts from a checkpoint after being down for a while. > It recreates missing batch since the down time. > When there are few missing batches, spark creates new incoming batch every > batchTime, but when there is enough missing time to create 1000 batches no > new batch is created. > So when all these batch are completed the stream is idle ... > I think there is a rigid limit set somewhere. > I was expecting that spark continue to recreate missed batches, maybe not all > at once ( because it's look like it's cause driver memory problem ), and then > recreate batches each batchTime. > Another solution would be to not create missing batches but still restart the > direct input. > Right know for me the only solution to restart a stream after a long break it > to remove the checkpoint to allow the creation of a new stream. But losing > all my states. > ps : I'm speaking about direct Kafka input because it's the source I'm > currently using, I don't know what happens with other sources. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org