Pinging again to see if anyone has any thoughts or prior experience with this issue.
On Wed, Nov 25, 2015 at 3:56 PM, Spark Newbie <sparknewbie1...@gmail.com> wrote: > Hi Spark users, > > I have been seeing this issue where receivers enter a "stuck" state after > it encounters a the following exception "Error in block pushing thread - > java.util.concurrent.TimeoutException: Futures timed out". > I am running the application on spark-1.4.1 and using kinesis-asl-1.4. > > When this happens, the observation is that the > Kinesis.ProcessTask.shardxxxx.MillisBehindLatest metric does not get > published anymore, when I look at cloudwatch, which indicates that the > workers associated with the receiver are not checkpointing any more for the > shards that they were reading from. > > This seems like a bug in to BlockGenerator code , here - > https://github.com/apache/spark/blob/branch-1.4/streaming/src/main/scala/org/apache/spark/streaming/receiver/BlockGenerator.scala#L171 > when pushBlock encounters an exception, in this case the TimeoutException, > it stops pushing blocks. Is this really expected behavior? > > Has anyone else seen this error and have you also seen the issue where > receivers stop receiving records? I'm also trying to find the root cause > for the TimeoutException. If anyone has an idea on this please share. > > Thanks, > > Bharath >
