Is the exception on the driver or the executor? To be clear, you're going to see a task fail if a partition changes leader while the task is running, regardless of configuration settings. The task should be retried up the maxFailures though.
What are maxRetries and maxFailures set to? How long does the leader change take? How many task retries happen during that time (assuming it's an exception on the executor)? On Mon, Jun 13, 2016 at 10:40 AM, Bryan Jeffrey <bryan.jeff...@gmail.com> wrote: > Cody, > > We already set the maxRetries. We're still seeing issue - when leader is > shifted, for example, it does not appear that direct stream reader correctly > handles this. We're running 1.6.1. > > Bryan Jeffrey > > On Mon, Jun 13, 2016 at 10:37 AM, Cody Koeninger <c...@koeninger.org> wrote: >> >> http://spark.apache.org/docs/latest/configuration.html >> >> spark.streaming.kafka.maxRetries >> >> spark.task.maxFailures >> >> On Mon, Jun 13, 2016 at 8:25 AM, Bryan Jeffrey <bryan.jeff...@gmail.com> >> wrote: >> > All, >> > >> > We're running a Spark job that is consuming data from a large Kafka >> > cluster >> > using the Direct Stream receiver. We're seeing intermittent >> > NotLeaderForPartitionExceptions when the leader is moved to another >> > broker. >> > Currently even with retry enabled we're seeing the job fail at this >> > exception. Is there a configuration setting I am missing? How are >> > these >> > issues typically handled? >> > >> > User class threw exception: org.apache.spark.SparkException: >> > ArrayBuffer(kafka.common.NotLeaderForPartitionException, >> > org.apache.spark.SparkException: Couldn't find leader offsets for >> > Set([MyTopic,43])) >> > >> > Thank you, >> > >> > Bryan Jeffrey >> > > > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
