I've created a ticket here: https://issues.apache.org/jira/browse/SPARK-19888 <https://issues.apache.org/jira/browse/SPARK-19888>
Thanks, Justin > On Mar 10, 2017, at 1:14 PM, Michael Armbrust <[email protected]> wrote: > > If you have a reproduction you should open a JIRA. It would be great if > there is a fix. I'm just saying I know a similar issue does not exist in > structured streaming. > > On Fri, Mar 10, 2017 at 7:46 AM, Justin Miller <[email protected] > <mailto:[email protected]>> wrote: > Hi Michael, > > I'm experiencing a similar issue. Will this not be fixed in Spark Streaming? > > Best, > Justin > >> On Mar 10, 2017, at 8:34 AM, Michael Armbrust <[email protected] >> <mailto:[email protected]>> wrote: >> >> One option here would be to try Structured Streaming. We've added an option >> "failOnDataLoss" that will cause Spark to just skip a head when this >> exception is encountered (its off by default though so you don't silently >> miss data). >> >> On Fri, Mar 18, 2016 at 4:16 AM, Ramkumar Venkataraman >> <[email protected] <mailto:[email protected]>> wrote: >> I am using Spark streaming and reading data from Kafka using >> KafkaUtils.createDirectStream. I have the "auto.offset.reset" set to >> smallest. >> >> But in some Kafka partitions, I get kafka.common.OffsetOutOfRangeException >> and my spark job crashes. >> >> I want to understand if there is a graceful way to handle this failure and >> not kill the job. I want to keep ignoring these exceptions, as some other >> partitions are fine and I am okay with data loss. >> >> Is there any way to handle this and not have my spark job crash? I have no >> option of increasing the kafka retention period. >> >> I tried to have the DStream returned by createDirectStream() wrapped in a >> Try construct, but since the exception happens in the executor, the Try >> construct didn't take effect. Do you have any ideas of how to handle this? >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-gracefully-handle-Kafka-OffsetOutOfRangeException-tp26534.html >> >> <http://apache-spark-user-list.1001560.n3.nabble.com/How-to-gracefully-handle-Kafka-OffsetOutOfRangeException-tp26534.html> >> Sent from the Apache Spark User List mailing list archive at Nabble.com >> <http://nabble.com/>. >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> <mailto:[email protected]> >> For additional commands, e-mail: [email protected] >> <mailto:[email protected]> >> >> > >
