Re: How to gracefully handle Kafka OffsetOutOfRangeException

2017-03-10 Thread Justin Miller
o handle this and not have my spark job crash? I have no >> option of increasing the kafka retention period. >> >> I tried to have the DStream returned by createDirectStream() wrapped in a >> Try construct, but since the exception happens in the executor, the Try >> cons

Re: How to gracefully handle Kafka OffsetOutOfRangeException

2017-03-10 Thread Michael Armbrust
If you have a reproduction you should open a JIRA. It would be great if there is a fix. I'm just saying I know a similar issue does not exist in structured streaming. On Fri, Mar 10, 2017 at 7:46 AM, Justin Miller < justin.mil...@protectwise.com> wrote: > Hi Michael, > > I'm experiencing a simi

Re: How to gracefully handle Kafka OffsetOutOfRangeException

2017-03-10 Thread Justin Miller
tion period. > > I tried to have the DStream returned by createDirectStream() wrapped in a > Try construct, but since the exception happens in the executor, the Try > construct didn't take effect. Do you have any ideas of how to handle this? > > > > -- > View this

Re: How to gracefully handle Kafka OffsetOutOfRangeException

2017-03-10 Thread Michael Armbrust
ve any ideas of how to handle this? > > > > -- > View this message in context: http://apache-spark-user-list. > 1001560.n3.nabble.com/How-to-gracefully-handle-Kafka- > OffsetOutOfRangeException-tp26534.html > Sent from the Apache Spark User List mailing list archive at

Re: How to gracefully handle Kafka OffsetOutOfRangeException

2017-03-10 Thread Ramkumar Venkataraman
@n3.nabble.com> wrote: > Did you find out how ? > > -- > If you reply to this email, your message will be added to the discussion > below: > http://apache-spark-user-list.1001560.n3.nabble.com/How-to- > gracefully-handle-Kafka-OffsetOutOfRangeExcept

Re: How to gracefully handle Kafka OffsetOutOfRangeException

2016-03-21 Thread Ramkumar Venkataraman
; and > >> > not kill the job. I want to keep ignoring these exceptions, as some > >> > other > >> > partitions are fine and I am okay with data loss. > >> > > >> > Is there any way to handle this and not have my spark job crash? I > ha

Re: How to gracefully handle Kafka OffsetOutOfRangeException

2016-03-21 Thread Cody Koeninger
not have my spark job crash? I have >> > no >> > option of increasing the kafka retention period. >> > >> > I tried to have the DStream returned by createDirectStream() wrapped in >> > a >> > Try construct, b

Re: How to gracefully handle Kafka OffsetOutOfRangeException

2016-03-21 Thread Ramkumar Venkataraman
Try construct, but since the exception happens in the executor, the Try > > construct didn't take effect. Do you have any ideas of how to handle > this? > > > > > > > > -- > > View this message in context: > http://apache-spark-user-list.1001560.n3.nabbl

Re: How to gracefully handle Kafka OffsetOutOfRangeException

2016-03-19 Thread Sebastian Piu
t; I tried to have the DStream returned by createDirectStream() wrapped in a > Try construct, but since the exception happens in the executor, the Try > construct didn't take effect. Do you have any ideas of how to handle this? > > > > -- > View this message in contex

How to gracefully handle Kafka OffsetOutOfRangeException

2016-03-19 Thread Ramkumar Venkataraman
he DStream returned by createDirectStream() wrapped in a Try construct, but since the exception happens in the executor, the Try construct didn't take effect. Do you have any ideas of how to handle this? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Ho

Re: How to gracefully handle Kafka OffsetOutOfRangeException

2016-03-19 Thread Cody Koeninger
t take effect. Do you have any ideas of how to handle this? > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/How-to-gracefully-handle-Kafka-OffsetOutOfRangeException-tp26534.html > Sent from the Apache Spark User List mailing list archi