[
https://issues.apache.org/jira/browse/SPARK-12693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15088725#comment-15088725
]
Cody Koeninger commented on SPARK-12693:
----------------------------------------
What is your actual use case for changing the log retention check from the
default 5 minutes down to 100 millis?
As I said originally, if you have such a short retention time, and want to
avoid the issue, use offset reset largest rather than smallest. Even then
you need to make sure your batch time is small enough and you never lag
behind on processing.
On Thu, Jan 7, 2016 at 10:51 PM, Rado Buransky (JIRA) <j...@apache.org>
> OffsetOutOfRangeException cause by retention
> --------------------------------------------
>
> Key: SPARK-12693
> URL: https://issues.apache.org/jira/browse/SPARK-12693
> Project: Spark
> Issue Type: Bug
> Components: Streaming
> Affects Versions: 1.6.0
> Environment: Ubuntu 64bit, Intel i7
> Reporter: Rado Buransky
> Priority: Minor
> Labels: kafka
> Attachments: kafka-log.txt, log.txt
>
>
> I am running Kafka server locally with extremely low retention of 3 seconds
> and with 1 second segmentation. I create direct Kafka stream with
> auto.offset.reset = smallest.
> In case of bad luck (happens actually quite often in my case) the smallest
> offset retrieved druing stream initialization doesn't already exists when
> streaming actually starts.
> Complete source code of the Spark Streaming application is here:
> https://github.com/pygmalios/spark-checkpoint-experience/blob/cb27ab83b7a29e619386b56e68a755d7bd73fc46/src/main/scala/com/pygmalios/sparkCheckpointExperience/spark/SparkApp.scala
> The application ends in an endless loop trying to get that non-existing
> offset and has to be killed. Check attached logs from Spark and also from
> Kafka server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
