[ https://issues.apache.org/jira/browse/SPARK-22963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-22963. ---------------------------------- Resolution: Incomplete > Make failure recovery global and automatic for continuous processing. > --------------------------------------------------------------------- > > Key: SPARK-22963 > URL: https://issues.apache.org/jira/browse/SPARK-22963 > Project: Spark > Issue Type: Improvement > Components: Structured Streaming > Affects Versions: 2.3.0 > Reporter: Jose Torres > Priority: Major > Labels: bulk-closed > > Spark native task restarts don't work well for continuous processing. They > will process all data from the task's original start offset - even data which > has already been committed. This is not semantically incorrect under at least > once semantics, but it's awkward and bad. > Fortunately, they're also not necessary; the central coordinator can restart > every task from the checkpointed offsets without losing much. So we should > make that happen automatically on task failures instead. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org