[
https://issues.apache.org/jira/browse/SPARK-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Patrick Wendell updated SPARK-6222:
-----------------------------------
Fix Version/s: 1.4.0
1.3.1
> [STREAMING] All data may not be recovered from WAL when driver is killed
> ------------------------------------------------------------------------
>
> Key: SPARK-6222
> URL: https://issues.apache.org/jira/browse/SPARK-6222
> Project: Spark
> Issue Type: Bug
> Components: Streaming
> Affects Versions: 1.3.0
> Reporter: Hari Shreedharan
> Priority: Blocker
> Fix For: 1.3.1, 1.4.0
>
> Attachments: AfterPatch.txt, CleanWithoutPatch.txt, SPARK-6122.patch
>
>
> When testing for our next release, our internal tests written by [~wypoon]
> caught a regression in Spark Streaming between 1.2.0 and 1.3.0. The test runs
> FlumePolling stream to read data from Flume, then kills the Application
> Master. Once YARN restarts it, the test waits until no more data is to be
> written and verifies the original against the data on HDFS. This was passing
> in 1.2.0, but is failing now.
> Since the test ties into Cloudera's internal infrastructure and build
> process, it cannot be directly run on an Apache build. But I have been
> working on isolating the commit that may have caused the regression. I have
> confirmed that it was caused by SPARK-5147 (PR #
> [4149|https://github.com/apache/spark/pull/4149]). I confirmed this several
> times using the test and the failure is consistently reproducible.
> To re-confirm, I reverted just this one commit (and Clock consolidation one
> to avoid conflicts), and the issue was no longer reproducible.
> Since this is a data loss issue, I believe this is a blocker for Spark 1.3.0
> /cc [~tdas], [~pwendell]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
