GitHub user xuanyuanking opened a pull request:
https://github.com/apache/spark/pull/20675
[SPARK-23033][SS][Follow Up] Task level retry for continuous processing
## What changes were proposed in this pull request?
Here we want to reimplement the task level retry for continuous processing,
changes include:
1. Add a new `EpochCoordinatorMessage` named `GetLastEpochAndOffset`, it is
used for getting last epoch and offset of particular partition while task
restarted.
2. Add function setOffset for `ContinuousDataReader`, it supported
BaseReader can restart from given offset.
## How was this patch tested?
Add new UT in `ContinuousSuite` and new `StreamAction` named
`CheckAnswerRowsContainsOnlyOnce` for more accurate result checking.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/xuanyuanking/spark SPARK-23033
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20675.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20675
----
commit 21f574e2a3ad3c8e68b92776d2a141d7fcb90502
Author: Yuanjian Li <xyliyuanjian@...>
Date: 2018-02-26T07:27:10Z
[SPARK-23033][SS][Follow Up] Task level retry for continuous processing
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
