GitHub user efimpoberezkin opened a pull request:
https://github.com/apache/spark/pull/20936
[SPARK-23503][Structured Streaming] Enforcing sequencing of committed
epochs for Continuous Execution
## What changes were proposed in this pull request?
Made changes to EpochCoordinator so that it enforces a commit order. In
case a message for epoch n is lost and epoch (n + 1) is ready for commit before
epoch n is, epoch (n + 1) will wait for epoch n to be committed first.
## How was this patch tested?
Existing tests in ContinuousSuite.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/efimpoberezkin/spark
pr/sequence-commited-epochs
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20936.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20936
----
commit 35669933b655e7b9b291786097f41ee6359e02e8
Author: Efim Poberezkin <efim@...>
Date: 2018-03-29T14:24:06Z
[SPARK-23503][Structured Streaming] Continuous Execution should sequence
committed epochs
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
