GitHub user jose-torres opened a pull request:
https://github.com/apache/spark/pull/21384
[SPARK-23416][SS] Add a specific stop method for ContinuousExecution.
## What changes were proposed in this pull request?
Add a specific stop method for ContinuousExecution. The previous
StreamExecution.stop() method had a race condition as applied to continuous
processing: if the cancellation was round-tripped to the driver too quickly,
the generic SparkException it caused would be reported as the query death
cause. We earlier decided that SparkException should not be added to the
StreamExecution.isInterruptionException() whitelist, so we need to ensure this
never happens instead.
## How was this patch tested?
Existing tests. I could consistently reproduce the previous flakiness by
putting Thread.sleep(1000) between the first job cancellation and thread
interruption in StreamExecution.stop().
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jose-torres/spark fixKafka
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21384.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21384
----
commit 61f691d3112e7b2978e845bf16680f6fbf27d4d5
Author: Jose Torres <torres.joseph.f+github@...>
Date: 2018-05-21T19:03:38Z
specific stop method
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
