GitHub user brkyvz opened a pull request:
https://github.com/apache/spark/pull/22478
[SPARK-25472] Don't have legitimate stops of streams cause stream exceptions
## What changes were proposed in this pull request?
Legitimate stops of streams may actually cause an exception to be captured
by stream execution, because the job throws a SparkException regarding job
cancellation during a stop. This PR makes the stop more graceful by swallowing
this cancellation error.
## How was this patch tested?
This is pretty hard to test. The existing tests should make sure that we're
not swallowing other specific SparkExceptions. I've also run the
`KafkaSourceStressForDontFailOnDataLossSuite`100 times, and it didn't fail,
whereas it used to be flaky.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/brkyvz/spark SPARK-25472
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22478.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22478
----
commit dcfc20b291affa6dcc396cdca678ce1f52184dce
Author: Burak Yavuz <brkyvz@...>
Date: 2018-09-19T23:43:57Z
Don't have legitimate stops of streams cause stream exceptions
commit 3b8addb9cf02489978594505470fdd527a35c2a7
Author: Burak Yavuz <brkyvz@...>
Date: 2018-09-19T23:46:47Z
fix scalastyle
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
