GitHub user jiangxb1987 opened a pull request:
https://github.com/apache/spark/pull/20993
[SPARK-23881][CORE][TEST] Fix flaky test
JobCancellationSuite."interruptible iterator of shuffle reader"
## What changes were proposed in this pull request?
The test case JobCancellationSuite."interruptible iterator of shuffle
reader" has been flaky because `KillTask` event is handled asynchronously, so
it can happen that the semaphore is released but the task is still running.
Actually we only have to check the total number of processed elements if
less than the input elements number, so we know the task get cancelled.
## How was this patch tested?
The new test case still fails without the purposed patch, and succeeded in
current master.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jiangxb1987/spark JobCancellationSuite
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20993.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20993
----
commit 965a80f2d02aacff68bae096483c2809e1c9d67e
Author: Xingbo Jiang <xingbo.jiang@...>
Date: 2018-04-06T07:27:23Z
fix flaky test
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
