GitHub user brkyvz opened a pull request:
https://github.com/apache/spark/pull/9953
[STREAMING][FLAKY-TEST] Catch execution context race condition in
`FileBasedWriteAheadLog.close()`
There is a race condition in `FileBasedWriteAheadLog.close()`, where if
delete's of old log files are in progress, the write ahead log may close, and
result in a `RejectedExecutionException`. This is okay, and should be handled
gracefully.
Example test failures:
https://amplab.cs.berkeley.edu/jenkins/job/Spark-1.6-SBT/AMPLAB_JENKINS_BUILD_PROFILE=hadoop1.0,label=spark-test/95/testReport/junit/org.apache.spark.streaming.util/BatchedWriteAheadLogWithCloseFileAfterWriteSuite/BatchedWriteAheadLog___clean_old_logs/
The reason the test fails is in `afterEach`, `writeAheadLog.close` is
called, and there may still be async deletes in flight.
@tdas @zsxwing
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/brkyvz/spark flaky-ss
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/9953.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #9953
----
commit 890e6dc33b35866971e9fc896b06c8ed43b0933a
Author: Burak Yavuz <[email protected]>
Date: 2015-11-25T02:11:48Z
catch execution context race condition
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
