GitHub user zsxwing opened a pull request:
https://github.com/apache/spark/pull/12852
Use a fair lock to avoid thread starvation in StreamExecution
## What changes were proposed in this pull request?
Right now `StreamExecution.awaitBatchLock` uses an unfair lock.
`StreamExecution.awaitOffset` may run too long and fail some test because
`StreamExecution. constructNextBatch` keeps getting the lock.
See:
https://amplab.cs.berkeley.edu/jenkins/job/spark-master-test-sbt-hadoop-2.4/865/testReport/junit/org.apache.spark.sql.streaming/FileStreamSourceStressTestSuite/file_source_stress_test/
This PR uses a fair ReentrantLock to resolve the thread starvation issue.
## How was this patch tested?
Modified `FileStreamSourceStressTestSuite.test("file source stress test")`
to run the content 100 times locally. It always fails because of timeout
without this patch.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zsxwing/spark SPARK-15077
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/12852.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #12852
----
commit 53ab492f530c90073f648624db53c3bbe44c062c
Author: Shixiong Zhu <[email protected]>
Date: 2016-05-02T23:46:01Z
Use a fair lock to avoid thread starvation in StreamExecution
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
