GitHub user zsxwing opened a pull request:
https://github.com/apache/spark/pull/11940
[SPARK-14131][SQL]Add a workaround for HADOOP-10622 to fix
DataFrameReaderWriterSuite
## What changes were proposed in this pull request?
There is a potential dead-lock in Hadoop Shell.runCommand before 2.5.0
([HADOOP-10622](https://issues.apache.org/jira/browse/HADOOP-10622)). If we
interrupt some thread running Shell.runCommand, we may hit this issue.
This PR adds some protecion to prevent from interrupting the
microBatchThread when we may run into Shell.runCommand. There are two places
will call Shell.runCommand now:
- offsetLog.add
- FileStreamSource.getOffset
They will create a file using HDFS API and call Shell.runCommand to set the
file permission.
## How was this patch tested?
Existing unit tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zsxwing/spark workaround-for-HADOOP-10622
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11940.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11940
----
commit 8c0f5d18b69674ba0e214c10ca2588a0f20cd008
Author: Shixiong Zhu <[email protected]>
Date: 2016-03-24T17:42:57Z
Add a workaround for HADOOP-10622 to fix DataFrameReaderWriterSuite
There is a potential dead-lock in Hadoop Shell.runCommand before 2.5.0
([HADOOP-10622](https://issues.apache.org/jira/browse/HADOOP-10622)). If we
interrupt some thread running Shell.runCommand, we may hit this issue.
This PR adds some protecion to prevent from interrupting the
microBatchThread when we may run into Shell.runCommand. There are two places
will call Shell.runCommand now:
- offsetLog.add
- FileStreamSource.getOffset
They will create a file using HDFS API and call Shell.runCommand to set the
file permission.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
