[
https://issues.apache.org/jira/browse/HBASE-17970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15988199#comment-15988199
]
Duo Zhang commented on HBASE-17970:
-----------------------------------
And I've found a difference between master and branch-1. On master, we will
start MR cluster first but on branch-1 we will start HBase cluster first. The
difference is that if we start MR cluster first, we will use LocalFileSystem
rather than the mini DFS cluster as the working file system for MR cluster. And
if we use the mini DFS cluster then the problem is gone? The mini dfs cluster
is per instance, different tests will not share the same directory then.
Let me prepare a patch to try it.
> Set yarn.app.mapreduce.am.staging-dir when starting MiniMRCluster
> -----------------------------------------------------------------
>
> Key: HBASE-17970
> URL: https://issues.apache.org/jira/browse/HBASE-17970
> Project: HBase
> Issue Type: Bug
> Components: test
> Reporter: Duo Zhang
> Assignee: Duo Zhang
> Attachments: HBASE-17970.patch
>
>
> As discussed in mailing-list
> https://builds.apache.org/job/HBASE-Flaky-Tests/15333/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.snapshot.TestMobExportSnapshot-output.txt/*view*/
> Test*ExportSnapshot will fail if they are executed concurrently because of
> they share the same staging directory.
> {noformat}
> java.io.FileNotFoundException: File
> file:/tmp/hadoop-yarn/staging/jenkins/.staging/job_1492809330233_0001/job.split
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:606)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:819)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:596)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
> at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:253)
> at
> org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:63)
> at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:361)
> at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:359)
> at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> We need to set yarn.app.mapreduce.am.staging-dir to let hem use different
> directories.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
