[jira] [Commented] (MAPREDUCE-4987) TestMRJobs#testDistributedCache fails on Windows due to unexpected behavior of symlinks
[
https://issues.apache.org/jira/browse/MAPREDUCE-4987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13635825#comment-13635825
]
Bikas Saha commented on MAPREDUCE-4987:
---
I thought that static blocks are executed once per class loader. So I am not
sure why this one would be executed per inner class object creation. In any
case, moving the code to @BeforeClass is a right thing to do in general.
Are the test times similar in Linux and Windows or they are close to timing out
only on Windows. I would be wary of simply doubling the test timeouts.
this probably can be done once instead of multiple times right? I am assuming
this is a slow filesystem operation.
{code}
+if (new File(entry.getKey().toUri().getPath()).isDirectory()) {
THIS
+ newClassPath.append(Path.SEPARATOR);
+}
{code}
btw, there doesnt seem to be a test about explicitly adding local resources to
the classpath in this patch, right?
Finally, this will have to be split into common, mr and yarn jiras+patches,
though we will need a combined patch to get a successful jenkins run. we can
attach the combined patch to the common jira because that will be committed
first.
TestMRJobs#testDistributedCache fails on Windows due to unexpected behavior
of symlinks
---
Key: MAPREDUCE-4987
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4987
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distributed-cache, nodemanager
Affects Versions: 3.0.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: MAPREDUCE-4987.1.patch, MAPREDUCE-4987.2.patch,
MAPREDUCE-4987.3.patch, MAPREDUCE-4987.4.patch, MAPREDUCE-4987.5.patch
On Windows, {{TestMRJobs#testDistributedCache}} fails on an assertion while
checking the length of a symlink. It expects to see the length of the target
of the symlink, but Java 6 on Windows always reports that a symlink has
length 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4987) TestMRJobs#testDistributedCache fails on Windows due to unexpected behavior of symlinks
[
https://issues.apache.org/jira/browse/MAPREDUCE-4987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13609740#comment-13609740
]
Hadoop QA commented on MAPREDUCE-4987:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12574912/MAPREDUCE-4987.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 tests included appear to have a timeout.{color}
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3452//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3452//console
This message is automatically generated.
TestMRJobs#testDistributedCache fails on Windows due to unexpected behavior
of symlinks
---
Key: MAPREDUCE-4987
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4987
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distributed-cache, nodemanager
Affects Versions: 3.0.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: MAPREDUCE-4987.1.patch, MAPREDUCE-4987.2.patch,
MAPREDUCE-4987.3.patch
On Windows, {{TestMRJobs#testDistributedCache}} fails on an assertion while
checking the length of a symlink. It expects to see the length of the target
of the symlink, but Java 6 on Windows always reports that a symlink has
length 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4987) TestMRJobs#testDistributedCache fails on Windows due to unexpected behavior of symlinks
[
https://issues.apache.org/jira/browse/MAPREDUCE-4987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13606136#comment-13606136
]
Ivan Mitic commented on MAPREDUCE-4987:
---
bq. These tests were running pretty close to the timeouts in my environment,
even on Mac. Here is a new patch that increases the timeouts.
Thanks, I verified that the test now passes, +1 on the patch
TestMRJobs#testDistributedCache fails on Windows due to unexpected behavior
of symlinks
---
Key: MAPREDUCE-4987
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4987
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distributed-cache, nodemanager
Affects Versions: 3.0.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: MAPREDUCE-4987.1.patch, MAPREDUCE-4987.2.patch
On Windows, {{TestMRJobs#testDistributedCache}} fails on an assertion while
checking the length of a symlink. It expects to see the length of the target
of the symlink, but Java 6 on Windows always reports that a symlink has
length 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4987) TestMRJobs#testDistributedCache fails on Windows due to unexpected behavior of symlinks
[
https://issues.apache.org/jira/browse/MAPREDUCE-4987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13606146#comment-13606146
]
Hadoop QA commented on MAPREDUCE-4987:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12574326/MAPREDUCE-4987.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 tests included appear to have a timeout.{color}
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser
org.apache.hadoop.mapreduce.lib.db.TestDataDrivenDBInputFormat
org.apache.hadoop.mapred.TestFieldSelection
org.apache.hadoop.mapreduce.TestLocalRunner
org.apache.hadoop.mapred.TestUserDefinedCounters
org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution
org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
org.apache.hadoop.mapreduce.TestMROutputFormat
org.apache.hadoop.mapred.TestLineRecordReader
org.apache.hadoop.mapreduce.lib.fieldsel.TestMRFieldSelection
org.apache.hadoop.mapred.TestMiniMRClasspath
org.apache.hadoop.mapreduce.lib.map.TestMultithreadedMapper
org.apache.hadoop.mapred.TestReduceFetchFromPartialMem
org.apache.hadoop.mapred.lib.TestChainMapReduce
org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
org.apache.hadoop.mapreduce.TestMapReduce
org.apache.hadoop.mapred.TestLazyOutput
org.apache.hadoop.mapreduce.lib.join.TestJoinDatamerge
org.apache.hadoop.mapred.lib.TestKeyFieldBasedComparator
org.apache.hadoop.mapred.lib.TestMultithreadedMapRunner
org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService
org.apache.hadoop.mapreduce.TestChild
org.apache.hadoop.mapred.lib.TestMultipleOutputs
org.apache.hadoop.mapred.TestJavaSerialization
org.apache.hadoop.mapreduce.lib.input.TestLineRecordReader
org.apache.hadoop.mapreduce.security.ssl.TestEncryptedShuffle
org.apache.hadoop.mapreduce.lib.output.TestMRMultipleOutputs
org.apache.hadoop.mapred.TestClusterMapReduceTestCase
org.apache.hadoop.mapred.TestCollect
org.apache.hadoop.fs.slive.TestSlive
org.apache.hadoop.mapred.join.TestDatamerge
org.apache.hadoop.mapreduce.TestMapCollection
org.apache.hadoop.mapred.TestMiniMRClientCluster
org.apache.hadoop.fs.TestDFSIO
org.apache.hadoop.mapred.TestMapRed
org.apache.hadoop.mapred.TestFileOutputFormat
org.apache.hadoop.mapreduce.TestValueIterReset
org.apache.hadoop.mapreduce.v2.TestMROldApiJobs
org.apache.hadoop.mapred.TestJobCounters
org.apache.hadoop.fs.TestFileSystem
org.apache.hadoop.mapreduce.v2.TestUberAM
org.apache.hadoop.conf.TestNoDefaultsJobConf
org.apache.hadoop.mapred.TestReporter
org.apache.hadoop.mapred.TestJobName
org.apache.hadoop.mapreduce.lib.partition.TestMRKeyFieldBasedComparator
org.apache.hadoop.mapreduce.lib.chain.TestChainErrors
org.apache.hadoop.mapreduce.lib.chain.TestSingleElementChain
org.apache.hadoop.mapreduce.lib.input.TestMultipleInputs
org.apache.hadoop.mapreduce.v2.TestMRJobs
org.apache.hadoop.mapred.TestComparators
org.apache.hadoop.mapreduce.lib.chain.TestMapReduceChain
[jira] [Commented] (MAPREDUCE-4987) TestMRJobs#testDistributedCache fails on Windows due to unexpected behavior of symlinks
[
https://issues.apache.org/jira/browse/MAPREDUCE-4987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13606392#comment-13606392
]
Chris Nauroth commented on MAPREDUCE-4987:
--
The new test failures appear to be unrelated, and I believe that they were
introduced by the patch on MAPREDUCE-5028. I'm following up on that jira.
TestMRJobs#testDistributedCache fails on Windows due to unexpected behavior
of symlinks
---
Key: MAPREDUCE-4987
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4987
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distributed-cache, nodemanager
Affects Versions: 3.0.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: MAPREDUCE-4987.1.patch, MAPREDUCE-4987.2.patch
On Windows, {{TestMRJobs#testDistributedCache}} fails on an assertion while
checking the length of a symlink. It expects to see the length of the target
of the symlink, but Java 6 on Windows always reports that a symlink has
length 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4987) TestMRJobs#testDistributedCache fails on Windows due to unexpected behavior of symlinks
[
https://issues.apache.org/jira/browse/MAPREDUCE-4987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13606524#comment-13606524
]
Chris Nauroth commented on MAPREDUCE-4987:
--
This patch may cause a merge conflict with my patch on YARN-488, depending on
which one gets committed first. After one of these gets committed, I'll check
and rebase if necessary.
TestMRJobs#testDistributedCache fails on Windows due to unexpected behavior
of symlinks
---
Key: MAPREDUCE-4987
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4987
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distributed-cache, nodemanager
Affects Versions: 3.0.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: MAPREDUCE-4987.1.patch, MAPREDUCE-4987.2.patch
On Windows, {{TestMRJobs#testDistributedCache}} fails on an assertion while
checking the length of a symlink. It expects to see the length of the target
of the symlink, but Java 6 on Windows always reports that a symlink has
length 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4987) TestMRJobs#testDistributedCache fails on Windows due to unexpected behavior of symlinks
[
https://issues.apache.org/jira/browse/MAPREDUCE-4987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13605755#comment-13605755
]
Ivan Mitic commented on MAPREDUCE-4987:
---
Thanks Chris, patch looks good overall, +1
I noticed that TestMRJobs fails with timeout on my box. Does it consistently
succeed for you? I see that the timeouts are set quite high (5 minutes). This
is non blocking, I'll take a look when I get a chance, just thought I'll ask.
TestFileUtil passes fine.
TestMRJobs#testDistributedCache fails on Windows due to unexpected behavior
of symlinks
---
Key: MAPREDUCE-4987
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4987
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distributed-cache, nodemanager
Affects Versions: 3.0.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: MAPREDUCE-4987.1.patch
On Windows, {{TestMRJobs#testDistributedCache}} fails on an assertion while
checking the length of a symlink. It expects to see the length of the target
of the symlink, but Java 6 on Windows always reports that a symlink has
length 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4987) TestMRJobs#testDistributedCache fails on Windows due to unexpected behavior of symlinks
[
https://issues.apache.org/jira/browse/MAPREDUCE-4987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13605760#comment-13605760
]
Chris Nauroth commented on MAPREDUCE-4987:
--
Thanks, Ivan. TestMRJobs has been passing consistently for me. I haven't been
experiencing timeout failures. Is there a particular test within the suite
that is timing out consistently for you?
TestMRJobs#testDistributedCache fails on Windows due to unexpected behavior
of symlinks
---
Key: MAPREDUCE-4987
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4987
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distributed-cache, nodemanager
Affects Versions: 3.0.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: MAPREDUCE-4987.1.patch
On Windows, {{TestMRJobs#testDistributedCache}} fails on an assertion while
checking the length of a symlink. It expects to see the length of the target
of the symlink, but Java 6 on Windows always reports that a symlink has
length 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4987) TestMRJobs#testDistributedCache fails on Windows due to unexpected behavior of symlinks
[
https://issues.apache.org/jira/browse/MAPREDUCE-4987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13605803#comment-13605803
]
Ivan Mitic commented on MAPREDUCE-4987:
---
bq. Is there a particular test within the suite that is timing out consistently
for you?
I tried to remove all timeouts from the test and it is passing now. Let me find
the exact test case that is causing problems.
TestMRJobs#testDistributedCache fails on Windows due to unexpected behavior
of symlinks
---
Key: MAPREDUCE-4987
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4987
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distributed-cache, nodemanager
Affects Versions: 3.0.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: MAPREDUCE-4987.1.patch
On Windows, {{TestMRJobs#testDistributedCache}} fails on an assertion while
checking the length of a symlink. It expects to see the length of the target
of the symlink, but Java 6 on Windows always reports that a symlink has
length 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4987) TestMRJobs#testDistributedCache fails on Windows due to unexpected behavior of symlinks
[
https://issues.apache.org/jira/browse/MAPREDUCE-4987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13605921#comment-13605921
]
Ivan Mitic commented on MAPREDUCE-4987:
---
Hi Chris, I played around with the test a bit, and the following tests fail
because of the timeout on my box: testRandomWriter, testFailingMapper,
testSleepJobWithSecurityOn.
TestMRJobs#testDistributedCache fails on Windows due to unexpected behavior
of symlinks
---
Key: MAPREDUCE-4987
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4987
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distributed-cache, nodemanager
Affects Versions: 3.0.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: MAPREDUCE-4987.1.patch
On Windows, {{TestMRJobs#testDistributedCache}} fails on an assertion while
checking the length of a symlink. It expects to see the length of the target
of the symlink, but Java 6 on Windows always reports that a symlink has
length 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4987) TestMRJobs#testDistributedCache fails on Windows due to unexpected behavior of symlinks
[
https://issues.apache.org/jira/browse/MAPREDUCE-4987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13605988#comment-13605988
]
Ivan Mitic commented on MAPREDUCE-4987:
---
bq. I played around with the test a bit, and the following tests fail because
of the timeout on my box: testRandomWriter, testFailingMapper,
testSleepJobWithSecurityOn.
Increasing the test timeouts by the factor of 2 helped make the tests pass on
my box.
TestMRJobs#testDistributedCache fails on Windows due to unexpected behavior
of symlinks
---
Key: MAPREDUCE-4987
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4987
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distributed-cache, nodemanager
Affects Versions: 3.0.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: MAPREDUCE-4987.1.patch
On Windows, {{TestMRJobs#testDistributedCache}} fails on an assertion while
checking the length of a symlink. It expects to see the length of the target
of the symlink, but Java 6 on Windows always reports that a symlink has
length 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4987) TestMRJobs#testDistributedCache fails on Windows due to unexpected behavior of symlinks
[
https://issues.apache.org/jira/browse/MAPREDUCE-4987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13603700#comment-13603700
]
Hadoop QA commented on MAPREDUCE-4987:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12573897/MAPREDUCE-4987.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 tests included appear to have a timeout.{color}
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3420//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3420//console
This message is automatically generated.
TestMRJobs#testDistributedCache fails on Windows due to unexpected behavior
of symlinks
---
Key: MAPREDUCE-4987
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4987
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distributed-cache, nodemanager
Affects Versions: 3.0.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: MAPREDUCE-4987.1.patch
On Windows, {{TestMRJobs#testDistributedCache}} fails on an assertion while
checking the length of a symlink. It expects to see the length of the target
of the symlink, but Java 6 on Windows always reports that a symlink has
length 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4987) TestMRJobs#testDistributedCache fails on Windows due to unexpected behavior of symlinks
[
https://issues.apache.org/jira/browse/MAPREDUCE-4987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13603718#comment-13603718
]
Arpit Agarwal commented on MAPREDUCE-4987:
--
+1
Chris explained to me offline about the change in
FileUtil#createJarWithClassPath. Quoting here since I found it helpful to
understand the change.
{quote}
In the method sanitizeEnv, you'll see that nodemanager does various things to
set up a new environment for the container to be launched. The final state of
this environment will be different from the environment of the currently
running process (the nodemanager itself).
The most glaring problem with this bug was the setting of PWD to the new
container work directory. There are various classpath entries for the
distributed cache files that are of the form $PWD/file on Mac or %PWD%/file on
Windows, and FileUtil#createJarWithClassPath needs to expand this to
container_dir/file. Without this change, the variable expansion would be
incorrect: nodemanager_working_dir/file on Mac or just /file on Windows
(since Windows doesn't intrinsically have %PWD% defined until nodemanager sets
it in sanitizeEnv).
{quote}
TestMRJobs#testDistributedCache fails on Windows due to unexpected behavior
of symlinks
---
Key: MAPREDUCE-4987
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4987
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distributed-cache, nodemanager
Affects Versions: 3.0.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: MAPREDUCE-4987.1.patch
On Windows, {{TestMRJobs#testDistributedCache}} fails on an assertion while
checking the length of a symlink. It expects to see the length of the target
of the symlink, but Java 6 on Windows always reports that a symlink has
length 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4987) TestMRJobs#testDistributedCache fails on Windows due to unexpected behavior of symlinks
[
https://issues.apache.org/jira/browse/MAPREDUCE-4987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573742#comment-13573742
]
Vinod Kumar Vavilapalli commented on MAPREDUCE-4987:
But symlinks CAN be used for functional purposes i.e linking to libraries etc.
? If so we can just do the platform check in the test-case.
TestMRJobs#testDistributedCache fails on Windows due to unexpected behavior
of symlinks
---
Key: MAPREDUCE-4987
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4987
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distributed-cache, nodemanager
Affects Versions: trunk-win
Reporter: Chris Nauroth
On Windows, {{TestMRJobs#testDistributedCache}} fails on an assertion while
checking the length of a symlink. It expects to see the length of the target
of the symlink, but Java 6 on Windows always reports that a symlink has
length 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4987) TestMRJobs#testDistributedCache fails on Windows due to unexpected behavior of symlinks
[
https://issues.apache.org/jira/browse/MAPREDUCE-4987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13574097#comment-13574097
]
Ivan Mitic commented on MAPREDUCE-4987:
---
Thanks for reporting this Chris.
bq. But symlinks CAN be used for functional purposes i.e linking to libraries
etc. ?
Hi Vinod. I believe we'll have to port the branch-1-win semantic to trunk to
properly support symlinks on both Java6 and Java7 on Windows. Yes, symlinks can
be created to point to folders and files, however, Java6 does not interpret
them correctly. We've seen so many issues with symlinks on Java6, and the only
option that worked fine (and was signed off on) is to do a file copy in case of
Java6. HADOOP-9061 talks about some of these problems.
bq. If so we can just do the platform check in the test-case.
We also initially thought this would be fine (you can check thru branch-1-win
history :)). However, the real problem comes when someone tries to access the
symlink thru Java APIs. Examples of problems are, File#length on symlinks
returns zero. This means that RLFS does not work on top of symlinks.
Additionally, File#renameTo on symlink renames the target file instead of the
symlink (really strange I know :)).
Hope this helps
TestMRJobs#testDistributedCache fails on Windows due to unexpected behavior
of symlinks
---
Key: MAPREDUCE-4987
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4987
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distributed-cache, nodemanager
Affects Versions: trunk-win
Reporter: Chris Nauroth
On Windows, {{TestMRJobs#testDistributedCache}} fails on an assertion while
checking the length of a symlink. It expects to see the length of the target
of the symlink, but Java 6 on Windows always reports that a symlink has
length 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4987) TestMRJobs#testDistributedCache fails on Windows due to unexpected behavior of symlinks
[
https://issues.apache.org/jira/browse/MAPREDUCE-4987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573191#comment-13573191
]
Chris Nauroth commented on MAPREDUCE-4987:
--
In {{TestMRJobs}}, the last assertion in this code fragment fails:
{code}
// Check lengths of the files
MapString, Path filesMap = pathsToMap(files);
Assert.assertTrue(filesMap.containsKey(distributed.first.symlink));
Assert.assertEquals(1, localFs.getFileStatus(
filesMap.get(distributed.first.symlink)).getLen());
{code}
This is a known issue with Java 6 on Windows. It always reports a symlink as
having length zero instead of the length of the target file. This problem was
fixed on branch-1-win in HADOOP-9061 by detecting if the runtime environment is
Windows + Java 6, and if so, copying files into the symlink location instead of
actually creating a symlink. Applying the same logic to branch-trunk-win will
require different code changes. In YARN, the symlinks for the distributed
cache get generated by the container launch scripts. See
{{ContainerLaunch#WindowsShellScriptBuilder#link}}.
TestMRJobs#testDistributedCache fails on Windows due to unexpected behavior
of symlinks
---
Key: MAPREDUCE-4987
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4987
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distributed-cache, nodemanager
Affects Versions: trunk-win
Reporter: Chris Nauroth
On Windows, {{TestMRJobs#testDistributedCache}} fails on an assertion while
checking the length of a symlink. It expects to see the length of the target
of the symlink, but Java 6 on Windows always reports that a symlink has
length 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
