[jira] [Commented] (YARN-449) MRAppMaster classpath not set properly for unit tests in downstream projects
[
https://issues.apache.org/jira/browse/YARN-449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13600169#comment-13600169
]
Ted Yu commented on YARN-449:
-
On flubber, I installed protoc 2.4.1 but couldn't use it:
{code}
$ protoc --version
protoc: error while loading shared libraries: libprotobuf.so.7: cannot open
shared object file: No such file or directory
{code}
I applied minimr_randomdir-branch2.txt locally and ran the following command:
mt -Dhadoop.profile=2.0
-Dtest=org.apache.hadoop.hbase.mapreduce.TestTableMapReduce#testMultiRegionTable
The test passed.
MRAppMaster classpath not set properly for unit tests in downstream projects
Key: YARN-449
URL: https://issues.apache.org/jira/browse/YARN-449
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.0.3-alpha
Reporter: Siddharth Seth
Priority: Blocker
Attachments: 7904-v5.txt, hbase-7904-v3.txt,
hbase-TestHFileOutputFormat-wip.txt, hbase-TestingUtility-wip.txt,
minimr_randomdir-branch2.txt
Post YARN-429, unit tests for HBase continue to fail since the classpath for
the MRAppMaster is not being set correctly.
Reverting YARN-129 may fix this, but I'm not sure that's the correct
solution. My guess is, as Alexandro pointed out in YARN-129, maven
classloader magic is messing up java.class.path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-449) MRAppMaster classpath not set properly for unit tests in downstream projects
[
https://issues.apache.org/jira/browse/YARN-449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13600416#comment-13600416
]
Ted Yu commented on YARN-449:
-
From
https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/442/testReport/org.apache.hadoop.hbase.mapreduce/TestRowCounter/testRowCounterNoColumn/
(hadoop-2.0.2-alpha was used):
{code}
2013-03-12 05:44:18,139 WARN [DeletionService #1]
nodemanager.DefaultContainerExecutor(276): delete returned false for path:
[/home/jenkins/jenkins-slave/workspace/HBase-TRUNK-on-Hadoop-2.0.0/trunk/hbase-server/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-localDir-nm-1_0/usercache/jenkins/appcache/application_1363067018215_0001/container_1363067018215_0001_01_01]
2013-03-12 05:44:18,145 WARN [AsyncDispatcher event handler]
nodemanager.NMAuditLogger(150): USER=jenkins OPERATION=Container Finished
- Failed TARGET=ContainerImplRESULT=FAILURE DESCRIPTION=Container failed
with state: LOCALIZATION_FAILEDAPPID=application_1363067018215_0001
CONTAINERID=container_1363067018215_0001_01_01
2013-03-12 05:44:18,141 WARN [DeletionService #0]
nodemanager.DefaultContainerExecutor(276): delete returned false for path:
[/home/jenkins/jenkins-slave/workspace/HBase-TRUNK-on-Hadoop-2.0.0/trunk/hbase-server/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-localDir-nm-1_1/usercache/jenkins/appcache/application_1363067018215_0001/container_1363067018215_0001_01_01]
2013-03-12 05:44:18,220 WARN [DeletionService #0]
nodemanager.DefaultContainerExecutor(276): delete returned false for path:
[/home/jenkins/jenkins-slave/workspace/HBase-TRUNK-on-Hadoop-2.0.0/trunk/hbase-server/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-localDir-nm-1_2/usercache/jenkins/appcache/application_1363067018215_0001/container_1363067018215_0001_01_01]
2013-03-12 05:44:18,220 WARN [DeletionService #0]
nodemanager.DefaultContainerExecutor(276): delete returned false for path:
[/home/jenkins/jenkins-slave/workspace/HBase-TRUNK-on-Hadoop-2.0.0/trunk/hbase-server/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-localDir-nm-1_3/usercache/jenkins/appcache/application_1363067018215_0001/container_1363067018215_0001_01_01]
2013-03-12 05:44:18,865 WARN [AsyncDispatcher event handler]
resourcemanager.RMAuditLogger(255): USER=jenkins OPERATION=Application
Finished - Failed TARGET=RMAppManager RESULT=FAILURE DESCRIPTION=App
failed with state: FAILED PERMISSIONS=Application
application_1363067018215_0001 failed 1 times due to AM Container for
appattempt_1363067018215_0001_01 exited with exitCode: -1000 due to:
RemoteTrace:
java.io.IOException: Unable to rename file:
[/home/jenkins/jenkins-slave/workspace/HBase-TRUNK-on-Hadoop-2.0.0/trunk/hbase-server/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-localDir-nm-1_1/usercache/jenkins/filecache/5596410335910248146_tmp/hadoop-262140332608909552.jar.tmp]
to
[/home/jenkins/jenkins-slave/workspace/HBase-TRUNK-on-Hadoop-2.0.0/trunk/hbase-server/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-localDir-nm-1_1/usercache/jenkins/filecache/5596410335910248146_tmp/hadoop-262140332608909552.jar]
at org.apache.hadoop.yarn.util.FSDownload.unpack(FSDownload.java:162)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:205)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:50)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
at LocalTrace:
org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl:
Unable to rename file:
[/home/jenkins/jenkins-slave/workspace/HBase-TRUNK-on-Hadoop-2.0.0/trunk/hbase-server/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-localDir-nm-1_1/usercache/jenkins/filecache/5596410335910248146_tmp/hadoop-262140332608909552.jar.tmp]
to
[/home/jenkins/jenkins-slave/workspace/HBase-TRUNK-on-Hadoop-2.0.0/trunk/hbase-server/target/org.apache.hadoop.mapred.MiniMRCluster/org.apache.hadoop.mapred.MiniMRCluster-localDir-nm-1_1/usercache/jenkins/filecache/5596410335910248146_tmp/hadoop-262140332608909552.jar]
at
[jira] [Commented] (YARN-449) MRAppMaster classpath not set properly for unit tests in downstream projects
[
https://issues.apache.org/jira/browse/YARN-449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13600578#comment-13600578
]
Siddharth Seth commented on YARN-449:
-
{code}
013-03-12 18:53:39,275 WARN [Container Monitor]
monitor.ContainersMonitorImpl$MonitoringThread(444): Container
[pid=8438,containerID=container_1363114400920_0001_01_01] is running beyond
virtual memory limits. Current usage: 217.9 MB of 2 GB physical memory used;
6.5 GB of 4.2 GB virtual memory used. Killing container.
Dump of the process-tree for container_1363114400920_0001_01_01 :
|- PID PPID PGRPID SESSID CMD_NAME USER_MODE_TIME(MILLIS)
SYSTEM_TIME(MILLIS) VMEM_USAGE(BYTES) RSSMEM_USAGE(PAGES) FULL_CMD_LINE
|- 8438 7023 8438 8438 (bash) 1 0 108650496 310 /bin/bash -c
/usr/lib/jvm/java-1.6.0-sun-1.6.0.37.x86_64/bin/java
-Dlog4j.configuration=container-log4j.properties
-Dyarn.app.mapreduce.container.log.dir=/trunk/hbase-server/target/org.apache.hadoop.mapred.MiniMRCluster_1035429065/org.apache.hadoop.mapred.MiniMRCluster_1035429065-logDir-nm-1_2/application_1363114400920_0001/container_1363114400920_0001_01_01
-Dyarn.app.mapreduce.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
-Xmx1024m org.apache.hadoop.mapreduce.v2.app.MRAppMaster
1gt;/trunk/hbase-server/target/org.apache.hadoop.mapred.MiniMRCluster_1035429065/org.apache.hadoop.mapred.MiniMRCluster_1035429065-logDir-nm-1_2/application_1363114400920_0001/container_1363114400920_0001_01_01/stdout
2gt;/trunk/hbase-server/target/org.apache.hadoop.mapred.MiniMRCluster_1035429065/org.apache.hadoop.mapred.MiniMRCluster_1035429065-logDir-nm-1_2/application_1363114400920_0001/container_1363114400920_0001_01_01/stderr
{code}
This is what caused TestRowCounter to fail in the linux env. Not sure why the
Vmem is going that high. The hadoop-1 default config likely disables this
monitoring.
At this point there seem to be solutions for the original prolem the jira was
opened for, and this is really re-purposed to get HBase unit tests working with
Hadoop 2. Changing the title accordingly.
MRAppMaster classpath not set properly for unit tests in downstream projects
Key: YARN-449
URL: https://issues.apache.org/jira/browse/YARN-449
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.0.3-alpha
Reporter: Siddharth Seth
Priority: Blocker
Attachments: 7904-v5.txt, hbase-7904-v3.txt,
hbase-TestHFileOutputFormat-wip.txt, hbase-TestingUtility-wip.txt,
minimr_randomdir-branch2.txt
Post YARN-429, unit tests for HBase continue to fail since the classpath for
the MRAppMaster is not being set correctly.
Reverting YARN-129 may fix this, but I'm not sure that's the correct
solution. My guess is, as Alexandro pointed out in YARN-129, maven
classloader magic is messing up java.class.path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-449) MRAppMaster classpath not set properly for unit tests in downstream projects
[ https://issues.apache.org/jira/browse/YARN-449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599719#comment-13599719 ] Siddharth Seth commented on YARN-449: - Ted, are all these tests passing against 2.0.2-alpha ? That seems like the first thing that should be addressed - did something changed in 2.0.3 which causes them to fail. Also, does HBase run tests in parallel ? Three different tests failing in different test-runs on the build box could be caused by this. Looking at the way MiniMRCluster works, the work-dir ends up being the same across tests - so that can be problematic. (Attaching a wip patch for this which I've tested on a couple of HBase tests - running individually). Would it be possible for you to try the patch on a box where you're seeing failures. From running the three tests locally - TestImportExport consistently hangs. The other two seem to work ok. MRAppMaster classpath not set properly for unit tests in downstream projects Key: YARN-449 URL: https://issues.apache.org/jira/browse/YARN-449 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Siddharth Seth Priority: Blocker Attachments: 7904-v5.txt, hbase-7904-v3.txt, hbase-TestHFileOutputFormat-wip.txt, hbase-TestingUtility-wip.txt Post YARN-429, unit tests for HBase continue to fail since the classpath for the MRAppMaster is not being set correctly. Reverting YARN-129 may fix this, but I'm not sure that's the correct solution. My guess is, as Alexandro pointed out in YARN-129, maven classloader magic is messing up java.class.path. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-449) MRAppMaster classpath not set properly for unit tests in downstream projects
[ https://issues.apache.org/jira/browse/YARN-449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13599733#comment-13599733 ] Ted Yu commented on YARN-449: - Test failure couldn't be reproduced on Mac. On flubber machine, protoc 2.5 gave me error. I will try to build with your patch on flubber. Thanks MRAppMaster classpath not set properly for unit tests in downstream projects Key: YARN-449 URL: https://issues.apache.org/jira/browse/YARN-449 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Siddharth Seth Priority: Blocker Attachments: 7904-v5.txt, hbase-7904-v3.txt, hbase-TestHFileOutputFormat-wip.txt, hbase-TestingUtility-wip.txt, minimr_randomdir-branch2.txt Post YARN-429, unit tests for HBase continue to fail since the classpath for the MRAppMaster is not being set correctly. Reverting YARN-129 may fix this, but I'm not sure that's the correct solution. My guess is, as Alexandro pointed out in YARN-129, maven classloader magic is messing up java.class.path. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-449) MRAppMaster classpath not set properly for unit tests in downstream projects
[ https://issues.apache.org/jira/browse/YARN-449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598288#comment-13598288 ] Ted Yu commented on YARN-449: - I tried 3 methods which resulted in different mapreduce jobs to fail. 1. copy JobConf returned from MiniMRCluster.getJobConf() to HBaseTestingUtility.conf here is test result based on hadoop 1.0 (org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat failed): https://builds.apache.org/job/PreCommit-HBASE-Build/4719//testReport/ 2. extract config entries from MiniMRCluster.getJobConf() which are absent in HBaseTestingUtility.conf and put them in HBaseTestingUtility.conf mapreduce.TestRowCounter failed building against hadoop 2.0: https://builds.apache.org/job/PreCommit-HBASE-Build/4743//testReport/ 3. extract all config entries from MiniMRCluster.getJobConf() and put them in HBaseTestingUtility.conf mapreduce.TestImportExport failed building against hadoop 2.0: https://builds.apache.org/job/PreCommit-HBASE-Build/4746//testReport/ MRAppMaster classpath not set properly for unit tests in downstream projects Key: YARN-449 URL: https://issues.apache.org/jira/browse/YARN-449 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Siddharth Seth Priority: Blocker Attachments: 7904-v5.txt, hbase-7904-v3.txt, hbase-TestHFileOutputFormat-wip.txt, hbase-TestingUtility-wip.txt Post YARN-429, unit tests for HBase continue to fail since the classpath for the MRAppMaster is not being set correctly. Reverting YARN-129 may fix this, but I'm not sure that's the correct solution. My guess is, as Alexandro pointed out in YARN-129, maven classloader magic is messing up java.class.path. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-449) MRAppMaster classpath not set properly for unit tests in downstream projects
[
https://issues.apache.org/jira/browse/YARN-449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13597392#comment-13597392
]
Ted Yu commented on YARN-449:
-
TestRowCounter failed with hadoop 2.0.2-alpha on the above machine as well.
I saw the following in test output:
{code}
2013-03-08 18:39:07,078 WARN [Thread-58] datanode.BlockPoolSliceScanner(648):
Received exception:
java.io.FileNotFoundException:
/homes/hortonzy/trunk/hbase-server/target/test-data/30098d58-1320-499d-9cac-2025bd03f25d/dfscluster_19bfcc6a-b942-4692-ab78-8d55040f9fac/dfs/data/data1/current/BP-1507903350-68.142.244.61-1362767892061/dncp_block_verification.log.curr
(No such file or directory)
at java.io.FileOutputStream.openAppend(Native Method)
at java.io.FileOutputStream.init(FileOutputStream.java:192)
at
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.RollingLogsImpl.roll(RollingLogsImpl.java:111)
at
org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.rollVerificationLogs(BlockPoolSliceScanner.java:646)
at
org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.scan(BlockPoolSliceScanner.java:636)
at
org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.scanBlockPoolSlice(BlockPoolSliceScanner.java:599)
at
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:101)
at java.lang.Thread.run(Thread.java:662)
2013-03-08 18:39:16,015 WARN [AsyncDispatcher event handler]
event.AsyncDispatcher$1(71): AsyncDispatcher thread interrupted
java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1961)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1996)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:69)
at java.lang.Thread.run(Thread.java:662)
{code}
MRAppMaster classpath not set properly for unit tests in downstream projects
Key: YARN-449
URL: https://issues.apache.org/jira/browse/YARN-449
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.0.3-alpha
Reporter: Siddharth Seth
Priority: Blocker
Attachments: hbase-7904-v3.txt, hbase-TestHFileOutputFormat-wip.txt,
hbase-TestingUtility-wip.txt
Post YARN-429, unit tests for HBase continue to fail since the classpath for
the MRAppMaster is not being set correctly.
Reverting YARN-129 may fix this, but I'm not sure that's the correct
solution. My guess is, as Alexandro pointed out in YARN-129, maven
classloader magic is messing up java.class.path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-449) MRAppMaster classpath not set properly for unit tests in downstream projects
[ https://issues.apache.org/jira/browse/YARN-449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13595872#comment-13595872 ] Arun C Murthy commented on YARN-449: [~yuzhih...@gmail.com] Thanks a bunch for helping us debug this. Can you please verify that this is just a HBase issue and we can close this out our end? Thanks again! MRAppMaster classpath not set properly for unit tests in downstream projects Key: YARN-449 URL: https://issues.apache.org/jira/browse/YARN-449 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Siddharth Seth Priority: Blocker Attachments: hbase-7904-v3.txt, hbase-TestHFileOutputFormat-wip.txt, hbase-TestingUtility-wip.txt Post YARN-429, unit tests for HBase continue to fail since the classpath for the MRAppMaster is not being set correctly. Reverting YARN-129 may fix this, but I'm not sure that's the correct solution. My guess is, as Alexandro pointed out in YARN-129, maven classloader magic is messing up java.class.path. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-449) MRAppMaster classpath not set properly for unit tests in downstream projects
[ https://issues.apache.org/jira/browse/YARN-449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13595906#comment-13595906 ] Ted Yu commented on YARN-449: - Hbase can adjust as Sid suggested. What about Hive ? I didn't finish verification for Hive. MRAppMaster classpath not set properly for unit tests in downstream projects Key: YARN-449 URL: https://issues.apache.org/jira/browse/YARN-449 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Siddharth Seth Priority: Blocker Attachments: hbase-7904-v3.txt, hbase-TestHFileOutputFormat-wip.txt, hbase-TestingUtility-wip.txt Post YARN-429, unit tests for HBase continue to fail since the classpath for the MRAppMaster is not being set correctly. Reverting YARN-129 may fix this, but I'm not sure that's the correct solution. My guess is, as Alexandro pointed out in YARN-129, maven classloader magic is messing up java.class.path. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-449) MRAppMaster classpath not set properly for unit tests in downstream projects
[
https://issues.apache.org/jira/browse/YARN-449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13596441#comment-13596441
]
Ted Yu commented on YARN-449:
-
Assuming the JobConf returned by MiniMRCluster contains the conf parameters we
passed to MiniMRCluster at time of start, I don't know if
getMinimalConfiguration() is needed.
In HBase we use shim to hide the difference between hadoop 1.0 and 2.0:
{code}
JobConf jobConf = MapreduceTestingShim.getJobConf(mrCluster);
if (jobConf == null) {
jobConf = mrCluster.createJobConf();
}
{code}
MRAppMaster classpath not set properly for unit tests in downstream projects
Key: YARN-449
URL: https://issues.apache.org/jira/browse/YARN-449
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.0.3-alpha
Reporter: Siddharth Seth
Priority: Blocker
Attachments: hbase-7904-v3.txt, hbase-TestHFileOutputFormat-wip.txt,
hbase-TestingUtility-wip.txt
Post YARN-429, unit tests for HBase continue to fail since the classpath for
the MRAppMaster is not being set correctly.
Reverting YARN-129 may fix this, but I'm not sure that's the correct
solution. My guess is, as Alexandro pointed out in YARN-129, maven
classloader magic is messing up java.class.path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-449) MRAppMaster classpath not set properly for unit tests in downstream projects
[ https://issues.apache.org/jira/browse/YARN-449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13596658#comment-13596658 ] Ted Yu commented on YARN-449: - There was test failure using patch v3 against hadoop 1.0 See: https://builds.apache.org/job/PreCommit-HBASE-Build/4719/artifact/trunk/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat.txt MRAppMaster classpath not set properly for unit tests in downstream projects Key: YARN-449 URL: https://issues.apache.org/jira/browse/YARN-449 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Siddharth Seth Priority: Blocker Attachments: hbase-7904-v3.txt, hbase-TestHFileOutputFormat-wip.txt, hbase-TestingUtility-wip.txt Post YARN-429, unit tests for HBase continue to fail since the classpath for the MRAppMaster is not being set correctly. Reverting YARN-129 may fix this, but I'm not sure that's the correct solution. My guess is, as Alexandro pointed out in YARN-129, maven classloader magic is messing up java.class.path. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-449) MRAppMaster classpath not set properly for unit tests in downstream projects
[ https://issues.apache.org/jira/browse/YARN-449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13594332#comment-13594332 ] Siddharth Seth commented on YARN-449: - Ted, good catch on the isMiniYARNCluster property not being set. I assumed that would be in place. The MiniCluster, as part of it's startup process, sets parameters like the RM address in the configuration. This is then available via createJobConf in MiniMRCluster, or getConfig in MiniMRClientCluster. Instead of selectively copying out parameters, downstream projects should really be using the configuration objects returned by these APIs to submit jobs. That would allow things to keep working if parameters were changed. Looked at the PIG code, and that's exactly what it is doing - so the tests passing is expected. I'm not sure if Hive unit tests will work. In the test command you pasted, I believe TestCliDriver needs to be replaced with TestMinimrCliDriver to actually get it to use the MiniMRCluster. IAC, does it make sense for HBase to make use of config objects returned by the getConfig objects so that similar changes in the future don't break unit tests ? MRAppMaster classpath not set properly for unit tests in downstream projects Key: YARN-449 URL: https://issues.apache.org/jira/browse/YARN-449 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Siddharth Seth Priority: Blocker Attachments: hbase-TestHFileOutputFormat-wip.txt, hbase-TestingUtility-wip.txt Post YARN-429, unit tests for HBase continue to fail since the classpath for the MRAppMaster is not being set correctly. Reverting YARN-129 may fix this, but I'm not sure that's the correct solution. My guess is, as Alexandro pointed out in YARN-129, maven classloader magic is messing up java.class.path. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-449) MRAppMaster classpath not set properly for unit tests in downstream projects
[
https://issues.apache.org/jira/browse/YARN-449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13594377#comment-13594377
]
Ted Yu commented on YARN-449:
-
I tried the following change:
{code}
Index:
hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
===
--- hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
(revision 1453107)
+++ hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
(working copy)
@@ -1578,6 +1578,7 @@
mrCluster = new MiniMRCluster(servers,
FS_URI != null ? FS_URI : FileSystem.get(conf).getUri().toString(), 1,
null, null, new JobConf(this.conf));
+this.conf = mrCluster.createJobConf();
JobConf jobConf = MapreduceTestingShim.getJobConf(mrCluster);
if (jobConf == null) {
jobConf = mrCluster.createJobConf();
{code}
mapreduce.TestTableMapReduce#testMultiRegionTable hangs running against hadoop
1.0
MRAppMaster classpath not set properly for unit tests in downstream projects
Key: YARN-449
URL: https://issues.apache.org/jira/browse/YARN-449
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.0.3-alpha
Reporter: Siddharth Seth
Priority: Blocker
Attachments: hbase-TestHFileOutputFormat-wip.txt,
hbase-TestingUtility-wip.txt
Post YARN-429, unit tests for HBase continue to fail since the classpath for
the MRAppMaster is not being set correctly.
Reverting YARN-129 may fix this, but I'm not sure that's the correct
solution. My guess is, as Alexandro pointed out in YARN-129, maven
classloader magic is messing up java.class.path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-449) MRAppMaster classpath not set properly for unit tests in downstream projects
[ https://issues.apache.org/jira/browse/YARN-449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13592915#comment-13592915 ] Hitesh Shah commented on YARN-449: -- This probably will work for a short term until the internal implementation of MiniYarnCluster or any other minicluster for that matter introduces a new config property that it needs/refers to. Looking at the hbase tests, it seems like that instead of using the config object returned by the MiniMRCluster and building on top of it, it tries to do some form of a union between 2 confs. In such cases, chances of missing some internal settings are always likely. I believe there was an earlier fix to set the framework.name to 'yarn' to solve something similar to the current problem when hbase starting running tests against 0.23. [~te...@apache.org], do you have any comments on the above? Is it possible to change the base test class for hbase unit tests to build upon the config provided by the mini cluster? Any reason for not doing so? MRAppMaster classpath not set properly for unit tests in downstream projects Key: YARN-449 URL: https://issues.apache.org/jira/browse/YARN-449 Project: Hadoop YARN Issue Type: Bug Affects Versions: 2.0.3-alpha Reporter: Siddharth Seth Priority: Blocker Attachments: hbase-TestHFileOutputFormat-wip.txt Post YARN-429, unit tests for HBase continue to fail since the classpath for the MRAppMaster is not being set correctly. Reverting YARN-129 may fix this, but I'm not sure that's the correct solution. My guess is, as Alexandro pointed out in YARN-129, maven classloader magic is messing up java.class.path. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-449) MRAppMaster classpath not set properly for unit tests in downstream projects
[
https://issues.apache.org/jira/browse/YARN-449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13592981#comment-13592981
]
Ted Yu commented on YARN-449:
-
It is possible to change HBase test class.
I would spend some time tomorrow in understanding why the following code in
MiniYARNCluster doesn't give us expected effect:
{code}
public synchronized void start() {
try {
getConfig().setBoolean(YarnConfiguration.IS_MINI_YARN_CLUSTER, true);
{code}
MRAppMaster classpath not set properly for unit tests in downstream projects
Key: YARN-449
URL: https://issues.apache.org/jira/browse/YARN-449
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.0.3-alpha
Reporter: Siddharth Seth
Priority: Blocker
Attachments: hbase-TestHFileOutputFormat-wip.txt
Post YARN-429, unit tests for HBase continue to fail since the classpath for
the MRAppMaster is not being set correctly.
Reverting YARN-129 may fix this, but I'm not sure that's the correct
solution. My guess is, as Alexandro pointed out in YARN-129, maven
classloader magic is messing up java.class.path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-449) MRAppMaster classpath not set properly for unit tests in downstream projects
[
https://issues.apache.org/jira/browse/YARN-449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13593080#comment-13593080
]
Ted Yu commented on YARN-449:
-
The second patch made TestTableMapReduce pass based on 2.0.4-SNAPSHOT:
{code}
Running org.apache.hadoop.hbase.mapreduce.TestTableMapReduce
2013-03-04 21:09:47.027 java[28537:1203] Unable to load realm info from
SCDynamicStore
2013-03-04 21:09:47.166 java[28537:1203] Unable to load realm info from
SCDynamicStore
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 238.86 sec
{code}
MRAppMaster classpath not set properly for unit tests in downstream projects
Key: YARN-449
URL: https://issues.apache.org/jira/browse/YARN-449
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 2.0.3-alpha
Reporter: Siddharth Seth
Priority: Blocker
Attachments: hbase-TestHFileOutputFormat-wip.txt,
hbase-TestingUtility-wip.txt
Post YARN-429, unit tests for HBase continue to fail since the classpath for
the MRAppMaster is not being set correctly.
Reverting YARN-129 may fix this, but I'm not sure that's the correct
solution. My guess is, as Alexandro pointed out in YARN-129, maven
classloader magic is messing up java.class.path.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
