[jira] [Updated] (MAPREDUCE-5783) web UI update to allow people to request thread dump of a running task.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Junjun updated MAPREDUCE-5783: - Summary: web UI update to allow people to request thread dump of a running task. (was: web UI update to allow people to request thread dump of a running task) web UI update to allow people to request thread dump of a running task. --- Key: MAPREDUCE-5783 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5783 Project: Hadoop Map/Reduce Issue Type: Task Components: webapps Reporter: Ming Ma This depends on https://issues.apache.org/jira/browse/YARN-445. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5818) hsadmin cmd is missing in mapred.cmd
[ https://issues.apache.org/jira/browse/MAPREDUCE-5818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated MAPREDUCE-5818: - Hadoop Flags: Reviewed +1 for patch v3. Thanks, Jian! hsadmin cmd is missing in mapred.cmd Key: MAPREDUCE-5818 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5818 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Jian He Assignee: Jian He Attachments: MAPREDUCE-5818.1.patch, MAPREDUCE-5818.3.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-2980) Fetch failures and other related issues in Jetty 6.1.26
[ https://issues.apache.org/jira/browse/MAPREDUCE-2980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13958689#comment-13958689 ] Fengdong Yu commented on MAPREDUCE-2980: why don't upgrade to the latest version of Jetty? it's incompatible? Fetch failures and other related issues in Jetty 6.1.26 --- Key: MAPREDUCE-2980 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2980 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.20.205.0, 0.23.0 Reporter: Todd Lipcon Since upgrading Jetty from 6.1.14 to 6.1.26 we've had a ton of HTTP-related issues, including: - Much higher incidence of fetch failures - A few strange file-descriptor related bugs (eg MAPREDUCE-2389) - A few unexplained issues where long fscks on the NameNode drop out halfway through with a ClosedChannelException Stress tests with 1Map x 1Reduce sleep jobs reliably reproduce fetch failures at a rate of about 1 per million on a 25 node test cluster. These problems are all new since the upgrade from 6.1.14. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5014) Extending DistCp through a custom CopyListing is not possible
[ https://issues.apache.org/jira/browse/MAPREDUCE-5014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13958726#comment-13958726 ] Hudson commented on MAPREDUCE-5014: --- SUCCESS: Integrated in Hadoop-Yarn-trunk #528 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/528/]) Move MAPREDUCE-5014 to the right section now that it's been merged to branch-2. (atm: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1584234) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt Extending DistCp through a custom CopyListing is not possible - Key: MAPREDUCE-5014 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5014 Project: Hadoop Map/Reduce Issue Type: Improvement Components: distcp Affects Versions: 0.23.0, 0.23.1, 0.23.3, trunk, 0.23.4, 0.23.5 Reporter: Srikanth Sundarrajan Assignee: Srikanth Sundarrajan Fix For: 3.0.0, 2.5.0 Attachments: MAPREDUCE-5014.patch, MAPREDUCE-5014.patch, m5014_20130322.patch, m5014_20130322_b-1.patch, m5014_20130322b.patch, m5014_20130322b_b-1.patch Original Estimate: 24h Remaining Estimate: 24h * While it is possible to implement a custom CopyListing in DistCp, DistCp driver class doesn't allow for using this custom CopyListing. * Allow SimpleCopyListing to provide an option to exclude files (For instance it is useful to exclude FileOutputCommiter.SUCCEEDED_FILE_NAME during copy as premature copy can indicate that the entire data is available at the destination) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5014) Extending DistCp through a custom CopyListing is not possible
[ https://issues.apache.org/jira/browse/MAPREDUCE-5014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13958807#comment-13958807 ] Hudson commented on MAPREDUCE-5014: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1746 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1746/]) Move MAPREDUCE-5014 to the right section now that it's been merged to branch-2. (atm: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1584234) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt Extending DistCp through a custom CopyListing is not possible - Key: MAPREDUCE-5014 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5014 Project: Hadoop Map/Reduce Issue Type: Improvement Components: distcp Affects Versions: 0.23.0, 0.23.1, 0.23.3, trunk, 0.23.4, 0.23.5 Reporter: Srikanth Sundarrajan Assignee: Srikanth Sundarrajan Fix For: 3.0.0, 2.5.0 Attachments: MAPREDUCE-5014.patch, MAPREDUCE-5014.patch, m5014_20130322.patch, m5014_20130322_b-1.patch, m5014_20130322b.patch, m5014_20130322b_b-1.patch Original Estimate: 24h Remaining Estimate: 24h * While it is possible to implement a custom CopyListing in DistCp, DistCp driver class doesn't allow for using this custom CopyListing. * Allow SimpleCopyListing to provide an option to exclude files (For instance it is useful to exclude FileOutputCommiter.SUCCEEDED_FILE_NAME during copy as premature copy can indicate that the entire data is available at the destination) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5014) Extending DistCp through a custom CopyListing is not possible
[ https://issues.apache.org/jira/browse/MAPREDUCE-5014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13958836#comment-13958836 ] Hudson commented on MAPREDUCE-5014: --- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1720 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1720/]) Move MAPREDUCE-5014 to the right section now that it's been merged to branch-2. (atm: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1584234) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt Extending DistCp through a custom CopyListing is not possible - Key: MAPREDUCE-5014 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5014 Project: Hadoop Map/Reduce Issue Type: Improvement Components: distcp Affects Versions: 0.23.0, 0.23.1, 0.23.3, trunk, 0.23.4, 0.23.5 Reporter: Srikanth Sundarrajan Assignee: Srikanth Sundarrajan Fix For: 3.0.0, 2.5.0 Attachments: MAPREDUCE-5014.patch, MAPREDUCE-5014.patch, m5014_20130322.patch, m5014_20130322_b-1.patch, m5014_20130322b.patch, m5014_20130322b_b-1.patch Original Estimate: 24h Remaining Estimate: 24h * While it is possible to implement a custom CopyListing in DistCp, DistCp driver class doesn't allow for using this custom CopyListing. * Allow SimpleCopyListing to provide an option to exclude files (For instance it is useful to exclude FileOutputCommiter.SUCCEEDED_FILE_NAME during copy as premature copy can indicate that the entire data is available at the destination) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5818) hsadmin cmd is missing in mapred.cmd
[ https://issues.apache.org/jira/browse/MAPREDUCE-5818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13959041#comment-13959041 ] Zhijie Shen commented on MAPREDUCE-5818: +1, will commit it once ASF LDAP is back. hsadmin cmd is missing in mapred.cmd Key: MAPREDUCE-5818 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5818 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Jian He Assignee: Jian He Attachments: MAPREDUCE-5818.1.patch, MAPREDUCE-5818.3.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5818) hsadmin cmd is missing in mapred.cmd
[ https://issues.apache.org/jira/browse/MAPREDUCE-5818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated MAPREDUCE-5818: --- Status: Patch Available (was: Open) hsadmin cmd is missing in mapred.cmd Key: MAPREDUCE-5818 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5818 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Jian He Assignee: Jian He Attachments: MAPREDUCE-5818.1.patch, MAPREDUCE-5818.3.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5815) Fix NPE in TestMRAppMaster
[ https://issues.apache.org/jira/browse/MAPREDUCE-5815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13959053#comment-13959053 ] Mit Desai commented on MAPREDUCE-5815: -- I agree with [~acmurthy] that TestMRAppMaster should set a proper queue name. On the other note, instead of ignoring the queue name, I think it would be a good idea to have the null check and throw an appropriate exception. What do you guys think? Fix NPE in TestMRAppMaster -- Key: MAPREDUCE-5815 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5815 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, mrv2 Reporter: Gera Shegalov Assignee: Gera Shegalov Priority: Blocker Attachments: MAPREDUCE-5815.v01.patch Working MAPREDUCE-5813 I stumbled on NPE's in TestMRAppMaster. They seem to be introduced by MAPREDUCE-5805. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5818) hsadmin cmd is missing in mapred.cmd
[
https://issues.apache.org/jira/browse/MAPREDUCE-5818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13959071#comment-13959071
]
Hadoop QA commented on MAPREDUCE-5818:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12638368/MAPREDUCE-5818.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4484//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4484//console
This message is automatically generated.
hsadmin cmd is missing in mapred.cmd
Key: MAPREDUCE-5818
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5818
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Jian He
Assignee: Jian He
Attachments: MAPREDUCE-5818.1.patch, MAPREDUCE-5818.3.patch
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (MAPREDUCE-5818) hsadmin cmd is missing in mapred.cmd
[ https://issues.apache.org/jira/browse/MAPREDUCE-5818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5818: --- Resolution: Fixed Fix Version/s: 2.4.1 Status: Resolved (was: Patch Available) hsadmin cmd is missing in mapred.cmd Key: MAPREDUCE-5818 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5818 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Jian He Assignee: Jian He Fix For: 2.4.1 Attachments: MAPREDUCE-5818.1.patch, MAPREDUCE-5818.3.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5818) hsadmin cmd is missing in mapred.cmd
[ https://issues.apache.org/jira/browse/MAPREDUCE-5818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13959170#comment-13959170 ] Zhijie Shen commented on MAPREDUCE-5818: Committed to trunk, branch-2 and branch-2.4. Thanks, Jian! hsadmin cmd is missing in mapred.cmd Key: MAPREDUCE-5818 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5818 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Jian He Assignee: Jian He Fix For: 2.4.1 Attachments: MAPREDUCE-5818.1.patch, MAPREDUCE-5818.3.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5818) hsadmin cmd is missing in mapred.cmd
[ https://issues.apache.org/jira/browse/MAPREDUCE-5818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13959182#comment-13959182 ] Hudson commented on MAPREDUCE-5818: --- SUCCESS: Integrated in Hadoop-trunk-Commit #5452 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5452/]) MAPREDUCE-5818. Added hsadmin command into mapred.cmd. Contributed by Jian He. (zjshen: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1584378) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/bin/mapred.cmd hsadmin cmd is missing in mapred.cmd Key: MAPREDUCE-5818 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5818 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Jian He Assignee: Jian He Fix For: 2.4.1 Attachments: MAPREDUCE-5818.1.patch, MAPREDUCE-5818.3.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5785) Derive task attempt JVM max heap size and io.sort.mb automatically from mapreduce.*.memory.mb
[ https://issues.apache.org/jira/browse/MAPREDUCE-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gera Shegalov updated MAPREDUCE-5785: - Summary: Derive task attempt JVM max heap size and io.sort.mb automatically from mapreduce.*.memory.mb (was: Derive task attempt JVM max heap size automatically from mapreduce.*.memory.mb) Derive task attempt JVM max heap size and io.sort.mb automatically from mapreduce.*.memory.mb - Key: MAPREDUCE-5785 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5785 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mr-am, task Reporter: Gera Shegalov Assignee: Gera Shegalov Attachments: MAPREDUCE-5785.v01.patch, MAPREDUCE-5785.v02.patch Currently users have to set 2 memory-related configs per Job / per task type. One first chooses some container size map reduce.\*.memory.mb and then a corresponding maximum Java heap size Xmx map reduce.\*.memory.mb. This makes sure that the JVM's C-heap (native memory + Java heap) does not exceed this mapreduce.*.memory.mb. If one forgets to tune Xmx, MR-AM might be - allocating big containers whereas the JVM will only use the default -Xmx200m. - allocating small containers that will OOM because Xmx is too high. With this JIRA, we propose to set Xmx automatically based on an empirical ratio that can be adjusted. Xmx is not changed automatically if provided by the user. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5785) Derive task attempt JVM max heap size and io.sort.mb automatically from mapreduce.*.memory.mb
[ https://issues.apache.org/jira/browse/MAPREDUCE-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13959433#comment-13959433 ] Gera Shegalov commented on MAPREDUCE-5785: -- [~kasha], can you take a look regarding your suggestion of including sort buffer size? Derive task attempt JVM max heap size and io.sort.mb automatically from mapreduce.*.memory.mb - Key: MAPREDUCE-5785 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5785 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mr-am, task Reporter: Gera Shegalov Assignee: Gera Shegalov Attachments: MAPREDUCE-5785.v01.patch, MAPREDUCE-5785.v02.patch Currently users have to set 2 memory-related configs per Job / per task type. One first chooses some container size map reduce.\*.memory.mb and then a corresponding maximum Java heap size Xmx map reduce.\*.memory.mb. This makes sure that the JVM's C-heap (native memory + Java heap) does not exceed this mapreduce.*.memory.mb. If one forgets to tune Xmx, MR-AM might be - allocating big containers whereas the JVM will only use the default -Xmx200m. - allocating small containers that will OOM because Xmx is too high. With this JIRA, we propose to set Xmx automatically based on an empirical ratio that can be adjusted. Xmx is not changed automatically if provided by the user. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5812) Make task context available to OutputCommitter.isRecoverySupported()
[ https://issues.apache.org/jira/browse/MAPREDUCE-5812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13959521#comment-13959521 ] Mohammad Kamrul Islam commented on MAPREDUCE-5812: -- Test failure is unrelated and it is being addressed at : https://issues.apache.org/jira/browse/MAPREDUCE-5816 Make task context available to OutputCommitter.isRecoverySupported() - Key: MAPREDUCE-5812 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5812 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am Affects Versions: 2.3.0 Reporter: Mohammad Kamrul Islam Assignee: Mohammad Kamrul Islam Attachments: MAPREDUCE-5812.1.patch Background == The system like Hive provides its version of OutputCommitter. The custom implementation of isRecoverySupported() requires task context. From taskContext:getConfiguration(), hive checks if hive-defined specific property is set or not. Based on the property value, it returns true or false. However, in the current OutputCommitter:isRecoverySupported(), there is no way of getting task config. As a result, user can't turn on/off the MRAM recovery feature. Proposed resolution: === 1. Pass Task Context into isRecoverySupported() method. Pros: Easy and clean Cons: Possible backward compatibility issue due to aPI changes. (Is it true?) 2. Call outputCommitter.setupTask(taskContext) from MRAM: The new OutputCommitter will store the context in the class level variable and use it from isRecoverySupported() Props: No API changes. No backward compatibility issue. This call can be made from MRAppMaster.getOutputCommitter() method for old API case. Cons: Might not be very clean solution due to class level variable. Please give your comments. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5655) Remote job submit from windows to a linux hadoop cluster fails due to wrong classpath
[ https://issues.apache.org/jira/browse/MAPREDUCE-5655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13959593#comment-13959593 ] Daniel Crabtree commented on MAPREDUCE-5655: This bug also affects 2.3.0. I've tested this patch on 2.3.0 and it fixes the problem. However, in addition to applying attached patches and recompiling appropriate client jars, and the additions mentioned by Igor Rudyak, I also had to create a jar for the client side map reduce job and set mapreduce.job.jar to that jar on the client side. This can be done programatically on a Configuration object as follows: conf.set(mapreduce.job.jar,C:\application.jar); Remote job submit from windows to a linux hadoop cluster fails due to wrong classpath - Key: MAPREDUCE-5655 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5655 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, job submission Affects Versions: 2.2.0 Environment: Client machine is a Windows 7 box, with Eclipse Remote: there is a multi node hadoop cluster, installed on Ubuntu boxes (any linux) Reporter: Attila Pados Attachments: MRApps.patch, YARNRunner.patch I was trying to run a java class on my client, windows 7 developer environment, which submits a job to the remote Hadoop cluster, initiates a mapreduce there, and then downloads the results back to the local machine. General use case is to use hadoop services from a web application installed on a non-cluster computer, or as part of a developer environment. The problem was, that the ApplicationMaster's startup shell script (launch_container.sh) was generated with wrong CLASSPATH entry. Together with the java process call on the bottom of the file, these entries were generated in windows style, using % as shell variable marker and ; as the CLASSPATH delimiter. I tracked down the root cause, and found that the MrApps.java, and the YarnRunner.java classes create these entries, and is passed forward to the ApplicationMaster, assuming that the OS that runs these classes will match the one running the ApplicationMaster. But it's not the case, these are in 2 different jvm, and also the OS can be different, the strings are generated based on the client/submitter side's OS. I made some workaround changes to these 2 files, so i could launch my job, however there may be more problems ahead. update error message: 13/12/04 16:33:15 INFO mapreduce.Job: Job job_1386170530016_0001 failed with state FAILED due to: Application application_1386170530016_0001 failed 2 times due to AM Container for appattempt_1386170530016_0001_02 exited with exitCode: 1 due to: Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException: /bin/bash: line 0: fg: no job control at org.apache.hadoop.util.Shell.runCommand(Shell.java:464) at org.apache.hadoop.util.Shell.run(Shell.java:379) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) update2: It also reqires to add the following property to mapred-site.xml (or mapred-default.xml), on the windows box, so that the job launcher knows, that the job runner will be a linux: property namemapred.remote.os/name valueLinux/value descriptionRemote MapReduce framework's OS, can be either Linux or Windows/description /property without this entry, the patched jar does the same as the unpatched, so it's required to work! -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5655) Remote job submit from windows to a linux hadoop cluster fails due to wrong classpath
[ https://issues.apache.org/jira/browse/MAPREDUCE-5655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Crabtree updated MAPREDUCE-5655: --- Affects Version/s: 2.3.0 Remote job submit from windows to a linux hadoop cluster fails due to wrong classpath - Key: MAPREDUCE-5655 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5655 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, job submission Affects Versions: 2.2.0, 2.3.0 Environment: Client machine is a Windows 7 box, with Eclipse Remote: there is a multi node hadoop cluster, installed on Ubuntu boxes (any linux) Reporter: Attila Pados Attachments: MRApps.patch, YARNRunner.patch I was trying to run a java class on my client, windows 7 developer environment, which submits a job to the remote Hadoop cluster, initiates a mapreduce there, and then downloads the results back to the local machine. General use case is to use hadoop services from a web application installed on a non-cluster computer, or as part of a developer environment. The problem was, that the ApplicationMaster's startup shell script (launch_container.sh) was generated with wrong CLASSPATH entry. Together with the java process call on the bottom of the file, these entries were generated in windows style, using % as shell variable marker and ; as the CLASSPATH delimiter. I tracked down the root cause, and found that the MrApps.java, and the YarnRunner.java classes create these entries, and is passed forward to the ApplicationMaster, assuming that the OS that runs these classes will match the one running the ApplicationMaster. But it's not the case, these are in 2 different jvm, and also the OS can be different, the strings are generated based on the client/submitter side's OS. I made some workaround changes to these 2 files, so i could launch my job, however there may be more problems ahead. update error message: 13/12/04 16:33:15 INFO mapreduce.Job: Job job_1386170530016_0001 failed with state FAILED due to: Application application_1386170530016_0001 failed 2 times due to AM Container for appattempt_1386170530016_0001_02 exited with exitCode: 1 due to: Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException: /bin/bash: line 0: fg: no job control at org.apache.hadoop.util.Shell.runCommand(Shell.java:464) at org.apache.hadoop.util.Shell.run(Shell.java:379) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:724) update2: It also reqires to add the following property to mapred-site.xml (or mapred-default.xml), on the windows box, so that the job launcher knows, that the job runner will be a linux: property namemapred.remote.os/name valueLinux/value descriptionRemote MapReduce framework's OS, can be either Linux or Windows/description /property without this entry, the patched jar does the same as the unpatched, so it's required to work! -- This message was sent by Atlassian JIRA (v6.2#6252)
