[jira] [Created] (MAPREDUCE-5927) Getting following error
Kedar Dixit created MAPREDUCE-5927: -- Summary: Getting following error Key: MAPREDUCE-5927 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5927 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Reporter: Kedar Dixit Priority: Blocker -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (MAPREDUCE-5928) Deadlock allocating containers for mappers and reducers
Niels Basjes created MAPREDUCE-5928:
---
Summary: Deadlock allocating containers for mappers and reducers
Key: MAPREDUCE-5928
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5928
Project: Hadoop Map/Reduce
Issue Type: Bug
Environment: Hadoop 2.4.0 (as packaged by HortonWorks in HDP 2.1.2)
Reporter: Niels Basjes
I have a small cluster consisting of 8 desktop class systems (1 master + 7
workers).
Due to the small memory of these systems I configured yarn as follows:
{quote}
yarn.nodemanager.resource.memory-mb = 2200
yarn.scheduler.minimum-allocation-mb = 250
{quote}
On my client I did
{quote}
mapreduce.map.memory.mb = 512
mapreduce.reduce.memory.mb = 512
{quote}
Now I run a job with 27 mappers and 32 reducers.
After a while I saw this deadlock occur:
- All nodes had been filled to their maximum capacity with reducers.
- 1 Mapper was waiting for a container slot to start in.
I tried killing reducer attempts but that didn't help (new reducer attempts
simply took the existing container).
*Workaround*:
I set this value from my job. The default value is 0.05 (= 5%)
{quote}
mapreduce.job.reduce.slowstart.completedmaps = 0.99f
{quote}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Reopened] (MAPREDUCE-5927) Getting following error
[ https://issues.apache.org/jira/browse/MAPREDUCE-5927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe reopened MAPREDUCE-5927: --- Getting following error --- Key: MAPREDUCE-5927 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5927 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Reporter: Kedar Dixit Assignee: Vinod Kumar Vavilapalli Priority: Blocker Hi, I am getting following error, while running application on cluser - 14/06/16 16:21:48 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 14/06/16 16:21:49 INFO input.FileInputFormat: Total input paths to process : 1 14/06/16 16:21:49 INFO mapreduce.JobSubmitter: number of splits:1 14/06/16 16:21:49 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class 14/06/16 16:21:49 INFO Configuration.deprecation: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name 14/06/16 16:21:49 INFO Configuration.deprecation: mapreduce.inputformat.class is deprecated. Instead, use mapreduce.job.inputformat.class 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir 14/06/16 16:21:49 INFO Configuration.deprecation: mapreduce.outputformat.class is deprecated. Instead, use mapreduce.job.outputformat.class 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir 14/06/16 16:21:49 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1402913701967_0006 14/06/16 16:21:49 INFO impl.YarnClientImpl: Submitted application application_1402913701967_0006 to ResourceManager at master/10.71.71.110:8032 14/06/16 16:21:49 INFO mapreduce.Job: The url to track the job: http://gs-1695:8088/proxy/application_1402913701967_0006/ 14/06/16 16:21:49 INFO mapreduce.Job: Running job: job_1402913701967_0006 14/06/16 16:21:54 INFO mapreduce.Job: Job job_1402913701967_0006 running in uber mode : false 14/06/16 16:21:54 INFO mapreduce.Job: map 0% reduce 0% 14/06/16 16:21:54 INFO mapreduce.Job: Job job_1402913701967_0006 failed with state FAILED due to: Application application_1402913701967_0006 failed 2 times due to AM Container for appattempt_1402913701967_0006_02 exited with exitCode: 1 due to: Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException: at org.apache.hadoop.util.Shell.runCommand(Shell.java:464) at org.apache.hadoop.util.Shell.run(Shell.java:379) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) .Failing this attempt.. Failing the application. 14/06/16 16:21:54 INFO mapreduce.Job: Counters: 0 Can you please help me in fixing this ? Thanks, ~Kedar -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (MAPREDUCE-5927) Getting following error
[ https://issues.apache.org/jira/browse/MAPREDUCE-5927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe resolved MAPREDUCE-5927. --- Resolution: Fixed Getting following error --- Key: MAPREDUCE-5927 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5927 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Reporter: Kedar Dixit Assignee: Vinod Kumar Vavilapalli Priority: Blocker Hi, I am getting following error, while running application on cluser - 14/06/16 16:21:48 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 14/06/16 16:21:49 INFO input.FileInputFormat: Total input paths to process : 1 14/06/16 16:21:49 INFO mapreduce.JobSubmitter: number of splits:1 14/06/16 16:21:49 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class 14/06/16 16:21:49 INFO Configuration.deprecation: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name 14/06/16 16:21:49 INFO Configuration.deprecation: mapreduce.inputformat.class is deprecated. Instead, use mapreduce.job.inputformat.class 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir 14/06/16 16:21:49 INFO Configuration.deprecation: mapreduce.outputformat.class is deprecated. Instead, use mapreduce.job.outputformat.class 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir 14/06/16 16:21:49 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1402913701967_0006 14/06/16 16:21:49 INFO impl.YarnClientImpl: Submitted application application_1402913701967_0006 to ResourceManager at master/10.71.71.110:8032 14/06/16 16:21:49 INFO mapreduce.Job: The url to track the job: http://gs-1695:8088/proxy/application_1402913701967_0006/ 14/06/16 16:21:49 INFO mapreduce.Job: Running job: job_1402913701967_0006 14/06/16 16:21:54 INFO mapreduce.Job: Job job_1402913701967_0006 running in uber mode : false 14/06/16 16:21:54 INFO mapreduce.Job: map 0% reduce 0% 14/06/16 16:21:54 INFO mapreduce.Job: Job job_1402913701967_0006 failed with state FAILED due to: Application application_1402913701967_0006 failed 2 times due to AM Container for appattempt_1402913701967_0006_02 exited with exitCode: 1 due to: Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException: at org.apache.hadoop.util.Shell.runCommand(Shell.java:464) at org.apache.hadoop.util.Shell.run(Shell.java:379) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) .Failing this attempt.. Failing the application. 14/06/16 16:21:54 INFO mapreduce.Job: Counters: 0 Can you please help me in fixing this ? Thanks, ~Kedar -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (MAPREDUCE-5927) Getting following error
[ https://issues.apache.org/jira/browse/MAPREDUCE-5927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe resolved MAPREDUCE-5927. --- Resolution: Invalid This is a general support question better asked on the u...@hadoop.apache.org list. JIRA is for tracking bugs and features in Hadoop and not a general user support channel. In this case the ApplicationMaster is crashing shortly after startup. You'll need to examine the ApplicationMaster log to determine what happened -- click on the tracking URL and then from there go to the AM logs link or you can also use the yarn logs command if log aggregation is enabled on your cluster. Getting following error --- Key: MAPREDUCE-5927 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5927 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Reporter: Kedar Dixit Assignee: Vinod Kumar Vavilapalli Priority: Blocker Hi, I am getting following error, while running application on cluser - 14/06/16 16:21:48 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this. 14/06/16 16:21:49 INFO input.FileInputFormat: Total input paths to process : 1 14/06/16 16:21:49 INFO mapreduce.JobSubmitter: number of splits:1 14/06/16 16:21:49 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class 14/06/16 16:21:49 INFO Configuration.deprecation: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name 14/06/16 16:21:49 INFO Configuration.deprecation: mapreduce.inputformat.class is deprecated. Instead, use mapreduce.job.inputformat.class 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir 14/06/16 16:21:49 INFO Configuration.deprecation: mapreduce.outputformat.class is deprecated. Instead, use mapreduce.job.outputformat.class 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class 14/06/16 16:21:49 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir 14/06/16 16:21:49 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1402913701967_0006 14/06/16 16:21:49 INFO impl.YarnClientImpl: Submitted application application_1402913701967_0006 to ResourceManager at master/10.71.71.110:8032 14/06/16 16:21:49 INFO mapreduce.Job: The url to track the job: http://gs-1695:8088/proxy/application_1402913701967_0006/ 14/06/16 16:21:49 INFO mapreduce.Job: Running job: job_1402913701967_0006 14/06/16 16:21:54 INFO mapreduce.Job: Job job_1402913701967_0006 running in uber mode : false 14/06/16 16:21:54 INFO mapreduce.Job: map 0% reduce 0% 14/06/16 16:21:54 INFO mapreduce.Job: Job job_1402913701967_0006 failed with state FAILED due to: Application application_1402913701967_0006 failed 2 times due to AM Container for appattempt_1402913701967_0006_02 exited with exitCode: 1 due to: Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException: at org.apache.hadoop.util.Shell.runCommand(Shell.java:464) at org.apache.hadoop.util.Shell.run(Shell.java:379) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
Hadoop-Mapreduce-trunk - Build # 1803 - Still Failing
See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1803/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 36152 lines...] Running org.apache.hadoop.mapreduce.TestChild Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 82.683 sec - in org.apache.hadoop.mapreduce.TestChild Running org.apache.hadoop.mapreduce.filecache.TestURIFragments Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.064 sec - in org.apache.hadoop.mapreduce.filecache.TestURIFragments Running org.apache.hadoop.mapreduce.TestMapReduce Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.06 sec - in org.apache.hadoop.mapreduce.TestMapReduce Running org.apache.hadoop.mapreduce.TestNewCombinerGrouping Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.609 sec - in org.apache.hadoop.mapreduce.TestNewCombinerGrouping Results : Tests run: 494, Failures: 0, Errors: 0, Skipped: 11 [INFO] [INFO] Reactor Summary: [INFO] [INFO] hadoop-mapreduce-client ... SUCCESS [2.645s] [INFO] hadoop-mapreduce-client-core .. SUCCESS [53.714s] [INFO] hadoop-mapreduce-client-common SUCCESS [26.303s] [INFO] hadoop-mapreduce-client-shuffle ... SUCCESS [4.282s] [INFO] hadoop-mapreduce-client-app ... SUCCESS [7:03.138s] [INFO] hadoop-mapreduce-client-hs SUCCESS [4:43.014s] [INFO] hadoop-mapreduce-client-jobclient . FAILURE [1:37:38.098s] [INFO] hadoop-mapreduce-client-hs-plugins SKIPPED [INFO] Apache Hadoop MapReduce Examples .. SKIPPED [INFO] hadoop-mapreduce .. SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 1:50:52.230s [INFO] Finished at: Mon Jun 16 15:17:22 UTC 2014 [INFO] Final Memory: 29M/73M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.16:test (default-test) on project hadoop-mapreduce-client-jobclient: There was a timeout or other error in the fork - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn goals -rf :hadoop-mapreduce-client-jobclient Build step 'Execute shell' marked build as failure [FINDBUGS] Skipping publisher since build result is FAILURE Archiving artifacts Sending artifact delta relative to Hadoop-Mapreduce-trunk #1765 Archived 2 artifacts Archive block size is 32768 Received 0 blocks and 16948207 bytes Compression is 0.0% Took 12 sec Updating HDFS-2006 Updating MAPREDUCE-5898 Updating HDFS-6464 Updating HDFS-6375 Updating MAPREDUCE-5920 Email was triggered for: Failure Sending email for trigger: Failure ### ## FAILED TESTS (if any) ## No tests ran.
[VOTE] Release Apache Hadoop 2.4.1
Folks, I've created a release candidate (rc0) for hadoop-2.4.1 (bug-fix release) that I would like to push out. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.4.1-rc0 The RC tag in svn is here: https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.4.1-rc0 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/hdp/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[jira] [Created] (MAPREDUCE-5929) YARNRunner.java, path for jobJarPath not set correctly
Chao Tian created MAPREDUCE-5929: Summary: YARNRunner.java, path for jobJarPath not set correctly Key: MAPREDUCE-5929 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5929 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.2.0 Reporter: Chao Tian In YARNRunner.java, line 357, Path jobJarPath = new Path(jobConf.get(MRJobConfig.JAR)); This causes the job.jar file to miss scheme, host and port number on distributed file systems other than hdfs. If we compare line 357 with line 344, there job.xml is actually set as Path jobConfPath = new Path(jobSubmitDir,MRJobConfig.JOB_CONF_FILE); It appears jobSubmitDir is missing on line 357, which causes this problem. In hdfs, the additional qualify process will correct this problem, but not other generic distributed file systems. The proposed change is to replace 35 7 with Path jobJarPath = new Path(jobConf.get(jobSubmitDir,MRJobConfig.JAR)); -- This message was sent by Atlassian JIRA (v6.2#6252)
