[jira] [Created] (MAPREDUCE-3409) Incorrect custom task status when running on MR2
Incorrect custom task status when running on MR2
Key: MAPREDUCE-3409
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3409
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Ahmed Radwan
To reproduce this problem:
1- In your mapper setup() set:
{code}
context.setStatus("myStatus")
{code}
2- When the job finishes:
{code}
TaskReport[] reports = job.getTaskReports(TaskType.MAP);
assertEquals("myStatus", reports[0].getState());
{code}
The returned status from reports[0].getState() is "SUCCEEDED" as opposed to the
expected "myStatus" value.
This exact code work fine on MR1. I saw this issue when tried running the
TestTaskContext test cases on MR2.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3396) Mumak compilation is broken (for a while?)
[
https://issues.apache.org/jira/browse/MAPREDUCE-3396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Boudnik resolved MAPREDUCE-3396.
---
Resolution: Duplicate
It really dups MAPREDUCE-3311, so closing this one.
> Mumak compilation is broken (for a while?)
> --
>
> Key: MAPREDUCE-3396
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3396
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/mumak
>Affects Versions: 0.22.0
>Reporter: Konstantin Boudnik
>Assignee: Konstantin Boudnik
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-3396.patch
>
>
> {{contrib/mumak}} compilation is broken because of the missing
> {{commons-lang}} dependency
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3408) yarn-daemon.sh unconditionnaly sets yarn.root.logger
yarn-daemon.sh unconditionnaly sets yarn.root.logger
Key: MAPREDUCE-3408
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3408
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, nodemanager, resourcemanager
Affects Versions: 0.23.0, 0.23.1
Reporter: Bruno Mahé
Assignee: Bruno Mahé
yarn-daemon.sh unconditionnaly sets yarn.root.logger which then prevent any
override from happening.
>From ./hadoop-mapreduce-project/hadoop-yarn/bin/yarn-daemon.sh:
> export YARN_ROOT_LOGGER="INFO,DRFA"
> export YARN_JHS_LOGGER="INFO,JSA"
and then yarn-daemon.sh will call "$YARN_HOME"/bin/yarn which does the
following:
> YARN_OPTS="$YARN_OPTS -Dhadoop.root.logger=${YARN_ROOT_LOGGER:-INFO,console}"
> YARN_OPTS="$YARN_OPTS -Dyarn.root.logger=${YARN_ROOT_LOGGER:-INFO,console}"
This has at least 2 issues:
* I cannot override hadoop.root.logger when using the yarn-daemon.sh script
* I cannot have different values for hadoop.root.logger and yarn.root.logger
I currently see two different ways to proceed forward:
1/ Make the script yarn-daemon.sh only sets a default value for
YARN_ROOT_LOGGER if this variable is not defined
2/ Remove the quoted code from yarn-daemon.sh since yarn already does something
similar
3/ Entirely remove that chunk and let people define their logging however they
want through some properties files (see log4j.properties in the conf
directories for instance)
I would also use the variable HADOOP_ROOT_LOGGER for hadoop.root.logger if
either option 1/ or 2/ would be taken.
I don't really have any preference toward any of these solutions. What would
you recommend? What is the Apache Hadoop way for this matter?
Note: This is probably happening as well for the other daemons, and I will take
a look at it once this issue is resolved.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3407) Wrong jar getting used in TestMR*Jobs* for MiniMRYarnCluster
Wrong jar getting used in TestMR*Jobs* for MiniMRYarnCluster Key: MAPREDUCE-3407 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3407 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: Hitesh Shah Assignee: Hitesh Shah Priority: Minor Fix For: 0.23.1 pom for mapreduce-client-jobclient sets system property to incorrect jar name. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3406) Add node information to bin/mapred job -list-attempt-ids and other improvements
Add node information to bin/mapred job -list-attempt-ids and other improvements --- Key: MAPREDUCE-3406 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3406 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Ravi Prakash Assignee: Ravi Prakash Fix For: 0.23.1 >From [~rramya] Providing the NM information where the containers are scheduled in bin/mapred job -list-attempt-ids will be helpful in automation, debugging and to avoid grepping through the AM logs. >From my own observation, the list-attempt-ids should list the attempt ids and >not require the arguments. The arguments if given, can be used to filter the >results. From the usage: bq. [-list-attempt-ids ]. Valid values for are MAP REDUCE JOB_SETUP JOB_CLEANUP TASK_CLEANUP. Valid values for are running, completed -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3405) MAPREDUCE-3015 broke compilation of contrib scheduler tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-3405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon resolved MAPREDUCE-3405. Resolution: Fixed Fix Version/s: 0.20.206.0 > MAPREDUCE-3015 broke compilation of contrib scheduler tests > --- > > Key: MAPREDUCE-3405 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3405 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/capacity-sched, contrib/fair-share >Affects Versions: 0.20.206.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Critical > Fix For: 0.20.206.0 > > Attachments: mr-3405.txt > > > MAPREDUCE-3015 added a new argument to the TaskTrackerStatus constructor, > which is used by a few of the scheduler tests, but didn't update those tests. > So, the contrib test build is now failing on 0.20-security -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3027) MR-279: Completed container exit state needs to be enhanced to differentiate between container aborts/failures and actual application process exit codes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah resolved MAPREDUCE-3027. Resolution: Invalid > MR-279: Completed container exit state needs to be enhanced to differentiate > between container aborts/failures and actual application process exit codes > > > Key: MAPREDUCE-3027 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3027 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Hitesh Shah >Assignee: Hitesh Shah > Fix For: 0.24.0 > > > Currently, a completed container's exit status is set to -100 to denote the > container being killed by the framework either as a result of the application > releasing the container or a node failure. An application process may also > return an exit code of -100 creating an ambiguity. > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3405) MAPREDUCE-3015 broke compilation of contrib scheduler tests
MAPREDUCE-3015 broke compilation of contrib scheduler tests --- Key: MAPREDUCE-3405 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3405 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/capacity-sched, contrib/fair-share Affects Versions: 0.20.206.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Critical MAPREDUCE-3015 added a new argument to the TaskTrackerStatus constructor, which is used by a few of the scheduler tests, but didn't update those tests. So, the contrib test build is now failing on 0.20-security -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Purpose of COMMIT_PENDING
Pedro, Simply put: Speculative execution. When a task enters that state, it means that it has completed the M/R execution, and its awaiting the tracker to commit it so that it can run the OutputCommitter process and finalize the outputs (outputs lie in temporary directories until committed, if you check with FileOutputCommitter, the default OutputCommitter in Hadoop MR). This is to avoid conflicting outputs when you have speculatives turned on. Two tasks can complete at the same time and you do not want both to be committed. So the TT will commit the first one that reports back, and kill away the other COMMIT_PENDING waiting one in this case. You might notice (3) cause speculative execution does affect the tail of a job run. On 15-Nov-2011, at 10:35 PM, Pedro Costa wrote: > Hi, > > Hadoop MR tasks can have the state COMMIT_PENDING. > > 1- What's the purpose of that state? > 2- What's the reason for a task being in this state? > 3- It's only the last task before finishing a job that enters this state? > > -- > Thanks,
[jira] [Created] (MAPREDUCE-3404) Speculative Execution: speculative map tasks launched even if -Dmapreduce.job.maps.speculative.execution=false
Speculative Execution: speculative map tasks launched even if -Dmapreduce.job.maps.speculative.execution=false -- Key: MAPREDUCE-3404 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3404 Project: Hadoop Map/Reduce Issue Type: Bug Components: job submission Affects Versions: 0.23.0 Environment: Hadoop version is: Hadoop 0.23.0.1110031628 10 node test cluster Reporter: patrick white When forcing a mapper to take significantly longer than other map tasks, speculative map tasks are launched even if the mapreduce.job.maps.speculative.execution parameter is set to 'false'. Testcase: ran default WordCount job with spec execution set to false for both map and reduce but still saw a fifth mapper task launch, ran job as follows: hadoop --config jar /tmp/testphw/wordcount.jar WordCount -Dmapreduce.job.maps.speculative.execution=false -Dmapreduce.job.reduces.speculative.execution=false /tmp/test_file_of_words* /tmp/file_of_words.out Input data was 4 text files >hdfs blocksize, with same word pattern plus one diff text line in each file, fourth file was 4 times as large as others: hadoop --config fs -ls /tmp Found 5 items drwxr-xr-x - user hdfs 0 2011-10-20 16:17 /tmp/file_of_words.out -rw-r--r-- 3 user hdfs 62800021 2011-10-20 14:45 /tmp/test_file_of_words1 -rw-r--r-- 3 user hdfs 62800024 2011-10-20 14:46 /tmp/test_file_of_words2 -rw-r--r-- 3 user hdfs 62800024 2011-10-20 14:46 /tmp/test_file_of_words3 -rw-r--r-- 3 user hdfs 271708312 2011-10-20 15:50 /tmp/test_file_of_words4 Job launched 5 mappers despite spec exec set to false, output snippet: org.apache.hadoop.mapreduce.JobCounter NUM_FAILED_MAPS=1 TOTAL_LAUNCHED_MAPS=5 TOTAL_LAUNCHED_REDUCES=1 RACK_LOCAL_MAPS=5 SLOTS_MILLIS_MAPS=273540 SLOTS_MILLIS_REDUCES=212876 Reran same case as above only set both spec exec params to 'true', same results only this time the fifth task being launched is expected since spec exec = true. job run: hadoop --config jar /tmp/testphw/wordcount.jar WordCount -Dmapreduce.job.maps.speculative.execution=true -Dmapreduce.job.reduces.speculative.execution=true /tmp/test_file_of_words* /tmp/file_of_words.out output snippet: org.apache.hadoop.mapreduce.JobCounter NUM_FAILED_MAPS=1 TOTAL_LAUNCHED_MAPS=5 TOTAL_LAUNCHED_REDUCES=1 RACK_LOCAL_MAPS=5 SLOTS_MILLIS_MAPS=279653 SLOTS_MILLIS_REDUCES=211474 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3403) Speculative Execution: enabling multiple reduce tasks inhibit spec exec launch of mappers
Speculative Execution: enabling multiple reduce tasks inhibit spec exec launch
of mappers
--
Key: MAPREDUCE-3403
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3403
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: job submission
Affects Versions: 0.23.0
Environment: Hadoop version is: Hadoop 0.23.0.1110031628
10 node test cluster
Reporter: patrick white
When forcing multiple reduce tasks to be launched by applying the
setNumReduceTasks() method on a Job object, and
running on input data which has one significantly longer map (and consequently
reduce) task;
- a speculative reduce task was not launched, even with a longer running
reducer only 4 reduce tasks were launched
- the spec launch of map tasks was inhibited by the setNumReduceTasks() method
applied, so even with
-Dmapreduce.job.maps.speculative.execution=true we only had 4 map tasks
launched. The exact same code with the
setNumReduceTasks() method taken out, and on the same input data set,
consistently launched 5 mappers as expected.
Testing info:
3. modified WordCount to force 4 reducers being launched, by adding:
job.setNumReduceTasks(4); // hardwire 4 reducers for now
System.out.println("\nTESTDEBUG: using 4 reduce tasks for now\n\n");
to the Job object. This causes 4 reduce tasks to be launched, oddly though it
inhibits the map task from speculative
launch. So the same job code, without the setNumReduceTasks() method, will
launch 5 mappers as described in case #2.
When this method is added, that same job will only launch 4 mappers, as well as
4 reducers, otherwise the job
successfully completes.
output snippet with setNumReduceTasks():
org.apache.hadoop.mapreduce.JobCounter
TOTAL_LAUNCHED_MAPS=4
TOTAL_LAUNCHED_REDUCES=4
RACK_LOCAL_MAPS=4
SLOTS_MILLIS_MAPS=190787
SLOTS_MILLIS_REDUCES=572554
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
Purpose of COMMIT_PENDING
Hi, Hadoop MR tasks can have the state COMMIT_PENDING. 1- What's the purpose of that state? 2- What's the reason for a task being in this state? 3- It's only the last task before finishing a job that enters this state? -- Thanks,
[jira] [Created] (MAPREDUCE-3402) AMScalability test of Sleep job with 100K 1-sec maps regressed into running very slowly
AMScalability test of Sleep job with 100K 1-sec maps regressed into running very slowly --- Key: MAPREDUCE-3402 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3402 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Fix For: 0.23.1 The world was rosier before October 19-25, [~karams] says. The 100K 1 second sleep job used to take around 800mins or 13-14 mins. It now runs till 45 mins and still manages to complete only about 45K tasks. One/more of the flurry of commits for 0.23.0 deserve(s) the blame. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3401) Make single node secure cluster setup documentation for 0.23
Make single node secure cluster setup documentation for 0.23 Key: MAPREDUCE-3401 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3401 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: Anupam Seth Assignee: Anupam Seth Priority: Minor Fix For: 0.23.1 This JIRA is to track some minor corrections and suggestions for improvement for the documentation for the setup of a single node cluster using 0.23 currently available at http://people.apache.org/~acmurthy/hadoop-0.23/hadoop-yarn/hadoop-yarn-site/SingleCluster.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
