[jira] [Updated] (MAPREDUCE-6462) JobHistoryServer to support JvmPauseMonitor as a service

[Sunil G (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sunil G updated MAPREDUCE-6462:
---
Status: Open  (was: Patch Available)

Cancelling patch to upload MapReduce related patch and aggregated patch to 
check Jenkins

> JobHistoryServer to support JvmPauseMonitor as a service
> 
>
> Key: MAPREDUCE-6462
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6462
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: jobhistoryserver
>Affects Versions: 2.8.0
>Reporter: Sunil G
>Assignee: Sunil G
>Priority: Minor
> Attachments: 0001-MAPREDUCE-6462.patch, HADOOP-12321-003.patch
>
>
> As JvmPauseMonitor is made as an AbstractService, subsequent method changes 
> are needed in all places which uses the monitor.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MAPREDUCE-6472) MapReduce AM should have java.io.tmpdir=./tmp to be consistent with tasks

[Naganarasimha G R (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R updated MAPREDUCE-6472:
-
Status: Patch Available  (was: Open)

> MapReduce AM should have java.io.tmpdir=./tmp to be consistent with tasks
> -
>
> Key: MAPREDUCE-6472
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6472
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mr-am
>Affects Versions: 2.6.0
>Reporter: Jason Lowe
>Assignee: Naganarasimha G R
> Attachments: MAPREDUCE-6472.20150914-1.patch
>
>
> MapReduceChildJVM.getVMCommand ensures that all tasks have 
> -Djava.io.tmpdir=./tmp set as part of the task command-line, but this is only 
> used for tasks.  The AM itself does not have a corresponding java.io.tmpdir 
> setting.  It should also use the same tmpdir setting to avoid cases where the 
> AM JVM wants to place files in /tmp by default.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MAPREDUCE-6462) JobHistoryServer to support JvmPauseMonitor as a service

[Sunil G (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sunil G updated MAPREDUCE-6462:
---
Attachment: HADOOP-12321-005-aggregated.patch

Thank you [~steve_l]
Yes. I also feel we can take option 2. I feel its adding not much value to know 
the count of child services. Attaching an aggregated patch to check Jenkins.

> JobHistoryServer to support JvmPauseMonitor as a service
> 
>
> Key: MAPREDUCE-6462
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6462
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: jobhistoryserver
>Affects Versions: 2.8.0
>Reporter: Sunil G
>Assignee: Sunil G
>Priority: Minor
> Attachments: 0001-MAPREDUCE-6462.patch, 0002-MAPREDUCE-6462.patch, 
> HADOOP-12321-003.patch, HADOOP-12321-005-aggregated.patch
>
>
> As JvmPauseMonitor is made as an AbstractService, subsequent method changes 
> are needed in all places which uses the monitor.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MAPREDUCE-6472) MapReduce AM should have java.io.tmpdir=./tmp to be consistent with tasks

[Naganarasimha G R (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R updated MAPREDUCE-6472:
-
Attachment: MAPREDUCE-6472.20150914-1.patch

Hi [~jlowe], attaching a patch to this issue please check

> MapReduce AM should have java.io.tmpdir=./tmp to be consistent with tasks
> -
>
> Key: MAPREDUCE-6472
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6472
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mr-am
>Affects Versions: 2.6.0
>Reporter: Jason Lowe
>Assignee: Naganarasimha G R
> Attachments: MAPREDUCE-6472.20150914-1.patch
>
>
> MapReduceChildJVM.getVMCommand ensures that all tasks have 
> -Djava.io.tmpdir=./tmp set as part of the task command-line, but this is only 
> used for tasks.  The AM itself does not have a corresponding java.io.tmpdir 
> setting.  It should also use the same tmpdir setting to avoid cases where the 
> AM JVM wants to place files in /tmp by default.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MAPREDUCE-6462) JobHistoryServer to support JvmPauseMonitor as a service

[Sunil G (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sunil G updated MAPREDUCE-6462:
---
Attachment: 0002-MAPREDUCE-6462.patch

> JobHistoryServer to support JvmPauseMonitor as a service
> 
>
> Key: MAPREDUCE-6462
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6462
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: jobhistoryserver
>Affects Versions: 2.8.0
>Reporter: Sunil G
>Assignee: Sunil G
>Priority: Minor
> Attachments: 0001-MAPREDUCE-6462.patch, 0002-MAPREDUCE-6462.patch, 
> HADOOP-12321-003.patch
>
>
> As JvmPauseMonitor is made as an AbstractService, subsequent method changes 
> are needed in all places which uses the monitor.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MAPREDUCE-6462) JobHistoryServer to support JvmPauseMonitor as a service

[Sunil G (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sunil G updated MAPREDUCE-6462:
---
Status: Patch Available  (was: Open)

> JobHistoryServer to support JvmPauseMonitor as a service
> 
>
> Key: MAPREDUCE-6462
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6462
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: jobhistoryserver
>Affects Versions: 2.8.0
>Reporter: Sunil G
>Assignee: Sunil G
>Priority: Minor
> Attachments: 0001-MAPREDUCE-6462.patch, 0002-MAPREDUCE-6462.patch, 
> HADOOP-12321-003.patch, HADOOP-12321-005-aggregated.patch
>
>
> As JvmPauseMonitor is made as an AbstractService, subsequent method changes 
> are needed in all places which uses the monitor.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MAPREDUCE-6344) Inconsistent classpath/classloading from DistributedCache archives

[Preston Koprivica (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Preston Koprivica updated MAPREDUCE-6344:
-
Attachment: MAPREDUCE-6344.patch

I have attached a proposed fix for this issue.   The basic approach was to: 

* Modify the addToClassPathIfNotJar method to simply be: addToClassPath.  This 
should retain the order of both archives and non-archives when pulling from 
distributed cache.

* Moved the "glob" to be the last thing appended to the appended to the 
classpath.  (This is only needed to ensure a non-breaking change.  This will 
catch anything that was put in the container but not explicitly enumerated).

> Inconsistent classpath/classloading from DistributedCache archives
> --
>
> Key: MAPREDUCE-6344
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6344
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Affects Versions: 2.5.0, 2.6.0, 2.7.0, 2.5.1, 2.5.2
>Reporter: Preston Koprivica
> Attachments: MAPREDUCE-6344.patch
>
>
> We recently upgraded to MRv2 on YARN and have been noticing very inconsistent 
> classloading between the job submission client and the tasks as they start 
> up. 
> I've tracked the issue to this method:
> https://github.com/apache/hadoop/blob/release-2.5.0/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java#L264
> It appears that the classpath is simply "wild carded".  According the javase 
> 7&8 docs, the order of enumeration is not specified and may differ from 
> moment to moment [1][2].  This is a problem for applications that rely on 
> strict ordering, which the MRv1 DistributedCache used to provide.
> I'm unable to track down all the things that are linked or landed into the 
> $PWD of the container, but assuming we can't account for all these things, a 
> simple solution could be to explicitly enumerate the files in 
> DistributedCache - similar to the "non jar" case [3] - and then add the "*" 
> for passivity.  
> [1] http://docs.oracle.com/javase/7/docs/technotes/tools/windows/classpath.htm
> [2] 
> http://docs.oracle.com/javase/8/docs/technotes/tools/windows/classpath.html#A1100762
> [3] 
> https://github.com/apache/hadoop/blob/release-2.5.0/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java#L270



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6472) MapReduce AM should have java.io.tmpdir=./tmp to be consistent with tasks

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14743704#comment-14743704
 ] 

Hadoop QA commented on MAPREDUCE-6472:
--

\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch |  16m 16s | Pre-patch trunk compilation is 
healthy. |
| {color:green}+1{color} | @author |   0m  0s | The patch does not contain any 
@author tags. |
| {color:green}+1{color} | tests included |   0m  0s | The patch appears to 
include 1 new or modified test files. |
| {color:green}+1{color} | javac |   7m 52s | There were no new javac warning 
messages. |
| {color:green}+1{color} | javadoc |  10m  4s | There were no new javadoc 
warning messages. |
| {color:green}+1{color} | release audit |   0m 23s | The applied patch does 
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle |   0m 35s | The applied patch generated  1 
new checkstyle issues (total was 28, now 28). |
| {color:red}-1{color} | whitespace |   0m  0s | The patch has 2  line(s) that 
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install |   1m 30s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse |   0m 34s | The patch built with 
eclipse:eclipse. |
| {color:green}+1{color} | findbugs |   0m 51s | The patch does not introduce 
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | mapreduce tests |  32m 30s | Tests failed in 
hadoop-mapreduce-client-jobclient. |
| | |  70m 39s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.mapred.TestMRIntermediateDataEncryption |
|   | hadoop.mapred.TestReduceFetch |
|   | hadoop.mapreduce.lib.output.TestJobOutputCommitter |
|   | hadoop.mapred.join.TestDatamerge |
|   | hadoop.mapreduce.v2.TestSpeculativeExecution |
|   | hadoop.mapreduce.lib.input.TestCombineFileInputFormat |
|   | hadoop.mapreduce.v2.TestMROldApiJobs |
|   | hadoop.mapreduce.v2.TestMRJobsWithProfiler |
|   | hadoop.mapreduce.TestMapReduceLazyOutput |
|   | hadoop.mapreduce.security.TestMRCredentials |
|   | hadoop.ipc.TestMRCJCSocketFactory |
|   | hadoop.mapred.TestMiniMRBringup |
|   | hadoop.conf.TestNoDefaultsJobConf |
|   | hadoop.mapreduce.TestChild |
|   | hadoop.mapred.TestMRTimelineEventHandling |
|   | hadoop.mapred.TestJobCleanup |
|   | hadoop.mapreduce.lib.join.TestJoinProperties |
|   | hadoop.mapreduce.v2.TestMiniMRProxyUser |
|   | hadoop.mapreduce.v2.TestUberAM |
|   | hadoop.fs.TestDFSIO |
|   | hadoop.mapreduce.v2.TestMRJobsWithHistoryService |
|   | hadoop.mapreduce.v2.TestMRAppWithCombiner |
|   | hadoop.mapreduce.v2.TestRMNMInfo |
|   | hadoop.mapred.TestMRCJCFileInputFormat |
|   | hadoop.mapreduce.v2.TestNonExistentJob |
|   | hadoop.mapred.TestReduceFetchFromPartialMem |
|   | hadoop.mapred.TestClusterMapReduceTestCase |
|   | hadoop.mapreduce.security.TestBinaryTokenFile |
|   | hadoop.mapred.TestJobSysDirWithDFS |
|   | hadoop.mapred.lib.TestDelegatingInputFormat |
|   | hadoop.mapred.TestMiniMRClientCluster |
|   | hadoop.mapreduce.TestLargeSort |
|   | hadoop.mapreduce.TestMRJobClient |
|   | hadoop.mapreduce.v2.TestMRJobs |
|   | hadoop.mapreduce.lib.input.TestDelegatingInputFormat |
|   | hadoop.mapreduce.v2.TestMRAMWithNonNormalizedCapabilities |
|   | hadoop.mapred.TestClusterMRNotification |
|   | hadoop.mapreduce.lib.join.TestJoinDatamerge |
|   | hadoop.mapred.TestNetworkedJob |
|   | hadoop.mapreduce.security.TestJHSSecurity |
|   | hadoop.fs.TestFileSystem |
|   | hadoop.mapreduce.security.ssl.TestEncryptedShuffle |
|   | hadoop.mapred.TestSpecialCharactersInOutputPath |
| Timed out tests | org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers 
|
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL | 
http://issues.apache.org/jira/secure/attachment/12755743/MAPREDUCE-6472.20150914-1.patch
 |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 6955771 |
| checkstyle |  
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5987/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-jobclient.txt
 |
| whitespace | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5987/artifact/patchprocess/whitespace.txt
 |
| hadoop-mapreduce-client-jobclient test log | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5987/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
 |
| Test Results | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5987/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP 
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5987/console |


This message was automatically generated.

> MapReduce AM should have java.io.tmpdir=./tmp to be consistent with tasks
> 

[jira] [Commented] (MAPREDUCE-6472) MapReduce AM should have java.io.tmpdir=./tmp to be consistent with tasks

[Naganarasimha G R (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14743651#comment-14743651
 ] 

Naganarasimha G R commented on MAPREDUCE-6472:
--

Basically  {{-Djava.io.tmpdir=$PWD/tmp}} will be configured similar to the 
MapReduceChildJVM.getVMCommand 

> MapReduce AM should have java.io.tmpdir=./tmp to be consistent with tasks
> -
>
> Key: MAPREDUCE-6472
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6472
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mr-am
>Affects Versions: 2.6.0
>Reporter: Jason Lowe
>Assignee: Naganarasimha G R
> Attachments: MAPREDUCE-6472.20150914-1.patch
>
>
> MapReduceChildJVM.getVMCommand ensures that all tasks have 
> -Djava.io.tmpdir=./tmp set as part of the task command-line, but this is only 
> used for tasks.  The AM itself does not have a corresponding java.io.tmpdir 
> setting.  It should also use the same tmpdir setting to avoid cases where the 
> AM JVM wants to place files in /tmp by default.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6344) Inconsistent classpath/classloading from DistributedCache archives

[Preston Koprivica (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14743809#comment-14743809
 ] 

Preston Koprivica commented on MAPREDUCE-6344:
--

Just an FYI, the patch was created off the release-2.5.0 branch.

> Inconsistent classpath/classloading from DistributedCache archives
> --
>
> Key: MAPREDUCE-6344
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6344
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Affects Versions: 2.5.0, 2.6.0, 2.7.0, 2.5.1, 2.5.2
>Reporter: Preston Koprivica
> Attachments: MAPREDUCE-6344.patch
>
>
> We recently upgraded to MRv2 on YARN and have been noticing very inconsistent 
> classloading between the job submission client and the tasks as they start 
> up. 
> I've tracked the issue to this method:
> https://github.com/apache/hadoop/blob/release-2.5.0/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java#L264
> It appears that the classpath is simply "wild carded".  According the javase 
> 7&8 docs, the order of enumeration is not specified and may differ from 
> moment to moment [1][2].  This is a problem for applications that rely on 
> strict ordering, which the MRv1 DistributedCache used to provide.
> I'm unable to track down all the things that are linked or landed into the 
> $PWD of the container, but assuming we can't account for all these things, a 
> simple solution could be to explicitly enumerate the files in 
> DistributedCache - similar to the "non jar" case [3] - and then add the "*" 
> for passivity.  
> [1] http://docs.oracle.com/javase/7/docs/technotes/tools/windows/classpath.htm
> [2] 
> http://docs.oracle.com/javase/8/docs/technotes/tools/windows/classpath.html#A1100762
> [3] 
> https://github.com/apache/hadoop/blob/release-2.5.0/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/MRApps.java#L270



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MAPREDUCE-6473) Job submission can take a long time during Cluster initialization

[Kuhu Shukla (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kuhu Shukla updated MAPREDUCE-6473:
---
Status: Patch Available  (was: In Progress)

> Job submission can take a long time during Cluster initialization
> -
>
> Key: MAPREDUCE-6473
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6473
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: performance
>Reporter: Kuhu Shukla
>Assignee: Kuhu Shukla
> Attachments: 99%ile.png, MAPREDUCE-6473-PerfTest.txt, 
> MAPREDUCE-6473-v1.patch, avgtime.png
>
>
> During initialization in Cluster.java, the framework provider classes are 
> loaded inside a sync block which can considerably increase job submission 
> time when the number of submissions are high. The motive is to reduce time 
> spent in this sync block safely to improve performance.
> {noformat}
> synchronized (frameworkLoader) {
>   for (ClientProtocolProvider provider : frameworkLoader) {
> LOG.debug("Trying ClientProtocolProvider : "
> + provider.getClass().getName());
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-5003) AM recovery should recreate records for attempts that were incomplete

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14744366#comment-14744366
 ] 

Hadoop QA commented on MAPREDUCE-5003:
--

\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch |  18m 18s | Pre-patch trunk compilation is 
healthy. |
| {color:green}+1{color} | @author |   0m  0s | The patch does not contain any 
@author tags. |
| {color:green}+1{color} | tests included |   0m  0s | The patch appears to 
include 3 new or modified test files. |
| {color:green}+1{color} | javac |   8m  8s | There were no new javac warning 
messages. |
| {color:green}+1{color} | javadoc |  10m  4s | There were no new javadoc 
warning messages. |
| {color:green}+1{color} | release audit |   0m 24s | The applied patch does 
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle |   1m 15s | The applied patch generated  5 
new checkstyle issues (total was 175, now 180). |
| {color:red}-1{color} | whitespace |   0m  4s | The patch has 1  line(s) that 
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install |   1m 31s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse |   0m 34s | The patch built with 
eclipse:eclipse. |
| {color:green}+1{color} | findbugs |   2m 33s | The patch does not introduce 
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | mapreduce tests |   9m 17s | Tests passed in 
hadoop-mapreduce-client-app. |
| {color:green}+1{color} | mapreduce tests |   1m 48s | Tests passed in 
hadoop-mapreduce-client-core. |
| | |  54m  1s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL | 
http://issues.apache.org/jira/secure/attachment/12755833/MAPREDUCE-5003.7.patch 
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 6955771 |
| checkstyle |  
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5991/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
 |
| whitespace | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5991/artifact/patchprocess/whitespace.txt
 |
| hadoop-mapreduce-client-app test log | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5991/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
 |
| hadoop-mapreduce-client-core test log | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5991/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
 |
| Test Results | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5991/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP 
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5991/console |


This message was automatically generated.

> AM recovery should recreate records for attempts that were incomplete
> -
>
> Key: MAPREDUCE-5003
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5003
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: mr-am
>Reporter: Jason Lowe
>Assignee: Chang Li
> Attachments: MAPREDUCE-5003.1.patch, MAPREDUCE-5003.2.patch, 
> MAPREDUCE-5003.3.patch, MAPREDUCE-5003.4.patch, MAPREDUCE-5003.5.patch, 
> MAPREDUCE-5003.5.patch, MAPREDUCE-5003.6.patch, MAPREDUCE-5003.7.patch
>
>
> As discussed in MAPREDUCE-4992, it would be nice if the AM recovered task 
> attempt entries for *all* task attempts launched by the prior app attempt 
> even if those task attempts did not complete.  The attempts would have to be 
> marked as killed or something similar to indicate it is no longer running.  
> Having records for the task attempts enables the user to see what nodes were 
> associated with the attempts and potentially access their logs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MAPREDUCE-6477) Replace usage of deprecated NameNode.DEFAULT_PORT with HdfsClientConfigKeys.DFS_NAMENODE_RPC_PORT_DEFAULT

[Mingliang Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingliang Liu updated MAPREDUCE-6477:
-
Attachment: MAPREDUCE-6477.000.patch

> Replace usage of deprecated NameNode.DEFAULT_PORT with 
> HdfsClientConfigKeys.DFS_NAMENODE_RPC_PORT_DEFAULT
> -
>
> Key: MAPREDUCE-6477
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6477
> Project: Hadoop Map/Reduce
>  Issue Type: Task
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: MAPREDUCE-6477.000.patch
>
>
> The {{NameNode.DEFAULT_PORT}} static attribute is stale as we use 
> {{HdfsClientConfigKeys.DFS_NAMENODE_RPC_PORT_DEFAULT}} config value. Jira 
> [HDFS-9010] marks it as {{@Deprecated}} and refactored its usages in 
> {{hadoop-hdfs}} module. This jira tracks the effort of replacing its usage in 
> {{org.apache.hadoop.fs.TestFileSystem}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6477) Replace usage of deprecated NameNode.DEFAULT_PORT with HdfsClientConfigKeys.DFS_NAMENODE_RPC_PORT_DEFAULT

[Mingliang Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14744551#comment-14744551
 ] 

Mingliang Liu commented on MAPREDUCE-6477:
--

{{TestNetworkedJob}} fails on trunk, see [MAPREDUCE-6201] (can be reproduced 
locally on Mac). {{hadoop.mapreduce.TestLargeSort}} failure can not be 
reproduced on either Mac or Gentoo.

> Replace usage of deprecated NameNode.DEFAULT_PORT with 
> HdfsClientConfigKeys.DFS_NAMENODE_RPC_PORT_DEFAULT
> -
>
> Key: MAPREDUCE-6477
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6477
> Project: Hadoop Map/Reduce
>  Issue Type: Task
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: MAPREDUCE-6477.000.patch
>
>
> The {{NameNode.DEFAULT_PORT}} static attribute is stale as we use 
> {{HdfsClientConfigKeys.DFS_NAMENODE_RPC_PORT_DEFAULT}} config value. Jira 
> [HDFS-9010] marks it as {{@Deprecated}} and refactored its usages in 
> {{hadoop-hdfs}} module. This jira tracks the effort of replacing its usage in 
> {{org.apache.hadoop.fs.TestFileSystem}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (MAPREDUCE-6477) Replace usage of deprecated NameNode.DEFAULT_PORT with HdfsClientConfigKeys.DFS_NAMENODE_RPC_PORT_DEFAULT

[Mingliang Liu (JIRA)]

Mingliang Liu created MAPREDUCE-6477:


 Summary: Replace usage of deprecated NameNode.DEFAULT_PORT with 
HdfsClientConfigKeys.DFS_NAMENODE_RPC_PORT_DEFAULT
 Key: MAPREDUCE-6477
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6477
 Project: Hadoop Map/Reduce
  Issue Type: Task
Reporter: Mingliang Liu
Assignee: Mingliang Liu


The {{NameNode.DEFAULT_PORT}} static attribute is stale as we use 
{{HdfsClientConfigKeys.DFS_NAMENODE_RPC_PORT_DEFAULT}} config value. Jira 
[HDFS-9010] marks it as {{@Deprecated}} and refactored its usages in 
{{hadoop-hdfs}} module. This jira tracks the effort of replacing its usage in 
{{org.apache.hadoop.fs.TestFileSystem}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MAPREDUCE-5003) AM recovery should recreate records for attempts that were incomplete

[Chang Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chang Li updated MAPREDUCE-5003:

Attachment: MAPREDUCE-5003.7.patch

Thanks [~jlowe] for review! Have reproduced the problem about broken link for 
recovered incomplete task attempts. .7 fixed this issue. Also addressed your 
other suggestions. Because I touched TaskAttemptStarted event, I have checked 
and make sure that there is no backward compatibility issue.

> AM recovery should recreate records for attempts that were incomplete
> -
>
> Key: MAPREDUCE-5003
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5003
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: mr-am
>Reporter: Jason Lowe
>Assignee: Chang Li
> Attachments: MAPREDUCE-5003.1.patch, MAPREDUCE-5003.2.patch, 
> MAPREDUCE-5003.3.patch, MAPREDUCE-5003.4.patch, MAPREDUCE-5003.5.patch, 
> MAPREDUCE-5003.5.patch, MAPREDUCE-5003.6.patch, MAPREDUCE-5003.7.patch
>
>
> As discussed in MAPREDUCE-4992, it would be nice if the AM recovered task 
> attempt entries for *all* task attempts launched by the prior app attempt 
> even if those task attempts did not complete.  The attempts would have to be 
> marked as killed or something similar to indicate it is no longer running.  
> Having records for the task attempts enables the user to see what nodes were 
> associated with the attempts and potentially access their logs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6472) MapReduce AM should have java.io.tmpdir=./tmp to be consistent with tasks

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14744418#comment-14744418
 ] 

Hadoop QA commented on MAPREDUCE-6472:
--

\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch |  16m 15s | Findbugs (version ) appears to 
be broken on trunk. |
| {color:green}+1{color} | @author |   0m  0s | The patch does not contain any 
@author tags. |
| {color:green}+1{color} | tests included |   0m  0s | The patch appears to 
include 1 new or modified test files. |
| {color:green}+1{color} | javac |   7m 53s | There were no new javac warning 
messages. |
| {color:green}+1{color} | javadoc |   9m 57s | There were no new javadoc 
warning messages. |
| {color:green}+1{color} | release audit |   0m 24s | The applied patch does 
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle |   0m 25s | There were no new checkstyle 
issues. |
| {color:green}+1{color} | whitespace |   0m  0s | The patch has no lines that 
end in whitespace. |
| {color:green}+1{color} | install |   1m 37s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse |   0m 35s | The patch built with 
eclipse:eclipse. |
| {color:green}+1{color} | findbugs |   0m 50s | The patch does not introduce 
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | mapreduce tests | 116m 10s | Tests failed in 
hadoop-mapreduce-client-jobclient. |
| | | 154m 13s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.mapred.TestNetworkedJob |
| Timed out tests | 
org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL | 
http://issues.apache.org/jira/secure/attachment/12755788/MAPREDUCE-6472.20150914-2.patch
 |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 6955771 |
| hadoop-mapreduce-client-jobclient test log | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5989/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
 |
| Test Results | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5989/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP 
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5989/console |


This message was automatically generated.

> MapReduce AM should have java.io.tmpdir=./tmp to be consistent with tasks
> -
>
> Key: MAPREDUCE-6472
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6472
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mr-am
>Affects Versions: 2.6.0
>Reporter: Jason Lowe
>Assignee: Naganarasimha G R
> Attachments: MAPREDUCE-6472.20150914-1.patch, 
> MAPREDUCE-6472.20150914-2.patch
>
>
> MapReduceChildJVM.getVMCommand ensures that all tasks have 
> -Djava.io.tmpdir=./tmp set as part of the task command-line, but this is only 
> used for tasks.  The AM itself does not have a corresponding java.io.tmpdir 
> setting.  It should also use the same tmpdir setting to avoid cases where the 
> AM JVM wants to place files in /tmp by default.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6477) Replace usage of deprecated NameNode.DEFAULT_PORT with HdfsClientConfigKeys.DFS_NAMENODE_RPC_PORT_DEFAULT

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1477#comment-1477
 ] 

Hadoop QA commented on MAPREDUCE-6477:
--

\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch |   6m  7s | Pre-patch trunk compilation is 
healthy. |
| {color:green}+1{color} | @author |   0m  0s | The patch does not contain any 
@author tags. |
| {color:green}+1{color} | tests included |   0m  0s | The patch appears to 
include 1 new or modified test files. |
| {color:green}+1{color} | javac |   7m 47s | There were no new javac warning 
messages. |
| {color:green}+1{color} | release audit |   0m 19s | The applied patch does 
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle |   0m 31s | There were no new checkstyle 
issues. |
| {color:green}+1{color} | whitespace |   0m  0s | The patch has no lines that 
end in whitespace. |
| {color:green}+1{color} | install |   1m 27s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse |   0m 32s | The patch built with 
eclipse:eclipse. |
| {color:green}+1{color} | findbugs |   0m 48s | The patch does not introduce 
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | mapreduce tests | 101m 26s | Tests failed in 
hadoop-mapreduce-client-jobclient. |
| | | 119m  1s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.mapreduce.TestLargeSort |
|   | hadoop.mapred.TestNetworkedJob |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL | 
http://issues.apache.org/jira/secure/attachment/12755829/MAPREDUCE-6477.000.patch
 |
| Optional Tests | javac unit findbugs checkstyle |
| git revision | trunk / 6955771 |
| hadoop-mapreduce-client-jobclient test log | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5990/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
 |
| Test Results | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5990/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP 
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5990/console |


This message was automatically generated.

> Replace usage of deprecated NameNode.DEFAULT_PORT with 
> HdfsClientConfigKeys.DFS_NAMENODE_RPC_PORT_DEFAULT
> -
>
> Key: MAPREDUCE-6477
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6477
> Project: Hadoop Map/Reduce
>  Issue Type: Task
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: MAPREDUCE-6477.000.patch
>
>
> The {{NameNode.DEFAULT_PORT}} static attribute is stale as we use 
> {{HdfsClientConfigKeys.DFS_NAMENODE_RPC_PORT_DEFAULT}} config value. Jira 
> [HDFS-9010] marks it as {{@Deprecated}} and refactored its usages in 
> {{hadoop-hdfs}} module. This jira tracks the effort of replacing its usage in 
> {{org.apache.hadoop.fs.TestFileSystem}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6473) Job submission can take a long time during Cluster initialization

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14744208#comment-14744208
 ] 

Hadoop QA commented on MAPREDUCE-6473:
--

\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch |  20m 39s | Pre-patch trunk compilation is 
healthy. |
| {color:green}+1{color} | @author |   0m  0s | The patch does not contain any 
@author tags. |
| {color:red}-1{color} | tests included |   0m  0s | The patch doesn't appear 
to include any new or modified tests.  Please justify why no new tests are 
needed for this patch. Also please list what manual steps were performed to 
verify this patch. |
| {color:green}+1{color} | javac |  10m  0s | There were no new javac warning 
messages. |
| {color:green}+1{color} | javadoc |  15m 27s | There were no new javadoc 
warning messages. |
| {color:green}+1{color} | release audit |   0m 44s | The applied patch does 
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle |   1m 19s | The applied patch generated  
23 new checkstyle issues (total was 15, now 38). |
| {color:green}+1{color} | whitespace |   0m  0s | The patch has no lines that 
end in whitespace. |
| {color:green}+1{color} | install |   2m 16s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse |   0m 38s | The patch built with 
eclipse:eclipse. |
| {color:green}+1{color} | findbugs |   1m 41s | The patch does not introduce 
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | mapreduce tests |   2m  3s | Tests passed in 
hadoop-mapreduce-client-core. |
| | |  54m 51s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL | 
http://issues.apache.org/jira/secure/attachment/12755512/MAPREDUCE-6473-v1.patch
 |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 6955771 |
| checkstyle |  
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5988/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
 |
| hadoop-mapreduce-client-core test log | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5988/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
 |
| Test Results | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5988/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP 
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5988/console |


This message was automatically generated.

> Job submission can take a long time during Cluster initialization
> -
>
> Key: MAPREDUCE-6473
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6473
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: performance
>Reporter: Kuhu Shukla
>Assignee: Kuhu Shukla
> Attachments: 99%ile.png, MAPREDUCE-6473-PerfTest.txt, 
> MAPREDUCE-6473-v1.patch, avgtime.png
>
>
> During initialization in Cluster.java, the framework provider classes are 
> loaded inside a sync block which can considerably increase job submission 
> time when the number of submissions are high. The motive is to reduce time 
> spent in this sync block safely to improve performance.
> {noformat}
> synchronized (frameworkLoader) {
>   for (ClientProtocolProvider provider : frameworkLoader) {
> LOG.debug("Trying ClientProtocolProvider : "
> + provider.getClass().getName());
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-5003) AM recovery should recreate records for attempts that were incomplete

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14744496#comment-14744496
 ] 

Hadoop QA commented on MAPREDUCE-5003:
--

\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch |  17m 41s | Pre-patch trunk compilation is 
healthy. |
| {color:green}+1{color} | @author |   0m  0s | The patch does not contain any 
@author tags. |
| {color:green}+1{color} | tests included |   0m  0s | The patch appears to 
include 3 new or modified test files. |
| {color:green}+1{color} | javac |   7m 49s | There were no new javac warning 
messages. |
| {color:green}+1{color} | javadoc |   9m 57s | There were no new javadoc 
warning messages. |
| {color:green}+1{color} | release audit |   0m 22s | The applied patch does 
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle |   1m 13s | The applied patch generated  3 
new checkstyle issues (total was 175, now 178). |
| {color:green}+1{color} | whitespace |   0m  3s | The patch has no lines that 
end in whitespace. |
| {color:green}+1{color} | install |   1m 31s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse |   0m 32s | The patch built with 
eclipse:eclipse. |
| {color:green}+1{color} | findbugs |   2m 32s | The patch does not introduce 
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | mapreduce tests |   9m 16s | Tests failed in 
hadoop-mapreduce-client-app. |
| {color:green}+1{color} | mapreduce tests |   1m 46s | Tests passed in 
hadoop-mapreduce-client-core. |
| | |  52m 45s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.mapreduce.v2.app.TestMRApp |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL | 
http://issues.apache.org/jira/secure/attachment/12755851/MAPREDUCE-5003.8.patch 
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 6955771 |
| checkstyle |  
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5992/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
 |
| hadoop-mapreduce-client-app test log | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5992/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
 |
| hadoop-mapreduce-client-core test log | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5992/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
 |
| Test Results | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5992/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP 
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5992/console |


This message was automatically generated.

> AM recovery should recreate records for attempts that were incomplete
> -
>
> Key: MAPREDUCE-5003
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5003
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: mr-am
>Reporter: Jason Lowe
>Assignee: Chang Li
> Attachments: MAPREDUCE-5003.1.patch, MAPREDUCE-5003.2.patch, 
> MAPREDUCE-5003.3.patch, MAPREDUCE-5003.4.patch, MAPREDUCE-5003.5.patch, 
> MAPREDUCE-5003.5.patch, MAPREDUCE-5003.6.patch, MAPREDUCE-5003.7.patch, 
> MAPREDUCE-5003.8.patch
>
>
> As discussed in MAPREDUCE-4992, it would be nice if the AM recovered task 
> attempt entries for *all* task attempts launched by the prior app attempt 
> even if those task attempts did not complete.  The attempts would have to be 
> marked as killed or something similar to indicate it is no longer running.  
> Having records for the task attempts enables the user to see what nodes were 
> associated with the attempts and potentially access their logs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MAPREDUCE-6477) Replace usage of deprecated NameNode.DEFAULT_PORT with HdfsClientConfigKeys.DFS_NAMENODE_RPC_PORT_DEFAULT

[Mingliang Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingliang Liu updated MAPREDUCE-6477:
-
Status: Patch Available  (was: Open)

> Replace usage of deprecated NameNode.DEFAULT_PORT with 
> HdfsClientConfigKeys.DFS_NAMENODE_RPC_PORT_DEFAULT
> -
>
> Key: MAPREDUCE-6477
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6477
> Project: Hadoop Map/Reduce
>  Issue Type: Task
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: MAPREDUCE-6477.000.patch
>
>
> The {{NameNode.DEFAULT_PORT}} static attribute is stale as we use 
> {{HdfsClientConfigKeys.DFS_NAMENODE_RPC_PORT_DEFAULT}} config value. Jira 
> [HDFS-9010] marks it as {{@Deprecated}} and refactored its usages in 
> {{hadoop-hdfs}} module. This jira tracks the effort of replacing its usage in 
> {{org.apache.hadoop.fs.TestFileSystem}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MAPREDUCE-6477) Replace usage of deprecated NameNode.DEFAULT_PORT in TestFileSystem

[Mingliang Liu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mingliang Liu updated MAPREDUCE-6477:
-
Summary: Replace usage of deprecated NameNode.DEFAULT_PORT in 
TestFileSystem  (was: Replace usage of deprecated NameNode.DEFAULT_PORT with 
HdfsClientConfigKeys.DFS_NAMENODE_RPC_PORT_DEFAULT)

> Replace usage of deprecated NameNode.DEFAULT_PORT in TestFileSystem
> ---
>
> Key: MAPREDUCE-6477
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6477
> Project: Hadoop Map/Reduce
>  Issue Type: Task
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: MAPREDUCE-6477.000.patch
>
>
> The {{NameNode.DEFAULT_PORT}} static attribute is stale as we use 
> {{HdfsClientConfigKeys.DFS_NAMENODE_RPC_PORT_DEFAULT}} config value. Jira 
> [HDFS-9010] marks it as {{@Deprecated}} and refactored its usages in 
> {{hadoop-hdfs}} module. This jira tracks the effort of replacing its usage in 
> {{org.apache.hadoop.fs.TestFileSystem}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (MAPREDUCE-6471) Document distcp incremental copy

[nijel (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

nijel reassigned MAPREDUCE-6471:


Assignee: Neelesh Srinivas Salian  (was: nijel)

yes.. assigned to you. please go ahead.

> Document distcp incremental copy 
> -
>
> Key: MAPREDUCE-6471
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6471
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: distcp
>Affects Versions: 2.7.1
>Reporter: Arpit Agarwal
>Assignee: Neelesh Srinivas Salian
>  Labels: newbie
>
> MAPREDUCE-5899 added distcp support for incremental copy with a new 
> {{append}} flag.
> It should be documented.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6472) MapReduce AM should have java.io.tmpdir=./tmp to be consistent with tasks

[Naganarasimha G R (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14744749#comment-14744749
 ] 

Naganarasimha G R commented on MAPREDUCE-6472:
--

[~jlowe], test case failures are not related to patch. Seems like jenkins 
report for the patch is fine can you take a look at the patch ?

> MapReduce AM should have java.io.tmpdir=./tmp to be consistent with tasks
> -
>
> Key: MAPREDUCE-6472
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6472
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mr-am
>Affects Versions: 2.6.0
>Reporter: Jason Lowe
>Assignee: Naganarasimha G R
> Attachments: MAPREDUCE-6472.20150914-1.patch, 
> MAPREDUCE-6472.20150914-2.patch
>
>
> MapReduceChildJVM.getVMCommand ensures that all tasks have 
> -Djava.io.tmpdir=./tmp set as part of the task command-line, but this is only 
> used for tasks.  The AM itself does not have a corresponding java.io.tmpdir 
> setting.  It should also use the same tmpdir setting to avoid cases where the 
> AM JVM wants to place files in /tmp by default.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6477) Replace usage of deprecated NameNode.DEFAULT_PORT in TestFileSystem

[Haohui Mai (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14744848#comment-14744848
 ] 

Haohui Mai commented on MAPREDUCE-6477:
---

+1. Will commit shortly.

> Replace usage of deprecated NameNode.DEFAULT_PORT in TestFileSystem
> ---
>
> Key: MAPREDUCE-6477
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6477
> Project: Hadoop Map/Reduce
>  Issue Type: Task
>Reporter: Mingliang Liu
>Assignee: Mingliang Liu
> Attachments: MAPREDUCE-6477.000.patch
>
>
> The {{NameNode.DEFAULT_PORT}} static attribute is stale as we use 
> {{HdfsClientConfigKeys.DFS_NAMENODE_RPC_PORT_DEFAULT}} config value. Jira 
> [HDFS-9010] marks it as {{@Deprecated}} and refactored its usages in 
> {{hadoop-hdfs}} module. This jira tracks the effort of replacing its usage in 
> {{org.apache.hadoop.fs.TestFileSystem}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-5003) AM recovery should recreate records for attempts that were incomplete

[Chang Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14744644#comment-14744644
 ] 

Chang Li commented on MAPREDUCE-5003:
-

The broken unit test is unrelated to my change. Have test this test on my local 
machine with my patch on and the test passed. [~jlowe] please help review the 
latest patch. Thanks!

> AM recovery should recreate records for attempts that were incomplete
> -
>
> Key: MAPREDUCE-5003
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5003
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: mr-am
>Reporter: Jason Lowe
>Assignee: Chang Li
> Attachments: MAPREDUCE-5003.1.patch, MAPREDUCE-5003.2.patch, 
> MAPREDUCE-5003.3.patch, MAPREDUCE-5003.4.patch, MAPREDUCE-5003.5.patch, 
> MAPREDUCE-5003.5.patch, MAPREDUCE-5003.6.patch, MAPREDUCE-5003.7.patch, 
> MAPREDUCE-5003.8.patch
>
>
> As discussed in MAPREDUCE-4992, it would be nice if the AM recovered task 
> attempt entries for *all* task attempts launched by the prior app attempt 
> even if those task attempts did not complete.  The attempts would have to be 
> marked as killed or something similar to indicate it is no longer running.  
> Having records for the task attempts enables the user to see what nodes were 
> associated with the attempts and potentially access their logs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side

[Jian He (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14744709#comment-14744709
 ] 

Jian He commented on MAPREDUCE-5870:


I see, I'm ok to keep it supporting the enum as that's what I originally 
thought.  Just want to bring this up.

> Support for passing Job priority through Application Submission Context in 
> Mapreduce Side
> -
>
> Key: MAPREDUCE-5870
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: client
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch, 
> 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch, 
> 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch, Yarn-2002.1.patch
>
>
> Job Prioirty can be set from client side as below [Configuration and api].
>   a.  JobConf.getJobPriority() and 
> Job.setPriority(JobPriority priority) 
>   b.  We can also use configuration 
> "mapreduce.job.priority".
>   Now this Job priority can be passed in Application Submission 
> context from Client side.
>   Here we can reuse the MRJobConfig.PRIORITY configuration. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (MAPREDUCE-5003) AM recovery should recreate records for attempts that were incomplete

[Chang Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-5003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chang Li updated MAPREDUCE-5003:

Attachment: MAPREDUCE-5003.8.patch

fix whitespace and some checkstyle issues

> AM recovery should recreate records for attempts that were incomplete
> -
>
> Key: MAPREDUCE-5003
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5003
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: mr-am
>Reporter: Jason Lowe
>Assignee: Chang Li
> Attachments: MAPREDUCE-5003.1.patch, MAPREDUCE-5003.2.patch, 
> MAPREDUCE-5003.3.patch, MAPREDUCE-5003.4.patch, MAPREDUCE-5003.5.patch, 
> MAPREDUCE-5003.5.patch, MAPREDUCE-5003.6.patch, MAPREDUCE-5003.7.patch, 
> MAPREDUCE-5003.8.patch
>
>
> As discussed in MAPREDUCE-4992, it would be nice if the AM recovered task 
> attempt entries for *all* task attempts launched by the prior app attempt 
> even if those task attempts did not complete.  The attempts would have to be 
> marked as killed or something similar to indicate it is no longer running.  
> Having records for the task attempts enables the user to see what nodes were 
> associated with the attempts and potentially access their logs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-6462) JobHistoryServer to support JvmPauseMonitor as a service

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14743981#comment-14743981
 ] 

Hadoop QA commented on MAPREDUCE-6462:
--

\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch |  23m 30s | Findbugs (version 3.0.0) 
appears to be broken on trunk. |
| {color:green}+1{color} | @author |   0m  0s | The patch does not contain any 
@author tags. |
| {color:green}+1{color} | tests included |   0m  0s | The patch appears to 
include 3 new or modified test files. |
| {color:green}+1{color} | javac |   7m 49s | There were no new javac warning 
messages. |
| {color:green}+1{color} | javadoc |  10m  1s | There were no new javadoc 
warning messages. |
| {color:green}+1{color} | release audit |   0m 23s | The applied patch does 
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle |   4m 16s | There were no new checkstyle 
issues. |
| {color:green}+1{color} | whitespace |   0m  1s | The patch has no lines that 
end in whitespace. |
| {color:green}+1{color} | install |   1m 37s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse |   0m 34s | The patch built with 
eclipse:eclipse. |
| {color:green}+1{color} | findbugs |  10m 17s | The patch does not introduce 
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | common tests |  23m 10s | Tests passed in 
hadoop-common. |
| {color:green}+1{color} | mapreduce tests |   6m 13s | Tests passed in 
hadoop-mapreduce-client-hs. |
| {color:green}+1{color} | yarn tests |   3m 17s | Tests passed in 
hadoop-yarn-server-applicationhistoryservice. |
| {color:green}+1{color} | yarn tests |   8m 12s | Tests passed in 
hadoop-yarn-server-nodemanager. |
| {color:green}+1{color} | yarn tests |  56m 18s | Tests passed in 
hadoop-yarn-server-resourcemanager. |
| {color:green}+1{color} | yarn tests |   0m 27s | Tests passed in 
hadoop-yarn-server-web-proxy. |
| {color:red}-1{color} | hdfs tests |  88m 33s | Tests failed in hadoop-hdfs. |
| {color:green}+1{color} | hdfs tests |   1m 46s | Tests passed in 
hadoop-hdfs-nfs. |
| | | 246m 28s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | 
hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistLockedMemory |
|   | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyWriter |
|   | hadoop.tools.TestJMXGet |
|   | 
hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement |
| Timed out tests | org.apache.hadoop.hdfs.server.namenode.TestAuditLogger |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL | 
http://issues.apache.org/jira/secure/attachment/12755738/HADOOP-12321-005-aggregated.patch
 |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 6955771 |
| hadoop-common test log | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5986/artifact/patchprocess/testrun_hadoop-common.txt
 |
| hadoop-mapreduce-client-hs test log | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5986/artifact/patchprocess/testrun_hadoop-mapreduce-client-hs.txt
 |
| hadoop-yarn-server-applicationhistoryservice test log | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5986/artifact/patchprocess/testrun_hadoop-yarn-server-applicationhistoryservice.txt
 |
| hadoop-yarn-server-nodemanager test log | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5986/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
 |
| hadoop-yarn-server-resourcemanager test log | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5986/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
 |
| hadoop-yarn-server-web-proxy test log | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5986/artifact/patchprocess/testrun_hadoop-yarn-server-web-proxy.txt
 |
| hadoop-hdfs test log | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5986/artifact/patchprocess/testrun_hadoop-hdfs.txt
 |
| hadoop-hdfs-nfs test log | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5986/artifact/patchprocess/testrun_hadoop-hdfs-nfs.txt
 |
| Test Results | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5986/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP 
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5986/console |


This message was automatically generated.

> JobHistoryServer to support JvmPauseMonitor as a service
> 
>
> Key: MAPREDUCE-6462
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6462
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: jobhistoryserver
>Affects Versions: 2.8.0
>

[jira] [Updated] (MAPREDUCE-6472) MapReduce AM should have java.io.tmpdir=./tmp to be consistent with tasks

[Naganarasimha G R (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-6472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R updated MAPREDUCE-6472:
-
Attachment: MAPREDUCE-6472.20150914-2.patch

Reuploading the patch with white space fixes. Checkstyle cannot be addressed as 
part of this patch

> MapReduce AM should have java.io.tmpdir=./tmp to be consistent with tasks
> -
>
> Key: MAPREDUCE-6472
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6472
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mr-am
>Affects Versions: 2.6.0
>Reporter: Jason Lowe
>Assignee: Naganarasimha G R
> Attachments: MAPREDUCE-6472.20150914-1.patch, 
> MAPREDUCE-6472.20150914-2.patch
>
>
> MapReduceChildJVM.getVMCommand ensures that all tasks have 
> -Djava.io.tmpdir=./tmp set as part of the task command-line, but this is only 
> used for tasks.  The AM itself does not have a corresponding java.io.tmpdir 
> setting.  It should also use the same tmpdir setting to avoid cases where the 
> AM JVM wants to place files in /tmp by default.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side

[Jason Lowe (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14743954#comment-14743954
 ] 

Jason Lowe commented on MAPREDUCE-5870:
---

I honestly don't know how much the priority APIs were used from MRv1.  However 
given that there were priority APIs in MRv1 that's the most logical thing 
people migrating from Hadoop 1 will expect to use when they hear priorities are 
finally working in Hadoop 2.

If we decide to deprecate the old MRv1 priority API, should we avoid using 
mapreduce.job.priority (was mapred.job.priority)?  Wondering if jobs that were 
working in Hadoop 2 suddenly start failing during job submit because someone 
was specifying something for that property that was always ignored and now the 
new code barfs on any non-integer found there.  However I'm not excited about 
the confusion of having two similar properties to set the job priority, one 
that works and one that does not.

> Support for passing Job priority through Application Submission Context in 
> Mapreduce Side
> -
>
> Key: MAPREDUCE-5870
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: client
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch, 
> 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch, 
> 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch, Yarn-2002.1.patch
>
>
> Job Prioirty can be set from client side as below [Configuration and api].
>   a.  JobConf.getJobPriority() and 
> Job.setPriority(JobPriority priority) 
>   b.  We can also use configuration 
> "mapreduce.job.priority".
>   Now this Job priority can be passed in Application Submission 
> context from Client side.
>   Here we can reuse the MRJobConfig.PRIORITY configuration. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (MAPREDUCE-6476) InvalidResourceException when Nodelabel don't have access to queue should be handled

[Bibin A Chundatt (JIRA)]

Bibin A Chundatt created MAPREDUCE-6476:
---

 Summary: InvalidResourceException when Nodelabel don't have access 
to queue should be handled
 Key: MAPREDUCE-6476
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6476
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: applicationmaster
Reporter: Bibin A Chundatt
Assignee: Bibin A Chundatt


Steps to reproduce
===
Submit  mapreduce job 
# map to label x
# reduce to label y

Precondition
# Queue b to which reduce is submitted not having access to label specified

*Impact*
# Jobs fail only of the RM-AM comunication timeout
(About 10 mins i think)

Should kill the job immediately when InvalidResourceException is received on 
{{RMContainerRequestor#makeRemoteRequest}}
*Logs*

{noformat}
2015-09-11 16:44:30,116 ERROR [RMCommunicator Allocator] 
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: ERROR IN CONTACTING RM. 
org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid 
resource request, queue=b1 doesn't have permission to access all labels in 
resource request. labelExpression of resource request=1. Queue labels=3
at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:304)
at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:234)
at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndvalidateRequest(SchedulerUtils.java:250)
at 
org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.normalizeAndValidateRequests(RMServerUtils.java:106)
at 
org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:457)
at 
org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
at 
org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:636)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:976)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2230)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2226)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1667)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2224)

at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown 
Source)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at 
org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
at 
org.apache.hadoop.yarn.ipc.RPCUtil.instantiateYarnException(RPCUtil.java:75)
at 
org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:116)
at 
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:79)
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:251)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
at com.sun.proxy.$Proxy37.allocate(Unknown Source)
at 
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor.makeRemoteRequest(RMContainerRequestor.java:203)
at 
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.getResources(RMContainerAllocator.java:694)
at 
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.heartbeat(RMContainerAllocator.java:263)
at 
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator$AllocatorRunnable.run(RMCommunicator.java:281)
at java.lang.Thread.run(Thread.java:745)
Caused by: 
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException):
 Invalid resource request, queue=b1 doesn't have permission to access all 
labels in resource request. labelExpression of resource request=1. Queue 
labels=3
at 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:304)
at 

[jira] [Commented] (MAPREDUCE-6476) InvalidResourceException when Nodelabel don't have access to queue should be handled

[Bibin A Chundatt (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14743019#comment-14743019
 ] 

Bibin A Chundatt commented on MAPREDUCE-6476:
-

\cc [~leftnoteasy]

> InvalidResourceException when Nodelabel don't have access to queue should be 
> handled
> 
>
> Key: MAPREDUCE-6476
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6476
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: applicationmaster
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>
> Steps to reproduce
> ===
> Submit  mapreduce job 
> # map to label x
> # reduce to label y
> Precondition
> # Queue b to which reduce is submitted not having access to label specified
> *Impact*
> # Jobs fail only of the RM-AM comunication timeout
> (About 10 mins i think)
> Should kill the job immediately when InvalidResourceException is received on 
> {{RMContainerRequestor#makeRemoteRequest}}
> *Logs*
> {noformat}
> 2015-09-11 16:44:30,116 ERROR [RMCommunicator Allocator] 
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator: ERROR IN CONTACTING RM. 
> org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException: Invalid 
> resource request, queue=b1 doesn't have permission to access all labels in 
> resource request. labelExpression of resource request=1. Queue labels=3
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.validateResourceRequest(SchedulerUtils.java:304)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndValidateRequest(SchedulerUtils.java:234)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.normalizeAndvalidateRequest(SchedulerUtils.java:250)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.normalizeAndValidateRequests(RMServerUtils.java:106)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:457)
>   at 
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationMasterProtocolPBServiceImpl.allocate(ApplicationMasterProtocolPBServiceImpl.java:60)
>   at 
> org.apache.hadoop.yarn.proto.ApplicationMasterProtocol$ApplicationMasterProtocolService$2.callBlockingMethod(ApplicationMasterProtocol.java:99)
>   at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:636)
>   at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:976)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2230)
>   at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2226)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at javax.security.auth.Subject.doAs(Subject.java:422)
>   at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1667)
>   at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2224)
>   at sun.reflect.GeneratedConstructorAccessor39.newInstance(Unknown 
> Source)
>   at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>   at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
>   at 
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
>   at 
> org.apache.hadoop.yarn.ipc.RPCUtil.instantiateYarnException(RPCUtil.java:75)
>   at 
> org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:116)
>   at 
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:79)
>   at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:497)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:251)
>   at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:103)
>   at com.sun.proxy.$Proxy37.allocate(Unknown Source)
>   at 
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor.makeRemoteRequest(RMContainerRequestor.java:203)
>   at 
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.getResources(RMContainerAllocator.java:694)
>   at 
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.heartbeat(RMContainerAllocator.java:263)
>   at 
> org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator$AllocatorRunnable.run(RMCommunicator.java:281)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: 
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.yarn.exceptions.InvalidResourceRequestException):

[jira] [Updated] (MAPREDUCE-3919) Redirecting to job history server takes hours

[mingleizhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mingleizhang updated MAPREDUCE-3919:

Assignee: (was: mingleizhang)

> Redirecting to job history server takes hours
> -
>
> Key: MAPREDUCE-3919
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3919
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Daniel Dai
>Priority: Critical
>
> Saw the following message happening regularly, the job end up success, but 
> reconnecting job history server takes a long time (>10 hours sometimes).
> 2012-02-24 03:49:05,226 [main] INFO  org.apache.hadoop.ipc.Client - Retrying 
> connect to server: hrt11n31.cc1.ygridcore.net/98.137.234.159:44716. Already 
> tried 0 time(s).
> 2012-02-24 03:49:05,229 [main] INFO  
> org.apache.hadoop.mapred.ClientServiceDelegate - Application state is 
> completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
> 2012-02-24 03:49:06,233 [main] INFO  org.apache.hadoop.ipc.Client - Retrying 
> connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 0 time(s).
> 2012-02-24 03:49:07,236 [main] INFO  org.apache.hadoop.ipc.Client - Retrying 
> connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 1 time(s).
> 2012-02-24 03:49:08,239 [main] INFO  org.apache.hadoop.ipc.Client - Retrying 
> connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 2 time(s).
> 2012-02-24 03:49:09,242 [main] INFO  org.apache.hadoop.ipc.Client - Retrying 
> connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 3 time(s).
> 2012-02-24 03:49:10,245 [main] INFO  org.apache.hadoop.ipc.Client - Retrying 
> connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 4 time(s).
> 2012-02-24 03:49:11,248 [main] INFO  org.apache.hadoop.ipc.Client - Retrying 
> connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 5 time(s).
> 2012-02-24 03:49:12,251 [main] INFO  org.apache.hadoop.ipc.Client - Retrying 
> connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 6 time(s).
> 2012-02-24 03:49:13,254 [main] INFO  org.apache.hadoop.ipc.Client - Retrying 
> connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 7 time(s).
> 2012-02-24 03:49:14,257 [main] INFO  org.apache.hadoop.ipc.Client - Retrying 
> connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 8 time(s).
> 2012-02-24 03:49:15,260 [main] INFO  org.apache.hadoop.ipc.Client - Retrying 
> connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 9 time(s).
> ..
> 2012-02-24 18:10:35,711 [main] INFO  
> org.apache.hadoop.mapred.ClientServiceDelegate - Application state is 
> completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
> 2012-02-24 18:10:36,714 [main] INFO  org.apache.hadoop.ipc.Client - Retrying 
> connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 0 time(s).
> 2012-02-24 18:10:37,717 [main] INFO  org.apache.hadoop.ipc.Client - Retrying 
> connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 1 time(s).2012-02-24 
> 18:10:38,784 [main] INFO  org.apache.hadoop.ipc.Client - Retrying connect to 
> server: 0.0.0.0/0.0.0.0:10020. Already tried 2 time(s).2012-02-24 
> 18:10:39,787 [main] INFO  org.apache.hadoop.ipc.Client - Retrying connect to 
> server: 0.0.0.0/0.0.0.0:10020. Already tried 3 time(s).
> 2012-02-24 18:10:40,791 [main] INFO  org.apache.hadoop.ipc.Client - Retrying 
> connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 4 time(s).
> 2012-02-24 18:10:41,793 [main] INFO  org.apache.hadoop.ipc.Client - Retrying 
> connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 5 time(s).2012-02-24 
> 18:10:42,796 [main] INFO  org.apache.hadoop.ipc.Client - Retrying connect to 
> server: 0.0.0.0/0.0.0.0:10020. Already tried 6 time(s).2012-02-24 
> 18:10:43,799 [main] INFO  org.apache.hadoop.ipc.Client - Retrying connect to 
> server: 0.0.0.0/0.0.0.0:10020. Already tried 7 time(s).
> 2012-02-24 18:10:44,802 [main] INFO  org.apache.hadoop.ipc.Client - Retrying 
> connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 8 time(s).
> 2012-02-24 18:10:45,805 [main] INFO  org.apache.hadoop.ipc.Client - Retrying 
> connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 9 time(s).
> 2012-02-24 18:10:45,808 [main] INFO  
> org.apache.hadoop.mapred.ClientServiceDelegate - Application state is 
> completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
> 2012-02-24 18:10:46,810 [main] INFO  org.apache.hadoop.ipc.Client - Retrying 
> connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 0 time(s).2012-02-24 
> 18:10:47,813 [main] INFO  org.apache.hadoop.ipc.Client - Retrying connect to 
> server: 0.0.0.0/0.0.0.0:10020. Already tried 1 time(s).2012-02-24 
> 18:10:48,815 [main] INFO  org.apache.hadoop.ipc.Client - Retrying connect to 
> server: 0.0.0.0/0.0.0.0:10020. Already tried 2 

[jira] [Commented] (MAPREDUCE-5983) TestCommandLineJobSubmission assumes there is a /tmp dir that can be used for temp data

[Steve Loughran (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14743262#comment-14743262
 ] 

Steve Loughran commented on MAPREDUCE-5983:
---

the test can use {{FileContextTestHelper}} to pick up a test dir

> TestCommandLineJobSubmission assumes there is a /tmp dir that can be used for 
> temp data
> ---
>
> Key: MAPREDUCE-5983
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5983
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: test
>Reporter: Steve Loughran
>Assignee: Zhang Kun
>Priority: Minor
>  Labels: newbie
>
> the test case in TestCommandLineJobSubmission assumes that it can create 
> stuff in /tmp:
> File buildDir = new File(System.getProperty("test.build.data", "/tmp"));
> as well as being unix only, the result is a test case that doesnt work if >1 
> person is running the test on the same cluster. it should really be driven 
> off the java.io.tmpDir property



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side

[Jian He (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14743154#comment-14743154
 ] 

Jian He commented on MAPREDUCE-5870:


I earlier thought we can keep backward compatible with the enum priority, but 
now am thinking the value of doing this. This does bring extra complexity to 
support both. [~jlowe], do you know if there are many apps from MR1 are 
actually expecting this enum based priority to work ? Since priority is never 
supported since hadoop 2 for such a long time, I'm thinking if we can deprecate 
the old API and claim only support integers to be simple and clear.

> Support for passing Job priority through Application Submission Context in 
> Mapreduce Side
> -
>
> Key: MAPREDUCE-5870
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: client
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch, 
> 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch, 
> 0005-MAPREDUCE-5870.patch, 0006-MAPREDUCE-5870.patch, Yarn-2002.1.patch
>
>
> Job Prioirty can be set from client side as below [Configuration and api].
>   a.  JobConf.getJobPriority() and 
> Job.setPriority(JobPriority priority) 
>   b.  We can also use configuration 
> "mapreduce.job.priority".
>   Now this Job priority can be passed in Application Submission 
> context from Client side.
>   Here we can reuse the MRJobConfig.PRIORITY configuration. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (MAPREDUCE-3919) Redirecting to job history server takes hours

[mingleizhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

mingleizhang reassigned MAPREDUCE-3919:
---

Assignee: mingleizhang

> Redirecting to job history server takes hours
> -
>
> Key: MAPREDUCE-3919
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3919
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Daniel Dai
>Assignee: mingleizhang
>Priority: Critical
>
> Saw the following message happening regularly, the job end up success, but 
> reconnecting job history server takes a long time (>10 hours sometimes).
> 2012-02-24 03:49:05,226 [main] INFO  org.apache.hadoop.ipc.Client - Retrying 
> connect to server: hrt11n31.cc1.ygridcore.net/98.137.234.159:44716. Already 
> tried 0 time(s).
> 2012-02-24 03:49:05,229 [main] INFO  
> org.apache.hadoop.mapred.ClientServiceDelegate - Application state is 
> completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
> 2012-02-24 03:49:06,233 [main] INFO  org.apache.hadoop.ipc.Client - Retrying 
> connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 0 time(s).
> 2012-02-24 03:49:07,236 [main] INFO  org.apache.hadoop.ipc.Client - Retrying 
> connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 1 time(s).
> 2012-02-24 03:49:08,239 [main] INFO  org.apache.hadoop.ipc.Client - Retrying 
> connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 2 time(s).
> 2012-02-24 03:49:09,242 [main] INFO  org.apache.hadoop.ipc.Client - Retrying 
> connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 3 time(s).
> 2012-02-24 03:49:10,245 [main] INFO  org.apache.hadoop.ipc.Client - Retrying 
> connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 4 time(s).
> 2012-02-24 03:49:11,248 [main] INFO  org.apache.hadoop.ipc.Client - Retrying 
> connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 5 time(s).
> 2012-02-24 03:49:12,251 [main] INFO  org.apache.hadoop.ipc.Client - Retrying 
> connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 6 time(s).
> 2012-02-24 03:49:13,254 [main] INFO  org.apache.hadoop.ipc.Client - Retrying 
> connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 7 time(s).
> 2012-02-24 03:49:14,257 [main] INFO  org.apache.hadoop.ipc.Client - Retrying 
> connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 8 time(s).
> 2012-02-24 03:49:15,260 [main] INFO  org.apache.hadoop.ipc.Client - Retrying 
> connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 9 time(s).
> ..
> 2012-02-24 18:10:35,711 [main] INFO  
> org.apache.hadoop.mapred.ClientServiceDelegate - Application state is 
> completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
> 2012-02-24 18:10:36,714 [main] INFO  org.apache.hadoop.ipc.Client - Retrying 
> connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 0 time(s).
> 2012-02-24 18:10:37,717 [main] INFO  org.apache.hadoop.ipc.Client - Retrying 
> connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 1 time(s).2012-02-24 
> 18:10:38,784 [main] INFO  org.apache.hadoop.ipc.Client - Retrying connect to 
> server: 0.0.0.0/0.0.0.0:10020. Already tried 2 time(s).2012-02-24 
> 18:10:39,787 [main] INFO  org.apache.hadoop.ipc.Client - Retrying connect to 
> server: 0.0.0.0/0.0.0.0:10020. Already tried 3 time(s).
> 2012-02-24 18:10:40,791 [main] INFO  org.apache.hadoop.ipc.Client - Retrying 
> connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 4 time(s).
> 2012-02-24 18:10:41,793 [main] INFO  org.apache.hadoop.ipc.Client - Retrying 
> connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 5 time(s).2012-02-24 
> 18:10:42,796 [main] INFO  org.apache.hadoop.ipc.Client - Retrying connect to 
> server: 0.0.0.0/0.0.0.0:10020. Already tried 6 time(s).2012-02-24 
> 18:10:43,799 [main] INFO  org.apache.hadoop.ipc.Client - Retrying connect to 
> server: 0.0.0.0/0.0.0.0:10020. Already tried 7 time(s).
> 2012-02-24 18:10:44,802 [main] INFO  org.apache.hadoop.ipc.Client - Retrying 
> connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 8 time(s).
> 2012-02-24 18:10:45,805 [main] INFO  org.apache.hadoop.ipc.Client - Retrying 
> connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 9 time(s).
> 2012-02-24 18:10:45,808 [main] INFO  
> org.apache.hadoop.mapred.ClientServiceDelegate - Application state is 
> completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
> 2012-02-24 18:10:46,810 [main] INFO  org.apache.hadoop.ipc.Client - Retrying 
> connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 0 time(s).2012-02-24 
> 18:10:47,813 [main] INFO  org.apache.hadoop.ipc.Client - Retrying connect to 
> server: 0.0.0.0/0.0.0.0:10020. Already tried 1 time(s).2012-02-24 
> 18:10:48,815 [main] INFO  org.apache.hadoop.ipc.Client - Retrying connect to 
> server: 

[jira] [Commented] (MAPREDUCE-6462) JobHistoryServer to support JvmPauseMonitor as a service

[Steve Loughran (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-6462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14743205#comment-14743205
 ] 

Steve Loughran commented on MAPREDUCE-6462:
---

-1 —due to test assumptions

# {{TestApplicationHistoryServer}} failing as # of child services is 6, not 5
# same for {{TestJobHistoryServer}}

Two fixes come to mind
# increase the count, while adding an explanation message "child service count" 
to the asserts
# cull the check altogether. Why do the tests need to validate the no. of 
children. It's too brittle and delivers no apparent benefit.

> JobHistoryServer to support JvmPauseMonitor as a service
> 
>
> Key: MAPREDUCE-6462
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6462
> Project: Hadoop Map/Reduce
>  Issue Type: Improvement
>  Components: jobhistoryserver
>Affects Versions: 2.8.0
>Reporter: Sunil G
>Assignee: Sunil G
>Priority: Minor
> Attachments: 0001-MAPREDUCE-6462.patch, HADOOP-12321-003.patch
>
>
> As JvmPauseMonitor is made as an AbstractService, subsequent method changes 
> are needed in all places which uses the monitor.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)