[jira] [Commented] (MAPREDUCE-6284) Add a 'task attempt state' to MapReduce Application Master REST API
[ https://issues.apache.org/jira/browse/MAPREDUCE-6284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14532277#comment-14532277 ] Tsuyoshi Ozawa commented on MAPREDUCE-6284: --- OK, I'm checking... Add a 'task attempt state' to MapReduce Application Master REST API --- Key: MAPREDUCE-6284 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6284 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Ryu Kobayashi Assignee: Ryu Kobayashi Priority: Minor Labels: BB2015-05-TBR Attachments: MAPREDUCE-6284.1.patch, MAPREDUCE-6284.1.patch, MAPREDUCE-6284.2.patch, MAPREDUCE-6284.3.patch, MAPREDUCE-6284.3.patch, MAPREDUCE-6284.4.patch, MAPREDUCE-6284.5.patch It want to 'task attempt state' on the 'App state' similarly REST API. GET http://proxy http address:port/proxy/application _id/ws/v1/mapreduce/jobs/job_id/tasks/task_id/attempts/attempt_id/state PUT http://proxy http address:port/proxy/application _id/ws/v1/mapreduce/jobs/job_id/tasks/task_id/attempts/attempt_id/state -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6284) Add a 'task attempt state' to MapReduce Application Master REST API
[ https://issues.apache.org/jira/browse/MAPREDUCE-6284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14532265#comment-14532265 ] Ryu Kobayashi commented on MAPREDUCE-6284: -- [~ozawa] I fixed the checkstyle. and any more... Add a 'task attempt state' to MapReduce Application Master REST API --- Key: MAPREDUCE-6284 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6284 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Ryu Kobayashi Assignee: Ryu Kobayashi Priority: Minor Labels: BB2015-05-TBR Attachments: MAPREDUCE-6284.1.patch, MAPREDUCE-6284.1.patch, MAPREDUCE-6284.2.patch, MAPREDUCE-6284.3.patch, MAPREDUCE-6284.3.patch, MAPREDUCE-6284.4.patch, MAPREDUCE-6284.5.patch It want to 'task attempt state' on the 'App state' similarly REST API. GET http://proxy http address:port/proxy/application _id/ws/v1/mapreduce/jobs/job_id/tasks/task_id/attempts/attempt_id/state PUT http://proxy http address:port/proxy/application _id/ws/v1/mapreduce/jobs/job_id/tasks/task_id/attempts/attempt_id/state -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6356) Misspelling of threshold in log4j.properties for tests
[
https://issues.apache.org/jira/browse/MAPREDUCE-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14532104#comment-14532104
]
Hadoop QA commented on MAPREDUCE-6356:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 5m 13s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 7 new or modified test files. |
| {color:green}+1{color} | javac | 7m 38s | There were no new javac warning
messages. |
| {color:green}+1{color} | release audit | 0m 19s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | mapreduce tests | 9m 25s | Tests passed in
hadoop-mapreduce-client-app. |
| {color:green}+1{color} | mapreduce tests | 0m 46s | Tests passed in
hadoop-mapreduce-client-common. |
| {color:green}+1{color} | mapreduce tests | 1m 44s | Tests passed in
hadoop-mapreduce-client-core. |
| {color:green}+1{color} | mapreduce tests | 5m 36s | Tests passed in
hadoop-mapreduce-client-hs. |
| {color:red}-1{color} | mapreduce tests | 30m 28s | Tests failed in
hadoop-mapreduce-client-jobclient. |
| {color:green}+1{color} | mapreduce tests | 10m 10s | Tests passed in
hadoop-mapreduce-client-nativetask. |
| {color:green}+1{color} | mapreduce tests | 0m 20s | Tests passed in
hadoop-mapreduce-client-shuffle. |
| | | 73m 48s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.mapreduce.v2.TestSpeculativeExecution |
| | hadoop.mapreduce.security.ssl.TestEncryptedShuffle |
| | hadoop.mapred.TestMiniMRChildTask |
| | hadoop.mapred.TestMiniMRBringup |
| | hadoop.mapred.TestMiniMRClientCluster |
| | hadoop.mapred.TestClusterMRNotification |
| | hadoop.mapred.TestMRTimelineEventHandling |
| | hadoop.mapreduce.v2.TestMRJobsWithHistoryService |
| | hadoop.mapreduce.security.TestBinaryTokenFile |
| | hadoop.mapred.TestReduceFetch |
| | hadoop.mapreduce.v2.TestNonExistentJob |
| | hadoop.mapreduce.v2.TestMiniMRProxyUser |
| | hadoop.mapred.TestNetworkedJob |
| | hadoop.mapreduce.v2.TestMROldApiJobs |
| | hadoop.mapred.TestClusterMapReduceTestCase |
| | hadoop.mapred.TestMRIntermediateDataEncryption |
| | hadoop.mapred.TestReduceFetchFromPartialMem |
| | hadoop.mapreduce.lib.output.TestJobOutputCommitter |
| | hadoop.mapreduce.TestLargeSort |
| | hadoop.mapreduce.v2.TestMRAppWithCombiner |
| | hadoop.mapred.TestJobName |
| | hadoop.mapreduce.v2.TestMRJobsWithProfiler |
| | hadoop.mapred.TestLocalJobSubmission |
| | hadoop.mapreduce.TestMapReduceLazyOutput |
| | hadoop.mapred.TestSpecialCharactersInOutputPath |
| | hadoop.mapreduce.v2.TestUberAM |
| | hadoop.mapreduce.security.TestMRCredentials |
| | hadoop.mapred.TestLazyOutput |
| | hadoop.mapred.TestMerge |
| | hadoop.mapreduce.v2.TestMRAMWithNonNormalizedCapabilities |
| | hadoop.mapreduce.v2.TestRMNMInfo |
| | hadoop.mapred.TestJobCleanup |
| | hadoop.mapreduce.TestChild |
| | hadoop.mapreduce.TestMRJobClient |
| | hadoop.mapreduce.v2.TestMRJobs |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12731078/MAPREDUCE-6356-002.patch
|
| Optional Tests | javac unit |
| git revision | trunk / 918af8e |
| hadoop-mapreduce-client-app test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5656/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
|
| hadoop-mapreduce-client-common test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5656/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
|
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5656/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| hadoop-mapreduce-client-hs test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5656/artifact/patchprocess/testrun_hadoop-mapreduce-client-hs.txt
|
| hadoop-mapreduce-client-jobclient test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5656/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
|
| hadoop-mapreduce-client-nativetask test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5656/artifact/patchprocess/testrun_hadoop-mapreduce-client-nativetask.txt
|
| hadoop-mapreduce-client-shuffle test log |
[jira] [Commented] (MAPREDUCE-6284) Add a 'task attempt state' to MapReduce Application Master REST API
[
https://issues.apache.org/jira/browse/MAPREDUCE-6284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14532125#comment-14532125
]
Hadoop QA commented on MAPREDUCE-6284:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 36s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 3 new or modified test files. |
| {color:green}+1{color} | javac | 7m 27s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 34s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | site | 2m 58s | Site still builds. |
| {color:red}-1{color} | checkstyle | 1m 8s | The applied patch generated 9
new checkstyle issues (total was 36, now 45). |
| {color:red}-1{color} | checkstyle | 1m 42s | The applied patch generated 7
new checkstyle issues (total was 14, now 21). |
| {color:red}-1{color} | whitespace | 0m 5s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 42s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests | 9m 31s | Tests passed in
hadoop-mapreduce-client-app. |
| {color:green}+1{color} | mapreduce tests | 1m 33s | Tests passed in
hadoop-mapreduce-client-core. |
| {color:green}+1{color} | yarn tests | 0m 21s | Tests passed in
hadoop-yarn-server-web-proxy. |
| | | 56m 2s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12731084/MAPREDUCE-6284.4.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle site |
| git revision | trunk / 918af8e |
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5657/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-app.txt
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5657/artifact/patchprocess/diffcheckstylehadoop-yarn-server-web-proxy.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5657/artifact/patchprocess/whitespace.txt
|
| hadoop-mapreduce-client-app test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5657/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
|
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5657/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| hadoop-yarn-server-web-proxy test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5657/artifact/patchprocess/testrun_hadoop-yarn-server-web-proxy.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5657/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5657/console |
This message was automatically generated.
Add a 'task attempt state' to MapReduce Application Master REST API
---
Key: MAPREDUCE-6284
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6284
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Ryu Kobayashi
Assignee: Ryu Kobayashi
Priority: Minor
Labels: BB2015-05-TBR
Attachments: MAPREDUCE-6284.1.patch, MAPREDUCE-6284.1.patch,
MAPREDUCE-6284.2.patch, MAPREDUCE-6284.3.patch, MAPREDUCE-6284.3.patch,
MAPREDUCE-6284.4.patch
It want to 'task attempt state' on the 'App state' similarly REST API.
GET http://proxy http address:port/proxy/application
_id/ws/v1/mapreduce/jobs/job_id/tasks/task_id/attempts/attempt_id/state
PUT http://proxy http address:port/proxy/application
_id/ws/v1/mapreduce/jobs/job_id/tasks/task_id/attempts/attempt_id/state
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6356) Misspelling of threshold in log4j.properties for tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14532190#comment-14532190 ] Akira AJISAKA commented on MAPREDUCE-6356: -- Many tests are failing with java.lang.NoClassDefFoundError. Looks like the jars in the repository used by the tests are stale. Misspelling of threshold in log4j.properties for tests -- Key: MAPREDUCE-6356 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6356 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Reporter: Brahma Reddy Battula Assignee: Brahma Reddy Battula Priority: Minor Labels: BB2015-05-RFC Attachments: MAPREDUCE-6356-002.patch, MAPREDUCE-6356.patch log4j.properties file for test contains misspelling log4j.threshhold. We should use log4j.threshold correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6284) Add a 'task attempt state' to MapReduce Application Master REST API
[ https://issues.apache.org/jira/browse/MAPREDUCE-6284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryu Kobayashi updated MAPREDUCE-6284: - Attachment: MAPREDUCE-6284.5.patch Add a 'task attempt state' to MapReduce Application Master REST API --- Key: MAPREDUCE-6284 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6284 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Ryu Kobayashi Assignee: Ryu Kobayashi Priority: Minor Labels: BB2015-05-TBR Attachments: MAPREDUCE-6284.1.patch, MAPREDUCE-6284.1.patch, MAPREDUCE-6284.2.patch, MAPREDUCE-6284.3.patch, MAPREDUCE-6284.3.patch, MAPREDUCE-6284.4.patch, MAPREDUCE-6284.5.patch It want to 'task attempt state' on the 'App state' similarly REST API. GET http://proxy http address:port/proxy/application _id/ws/v1/mapreduce/jobs/job_id/tasks/task_id/attempts/attempt_id/state PUT http://proxy http address:port/proxy/application _id/ws/v1/mapreduce/jobs/job_id/tasks/task_id/attempts/attempt_id/state -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6284) Add a 'task attempt state' to MapReduce Application Master REST API
[
https://issues.apache.org/jira/browse/MAPREDUCE-6284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14532246#comment-14532246
]
Hadoop QA commented on MAPREDUCE-6284:
--
\\
\\
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 36s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 3 new or modified test files. |
| {color:green}+1{color} | javac | 7m 42s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 55s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | site | 2m 59s | Site still builds. |
| {color:green}+1{color} | checkstyle | 1m 25s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 5s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 35s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 47s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | mapreduce tests | 9m 58s | Tests passed in
hadoop-mapreduce-client-app. |
| {color:green}+1{color} | mapreduce tests | 1m 43s | Tests passed in
hadoop-mapreduce-client-core. |
| {color:green}+1{color} | yarn tests | 0m 22s | Tests passed in
hadoop-yarn-server-web-proxy. |
| | | 57m 6s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12731115/MAPREDUCE-6284.5.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle site |
| git revision | trunk / 449e442 |
| hadoop-mapreduce-client-app test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5658/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
|
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5658/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| hadoop-yarn-server-web-proxy test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5658/artifact/patchprocess/testrun_hadoop-yarn-server-web-proxy.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5658/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5658/console |
This message was automatically generated.
Add a 'task attempt state' to MapReduce Application Master REST API
---
Key: MAPREDUCE-6284
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6284
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Ryu Kobayashi
Assignee: Ryu Kobayashi
Priority: Minor
Labels: BB2015-05-TBR
Attachments: MAPREDUCE-6284.1.patch, MAPREDUCE-6284.1.patch,
MAPREDUCE-6284.2.patch, MAPREDUCE-6284.3.patch, MAPREDUCE-6284.3.patch,
MAPREDUCE-6284.4.patch, MAPREDUCE-6284.5.patch
It want to 'task attempt state' on the 'App state' similarly REST API.
GET http://proxy http address:port/proxy/application
_id/ws/v1/mapreduce/jobs/job_id/tasks/task_id/attempts/attempt_id/state
PUT http://proxy http address:port/proxy/application
_id/ws/v1/mapreduce/jobs/job_id/tasks/task_id/attempts/attempt_id/state
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6356) Misspelling of threshold in log4j.properties for tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14532191#comment-14532191 ] Akira AJISAKA commented on MAPREDUCE-6356: -- Submitted jenkins job: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5659/ Misspelling of threshold in log4j.properties for tests -- Key: MAPREDUCE-6356 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6356 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Reporter: Brahma Reddy Battula Assignee: Brahma Reddy Battula Priority: Minor Labels: BB2015-05-RFC Attachments: MAPREDUCE-6356-002.patch, MAPREDUCE-6356.patch log4j.properties file for test contains misspelling log4j.threshhold. We should use log4j.threshold correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6356) Misspelling of threshold in log4j.properties for tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14532082#comment-14532082 ] Akira AJISAKA commented on MAPREDUCE-6356: -- +1 pending Jenkins. Thanks [~brahmareddy]. Misspelling of threshold in log4j.properties for tests -- Key: MAPREDUCE-6356 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6356 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Reporter: Brahma Reddy Battula Assignee: Brahma Reddy Battula Priority: Minor Labels: BB2015-05-TBR Attachments: MAPREDUCE-6356-002.patch, MAPREDUCE-6356.patch log4j.properties file for test contains misspelling log4j.threshhold. We should use log4j.threshold correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6356) Misspelling of threshold in log4j.properties for tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated MAPREDUCE-6356: - Labels: BB2015-05-RFC (was: BB2015-05-TBR) Hadoop Flags: Reviewed Misspelling of threshold in log4j.properties for tests -- Key: MAPREDUCE-6356 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6356 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Reporter: Brahma Reddy Battula Assignee: Brahma Reddy Battula Priority: Minor Labels: BB2015-05-RFC Attachments: MAPREDUCE-6356-002.patch, MAPREDUCE-6356.patch log4j.properties file for test contains misspelling log4j.threshhold. We should use log4j.threshold correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6358) Document missing properties in mapred-default.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-6358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14533105#comment-14533105 ] Ray Chiang commented on MAPREDUCE-6358: --- Link to YARN equivalent. Document missing properties in mapred-default.xml - Key: MAPREDUCE-6358 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6358 Project: Hadoop Map/Reduce Issue Type: Bug Components: documentation Affects Versions: 2.7.0 Reporter: Ray Chiang Assignee: Ray Chiang The following properties are currently not defined in mapred-default.xml. These properties should either be A) documented in mapred-default.xml OR B) listed as an exception (with comments, e.g. for internal use) in the TestMapreduceConfigFields unit test -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6302) deadlock in a job between map and reduce cores allocation
[ https://issues.apache.org/jira/browse/MAPREDUCE-6302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated MAPREDUCE-6302: -- Status: Patch Available (was: Open) deadlock in a job between map and reduce cores allocation -- Key: MAPREDUCE-6302 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6302 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.6.0 Reporter: mai shurong Assignee: Karthik Kambatla Priority: Critical Attachments: AM_log_head10.txt.gz, AM_log_tail10.txt.gz, log.txt, mr-6302-prelim.patch, queue_with_max163cores.png, queue_with_max263cores.png, queue_with_max333cores.png I submit a big job, which has 500 maps and 350 reduce, to a queue(fairscheduler) with 300 max cores. When the big mapreduce job is running 100% maps, the 300 reduces have occupied 300 max cores in the queue. And then, a map fails and retry, waiting for a core, while the 300 reduces are waiting for failed map to finish. So a deadlock occur. As a result, the job is blocked, and the later job in the queue cannot run because no available cores in the queue. I think there is the similar issue for memory of a queue . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6302) deadlock in a job between map and reduce cores allocation
[
https://issues.apache.org/jira/browse/MAPREDUCE-6302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14533618#comment-14533618
]
Wangda Tan commented on MAPREDUCE-6302:
---
[~kasha],
Thanks for working on this.
Just take a look at your patch, overall approach looks good, some comments
about configuration:
{{MR_JOB_REDUCER_FORCE_PREEMPT_DELAY_SEC}}
It is actually not REDUCER_FORCE_PREEMPT_DELAY, it is timeout of mapper
allocation to start reducer preemption, I suggest to rename it to be:
mapreduce.job.mapper.timeout-to-start-reducer-preemption-ms. I think it's
better to use ms instead of sec to better control it.
In addition, do you think should we add a value to let user choose to disable
this? For example, -1.
And could you add some tests?
deadlock in a job between map and reduce cores allocation
--
Key: MAPREDUCE-6302
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6302
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: mai shurong
Assignee: Karthik Kambatla
Priority: Critical
Attachments: AM_log_head10.txt.gz, AM_log_tail10.txt.gz,
log.txt, mr-6302-prelim.patch, queue_with_max163cores.png,
queue_with_max263cores.png, queue_with_max333cores.png
I submit a big job, which has 500 maps and 350 reduce, to a
queue(fairscheduler) with 300 max cores. When the big mapreduce job is
running 100% maps, the 300 reduces have occupied 300 max cores in the queue.
And then, a map fails and retry, waiting for a core, while the 300 reduces
are waiting for failed map to finish. So a deadlock occur. As a result, the
job is blocked, and the later job in the queue cannot run because no
available cores in the queue.
I think there is the similar issue for memory of a queue .
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6302) deadlock in a job between map and reduce cores allocation
[ https://issues.apache.org/jira/browse/MAPREDUCE-6302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14533619#comment-14533619 ] Wangda Tan commented on MAPREDUCE-6302: --- Linked to YARN-1680, one is for more accurate calculation, one is to prevent inaccurate calculation. deadlock in a job between map and reduce cores allocation -- Key: MAPREDUCE-6302 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6302 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.6.0 Reporter: mai shurong Assignee: Karthik Kambatla Priority: Critical Attachments: AM_log_head10.txt.gz, AM_log_tail10.txt.gz, log.txt, mr-6302-prelim.patch, queue_with_max163cores.png, queue_with_max263cores.png, queue_with_max333cores.png I submit a big job, which has 500 maps and 350 reduce, to a queue(fairscheduler) with 300 max cores. When the big mapreduce job is running 100% maps, the 300 reduces have occupied 300 max cores in the queue. And then, a map fails and retry, waiting for a core, while the 300 reduces are waiting for failed map to finish. So a deadlock occur. As a result, the job is blocked, and the later job in the queue cannot run because no available cores in the queue. I think there is the similar issue for memory of a queue . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6302) deadlock in a job between map and reduce cores allocation
[ https://issues.apache.org/jira/browse/MAPREDUCE-6302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14533631#comment-14533631 ] zhihai xu commented on MAPREDUCE-6302: -- Also linked to YARN-3446 for discussion, YARN-1680 is for CapacityScheduler and YARN-3446 is for FairScheduler. It will be good to merge common functionality into AbstractYarnScheduler. deadlock in a job between map and reduce cores allocation -- Key: MAPREDUCE-6302 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6302 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.6.0 Reporter: mai shurong Assignee: Karthik Kambatla Priority: Critical Attachments: AM_log_head10.txt.gz, AM_log_tail10.txt.gz, log.txt, mr-6302-prelim.patch, queue_with_max163cores.png, queue_with_max263cores.png, queue_with_max333cores.png I submit a big job, which has 500 maps and 350 reduce, to a queue(fairscheduler) with 300 max cores. When the big mapreduce job is running 100% maps, the 300 reduces have occupied 300 max cores in the queue. And then, a map fails and retry, waiting for a core, while the 300 reduces are waiting for failed map to finish. So a deadlock occur. As a result, the job is blocked, and the later job in the queue cannot run because no available cores in the queue. I think there is the similar issue for memory of a queue . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6356) Misspelling of threshold in log4j.properties for tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14533396#comment-14533396 ] Hudson commented on MAPREDUCE-6356: --- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #188 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/188/]) MAPREDUCE-6356. Misspelling of threshold in log4j.properties for tests. Contributed by Brahma Reddy Battula. (aajisaka: rev d335071398fdb14153c854b740814b4fd385b658) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/resources/log4j.properties * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/resources/log4j.properties * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/resources/log4j.properties * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/resources/log4j.properties * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/log4j.properties * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/resources/log4j.properties * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/resources/log4j.properties Misspelling of threshold in log4j.properties for tests -- Key: MAPREDUCE-6356 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6356 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 2.7.0 Reporter: Brahma Reddy Battula Assignee: Brahma Reddy Battula Priority: Minor Fix For: 2.8.0 Attachments: MAPREDUCE-6356-002.patch, MAPREDUCE-6356.patch log4j.properties file for test contains misspelling log4j.threshhold. We should use log4j.threshold correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6279) AM should explicity exit JVM after all services have stopped
[ https://issues.apache.org/jira/browse/MAPREDUCE-6279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14533458#comment-14533458 ] Jason Lowe commented on MAPREDUCE-6279: --- +1 lgtm. Committing this. AM should explicity exit JVM after all services have stopped Key: MAPREDUCE-6279 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6279 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.5.0 Reporter: Jason Lowe Assignee: Eric Payne Labels: BB2015-05-TBR Attachments: MAPREDUCE-6279.v1.txt, MAPREDUCE-6279.v2.txt, MAPREDUCE-6279.v3.patch, MAPREDUCE-6279.v4.patch Occasionally the MapReduce AM can get stuck trying to shut down. MAPREDUCE-6049 and MAPREDUCE-5888 were specific instances that have been fixed, but this can also occur with uber jobs if the task code inadvertently leaves non-daemon threads lingering. We should explicitly shutdown the JVM after the MapReduce AM has unregistered and all services have been stopped. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6273) HistoryFileManager should check whether summaryFile exists to avoid FileNotFoundException causing HistoryFileInfo into MOVE_FAILED state
[
https://issues.apache.org/jira/browse/MAPREDUCE-6273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhihai xu updated MAPREDUCE-6273:
-
Attachment: (was: MAPREDUCE-6273.000.patch)
HistoryFileManager should check whether summaryFile exists to avoid
FileNotFoundException causing HistoryFileInfo into MOVE_FAILED state
Key: MAPREDUCE-6273
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6273
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver
Reporter: zhihai xu
Assignee: zhihai xu
Priority: Minor
Labels: BB2015-05-TBR
HistoryFileManager should check whether summaryFile exists to avoid
FileNotFoundException causing HistoryFileInfo into MOVE_FAILED state,
I saw the following error message:
{code}
2015-02-17 19:13:45,198 ERROR
org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager: Error while trying to
move a job to done
java.io.FileNotFoundException: File does not exist:
/user/history/done_intermediate/agd_laci-sluice/job_1423740288390_1884.summary
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:65)
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:55)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1878)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1819)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1799)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1771)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:527)
at
org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:85)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:356)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
at sun.reflect.GeneratedConstructorAccessor29.newInstance(Unknown
Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
at
org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1181)
at
org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1169)
at
org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1159)
at
org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:270)
at
org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:237)
at org.apache.hadoop.hdfs.DFSInputStream.init(DFSInputStream.java:230)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1457)
at org.apache.hadoop.fs.Hdfs.open(Hdfs.java:318)
at org.apache.hadoop.fs.Hdfs.open(Hdfs.java:59)
at
org.apache.hadoop.fs.AbstractFileSystem.open(AbstractFileSystem.java:621)
at org.apache.hadoop.fs.FileContext$6.next(FileContext.java:789)
at org.apache.hadoop.fs.FileContext$6.next(FileContext.java:785)
at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
at org.apache.hadoop.fs.FileContext.open(FileContext.java:785)
at
org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.getJobSummary(HistoryFileManager.java:953)
at
org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.access$400(HistoryFileManager.java:82)
at
[jira] [Updated] (MAPREDUCE-6079) Renaming JobImpl#username to reporterUserName
[
https://issues.apache.org/jira/browse/MAPREDUCE-6079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsuyoshi Ozawa updated MAPREDUCE-6079:
--
Attachment: MAPREDUCE-6079.2.patch
[~ajisakaa] Thank you for pinging me. Fixed the point.
Renaming JobImpl#username to reporterUserName
-
Key: MAPREDUCE-6079
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6079
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Tsuyoshi Ozawa
Assignee: Tsuyoshi Ozawa
Labels: BB2015-05-TBR
Attachments: MAPREDUCE-6079.1.patch, MAPREDUCE-6079.2.patch
On MAPREDUCE-6033, we found the bug because of confusing field names
{{userName}} and {{username}}. We should change the names to distinguish them
easily.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6342) Make POM project names consistent
[ https://issues.apache.org/jira/browse/MAPREDUCE-6342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith updated MAPREDUCE-6342: -- Labels: BB2015-05-RFC (was: BB2015-05-TBR) Make POM project names consistent - Key: MAPREDUCE-6342 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6342 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Reporter: Rohith Assignee: Rohith Priority: Minor Labels: BB2015-05-RFC Attachments: MAPREDUCE-6342.patch This is track MR changes for POM changes by name -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6273) HistoryFileManager should check whether summaryFile exists to avoid FileNotFoundException causing HistoryFileInfo into MOVE_FAILED state
[
https://issues.apache.org/jira/browse/MAPREDUCE-6273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhihai xu updated MAPREDUCE-6273:
-
Attachment: MAPREDUCE-6273.000.patch
HistoryFileManager should check whether summaryFile exists to avoid
FileNotFoundException causing HistoryFileInfo into MOVE_FAILED state
Key: MAPREDUCE-6273
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6273
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver
Reporter: zhihai xu
Assignee: zhihai xu
Priority: Minor
Labels: BB2015-05-TBR
Attachments: MAPREDUCE-6273.000.patch
HistoryFileManager should check whether summaryFile exists to avoid
FileNotFoundException causing HistoryFileInfo into MOVE_FAILED state,
I saw the following error message:
{code}
2015-02-17 19:13:45,198 ERROR
org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager: Error while trying to
move a job to done
java.io.FileNotFoundException: File does not exist:
/user/history/done_intermediate/agd_laci-sluice/job_1423740288390_1884.summary
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:65)
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:55)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1878)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1819)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1799)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1771)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:527)
at
org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:85)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:356)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
at sun.reflect.GeneratedConstructorAccessor29.newInstance(Unknown
Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
at
org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1181)
at
org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1169)
at
org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1159)
at
org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:270)
at
org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:237)
at org.apache.hadoop.hdfs.DFSInputStream.init(DFSInputStream.java:230)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1457)
at org.apache.hadoop.fs.Hdfs.open(Hdfs.java:318)
at org.apache.hadoop.fs.Hdfs.open(Hdfs.java:59)
at
org.apache.hadoop.fs.AbstractFileSystem.open(AbstractFileSystem.java:621)
at org.apache.hadoop.fs.FileContext$6.next(FileContext.java:789)
at org.apache.hadoop.fs.FileContext$6.next(FileContext.java:785)
at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
at org.apache.hadoop.fs.FileContext.open(FileContext.java:785)
at
org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.getJobSummary(HistoryFileManager.java:953)
at
org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.access$400(HistoryFileManager.java:82)
at
[jira] [Commented] (MAPREDUCE-6273) HistoryFileManager should check whether summaryFile exists to avoid FileNotFoundException causing HistoryFileInfo into MOVE_FAILED state
[
https://issues.apache.org/jira/browse/MAPREDUCE-6273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14533889#comment-14533889
]
zhihai xu commented on MAPREDUCE-6273:
--
update the patch based o latest code base.
HistoryFileManager should check whether summaryFile exists to avoid
FileNotFoundException causing HistoryFileInfo into MOVE_FAILED state
Key: MAPREDUCE-6273
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6273
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver
Reporter: zhihai xu
Assignee: zhihai xu
Priority: Minor
Labels: BB2015-05-TBR
Attachments: MAPREDUCE-6273.000.patch
HistoryFileManager should check whether summaryFile exists to avoid
FileNotFoundException causing HistoryFileInfo into MOVE_FAILED state,
I saw the following error message:
{code}
2015-02-17 19:13:45,198 ERROR
org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager: Error while trying to
move a job to done
java.io.FileNotFoundException: File does not exist:
/user/history/done_intermediate/agd_laci-sluice/job_1423740288390_1884.summary
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:65)
at
org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:55)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1878)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1819)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1799)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1771)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:527)
at
org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:85)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:356)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007)
at sun.reflect.GeneratedConstructorAccessor29.newInstance(Unknown
Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
at
org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1181)
at
org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1169)
at
org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1159)
at
org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:270)
at
org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:237)
at org.apache.hadoop.hdfs.DFSInputStream.init(DFSInputStream.java:230)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1457)
at org.apache.hadoop.fs.Hdfs.open(Hdfs.java:318)
at org.apache.hadoop.fs.Hdfs.open(Hdfs.java:59)
at
org.apache.hadoop.fs.AbstractFileSystem.open(AbstractFileSystem.java:621)
at org.apache.hadoop.fs.FileContext$6.next(FileContext.java:789)
at org.apache.hadoop.fs.FileContext$6.next(FileContext.java:785)
at org.apache.hadoop.fs.FSLinkResolver.resolve(FSLinkResolver.java:90)
at org.apache.hadoop.fs.FileContext.open(FileContext.java:785)
at
org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.getJobSummary(HistoryFileManager.java:953)
at
[jira] [Updated] (MAPREDUCE-3182) loadgen ignores -m command line when writing random data
[ https://issues.apache.org/jira/browse/MAPREDUCE-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated MAPREDUCE-3182: - Target Version/s: 2.8.0 (was: 2.6.0) Status: Open (was: Patch Available) loadgen ignores -m command line when writing random data Key: MAPREDUCE-3182 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3182 Project: Hadoop Map/Reduce Issue Type: Bug Components: documentation, mrv2, test Affects Versions: 2.3.0, 0.23.0 Reporter: Jonathan Eagles Assignee: Chen He Labels: BB2015-05-TBR Attachments: MAPREDUCE-3182.patch If no input directories are specified, loadgen goes into a special mode where random data is generated and written. In that mode, setting the number of mappers (-m command line option) is overridden by a calculation. Instead, it should take into consideration the user specified number of mappers and fall back to the calculation. In addition, update the documentation as well to match the new behavior in the code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-3182) loadgen ignores -m command line when writing random data
[
https://issues.apache.org/jira/browse/MAPREDUCE-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14533783#comment-14533783
]
Akira AJISAKA commented on MAPREDUCE-3182:
--
Mostly looks good to me. Two comments:
1. For mapred/GenericMRLoadGenerator, would you document that the number of map
tasks specified by -m option is overridden when -indir is not specified?
2.
{code}
+RandomWriter will be used to create input directory and data if
\-indir\ +
{code}
{code}
+RandomWriter will be used to create input directory and data if [-indir]
+
{code}
Is there any reason to use different form for -indir?
loadgen ignores -m command line when writing random data
Key: MAPREDUCE-3182
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3182
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, test
Affects Versions: 0.23.0, 2.3.0
Reporter: Jonathan Eagles
Assignee: Chen He
Labels: BB2015-05-TBR
Attachments: MAPREDUCE-3182.patch
If no input directories are specified, loadgen goes into a special mode where
random data is generated and written. In that mode, setting the number of
mappers (-m command line option) is overridden by a calculation. Instead, it
should take into consideration the user specified number of mappers and fall
back to the calculation. In addition, update the documentation as well to
match the new behavior in the code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-3182) loadgen ignores -m command line when writing random data
[ https://issues.apache.org/jira/browse/MAPREDUCE-3182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated MAPREDUCE-3182: - Component/s: documentation Priority: Minor (was: Major) loadgen ignores -m command line when writing random data Key: MAPREDUCE-3182 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3182 Project: Hadoop Map/Reduce Issue Type: Bug Components: documentation, mrv2, test Affects Versions: 0.23.0, 2.3.0 Reporter: Jonathan Eagles Assignee: Chen He Priority: Minor Labels: BB2015-05-TBR Attachments: MAPREDUCE-3182.patch If no input directories are specified, loadgen goes into a special mode where random data is generated and written. In that mode, setting the number of mappers (-m command line option) is overridden by a calculation. Instead, it should take into consideration the user specified number of mappers and fall back to the calculation. In addition, update the documentation as well to match the new behavior in the code. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-3384) Add warning message for org.apache.hadoop.mapreduce.lib.reduce.LongSumReducer
[ https://issues.apache.org/jira/browse/MAPREDUCE-3384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14533820#comment-14533820 ] Akira AJISAKA commented on MAPREDUCE-3384: -- I'm thinking adding warning message only in org.apache.hadoop.mapreduce.lib.reduce.LongSumReducer is inconsistent. We need to add warning messages in the following classes as well. * org.apache.hadoop.mapred.lib.LongSumReducer * org.apache.hadoop.mapreduce.lib.reduce.IntSumReducer Add warning message for org.apache.hadoop.mapreduce.lib.reduce.LongSumReducer - Key: MAPREDUCE-3384 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3384 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: JiangKai Priority: Minor Labels: BB2015-05-TBR Attachments: MAPREDUCE-3384.patch When we call the function reduce() of LongSumReducer,the result may overflow. We should send a warning message to users if overflow occurs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6356) Misspelling of threshold in log4j.properties for tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14532737#comment-14532737 ] Hudson commented on MAPREDUCE-6356: --- FAILURE: Integrated in Hadoop-Hdfs-trunk #2118 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2118/]) MAPREDUCE-6356. Misspelling of threshold in log4j.properties for tests. Contributed by Brahma Reddy Battula. (aajisaka: rev d335071398fdb14153c854b740814b4fd385b658) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/resources/log4j.properties * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/resources/log4j.properties * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/log4j.properties * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/resources/log4j.properties * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/resources/log4j.properties * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/resources/log4j.properties * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/resources/log4j.properties Misspelling of threshold in log4j.properties for tests -- Key: MAPREDUCE-6356 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6356 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 2.7.0 Reporter: Brahma Reddy Battula Assignee: Brahma Reddy Battula Priority: Minor Fix For: 2.8.0 Attachments: MAPREDUCE-6356-002.patch, MAPREDUCE-6356.patch log4j.properties file for test contains misspelling log4j.threshhold. We should use log4j.threshold correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6354) shuffle handler should log connection info
[ https://issues.apache.org/jira/browse/MAPREDUCE-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chang Li updated MAPREDUCE-6354: Attachment: MAPREDUCE-6354.6.patch shuffle handler should log connection info -- Key: MAPREDUCE-6354 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6354 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Chang Li Assignee: Chang Li Attachments: MAPREDUCE-6354.2.patch, MAPREDUCE-6354.3.patch, MAPREDUCE-6354.4.patch, MAPREDUCE-6354.5.patch, MAPREDUCE-6354.6.patch, MAPREDUCE-6354.patch currently, shuffle handler only log connection info in debug mode, we want to log that info in a more concise way -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6279) AM should explicity exit JVM after all services have stopped
[ https://issues.apache.org/jira/browse/MAPREDUCE-6279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14533509#comment-14533509 ] Hudson commented on MAPREDUCE-6279: --- SUCCESS: Integrated in Hadoop-trunk-Commit #7763 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7763/]) MAPREDUCE-6279. AM should explicity exit JVM after all services have stopped. Contributed by Eric Payne (jlowe: rev f30065c8b6099372f57015b505434120fe83c2b0) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java AM should explicity exit JVM after all services have stopped Key: MAPREDUCE-6279 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6279 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.5.0 Reporter: Jason Lowe Assignee: Eric Payne Fix For: 2.8.0 Attachments: MAPREDUCE-6279.v1.txt, MAPREDUCE-6279.v2.txt, MAPREDUCE-6279.v3.patch, MAPREDUCE-6279.v4.patch Occasionally the MapReduce AM can get stuck trying to shut down. MAPREDUCE-6049 and MAPREDUCE-5888 were specific instances that have been fixed, but this can also occur with uber jobs if the task code inadvertently leaves non-daemon threads lingering. We should explicitly shutdown the JVM after the MapReduce AM has unregistered and all services have been stopped. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6251) JobClient needs additional retries at a higher level to address not-immediately-consistent dfs corner cases
[ https://issues.apache.org/jira/browse/MAPREDUCE-6251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Craig Welch updated MAPREDUCE-6251: --- Attachment: MAPREDUCE-6251.8.patch ...add line failed to stage for last patch. JobClient needs additional retries at a higher level to address not-immediately-consistent dfs corner cases --- Key: MAPREDUCE-6251 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6251 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 2.6.0 Reporter: Craig Welch Assignee: Craig Welch Labels: BB2015-05-TBR Attachments: MAPREDUCE-6251.0.patch, MAPREDUCE-6251.1.patch, MAPREDUCE-6251.2.patch, MAPREDUCE-6251.3.patch, MAPREDUCE-6251.4.patch, MAPREDUCE-6251.6.patch, MAPREDUCE-6251.7.patch, MAPREDUCE-6251.8.patch The JobClient is used to get job status information for running and completed jobs. Final state and history for a job is communicated from the application master to the job history server via a distributed file system - where the history is uploaded by the application master to the dfs and then scanned/loaded by the jobhistory server. While HDFS has strong consistency guarantees not all Hadoop DFS's do. When used in conjunction with a distributed file system which does not have this guarantee there will be cases where the history server may not see an uploaded file, resulting in the dreaded no such job and a null value for the RunningJob in the client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6353) Divide by zero error in MR AM when calculating available containers
[
https://issues.apache.org/jira/browse/MAPREDUCE-6353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14533368#comment-14533368
]
Karthik Kambatla commented on MAPREDUCE-6353:
-
Patch looks reasonable to me. A couple of nits:
# The import removal looks unrelated. Let us leave it in.
# {{getLimitIfAny}} doesn't really say much about what the method does. Rename
it to a more descriptive name?
Divide by zero error in MR AM when calculating available containers
---
Key: MAPREDUCE-6353
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6353
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mr-am
Reporter: Anubhav Dhoot
Assignee: Anubhav Dhoot
Attachments: MAPREDUCE-6353.001.patch
When running a sleep job with zero CPU vcores i see the following exception
2015-04-30 06:41:06,954 ERROR [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: ERROR IN
CONTACTING RM.
java.lang.ArithmeticException: / by zero
at
org.apache.hadoop.mapreduce.v2.app.rm.ResourceCalculatorUtils.computeAvailableContainers(ResourceCalculatorUtils.java:38)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator$ScheduledRequests.assign(RMContainerAllocator.java:947)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator$ScheduledRequests.access$200(RMContainerAllocator.java:840)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.heartbeat(RMContainerAllocator.java:247)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator$1.run(RMCommunicator.java:282)
at java.lang.Thread.run(Thread.java:745)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6251) JobClient needs additional retries at a higher level to address not-immediately-consistent dfs corner cases
[ https://issues.apache.org/jira/browse/MAPREDUCE-6251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14533513#comment-14533513 ] Craig Welch commented on MAPREDUCE-6251: Uploaded patch to fix mistake in mapred-default.xml which caused TestNonExistentJob to fail. TestLineRecordReader passes on my box. Whitespace error is bogus (generally, and in this case specifically as it is a line moved not changed by the patch), but tried to fix it anyway. JobClient needs additional retries at a higher level to address not-immediately-consistent dfs corner cases --- Key: MAPREDUCE-6251 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6251 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 2.6.0 Reporter: Craig Welch Assignee: Craig Welch Labels: BB2015-05-TBR Attachments: MAPREDUCE-6251.0.patch, MAPREDUCE-6251.1.patch, MAPREDUCE-6251.2.patch, MAPREDUCE-6251.3.patch, MAPREDUCE-6251.4.patch, MAPREDUCE-6251.6.patch, MAPREDUCE-6251.7.patch The JobClient is used to get job status information for running and completed jobs. Final state and history for a job is communicated from the application master to the job history server via a distributed file system - where the history is uploaded by the application master to the dfs and then scanned/loaded by the jobhistory server. While HDFS has strong consistency guarantees not all Hadoop DFS's do. When used in conjunction with a distributed file system which does not have this guarantee there will be cases where the history server may not see an uploaded file, resulting in the dreaded no such job and a null value for the RunningJob in the client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6279) AM should explicity exit JVM after all services have stopped
[ https://issues.apache.org/jira/browse/MAPREDUCE-6279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-6279: -- Resolution: Fixed Fix Version/s: 2.8.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks, Eric! I committed this to trunk and branch-2. AM should explicity exit JVM after all services have stopped Key: MAPREDUCE-6279 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6279 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.5.0 Reporter: Jason Lowe Assignee: Eric Payne Fix For: 2.8.0 Attachments: MAPREDUCE-6279.v1.txt, MAPREDUCE-6279.v2.txt, MAPREDUCE-6279.v3.patch, MAPREDUCE-6279.v4.patch Occasionally the MapReduce AM can get stuck trying to shut down. MAPREDUCE-6049 and MAPREDUCE-5888 were specific instances that have been fixed, but this can also occur with uber jobs if the task code inadvertently leaves non-daemon threads lingering. We should explicitly shutdown the JVM after the MapReduce AM has unregistered and all services have been stopped. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6354) shuffle handler should log connection info
[
https://issues.apache.org/jira/browse/MAPREDUCE-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14532876#comment-14532876
]
Hadoop QA commented on MAPREDUCE-6354:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 14m 34s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 28s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 33s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 1m 27s | There were no new checkstyle
issues. |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 35s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 2m 17s | The patch does not introduce
any new Findbugs (version 2.0.3) warnings. |
| {color:green}+1{color} | common tests | 24m 27s | Tests passed in
hadoop-common. |
| {color:green}+1{color} | mapreduce tests | 0m 19s | Tests passed in
hadoop-mapreduce-client-shuffle. |
| | | 62m 38s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12731183/MAPREDUCE-6354.6.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 8e991f4 |
| whitespace |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5660/artifact/patchprocess/whitespace.txt
|
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5660/artifact/patchprocess/testrun_hadoop-common.txt
|
| hadoop-mapreduce-client-shuffle test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5660/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5660/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5660/console |
This message was automatically generated.
shuffle handler should log connection info
--
Key: MAPREDUCE-6354
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6354
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Chang Li
Assignee: Chang Li
Attachments: MAPREDUCE-6354.2.patch, MAPREDUCE-6354.3.patch,
MAPREDUCE-6354.4.patch, MAPREDUCE-6354.5.patch, MAPREDUCE-6354.6.patch,
MAPREDUCE-6354.patch
currently, shuffle handler only log connection info in debug mode, we want to
log that info in a more concise way
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6356) Misspelling of threshold in log4j.properties for tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14532885#comment-14532885 ] Hudson commented on MAPREDUCE-6356: --- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2136 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2136/]) MAPREDUCE-6356. Misspelling of threshold in log4j.properties for tests. Contributed by Brahma Reddy Battula. (aajisaka: rev d335071398fdb14153c854b740814b4fd385b658) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/resources/log4j.properties * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/resources/log4j.properties * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/resources/log4j.properties * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/resources/log4j.properties * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/log4j.properties * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/resources/log4j.properties * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/resources/log4j.properties Misspelling of threshold in log4j.properties for tests -- Key: MAPREDUCE-6356 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6356 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 2.7.0 Reporter: Brahma Reddy Battula Assignee: Brahma Reddy Battula Priority: Minor Fix For: 2.8.0 Attachments: MAPREDUCE-6356-002.patch, MAPREDUCE-6356.patch log4j.properties file for test contains misspelling log4j.threshhold. We should use log4j.threshold correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6354) shuffle handler should log connection info
[ https://issues.apache.org/jira/browse/MAPREDUCE-6354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14532904#comment-14532904 ] Chang Li commented on MAPREDUCE-6354: - That whitespace issue is not caused by my change. [~jlowe] could you help review my patch shuffle handler should log connection info -- Key: MAPREDUCE-6354 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6354 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Chang Li Assignee: Chang Li Attachments: MAPREDUCE-6354.2.patch, MAPREDUCE-6354.3.patch, MAPREDUCE-6354.4.patch, MAPREDUCE-6354.5.patch, MAPREDUCE-6354.6.patch, MAPREDUCE-6354.patch currently, shuffle handler only log connection info in debug mode, we want to log that info in a more concise way -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6356) Misspelling of threshold in log4j.properties for tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14532769#comment-14532769 ] Hudson commented on MAPREDUCE-6356: --- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #177 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/177/]) MAPREDUCE-6356. Misspelling of threshold in log4j.properties for tests. Contributed by Brahma Reddy Battula. (aajisaka: rev d335071398fdb14153c854b740814b4fd385b658) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/resources/log4j.properties * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/resources/log4j.properties * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/resources/log4j.properties * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/resources/log4j.properties * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/log4j.properties * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/resources/log4j.properties * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/resources/log4j.properties Misspelling of threshold in log4j.properties for tests -- Key: MAPREDUCE-6356 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6356 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 2.7.0 Reporter: Brahma Reddy Battula Assignee: Brahma Reddy Battula Priority: Minor Fix For: 2.8.0 Attachments: MAPREDUCE-6356-002.patch, MAPREDUCE-6356.patch log4j.properties file for test contains misspelling log4j.threshhold. We should use log4j.threshold correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6356) Misspelling of threshold in log4j.properties for tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14532811#comment-14532811 ] Hudson commented on MAPREDUCE-6356: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #187 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/187/]) MAPREDUCE-6356. Misspelling of threshold in log4j.properties for tests. Contributed by Brahma Reddy Battula. (aajisaka: rev d335071398fdb14153c854b740814b4fd385b658) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/resources/log4j.properties * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/resources/log4j.properties * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/resources/log4j.properties * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/log4j.properties * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/resources/log4j.properties * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/resources/log4j.properties * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/resources/log4j.properties * hadoop-mapreduce-project/CHANGES.txt Misspelling of threshold in log4j.properties for tests -- Key: MAPREDUCE-6356 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6356 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 2.7.0 Reporter: Brahma Reddy Battula Assignee: Brahma Reddy Battula Priority: Minor Fix For: 2.8.0 Attachments: MAPREDUCE-6356-002.patch, MAPREDUCE-6356.patch log4j.properties file for test contains misspelling log4j.threshhold. We should use log4j.threshold correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6356) Misspelling of threshold in log4j.properties for tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14532446#comment-14532446 ] Hudson commented on MAPREDUCE-6356: --- FAILURE: Integrated in Hadoop-trunk-Commit #7757 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7757/]) MAPREDUCE-6356. Misspelling of threshold in log4j.properties for tests. Contributed by Brahma Reddy Battula. (aajisaka: rev d335071398fdb14153c854b740814b4fd385b658) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/resources/log4j.properties * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/resources/log4j.properties * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/log4j.properties * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/resources/log4j.properties * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/resources/log4j.properties * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/resources/log4j.properties * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/test/resources/log4j.properties Misspelling of threshold in log4j.properties for tests -- Key: MAPREDUCE-6356 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6356 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 2.7.0 Reporter: Brahma Reddy Battula Assignee: Brahma Reddy Battula Priority: Minor Fix For: 2.8.0 Attachments: MAPREDUCE-6356-002.patch, MAPREDUCE-6356.patch log4j.properties file for test contains misspelling log4j.threshhold. We should use log4j.threshold correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6356) Misspelling of threshold in log4j.properties for tests
[
https://issues.apache.org/jira/browse/MAPREDUCE-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14532394#comment-14532394
]
Hadoop QA commented on MAPREDUCE-6356:
--
\\
\\
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 5m 9s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 7 new or modified test files. |
| {color:green}+1{color} | javac | 7m 28s | There were no new javac warning
messages. |
| {color:green}+1{color} | release audit | 0m 19s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 32s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | mapreduce tests | 9m 7s | Tests passed in
hadoop-mapreduce-client-app. |
| {color:green}+1{color} | mapreduce tests | 0m 44s | Tests passed in
hadoop-mapreduce-client-common. |
| {color:green}+1{color} | mapreduce tests | 1m 35s | Tests passed in
hadoop-mapreduce-client-core. |
| {color:green}+1{color} | mapreduce tests | 5m 39s | Tests passed in
hadoop-mapreduce-client-hs. |
| {color:green}+1{color} | mapreduce tests | 105m 41s | Tests passed in
hadoop-mapreduce-client-jobclient. |
| {color:green}+1{color} | mapreduce tests | 10m 17s | Tests passed in
hadoop-mapreduce-client-nativetask. |
| {color:green}+1{color} | mapreduce tests | 0m 24s | Tests passed in
hadoop-mapreduce-client-shuffle. |
| | | 148m 32s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12731078/MAPREDUCE-6356-002.patch
|
| Optional Tests | javac unit |
| git revision | trunk / 449e442 |
| hadoop-mapreduce-client-app test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5659/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
|
| hadoop-mapreduce-client-common test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5659/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
|
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5659/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| hadoop-mapreduce-client-hs test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5659/artifact/patchprocess/testrun_hadoop-mapreduce-client-hs.txt
|
| hadoop-mapreduce-client-jobclient test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5659/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
|
| hadoop-mapreduce-client-nativetask test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5659/artifact/patchprocess/testrun_hadoop-mapreduce-client-nativetask.txt
|
| hadoop-mapreduce-client-shuffle test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5659/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5659/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5659/console |
This message was automatically generated.
Misspelling of threshold in log4j.properties for tests
--
Key: MAPREDUCE-6356
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6356
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Reporter: Brahma Reddy Battula
Assignee: Brahma Reddy Battula
Priority: Minor
Labels: BB2015-05-RFC
Attachments: MAPREDUCE-6356-002.patch, MAPREDUCE-6356.patch
log4j.properties file for test contains misspelling log4j.threshhold.
We should use log4j.threshold correctly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6356) Misspelling of threshold in log4j.properties for tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated MAPREDUCE-6356: - Labels: (was: BB2015-05-RFC) Hadoop Flags: Reviewed Misspelling of threshold in log4j.properties for tests -- Key: MAPREDUCE-6356 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6356 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 2.7.0 Reporter: Brahma Reddy Battula Assignee: Brahma Reddy Battula Priority: Minor Fix For: 2.8.0 Attachments: MAPREDUCE-6356-002.patch, MAPREDUCE-6356.patch log4j.properties file for test contains misspelling log4j.threshhold. We should use log4j.threshold correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6356) Misspelling of threshold in log4j.properties for tests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated MAPREDUCE-6356: - Resolution: Fixed Fix Version/s: 2.8.0 Status: Resolved (was: Patch Available) +1, committed this to trunk and branch-2. Thanks [~brahmareddy] for the contribution. Misspelling of threshold in log4j.properties for tests -- Key: MAPREDUCE-6356 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6356 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 2.7.0 Reporter: Brahma Reddy Battula Assignee: Brahma Reddy Battula Priority: Minor Fix For: 2.8.0 Attachments: MAPREDUCE-6356-002.patch, MAPREDUCE-6356.patch log4j.properties file for test contains misspelling log4j.threshhold. We should use log4j.threshold correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
