[jira] [Commented] (MAPREDUCE-6426) TestShuffleHandler fails in trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14616926#comment-14616926
]
zhihai xu commented on MAPREDUCE-6426:
--
Yes, I will take a look at it, thanks for finding this issue [~devaraj.k]!
It looks like UserGroupInformation.isSecurityEnabled() is true for the test
testGetMapOutputInfo in this test environment(may be loaded from configuration
file). It looks like manually disabling security may fix this issue. I will
prepare a patch for this.
TestShuffleHandler fails in trunk
-
Key: MAPREDUCE-6426
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6426
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 2.8.0
Reporter: Devaraj K
Assignee: zhihai xu
{code:xml}
expected:1 but was:0
Stacktrace
java.lang.AssertionError: expected:1 but was:0
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.mapred.TestShuffleHandler.testGetMapOutputInfo(TestShuffleHandler.java:927)
{code}
https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2195/testReport/junit/org.apache.hadoop.mapred/TestShuffleHandler/testGetMapOutputInfo/
https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/247/testReport/org.apache.hadoop.mapred/TestShuffleHandler/testGetMapOutputInfo/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6038) A boolean may be set error in the Word Count v2.0 in MapReduce Tutorial
[
https://issues.apache.org/jira/browse/MAPREDUCE-6038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14617028#comment-14617028
]
Hudson commented on MAPREDUCE-6038:
---
FAILURE: Integrated in Hadoop-trunk-Commit #8126 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/8126/])
MAPREDUCE-6038. A boolean may be set error in the Word Count v2.0 in MapReduce
Tutorial. Contributed by Tsuyoshi Ozawa (cdouglas: rev
7e2fe8c9f28ec6fff32741ebf1bdbf47729d9eaf)
*
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/MapReduceTutorial.md
* hadoop-mapreduce-project/CHANGES.txt
A boolean may be set error in the Word Count v2.0 in MapReduce Tutorial
---
Key: MAPREDUCE-6038
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6038
Project: Hadoop Map/Reduce
Issue Type: Bug
Environment: java version 1.8.0_11 hostspot 64-bit
Reporter: Pei Ma
Assignee: Tsuyoshi Ozawa
Priority: Minor
Labels: BB2015-05-TBR
Fix For: 2.8.0
Attachments: MAPREDUCE-6038.1.patch, MAPREDUCE-6038.2.patch
As a beginner, when I learned about the basic of the mr, I found that I
cound't run the WordCount2 using the command bin/hadoop jar wc.jar
WordCount2 /user/joe/wordcount/input /user/joe/wordcount/output in the
Tutorial. The VM throwed the NullPoniterException at the line 47. In the line
45, the returned default value of conf.getBoolean is true. That is to say
when wordcount.skip.patterns is not set ,the WordCount2 will continue to
execute getCacheFiles.. Then patternsURIs gets the null value. When the
-skip option dosen't exist, wordcount.skip.patterns will not be set.
Then a NullPointerException come out.
At all, the block after the if-statement in line no. 45 shoudn't be executed
when the -skip option dosen't exist in command. Maybe the line 45 should
like that if (conf.getBoolean(wordcount.skip.patterns, false)) {
.Just change the boolean.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6426) TestShuffleHandler fails in trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhihai xu updated MAPREDUCE-6426:
-
Status: Patch Available (was: Open)
TestShuffleHandler fails in trunk
-
Key: MAPREDUCE-6426
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6426
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 2.8.0
Reporter: Devaraj K
Assignee: zhihai xu
Attachments: MAPREDUCE-6426.000.patch
{code:xml}
expected:1 but was:0
Stacktrace
java.lang.AssertionError: expected:1 but was:0
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.mapred.TestShuffleHandler.testGetMapOutputInfo(TestShuffleHandler.java:927)
{code}
https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2195/testReport/junit/org.apache.hadoop.mapred/TestShuffleHandler/testGetMapOutputInfo/
https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/247/testReport/org.apache.hadoop.mapred/TestShuffleHandler/testGetMapOutputInfo/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6426) TestShuffleHandler fails in trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhihai xu updated MAPREDUCE-6426:
-
Attachment: MAPREDUCE-6426.000.patch
TestShuffleHandler fails in trunk
-
Key: MAPREDUCE-6426
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6426
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 2.8.0
Reporter: Devaraj K
Assignee: zhihai xu
Attachments: MAPREDUCE-6426.000.patch
{code:xml}
expected:1 but was:0
Stacktrace
java.lang.AssertionError: expected:1 but was:0
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.mapred.TestShuffleHandler.testGetMapOutputInfo(TestShuffleHandler.java:927)
{code}
https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2195/testReport/junit/org.apache.hadoop.mapred/TestShuffleHandler/testGetMapOutputInfo/
https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/247/testReport/org.apache.hadoop.mapred/TestShuffleHandler/testGetMapOutputInfo/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6426) TestShuffleHandler fails in trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14617303#comment-14617303
]
zhihai xu commented on MAPREDUCE-6426:
--
{{testGetMapOutputInfo}} can only work with simple mode, If I enable security
by adding the following code to {{testGetMapOutputInfo}}
{code}
conf.set(CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHENTICATION,
kerberos);
UserGroupInformation.setConfiguration(conf);
{code}
I will get the same error:
{code}
2015-07-07 12:44:44,246 ERROR [New I/O worker #188] mapred.ShuffleHandler
(ShuffleHandler.java:messageReceived(829)) - Shuffle error :
java.io.IOException: Error Reading IndexFile
at
org.apache.hadoop.mapred.IndexCache.readIndexFileToCache(IndexCache.java:123)
at
org.apache.hadoop.mapred.IndexCache.getIndexInformation(IndexCache.java:68)
at
org.apache.hadoop.mapred.ShuffleHandler$Shuffle.getMapOutputInfo(ShuffleHandler.java:865)
at
org.apache.hadoop.mapred.ShuffleHandler$Shuffle.messageReceived(ShuffleHandler.java:818)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787)
at
org.jboss.netty.handler.stream.ChunkedWriteHandler.handleUpstream(ChunkedWriteHandler.java:142)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787)
at
org.jboss.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:148)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:787)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)
at
org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
at
org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:560)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:555)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:107)
at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
{code}
I also find out how security is enabled. Although {{testGetMapOutputInfo}}
didn't enable security, {{testMapFileAccess}} enabled security kerberos by
{{UserGroupInformation.setConfiguration(conf);}} which is a static member
function, if {{testMapFileAccess}} is running before {{testGetMapOutputInfo}},
the configuration used by UserGroupInformation is not changed by
{{testGetMapOutputInfo}}, So UserGroupInformation is still using the
configuration from {{testMapFileAccess}} for {{testGetMapOutputInfo}}.
I attached a patch MAPREDUCE-6426.000.patch, which manually disable security in
{testGetMapOutputInfo}}. I believe it should fix this issue.
TestShuffleHandler fails in trunk
-
Key: MAPREDUCE-6426
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6426
Project: Hadoop Map/Reduce
[jira] [Updated] (MAPREDUCE-6038) A boolean may be set error in the Word Count v2.0 in MapReduce Tutorial
[
https://issues.apache.org/jira/browse/MAPREDUCE-6038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Douglas updated MAPREDUCE-6038:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.8.0
Status: Resolved (was: Patch Available)
+1 I committed this. Thanks Tsuyoshi
A boolean may be set error in the Word Count v2.0 in MapReduce Tutorial
---
Key: MAPREDUCE-6038
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6038
Project: Hadoop Map/Reduce
Issue Type: Bug
Environment: java version 1.8.0_11 hostspot 64-bit
Reporter: Pei Ma
Assignee: Tsuyoshi Ozawa
Priority: Minor
Labels: BB2015-05-TBR
Fix For: 2.8.0
Attachments: MAPREDUCE-6038.1.patch, MAPREDUCE-6038.2.patch
As a beginner, when I learned about the basic of the mr, I found that I
cound't run the WordCount2 using the command bin/hadoop jar wc.jar
WordCount2 /user/joe/wordcount/input /user/joe/wordcount/output in the
Tutorial. The VM throwed the NullPoniterException at the line 47. In the line
45, the returned default value of conf.getBoolean is true. That is to say
when wordcount.skip.patterns is not set ,the WordCount2 will continue to
execute getCacheFiles.. Then patternsURIs gets the null value. When the
-skip option dosen't exist, wordcount.skip.patterns will not be set.
Then a NullPointerException come out.
At all, the block after the if-statement in line no. 45 shoudn't be executed
when the -skip option dosen't exist in command. Maybe the line 45 should
like that if (conf.getBoolean(wordcount.skip.patterns, false)) {
.Just change the boolean.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6426) TestShuffleHandler fails in trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14617485#comment-14617485
]
Hadoop QA commented on MAPREDUCE-6426:
--
\\
\\
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 6m 4s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 47s | There were no new javac warning
messages. |
| {color:green}+1{color} | release audit | 0m 21s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 21s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 0m 41s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | mapreduce tests | 0m 18s | Tests passed in
hadoop-mapreduce-client-shuffle. |
| | | 17m 41s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12744050/MAPREDUCE-6426.000.patch
|
| Optional Tests | javac unit findbugs checkstyle |
| git revision | trunk / c9dd2ca |
| hadoop-mapreduce-client-shuffle test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5873/artifact/patchprocess/testrun_hadoop-mapreduce-client-shuffle.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5873/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5873/console |
This message was automatically generated.
TestShuffleHandler fails in trunk
-
Key: MAPREDUCE-6426
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6426
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 2.8.0
Reporter: Devaraj K
Assignee: zhihai xu
Attachments: MAPREDUCE-6426.000.patch
{code:xml}
expected:1 but was:0
Stacktrace
java.lang.AssertionError: expected:1 but was:0
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.mapred.TestShuffleHandler.testGetMapOutputInfo(TestShuffleHandler.java:927)
{code}
https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2195/testReport/junit/org.apache.hadoop.mapred/TestShuffleHandler/testGetMapOutputInfo/
https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/247/testReport/org.apache.hadoop.mapred/TestShuffleHandler/testGetMapOutputInfo/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6394) Speed up Task processing loop in HsTasksBlock#render()
[ https://issues.apache.org/jira/browse/MAPREDUCE-6394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14617685#comment-14617685 ] Ray Chiang commented on MAPREDUCE-6394: --- [~rkanter], [~jlowe] any thoughts on the above suggested change? I don't think there will be any side effects outside of JHS, but this just occurred to me today so I thought I'd get feedback early. Speed up Task processing loop in HsTasksBlock#render() -- Key: MAPREDUCE-6394 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6394 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: jobhistoryserver Affects Versions: 2.7.0 Reporter: Ray Chiang Assignee: Ray Chiang Labels: supportability Attachments: MAPREDUCE-6394.001.patch, MAPREDUCE-6394.002.patch, MAPREDUCE-6394.003.patch In HsTasksBlock#render(), there is a loop to create a Javascript table which slows down immensely on jobs with a large number of tasks (200k or more). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6394) Speed up Task processing loop in HsTasksBlock#render()
[ https://issues.apache.org/jira/browse/MAPREDUCE-6394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14617678#comment-14617678 ] Ray Chiang commented on MAPREDUCE-6394: --- So, from everything I can tell, the HTML/Javascript Table generated in the for loop of HsTasksBlock#render() ends up being slow for really large numbers of Tasks (e.g. taking up 102 out of 109 seconds for my 751k tasks .jhist file). The bulk of that time taken up in the method with the signature: TypeConverter#toYarn(org.apache.hadoop.mapreduce.Counters counters) Given the CounterGroup/Counters are pretty simple and from what I can figure out through crude profiling is that the outer loop gets called some 4.5 million times and the inner loop nets out around 37 million times. Even as fast as it may be, the sheer number of counters adds up over so many tasks. The only solution I can see offhand would be to defer this conversion to when the user clicks on the Counters link in the Tasks page. This would mean substituting the calls like: report.setCounters(TypeConverter.toYarn(getCounters())); in CompletedTaskAttempt (and similar places if any) and adjusting the JHS method that accesses the counters to do the parsing at link click time. Speed up Task processing loop in HsTasksBlock#render() -- Key: MAPREDUCE-6394 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6394 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: jobhistoryserver Affects Versions: 2.7.0 Reporter: Ray Chiang Assignee: Ray Chiang Labels: supportability Attachments: MAPREDUCE-6394.001.patch, MAPREDUCE-6394.002.patch, MAPREDUCE-6394.003.patch In HsTasksBlock#render(), there is a loop to create a Javascript table which slows down immensely on jobs with a large number of tasks (200k or more). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6422) Add REST API for getting all attempts for all the tasks
[
https://issues.apache.org/jira/browse/MAPREDUCE-6422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14616549#comment-14616549
]
Hadoop QA commented on MAPREDUCE-6422:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 16m 28s | Pre-patch trunk has 1 extant
Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 7m 34s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 36s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 49s | The applied patch generated 5
new checkstyle issues (total was 25, now 24). |
| {color:red}-1{color} | whitespace | 0m 3s | The patch has 2 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 36s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 59s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | mapreduce tests | 9m 4s | Tests passed in
hadoop-mapreduce-client-app. |
| {color:red}-1{color} | mapreduce tests | 4m 59s | Tests failed in
hadoop-mapreduce-client-hs. |
| | | 53m 17s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.mapreduce.v2.hs.TestJobHistoryParsing |
| | hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesJobs |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12743930/MAPREDUCE-6422.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / af63427 |
| Pre-patch Findbugs warnings |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5872/artifact/patchprocess/trunkFindbugsWarningshadoop-mapreduce-client-app.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5872/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-app.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5872/artifact/patchprocess/whitespace.txt
|
| hadoop-mapreduce-client-app test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5872/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
|
| hadoop-mapreduce-client-hs test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5872/artifact/patchprocess/testrun_hadoop-mapreduce-client-hs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5872/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5872/console |
This message was automatically generated.
Add REST API for getting all attempts for all the tasks
---
Key: MAPREDUCE-6422
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6422
Project: Hadoop Map/Reduce
Issue Type: New Feature
Reporter: Lavkesh Lahngir
Assignee: Lavkesh Lahngir
Attachments: MAPREDUCE-6422.patch
Web UI has the feature where one can get all attempts for all map tasks or
reduce tasks.
REST api seems to be missing it.
Should we add this in both HsWebService and AMWebService ?
{code}
@GET
@Path(/mapreduce/jobs/{jobid}/tasks/attempts)
@Produces({ MediaType.APPLICATION_JSON, MediaType.APPLICATION_XML })
public JobTaskAttemptsInfo getAllJobTaskAttempts(@Context
HttpServletRequest hsr,
@PathParam(jobid) String jid, @QueryParam(type) String type) {
}
{code}
We might also add queryparam on state to filter by succeeded attempts etc.
Thoughts ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6422) Add REST API for getting all attempts for all the tasks
[
https://issues.apache.org/jira/browse/MAPREDUCE-6422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lavkesh Lahngir updated MAPREDUCE-6422:
---
Attachment: MAPREDUCE-6422.patch
Add REST API for getting all attempts for all the tasks
---
Key: MAPREDUCE-6422
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6422
Project: Hadoop Map/Reduce
Issue Type: New Feature
Reporter: Lavkesh Lahngir
Assignee: Lavkesh Lahngir
Attachments: MAPREDUCE-6422.patch
Web UI has the feature where one can get all attempts for all map tasks or
reduce tasks.
REST api seems to be missing it.
Should we add this in both HsWebService and AMWebService ?
{code}
@GET
@Path(/mapreduce/jobs/{jobid}/tasks/attempts)
@Produces({ MediaType.APPLICATION_JSON, MediaType.APPLICATION_XML })
public JobTaskAttemptsInfo getAllJobTaskAttempts(@Context
HttpServletRequest hsr,
@PathParam(jobid) String jid, @QueryParam(type) String type) {
}
{code}
We might also add queryparam on state to filter by succeeded attempts etc.
Thoughts ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6422) Add REST API for getting all attempts for all the tasks
[
https://issues.apache.org/jira/browse/MAPREDUCE-6422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lavkesh Lahngir updated MAPREDUCE-6422:
---
Status: Patch Available (was: Open)
Add REST API for getting all attempts for all the tasks
---
Key: MAPREDUCE-6422
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6422
Project: Hadoop Map/Reduce
Issue Type: New Feature
Reporter: Lavkesh Lahngir
Assignee: Lavkesh Lahngir
Attachments: MAPREDUCE-6422.patch
Web UI has the feature where one can get all attempts for all map tasks or
reduce tasks.
REST api seems to be missing it.
Should we add this in both HsWebService and AMWebService ?
{code}
@GET
@Path(/mapreduce/jobs/{jobid}/tasks/attempts)
@Produces({ MediaType.APPLICATION_JSON, MediaType.APPLICATION_XML })
public JobTaskAttemptsInfo getAllJobTaskAttempts(@Context
HttpServletRequest hsr,
@PathParam(jobid) String jid, @QueryParam(type) String type) {
}
{code}
We might also add queryparam on state to filter by succeeded attempts etc.
Thoughts ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6427) Fix Typo in JobHistoryEventHandler#processEventForTimelineServer
[
https://issues.apache.org/jira/browse/MAPREDUCE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14617908#comment-14617908
]
Brahma Reddy Battula commented on MAPREDUCE-6427:
-
[~rchiang] assigned to you as you worked on HADOOP-11854..Please check
[~chris.douglas] comment [here |
https://issues.apache.org/jira/browse/HADOOP-11854?focusedCommentId=14617212page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14617212]
for more info.
Fix Typo in JobHistoryEventHandler#processEventForTimelineServer
Key: MAPREDUCE-6427
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6427
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Brahma Reddy Battula
Assignee: Ray Chiang
Priority: Minor
JobHistoryEventHandler#processEventForTimelineServer
{code}tEvent.addEventInfo(WORKLFOW_ID, jse.getWorkflowId());{code}
*should be like below.*
{code}tEvent.addEventInfo(WORKFLOW_ID, jse.getWorkflowId()); {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6427) Fix Typo in JobHistoryEventHandler#processEventForTimelineServer
Brahma Reddy Battula created MAPREDUCE-6427:
---
Summary: Fix Typo in
JobHistoryEventHandler#processEventForTimelineServer
Key: MAPREDUCE-6427
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6427
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Brahma Reddy Battula
Assignee: Ray Chiang
Priority: Minor
JobHistoryEventHandler#processEventForTimelineServer
{code}tEvent.addEventInfo(WORKLFOW_ID, jse.getWorkflowId());{code}
*should be like below.*
{code}tEvent.addEventInfo(WORKFLOW_ID, jse.getWorkflowId()); {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6363) [NNBench] Lease mismatch error when running with multiple mappers
[
https://issues.apache.org/jira/browse/MAPREDUCE-6363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14617999#comment-14617999
]
Akira AJISAKA commented on MAPREDUCE-6363:
--
Thanks [~uladz] for the comment and the patch.
bq. Now, nor usage of random number (OMG!), not TaskID would generate a stable
output file names that can be (re-)used for following nnbench runs of READ and
DELETE operations.
Would you please tell me why the taskid does not generate a stable output? The
id is unique and sequential for each map task.
{code}
-String filePrefix = file_ + taskId + _;
+String fileName = file_ + value;
{code}
Nice catch. We should remove the {{_}}.
[NNBench] Lease mismatch error when running with multiple mappers
-
Key: MAPREDUCE-6363
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6363
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: benchmarks
Reporter: Brahma Reddy Battula
Assignee: Vlad Sharanhovich
Priority: Critical
Fix For: 2.8.0
Attachments: HDFS4929.patch, MAPREDUCE-6363-001.patch,
MAPREDUCE-6363-002.patch, MAPREDUCE-6363-003.patch
Command :
./yarn jar
../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.0.1-tests.jar
nnbench -operation create_write -numberOfFiles 1000 -blockSize 268435456
-bytesToWrite 102400 -baseDir /benchmarks/NNBench`hostname -s`
-replicationFactorPerFile 3 -maps 100 -reduces 10
Trace :
013-06-21 10:44:53,763 INFO org.apache.hadoop.ipc.Server: IPC Server handler
7 on 9005, call org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from
192.168.105.214:36320: error:
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: Lease mismatch
on /benchmarks/NNBenchlinux-185/data/file_linux-214__0 owned by
DFSClient_attempt_1371782327901_0001_m_48_0_1383437860_1 but is accessed
by DFSClient_attempt_1371782327901_0001_m_84_0_1880545303_1
org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: Lease mismatch
on /benchmarks/NNBenchlinux-185/data/file_linux-214__0 owned by
DFSClient_attempt_1371782327901_0001_m_48_0_1383437860_1 but is accessed
by DFSClient_attempt_1371782327901_0001_m_84_0_1880545303_1
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2351)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:2098)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2019)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:501)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:213)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:52012)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:435)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:925)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1710)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1706)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6426) TestShuffleHandler fails in trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14617953#comment-14617953
]
Devaraj K commented on MAPREDUCE-6426:
--
Thanks [~zxu] for the analysis and patch for the issue.
{code:xml}
Assert.assertEquals(failures.size(), 0);
{code}
Can you exchange the args here and also add a message for this assertion as
part of the patch?
TestShuffleHandler fails in trunk
-
Key: MAPREDUCE-6426
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6426
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 2.8.0
Reporter: Devaraj K
Assignee: zhihai xu
Attachments: MAPREDUCE-6426.000.patch
{code:xml}
expected:1 but was:0
Stacktrace
java.lang.AssertionError: expected:1 but was:0
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.mapred.TestShuffleHandler.testGetMapOutputInfo(TestShuffleHandler.java:927)
{code}
https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2195/testReport/junit/org.apache.hadoop.mapred/TestShuffleHandler/testGetMapOutputInfo/
https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/247/testReport/org.apache.hadoop.mapred/TestShuffleHandler/testGetMapOutputInfo/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6426) TestShuffleHandler fails in trunk
Devaraj K created MAPREDUCE-6426:
Summary: TestShuffleHandler fails in trunk
Key: MAPREDUCE-6426
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6426
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 2.8.0
Reporter: Devaraj K
Assignee: zhihai xu
{code:xml}
expected:1 but was:0
Stacktrace
java.lang.AssertionError: expected:1 but was:0
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.mapred.TestShuffleHandler.testGetMapOutputInfo(TestShuffleHandler.java:927)
{code}
https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2195/testReport/junit/org.apache.hadoop.mapred/TestShuffleHandler/testGetMapOutputInfo/
https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/247/testReport/org.apache.hadoop.mapred/TestShuffleHandler/testGetMapOutputInfo/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6426) TestShuffleHandler fails in trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-6426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14616812#comment-14616812
]
Devaraj K commented on MAPREDUCE-6426:
--
It is passing locally and failing in the Jenkins build.
[~zxu], I am assigning to you, Can you please have a look into this test
failure?
TestShuffleHandler fails in trunk
-
Key: MAPREDUCE-6426
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6426
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 2.8.0
Reporter: Devaraj K
Assignee: zhihai xu
{code:xml}
expected:1 but was:0
Stacktrace
java.lang.AssertionError: expected:1 but was:0
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.mapred.TestShuffleHandler.testGetMapOutputInfo(TestShuffleHandler.java:927)
{code}
https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2195/testReport/junit/org.apache.hadoop.mapred/TestShuffleHandler/testGetMapOutputInfo/
https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/247/testReport/org.apache.hadoop.mapred/TestShuffleHandler/testGetMapOutputInfo/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
