[jira] [Created] (MAPREDUCE-7232) Reduce excessive getFileLinkInfo calls in MR
Jonathan Eagles created MAPREDUCE-7232:
--
Summary: Reduce excessive getFileLinkInfo calls in MR
Key: MAPREDUCE-7232
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7232
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Jonathan Eagles
Assignee: Jonathan Eagles
The MR Client redundantly calls getFileLinkInfo for the same file sometimes
over 10 times. The files under question originate from
mapreduce.job.cache.files and mapreduce.job.classpath.files and their archive
equivalences.
Client Observation
Case 1 & 2
{code}
19/08/15 10:57:55 INFO hdfs.DFSClient:
getFileLinkInfo:/shared/tez-exploded/tez-0.9.2-SNAPSHOT-minimal/tez-runtime-internals-0.9.2-SNAPSHOT.jar
java.lang.Exception
at org.apache.hadoop.hdfs.DFSClient.getFileLinkInfo(DFSClient.java:1683)
at
org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1486)
at
org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1483)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileLinkStatus(DistributedFileSystem.java:1498)
at org.apache.hadoop.fs.FileSystem.resolvePath(FileSystem.java:772)
at
org.apache.hadoop.mapreduce.v2.util.MRApps.addToClasspathIfNotJar(MRApps.java:341)
at
org.apache.hadoop.mapreduce.v2.util.MRApps.addClasspathToEnv(MRApps.java:301)
at
org.apache.hadoop.mapreduce.v2.util.MRApps.setClasspath(MRApps.java:262)
at
org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:467)
at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:296)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:244)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1341)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1338)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1953)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1338)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1359)
at org.apache.hadoop.mapreduce.SleepJob.run(SleepJob.java:273)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.mapreduce.SleepJob.main(SleepJob.java:194)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
at
org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:136)
at
org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:144)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:239)
at org.apache.hadoop.util.RunJar.main(RunJar.java:153)
{code}
{code}
19/08/15 10:57:55 INFO hdfs.DFSClient:
getFileLinkInfo:/shared/tez-exploded/tez-0.9.2-SNAPSHOT-minimal/tez-runtime-internals-0.9.2-SNAPSHOT.jar
java.lang.Exception
at org.apache.hadoop.hdfs.DFSClient.getFileLinkInfo(DFSClient.java:1683)
at
org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1486)
at
org.apache.hadoop.hdfs.DistributedFileSystem$29.doCall(DistributedFileSystem.java:1483)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileLinkStatus(DistributedFileSystem.java:1498)
at org.apache.hadoop.fs.FileSystem.resolvePath(FileSystem.java:772)
at
org.apache.hadoop.mapreduce.v2.util.MRApps.addToClasspathIfNotJar(MRApps.java:341)
at
org.apache.hadoop.mapreduce.v2.util.MRApps.addClasspathToEnv(MRApps.java:301)
at
org.apache.hadoop.mapreduce.v2.util.MRApps.setClasspath(MRApps.java:263)
at
org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:467)
at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:296)
at
[jira] [Commented] (MAPREDUCE-6850) Shuffle Handler keep-alive connections are closed from the server side
[ https://issues.apache.org/jira/browse/MAPREDUCE-6850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15947963#comment-15947963 ] Jonathan Eagles commented on MAPREDUCE-6850: [~jlowe], posted patch that moves timeout to a private state final variable. > Shuffle Handler keep-alive connections are closed from the server side > -- > > Key: MAPREDUCE-6850 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6850 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: MAPREDUCE-6850.1.patch, MAPREDUCE-6850.2.patch, > MAPREDUCE-6850.3.patch, MAPREDUCE-6850.4.patch, MAPREDUCE-6850.5.patch, > MAPREDUCE-6850.6.patch, MAPREDUCE-6850.7.patch, With_Issue.png, > With_Patch.png, With_Patch_withData.png > > > When performance testing tez shuffle handler (TEZ-3334), it was noticed the > keep-alive connections are closed from the server-side. The client silently > recovers and logs the connection as keep-alive, despite reestablishing a > connection. This jira aims to remove the close from the server side, fixing > the bug preventing keep-alive connections. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6850) Shuffle Handler keep-alive connections are closed from the server side
[ https://issues.apache.org/jira/browse/MAPREDUCE-6850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-6850: --- Attachment: MAPREDUCE-6850.7.patch > Shuffle Handler keep-alive connections are closed from the server side > -- > > Key: MAPREDUCE-6850 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6850 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: MAPREDUCE-6850.1.patch, MAPREDUCE-6850.2.patch, > MAPREDUCE-6850.3.patch, MAPREDUCE-6850.4.patch, MAPREDUCE-6850.5.patch, > MAPREDUCE-6850.6.patch, MAPREDUCE-6850.7.patch, With_Issue.png, > With_Patch.png, With_Patch_withData.png > > > When performance testing tez shuffle handler (TEZ-3334), it was noticed the > keep-alive connections are closed from the server-side. The client silently > recovers and logs the connection as keep-alive, despite reestablishing a > connection. This jira aims to remove the close from the server side, fixing > the bug preventing keep-alive connections. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6850) Shuffle Handler keep-alive connections are closed from the server side
[ https://issues.apache.org/jira/browse/MAPREDUCE-6850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-6850: --- Attachment: MAPREDUCE-6850.6.patch > Shuffle Handler keep-alive connections are closed from the server side > -- > > Key: MAPREDUCE-6850 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6850 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: MAPREDUCE-6850.1.patch, MAPREDUCE-6850.2.patch, > MAPREDUCE-6850.3.patch, MAPREDUCE-6850.4.patch, MAPREDUCE-6850.5.patch, > MAPREDUCE-6850.6.patch, With_Issue.png, With_Patch.png, > With_Patch_withData.png > > > When performance testing tez shuffle handler (TEZ-3334), it was noticed the > keep-alive connections are closed from the server-side. The client silently > recovers and logs the connection as keep-alive, despite reestablishing a > connection. This jira aims to remove the close from the server side, fixing > the bug preventing keep-alive connections. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6850) Shuffle Handler keep-alive connections are closed from the server side
[ https://issues.apache.org/jira/browse/MAPREDUCE-6850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-6850: --- Attachment: MAPREDUCE-6850.5.patch [~jlowe], added state to the timeout handler (one per connection) that enables timeout once all writes are completed. Resets the state if the connection (and therefore pipeline's timer handler) is reused. > Shuffle Handler keep-alive connections are closed from the server side > -- > > Key: MAPREDUCE-6850 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6850 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: MAPREDUCE-6850.1.patch, MAPREDUCE-6850.2.patch, > MAPREDUCE-6850.3.patch, MAPREDUCE-6850.4.patch, MAPREDUCE-6850.5.patch, > With_Issue.png, With_Patch.png, With_Patch_withData.png > > > When performance testing tez shuffle handler (TEZ-3334), it was noticed the > keep-alive connections are closed from the server-side. The client silently > recovers and logs the connection as keep-alive, despite reestablishing a > connection. This jira aims to remove the close from the server side, fixing > the bug preventing keep-alive connections. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6850) Shuffle Handler keep-alive connections are closed from the server side
[ https://issues.apache.org/jira/browse/MAPREDUCE-6850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15888762#comment-15888762 ] Jonathan Eagles commented on MAPREDUCE-6850: [~jlowe], can you have a look at this patch to see if is ready to commit? [~rajesh.balamohan] and I are happy so far with the current patch but would like a second set of eyes. > Shuffle Handler keep-alive connections are closed from the server side > -- > > Key: MAPREDUCE-6850 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6850 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: MAPREDUCE-6850.1.patch, MAPREDUCE-6850.2.patch, > MAPREDUCE-6850.3.patch, MAPREDUCE-6850.4.patch, With_Issue.png, > With_Patch.png, With_Patch_withData.png > > > When performance testing tez shuffle handler (TEZ-3334), it was noticed the > keep-alive connections are closed from the server-side. The client silently > recovers and logs the connection as keep-alive, despite reestablishing a > connection. This jira aims to remove the close from the server side, fixing > the bug preventing keep-alive connections. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6850) Shuffle Handler keep-alive connections are closed from the server side
[ https://issues.apache.org/jira/browse/MAPREDUCE-6850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15886574#comment-15886574 ] Jonathan Eagles commented on MAPREDUCE-6850: [~rajesh.balamohan], posted v4 of the patch to address the extra Timer variable. Please review the latest version of the patch when you get a chance. > Shuffle Handler keep-alive connections are closed from the server side > -- > > Key: MAPREDUCE-6850 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6850 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: MAPREDUCE-6850.1.patch, MAPREDUCE-6850.2.patch, > MAPREDUCE-6850.3.patch, MAPREDUCE-6850.4.patch, With_Issue.png, > With_Patch.png, With_Patch_withData.png > > > When performance testing tez shuffle handler (TEZ-3334), it was noticed the > keep-alive connections are closed from the server-side. The client silently > recovers and logs the connection as keep-alive, despite reestablishing a > connection. This jira aims to remove the close from the server side, fixing > the bug preventing keep-alive connections. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6850) Shuffle Handler keep-alive connections are closed from the server side
[ https://issues.apache.org/jira/browse/MAPREDUCE-6850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-6850: --- Attachment: MAPREDUCE-6850.4.patch > Shuffle Handler keep-alive connections are closed from the server side > -- > > Key: MAPREDUCE-6850 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6850 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: MAPREDUCE-6850.1.patch, MAPREDUCE-6850.2.patch, > MAPREDUCE-6850.3.patch, MAPREDUCE-6850.4.patch, With_Issue.png, > With_Patch.png, With_Patch_withData.png > > > When performance testing tez shuffle handler (TEZ-3334), it was noticed the > keep-alive connections are closed from the server-side. The client silently > recovers and logs the connection as keep-alive, despite reestablishing a > connection. This jira aims to remove the close from the server side, fixing > the bug preventing keep-alive connections. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6850) Shuffle Handler keep-alive connections are closed from the server side
[ https://issues.apache.org/jira/browse/MAPREDUCE-6850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881892#comment-15881892 ] Jonathan Eagles commented on MAPREDUCE-6850: [~rajesh.balamohan], can you see if this patch fixes the keep-alive bug we were discussing and give a review? > Shuffle Handler keep-alive connections are closed from the server side > -- > > Key: MAPREDUCE-6850 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6850 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: MAPREDUCE-6850.1.patch, MAPREDUCE-6850.2.patch, > MAPREDUCE-6850.3.patch > > > When performance testing tez shuffle handler (TEZ-3334), it was noticed the > keep-alive connections are closed from the server-side. The client silently > recovers and logs the connection as keep-alive, despite reestablishing a > connection. This jira aims to remove the close from the server side, fixing > the bug preventing keep-alive connections. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6850) Shuffle Handler keep-alive connections are closed from the server side
[ https://issues.apache.org/jira/browse/MAPREDUCE-6850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-6850: --- Attachment: MAPREDUCE-6850.3.patch > Shuffle Handler keep-alive connections are closed from the server side > -- > > Key: MAPREDUCE-6850 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6850 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: MAPREDUCE-6850.1.patch, MAPREDUCE-6850.2.patch, > MAPREDUCE-6850.3.patch > > > When performance testing tez shuffle handler (TEZ-3334), it was noticed the > keep-alive connections are closed from the server-side. The client silently > recovers and logs the connection as keep-alive, despite reestablishing a > connection. This jira aims to remove the close from the server side, fixing > the bug preventing keep-alive connections. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6850) Shuffle Handler keep-alive connections are closed from the server side
[ https://issues.apache.org/jira/browse/MAPREDUCE-6850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-6850: --- Status: Patch Available (was: Open) > Shuffle Handler keep-alive connections are closed from the server side > -- > > Key: MAPREDUCE-6850 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6850 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: MAPREDUCE-6850.1.patch, MAPREDUCE-6850.2.patch > > > When performance testing tez shuffle handler (TEZ-3334), it was noticed the > keep-alive connections are closed from the server-side. The client silently > recovers and logs the connection as keep-alive, despite reestablishing a > connection. This jira aims to remove the close from the server side, fixing > the bug preventing keep-alive connections. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6850) Shuffle Handler keep-alive connections are closed from the server side
[ https://issues.apache.org/jira/browse/MAPREDUCE-6850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-6850: --- Attachment: MAPREDUCE-6850.2.patch > Shuffle Handler keep-alive connections are closed from the server side > -- > > Key: MAPREDUCE-6850 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6850 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: MAPREDUCE-6850.1.patch, MAPREDUCE-6850.2.patch > > > When performance testing tez shuffle handler (TEZ-3334), it was noticed the > keep-alive connections are closed from the server-side. The client silently > recovers and logs the connection as keep-alive, despite reestablishing a > connection. This jira aims to remove the close from the server side, fixing > the bug preventing keep-alive connections. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6850) Shuffle Handler keep-alive connections are closed from the server side
[ https://issues.apache.org/jira/browse/MAPREDUCE-6850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15881345#comment-15881345 ] Jonathan Eagles commented on MAPREDUCE-6850: Details regarding network trace showing keep-alive connections not being reused can be found in TEZ-3633. > Shuffle Handler keep-alive connections are closed from the server side > -- > > Key: MAPREDUCE-6850 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6850 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: MAPREDUCE-6850.1.patch > > > When performance testing tez shuffle handler (TEZ-3334), it was noticed the > keep-alive connections are closed from the server-side. The client silently > recovers and logs the connection as keep-alive, despite reestablishing a > connection. This jira aims to remove the close from the server side, fixing > the bug preventing keep-alive connections. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6850) Shuffle Handler keep-alive connections are closed from the server side
[ https://issues.apache.org/jira/browse/MAPREDUCE-6850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-6850: --- Attachment: MAPREDUCE-6850.1.patch > Shuffle Handler keep-alive connections are closed from the server side > -- > > Key: MAPREDUCE-6850 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6850 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: MAPREDUCE-6850.1.patch > > > When performance testing tez shuffle handler (TEZ-3334), it was noticed the > keep-alive connections are closed from the server-side. The client silently > recovers and logs the connection as keep-alive, despite reestablishing a > connection. This jira aims to remove the close from the server side, fixing > the bug preventing keep-alive connections. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6850) Shuffle Handler keep-alive connections are closed from the server side
Jonathan Eagles created MAPREDUCE-6850: -- Summary: Shuffle Handler keep-alive connections are closed from the server side Key: MAPREDUCE-6850 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6850 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Jonathan Eagles Assignee: Jonathan Eagles When performance testing tez shuffle handler (TEZ-3334), it was noticed the keep-alive connections are closed from the server-side. The client silently recovers and logs the connection as keep-alive, despite reestablishing a connection. This jira aims to remove the close from the server side, fixing the bug preventing keep-alive connections. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6808) Log map attempts as part of shuffle handler audit log
[ https://issues.apache.org/jira/browse/MAPREDUCE-6808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15761776#comment-15761776 ] Jonathan Eagles commented on MAPREDUCE-6808: Agree with [~kshukla]. Having this JIRA fixed would help me to better support tez. Any chance of a patch coming soon, [~haibochen], [~pairg]? > Log map attempts as part of shuffle handler audit log > - > > Key: MAPREDUCE-6808 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6808 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Gergő Pásztor > > With the introduction tez, multiple unrelated reducers in the same job will > have the same id. The audit log won't be able to distinguish which map > attempt the reduce was fetching. This jira is to discuss the possibility to > add map attempts logging to distinguish between the two. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-6808) Log map attempts as part of shuffle handler audit log
Jonathan Eagles created MAPREDUCE-6808: -- Summary: Log map attempts as part of shuffle handler audit log Key: MAPREDUCE-6808 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6808 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Jonathan Eagles With the introduction tez, multiple unrelated reducers in the same job will have the same id. The audit log won't be able to distinguish which map attempt the reduce was fetching. This jira is to discuss the possibility to add map attempts logging to distinguish between the two. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-5397) AM crashes because Webapp failed to start on multi node cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15246012#comment-15246012
]
Jonathan Eagles commented on MAPREDUCE-5397:
One thing else regarding this failure is that in some cases, node manager boxes
can be running a tmp directory cleaner process which can inadvertently delete
old entries in the /tmp directory. This could affect long running jobs
depending on the settings.
> AM crashes because Webapp failed to start on multi node cluster
> ---
>
> Key: MAPREDUCE-5397
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5397
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Jian He
> Attachments: MRAppMasterlog.txt, log.txt
>
>
> I set up a 12 nodes cluster and tried submitting jobs but get this exception.
> But job is able to succeed after AM crashes and retry a few times(2 or 3)
> {code}
> 2013-07-12 18:56:28,438 INFO [main] org.mortbay.log: Extract
> jar:file:/grid/0/dev/jhe/hadoop-2.1.0-beta/share/hadoop/yarn/hadoop-yarn-common-2.1.0-beta.jar!/webapps/mapreduce
> to /tmp/Jetty_0_0_0_0_43554_mapreduceljbmlg/webapp
> 2013-07-12 18:56:28,528 WARN [main] org.mortbay.log: Failed startup of
> context
> org.mortbay.jetty.webapp.WebAppContext@2726b2{/,jar:file:/grid/0/dev/jhe/hadoop-2.1.0-beta/share/hadoop/yarn/hadoop-yarn-common-2.1.0-beta.jar!/webapps/mapreduce}
> java.io.FileNotFoundException:
> /tmp/Jetty_0_0_0_0_43554_mapreduceljbmlg/webapp/webapps/mapreduce/.keep
> (No such file or directory)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.(FileOutputStream.java:194)
> at java.io.FileOutputStream.(FileOutputStream.java:145)
> at org.mortbay.resource.JarResource.extract(JarResource.java:215)
> at
> org.mortbay.jetty.webapp.WebAppContext.resolveWebApp(WebAppContext.java:974)
> at
> org.mortbay.jetty.webapp.WebAppContext.getWebInf(WebAppContext.java:832)
> at
> org.mortbay.jetty.webapp.WebInfConfiguration.configureClassLoader(WebInfConfiguration.java:62)
> at
> org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:489)
> at
> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
> at
> org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
> at
> org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
> at
> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
> at
> org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
> at org.mortbay.jetty.Server.doStart(Server.java:224)
> at
> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
> at org.apache.hadoop.http.HttpServer.start(HttpServer.java:684)
> at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:211)
> at
> org.apache.hadoop.mapreduce.v2.app.client.MRClientService.serviceStart(MRClientService.java:134)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:101)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1019)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1394)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1390)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5397) AM crashes because Webapp failed to start on multi node cluster
[
https://issues.apache.org/jira/browse/MAPREDUCE-5397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15245998#comment-15245998
]
Jonathan Eagles commented on MAPREDUCE-5397:
Couple of leads on a work-around.
MAPREDUCE-6472 (fixed in 2.7.2) which sets the java.io.tmpdir which is the
directory to the container's working directory
Reference Documentation
http://www.eclipse.org/jetty/documentation/current/ref-temporary-directories.html
In versions before 2.7.2, you can always suppliment the job specific
yarn.app.mapreduce.am.command-opts by adding -Djava.io.tmpdir=./tmp or add it
in the cluster wide admin settings yarn.app.mapreduce.am.admin-command-opts.
This has the added benefit of getting cleaned up when a job is finished. If a
job dies with data written to /tmp, it will be not be cleaned up and can
pollute the tmp file system and suffer the effects (jobs fail, slow jobs, etc).
> AM crashes because Webapp failed to start on multi node cluster
> ---
>
> Key: MAPREDUCE-5397
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5397
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Jian He
> Attachments: MRAppMasterlog.txt, log.txt
>
>
> I set up a 12 nodes cluster and tried submitting jobs but get this exception.
> But job is able to succeed after AM crashes and retry a few times(2 or 3)
> {code}
> 2013-07-12 18:56:28,438 INFO [main] org.mortbay.log: Extract
> jar:file:/grid/0/dev/jhe/hadoop-2.1.0-beta/share/hadoop/yarn/hadoop-yarn-common-2.1.0-beta.jar!/webapps/mapreduce
> to /tmp/Jetty_0_0_0_0_43554_mapreduceljbmlg/webapp
> 2013-07-12 18:56:28,528 WARN [main] org.mortbay.log: Failed startup of
> context
> org.mortbay.jetty.webapp.WebAppContext@2726b2{/,jar:file:/grid/0/dev/jhe/hadoop-2.1.0-beta/share/hadoop/yarn/hadoop-yarn-common-2.1.0-beta.jar!/webapps/mapreduce}
> java.io.FileNotFoundException:
> /tmp/Jetty_0_0_0_0_43554_mapreduceljbmlg/webapp/webapps/mapreduce/.keep
> (No such file or directory)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.(FileOutputStream.java:194)
> at java.io.FileOutputStream.(FileOutputStream.java:145)
> at org.mortbay.resource.JarResource.extract(JarResource.java:215)
> at
> org.mortbay.jetty.webapp.WebAppContext.resolveWebApp(WebAppContext.java:974)
> at
> org.mortbay.jetty.webapp.WebAppContext.getWebInf(WebAppContext.java:832)
> at
> org.mortbay.jetty.webapp.WebInfConfiguration.configureClassLoader(WebInfConfiguration.java:62)
> at
> org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:489)
> at
> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
> at
> org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
> at
> org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
> at
> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
> at
> org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
> at org.mortbay.jetty.Server.doStart(Server.java:224)
> at
> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
> at org.apache.hadoop.http.HttpServer.start(HttpServer.java:684)
> at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:211)
> at
> org.apache.hadoop.mapreduce.v2.app.client.MRClientService.serviceStart(MRClientService.java:134)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:101)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1019)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1394)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1390)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6385) mapreduce.map.env and mapreduce.reduce.env need to fall back to mapred.child.env for compatibility
Jonathan Eagles created MAPREDUCE-6385: -- Summary: mapreduce.map.env and mapreduce.reduce.env need to fall back to mapred.child.env for compatibility Key: MAPREDUCE-6385 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6385 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Jonathan Eagles -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6285) ClientServiceDelegate should not retry upon AuthenticationException
[ https://issues.apache.org/jira/browse/MAPREDUCE-6285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-6285: --- Attachment: MAPREDUCE-6285.2.patch ClientServiceDelegate should not retry upon AuthenticationException --- Key: MAPREDUCE-6285 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6285 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: MAPREDUCE-6285.1.patch, MAPREDUCE-6285.2.patch ClientServiceDelegate tries really hard to getJobStatus from a job through retries. However, if an AuthorizationException occurs, it will always continue to fail no matter how many retries it makes. AuthorizationException is also thrown when mapred job -status is call on unsupported services like tez jobs. This jira will limit the negative effect on AMs in both scenarios. Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): Protocol interface org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB is not known. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6285) ClientServiceDelegate should not retry upon AuthenticationException
[ https://issues.apache.org/jira/browse/MAPREDUCE-6285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-6285: --- Attachment: MAPREDUCE-6285.1.patch ClientServiceDelegate should not retry upon AuthenticationException --- Key: MAPREDUCE-6285 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6285 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: MAPREDUCE-6285.1.patch ClientServiceDelegate tries really hard to getJobStatus from a job through retries. However, if an AuthorizationException occurs, it will always continue to fail no matter how many retries it makes. AuthorizationException is also thrown when mapred job -status is call on unsupported services like tez jobs. This jira will limit the negative effect on AMs in both scenarios. Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): Protocol interface org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB is not known. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6285) ClientServiceDelegate should not retry upon AuthenticationException
[ https://issues.apache.org/jira/browse/MAPREDUCE-6285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-6285: --- Status: Patch Available (was: Open) ClientServiceDelegate should not retry upon AuthenticationException --- Key: MAPREDUCE-6285 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6285 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: MAPREDUCE-6285.1.patch ClientServiceDelegate tries really hard to getJobStatus from a job through retries. However, if an AuthorizationException occurs, it will always continue to fail no matter how many retries it makes. AuthorizationException is also thrown when mapred job -status is call on unsupported services like tez jobs. This jira will limit the negative effect on AMs in both scenarios. Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): Protocol interface org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB is not known. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6285) ClientServiceDelegate should not retry upon AuthenticationException
Jonathan Eagles created MAPREDUCE-6285: -- Summary: ClientServiceDelegate should not retry upon AuthenticationException Key: MAPREDUCE-6285 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6285 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Jonathan Eagles Assignee: Jonathan Eagles ClientServiceDelegate tries really hard to getJobStatus from a job through retries. However, if an AuthorizationException occurs, it will always continue to fail no matter how many retries it makes. AuthorizationException is also thrown when mapred job -status is call on unsupported services like tez jobs. This jira will limit the negative effect on AMs in both scenarios. Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.authorize.AuthorizationException): Protocol interface org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB is not known. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6141) History server leveldb recovery store
[ https://issues.apache.org/jira/browse/MAPREDUCE-6141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14190259#comment-14190259 ] Jonathan Eagles commented on MAPREDUCE-6141: Thanks, [~jlowe]. Everything looks good now. I'll wait a day or so to let others in the community. History server leveldb recovery store - Key: MAPREDUCE-6141 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6141 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobhistoryserver Reporter: Jason Lowe Assignee: Jason Lowe Attachments: MAPREDUCE-6141.patch, MAPREDUCE-6141v2.patch It would be nice to have a leveldb option to the job history server recovery store. Leveldb would provide some benefits over the existing filesystem store such as better support for atomic operations, fewer I/O ops per state update, and far fewer total files on the filesystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6141) History server leveldb recovery store
[ https://issues.apache.org/jira/browse/MAPREDUCE-6141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14189208#comment-14189208 ] Jonathan Eagles commented on MAPREDUCE-6141: [~jlowe]. Looks good overall. Couple of minor thoughts. - Should we consider making *mapreduce.jobhistory.recovery.store.class* default to levelb as part of a follow on jira - TestHistoryServerLeveldbStateStoreService - please document the test blocks testTokenStore with comments to help guide the reader of the test code (verify tokens stored can be read after reload, deleted tokens are really deleted) - HistoryServerLeveldbStateStoreService - if version in db in null upon initialization, I think we want to return this as the current version and not 1.0. I think the thought is that we are looking at an uninitialized db. Right now these values are the same, so I'm just trying to clarify the intention. - Is the mapreduce-default.xml sufficient documentation to use this feature? For me it is, but what is your take on it. For example what conditions would one want to use one over the other? History server leveldb recovery store - Key: MAPREDUCE-6141 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6141 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobhistoryserver Reporter: Jason Lowe Assignee: Jason Lowe Attachments: MAPREDUCE-6141.patch It would be nice to have a leveldb option to the job history server recovery store. Leveldb would provide some benefits over the existing filesystem store such as better support for atomic operations, fewer I/O ops per state update, and far fewer total files on the filesystem. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5933) Enable MR AM to post history events to the timeline server
[ https://issues.apache.org/jira/browse/MAPREDUCE-5933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14106289#comment-14106289 ] Jonathan Eagles commented on MAPREDUCE-5933: [~rkanter], Appoligies for not responding earlier to this JIRA. I've posted my design requests to MAPREDUCE-5858 to make sure existing can be kept happy. https://issues.apache.org/jira/browse/MAPREDUCE-5858?focusedCommentId=14085166page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14085166 Enable MR AM to post history events to the timeline server -- Key: MAPREDUCE-5933 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5933 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mr-am Reporter: Zhijie Shen Assignee: Robert Kanter Attachments: MAPREDUCE-5933.patch, MAPREDUCE-5933.patch, mr_timelineserver_response.txt Nowadays, MR AM collects the history events and writes it to HDFS for JHS to source. With the timeline server, MR AM can put these events there. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-6014) New task status field in task attempts table can lead to an empty web page
[ https://issues.apache.org/jira/browse/MAPREDUCE-6014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14085015#comment-14085015 ] Jonathan Eagles commented on MAPREDUCE-6014: +1. This will correct this issue New task status field in task attempts table can lead to an empty web page --- Key: MAPREDUCE-6014 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6014 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.5.0 Reporter: Mit Desai Assignee: Mit Desai Attachments: MAPREDUCE-6014.patch MAPREDUCE-5550 added a new task attempts field but didn't Javascript-escape the contents. Tasks with status messages that have newlines or other characters can then break the parsing of the web page and leave the user with a blank page. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5858) [Umbrella] MR should make use of the timeline server
[ https://issues.apache.org/jira/browse/MAPREDUCE-5858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14085166#comment-14085166 ] Jonathan Eagles commented on MAPREDUCE-5858: [~zjshen], we need to be careful when designing this. In particular, users of MR job history server will want to: - Continue using Job history server with no runtime dependency a timeline server being present - Enable or disable timeline server from the client side - automatic or manual pluggable configuration or disabling of an in memory solution for local mode testing - minicluster integration - Allow for history enabled for timeline server and job history server at the same time - What to do for the tracking url for different scenarios above - New MR UI using backing timeline backing store [Umbrella] MR should make use of the timeline server Key: MAPREDUCE-5858 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5858 Project: Hadoop Map/Reduce Issue Type: Task Reporter: Zhijie Shen Now MR relies on its own JobHistoryServer for MR specific history information. Given the timeline server is ready, we should gradually migrate MR historic data to it as well. relieving MR from maintaining its own history server daemon. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (MAPREDUCE-6018) Create a framework specific config to enable timeline server
Jonathan Eagles created MAPREDUCE-6018: -- Summary: Create a framework specific config to enable timeline server Key: MAPREDUCE-6018 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6018 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Jonathan Eagles -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-3418) If map output is not found, shuffle runs in tight loop
[ https://issues.apache.org/jira/browse/MAPREDUCE-3418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3418: --- Assignee: Chen He If map output is not found, shuffle runs in tight loop -- Key: MAPREDUCE-3418 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3418 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0, 2.3.0 Reporter: John George Assignee: Chen He Sharad Agarwal bumped into this while simulating fetch failures. Removed the map output directory. Shuffle runs in tight loop throwing : 2011-06-01 09:02:20,511 WARN org.apache.hadoop.mapreduce.task.reduce.Fetcher: Invalid map id java.lang.IllegalArgumentException: TaskAttemptId string : TTP/1.1 500 Internal Server Error Content-Type: text/plain; charset=UTF is not properly formed at org.apache.hadoop.mapreduce.TaskAttemptID.forName(TaskAttemptID.java:174) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyMapOutput(Fetcher.java:284) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:251) at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:149) Fetch failure is not triggered. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5854) Move the search box in UI from the right side to the left side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14044013#comment-14044013 ] Jonathan Eagles commented on MAPREDUCE-5854: +1. Looks good, [~lichangleo]. Let me leave this open a bit to let others chime on this cosmetic, but great change. Move the search box in UI from the right side to the left side -- Key: MAPREDUCE-5854 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5854 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.23.9 Reporter: Jinhui Liu Assignee: chang li Attachments: MAPREDUCE-5854.patch, MAPREDUCE-5854.patch In the UI for resoure manager, job history, and job configuration (this might not be a complete list), there is a search box at the top-right corner of the listed content. This search box is frequently used but it is usually not visible due to right-alignment. Extra scroll is needed to make it visable and it is not convenient. It would be good to move it to the left-side, next to the Show ... Entries drop-down box. In the same spirit, the First|Preious|...|Next|Last at the bottom-right corner of the listed content can also be moved to the left side. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5854) Move the search box in UI from the right side to the left side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14040885#comment-14040885 ] Jonathan Eagles commented on MAPREDUCE-5854: [~lichangleo], Thanks for the patch as this can be a nice fix. Can you address the release audit problems by including the Apache license header for the new css file added? Move the search box in UI from the right side to the left side -- Key: MAPREDUCE-5854 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5854 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.23.9 Reporter: Jinhui Liu Assignee: chang li Attachments: MAPREDUCE-5854.patch In the UI for resoure manager, job history, and job configuration (this might not be a complete list), there is a search box at the top-right corner of the listed content. This search box is frequently used but it is usually not visible due to right-alignment. Extra scroll is needed to make it visable and it is not convenient. It would be good to move it to the left-side, next to the Show ... Entries drop-down box. In the same spirit, the First|Preious|...|Next|Last at the bottom-right corner of the listed content can also be moved to the left side. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5928) Deadlock allocating containers for mappers and reducers
[
https://issues.apache.org/jira/browse/MAPREDUCE-5928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14032496#comment-14032496
]
Jonathan Eagles commented on MAPREDUCE-5928:
I think this is a case of task preemption not working since the headroom
calculation is not correct. Can you verify you are using the capacity scheduler.
YARN-1198
Deadlock allocating containers for mappers and reducers
---
Key: MAPREDUCE-5928
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5928
Project: Hadoop Map/Reduce
Issue Type: Bug
Environment: Hadoop 2.4.0 (as packaged by HortonWorks in HDP 2.1.2)
Reporter: Niels Basjes
Attachments: Cluster fully loaded.png.jpg, MR job stuck in
deadlock.png.jpg
I have a small cluster consisting of 8 desktop class systems (1 master + 7
workers).
Due to the small memory of these systems I configured yarn as follows:
{quote}
yarn.nodemanager.resource.memory-mb = 2200
yarn.scheduler.minimum-allocation-mb = 250
{quote}
On my client I did
{quote}
mapreduce.map.memory.mb = 512
mapreduce.reduce.memory.mb = 512
{quote}
Now I run a job with 27 mappers and 32 reducers.
After a while I saw this deadlock occur:
- All nodes had been filled to their maximum capacity with reducers.
- 1 Mapper was waiting for a container slot to start in.
I tried killing reducer attempts but that didn't help (new reducer attempts
simply took the existing container).
*Workaround*:
I set this value from my job. The default value is 0.05 (= 5%)
{quote}
mapreduce.job.reduce.slowstart.completedmaps = 0.99f
{quote}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5888) Failed job leaves hung AM after it unregisters
[ https://issues.apache.org/jira/browse/MAPREDUCE-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996619#comment-13996619 ] Jonathan Eagles commented on MAPREDUCE-5888: Agree with [~ozawa]. +1 binding. Thanks, [~jlowe]. Committing this to trunk and branch-2 Failed job leaves hung AM after it unregisters --- Key: MAPREDUCE-5888 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5888 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am Affects Versions: 2.2.0 Reporter: Jason Lowe Assignee: Jason Lowe Attachments: MAPREDUCE-5888.patch When a job fails the AM hangs during shutdown. A non-daemon thread pool executor thread prevents the JVM teardown from completing, and the AM lingers on the cluster for the AM expiry interval in the FINISHING state until eventually the RM expires it and kills the container. If application limits on the queue are relatively low (e.g.: small queue or small cluster) this can cause unnecessary delays in resource scheduling on the cluster. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5888) Failed job leaves hung AM after it unregisters
[ https://issues.apache.org/jira/browse/MAPREDUCE-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996739#comment-13996739 ] Jonathan Eagles commented on MAPREDUCE-5888: +1. No tests is expected in this case. Committing to branch-2 and trunk. Failed job leaves hung AM after it unregisters --- Key: MAPREDUCE-5888 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5888 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am Affects Versions: 2.2.0 Reporter: Jason Lowe Assignee: Jason Lowe Attachments: MAPREDUCE-5888.patch When a job fails the AM hangs during shutdown. A non-daemon thread pool executor thread prevents the JVM teardown from completing, and the AM lingers on the cluster for the AM expiry interval in the FINISHING state until eventually the RM expires it and kills the container. If application limits on the queue are relatively low (e.g.: small queue or small cluster) this can cause unnecessary delays in resource scheduling on the cluster. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5888) Failed job leaves hung AM after it unregisters
[ https://issues.apache.org/jira/browse/MAPREDUCE-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13996622#comment-13996622 ] Jonathan Eagles commented on MAPREDUCE-5888: pending Hadoop QA +1 Failed job leaves hung AM after it unregisters --- Key: MAPREDUCE-5888 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5888 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am Affects Versions: 2.2.0 Reporter: Jason Lowe Assignee: Jason Lowe Attachments: MAPREDUCE-5888.patch When a job fails the AM hangs during shutdown. A non-daemon thread pool executor thread prevents the JVM teardown from completing, and the AM lingers on the cluster for the AM expiry interval in the FINISHING state until eventually the RM expires it and kills the container. If application limits on the queue are relatively low (e.g.: small queue or small cluster) this can cause unnecessary delays in resource scheduling on the cluster. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5888) Failed job leaves hung AM after it unregisters
[ https://issues.apache.org/jira/browse/MAPREDUCE-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-5888: --- Resolution: Fixed Fix Version/s: 2.5.0 3.0.0 Status: Resolved (was: Patch Available) Failed job leaves hung AM after it unregisters --- Key: MAPREDUCE-5888 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5888 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am Affects Versions: 2.2.0 Reporter: Jason Lowe Assignee: Jason Lowe Fix For: 3.0.0, 2.5.0 Attachments: MAPREDUCE-5888.patch When a job fails the AM hangs during shutdown. A non-daemon thread pool executor thread prevents the JVM teardown from completing, and the AM lingers on the cluster for the AM expiry interval in the FINISHING state until eventually the RM expires it and kills the container. If application limits on the queue are relatively low (e.g.: small queue or small cluster) this can cause unnecessary delays in resource scheduling on the cluster. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5637) Convert Hadoop Streaming document to APT
[ https://issues.apache.org/jira/browse/MAPREDUCE-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-5637: --- Resolution: Fixed Fix Version/s: 2.5.0 3.0.0 Status: Resolved (was: Patch Available) +1. Thanks, again [~ajisakaa] Convert Hadoop Streaming document to APT Key: MAPREDUCE-5637 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5637 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: documentation Affects Versions: 2.2.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Fix For: 3.0.0, 2.5.0 Attachments: MAPREDUCE-5637.2.patch, MAPREDUCE-5637.patch Convert Hadoop Streaming document from forrest to APT. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5636) Convert MapReduce Tutorial document to APT
[ https://issues.apache.org/jira/browse/MAPREDUCE-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-5636: --- Resolution: Fixed Fix Version/s: 2.5.0 3.0.0 Status: Resolved (was: Patch Available) +1. It is great to see these finally get converted. Thanks, [~ajisakaa] Convert MapReduce Tutorial document to APT -- Key: MAPREDUCE-5636 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5636 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: documentation Affects Versions: 2.2.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Fix For: 3.0.0, 2.5.0 Attachments: MAPREDUCE-5636.2.patch, MAPREDUCE-5636.patch Convert MapReduce Tutorial document from forrest to APT. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-4282) Convert Forrest docs to APT
[ https://issues.apache.org/jira/browse/MAPREDUCE-4282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13990805#comment-13990805 ] Jonathan Eagles commented on MAPREDUCE-4282: Resolving this as all subtasks are complete. Thanks so much to everyone who contributed to this effort. Convert Forrest docs to APT --- Key: MAPREDUCE-4282 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4282 Project: Hadoop Map/Reduce Issue Type: Task Components: documentation Affects Versions: 2.0.0-alpha Reporter: Eli Collins Assignee: Akira AJISAKA Labels: newbie Attachments: MAPREDUCE-4282-1.patch, MAPREDUCE-4282-2.patch, MAPREDUCE-4282-streaming.patch, MAPREDUCE-4282-tutorial.2.patch, MAPREDUCE-4282-tutorial.3.patch, MAPREDUCE-4282-tutorial.patch, MapReduceTutorial.2.html, MapReduceTutorial.html, bad-forrest2apt-attempt1.tar MR side of HADOOP-8427. Not all of the old forrest docs in src/documentation/content/xdocs have been converted over to APT yet, let's do that and remove the forrest docs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-4282) Convert Forrest docs to APT
[ https://issues.apache.org/jira/browse/MAPREDUCE-4282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-4282: --- Resolution: Fixed Fix Version/s: 2.5.0 3.0.0 Target Version/s: (was: ) Status: Resolved (was: Patch Available) Convert Forrest docs to APT --- Key: MAPREDUCE-4282 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4282 Project: Hadoop Map/Reduce Issue Type: Task Components: documentation Affects Versions: 2.0.0-alpha Reporter: Eli Collins Assignee: Akira AJISAKA Labels: newbie Fix For: 3.0.0, 2.5.0 Attachments: MAPREDUCE-4282-1.patch, MAPREDUCE-4282-2.patch, MAPREDUCE-4282-streaming.patch, MAPREDUCE-4282-tutorial.2.patch, MAPREDUCE-4282-tutorial.3.patch, MAPREDUCE-4282-tutorial.patch, MapReduceTutorial.2.html, MapReduceTutorial.html, bad-forrest2apt-attempt1.tar MR side of HADOOP-8427. Not all of the old forrest docs in src/documentation/content/xdocs have been converted over to APT yet, let's do that and remove the forrest docs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5638) Port Hadoop Archives document to trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-5638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13984814#comment-13984814 ] Jonathan Eagles commented on MAPREDUCE-5638: +1. Good port from 1.x line. Checking into trunk and branch-2 Port Hadoop Archives document to trunk -- Key: MAPREDUCE-5638 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5638 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: documentation Reporter: Akira AJISAKA Assignee: Akira AJISAKA Attachments: MAPREDUCE-5638-md.patch, MAPREDUCE-5638.patch Now Hadoop Archive document exists only in branch-1. Let's port Hadoop Archives document to trunk. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5638) Port Hadoop Archives document to trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-5638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-5638: --- Resolution: Fixed Fix Version/s: 2.5.0 3.0.0 Status: Resolved (was: Patch Available) Port Hadoop Archives document to trunk -- Key: MAPREDUCE-5638 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5638 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: documentation Reporter: Akira AJISAKA Assignee: Akira AJISAKA Fix For: 3.0.0, 2.5.0 Attachments: MAPREDUCE-5638-md.patch, MAPREDUCE-5638.patch Now Hadoop Archive document exists only in branch-1. Let's port Hadoop Archives document to trunk. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (MAPREDUCE-5118) Inverted Index
[ https://issues.apache.org/jira/browse/MAPREDUCE-5118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles resolved MAPREDUCE-5118. Resolution: Later Target Version/s: 1.1.2, 2.0.3-alpha, 0.23.3, 0.20.205.0 (was: 0.20.205.0, 0.23.3, 2.0.3-alpha, 1.1.2) Inverted Index -- Key: MAPREDUCE-5118 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5118 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: benchmarks Reporter: Faraz Ahmad Assignee: Faraz Ahmad Attachments: MAPREDUCE-5116.patch Original Estimate: 0h Remaining Estimate: 0h Inverted-Index takes a list of documents as input and generates word-to-document indexing. Map emits word, docId tuples with each word emitted once per docId. Reduce combines all tuples on key word and emits word,docId tuples after removing duplicates. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (MAPREDUCE-5116) PUMA Benchmark Suite
[ https://issues.apache.org/jira/browse/MAPREDUCE-5116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles resolved MAPREDUCE-5116. Resolution: Later Target Version/s: 1.1.2, 0.23.5, 2.0.3-alpha, 0.20.205.0 (was: 0.20.205.0, 2.0.3-alpha, 0.23.5, 1.1.2) PUMA Benchmark Suite Key: MAPREDUCE-5116 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5116 Project: Hadoop Map/Reduce Issue Type: New Feature Components: benchmarks Reporter: Faraz Ahmad Assignee: Faraz Ahmad Original Estimate: 48h Time Spent: 48h Remaining Estimate: 0h A benchmark suite which represents a broad range of real-world MapReduce applications exhibiting application characteristics with high/low computation and high/low shuffle volumes. These benchmarks have been published as part of MaRCO (http://dx.doi.org/10.1016/j.jpdc.2012.12.012) project in JPDC '12. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5639) Port DistCp2 document to trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-5639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-5639: --- Resolution: Fixed Fix Version/s: 2.5.0 3.0.0 Status: Resolved (was: Patch Available) +1. Thanks [~ajisakaa] Port DistCp2 document to trunk -- Key: MAPREDUCE-5639 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5639 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: documentation Affects Versions: 2.2.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Fix For: 3.0.0, 2.5.0 Attachments: MAPREDUCE-5639.2.patch, MAPREDUCE-5639.patch Port DistCp2 document (http://hadoop.apache.org/docs/r1.2.1/distcp2.html) to trunk. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5639) Port DistCp2 document to trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-5639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13979768#comment-13979768 ] Jonathan Eagles commented on MAPREDUCE-5639: [~ajisakaa], the vm port looks to be accurate. Do we want to remove the hadoop-tools/hadoop-distcp/src/site directory as part of this jira? Port DistCp2 document to trunk -- Key: MAPREDUCE-5639 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5639 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: documentation Affects Versions: 2.2.0 Reporter: Akira AJISAKA Assignee: Akira AJISAKA Attachments: MAPREDUCE-5639.patch Port DistCp2 document (http://hadoop.apache.org/docs/r1.2.1/distcp2.html) to trunk. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-4065) Add .proto files to built tarball
[ https://issues.apache.org/jira/browse/MAPREDUCE-4065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-4065: --- Target Version/s: 2.5.0 (was: 0.23.0) Add .proto files to built tarball - Key: MAPREDUCE-4065 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4065 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.23.2 Reporter: Ralph H Castain Please add the .proto files to the built tarball so that users can build 3rd party tools that use protocol buffers without having to do an svn checkout of the source code. Sorry I don't know more about Maven, or I would provide a patch. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-4065) Add .proto files to built tarball
[ https://issues.apache.org/jira/browse/MAPREDUCE-4065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13976953#comment-13976953 ] Jonathan Eagles commented on MAPREDUCE-4065: As 0.23.x is going to maintenance mode, I have re-targeted this jira for an feature accepting active line. The issue of allowing 3rd party tools is especially compelling, but as Bobby mentioned, very difficult at this point. Add .proto files to built tarball - Key: MAPREDUCE-4065 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4065 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.23.2 Reporter: Ralph H Castain Please add the .proto files to the built tarball so that users can build 3rd party tools that use protocol buffers without having to do an svn checkout of the source code. Sorry I don't know more about Maven, or I would provide a patch. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5116) PUMA Benchmark Suite
[ https://issues.apache.org/jira/browse/MAPREDUCE-5116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13976976#comment-13976976 ] Jonathan Eagles commented on MAPREDUCE-5116: There hasn't been any traffic on this jira in some time. [~algol], are you still interested in getting this benchmark suite into Hadoop? Please comment on you interest in finishing up this work and I'll be happy to keep this ticket open. Otherwise, I'll close this ticket at the end of the week. PUMA Benchmark Suite Key: MAPREDUCE-5116 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5116 Project: Hadoop Map/Reduce Issue Type: New Feature Components: benchmarks Reporter: Faraz Ahmad Assignee: Faraz Ahmad Original Estimate: 48h Time Spent: 48h Remaining Estimate: 0h A benchmark suite which represents a broad range of real-world MapReduce applications exhibiting application characteristics with high/low computation and high/low shuffle volumes. These benchmarks have been published as part of MaRCO (http://dx.doi.org/10.1016/j.jpdc.2012.12.012) project in JPDC '12. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-3755) Add the equivalent of JobStatus to end of JobHistory file
[ https://issues.apache.org/jira/browse/MAPREDUCE-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13976993#comment-13976993 ] Jonathan Eagles commented on MAPREDUCE-3755: [~bikassaha], does this change still make sense with the addition of the ATS? I don't have much context for this issue, but I'm trying to cleanup some old tickets and not sure whether this ticket is still an issue we want to pursue. Add the equivalent of JobStatus to end of JobHistory file -- Key: MAPREDUCE-3755 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3755 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: jobhistoryserver, mrv2 Affects Versions: 0.23.0 Reporter: Arun C Murthy Assignee: Bikas Saha Fix For: 0.23.2 In MR1 we have the notion of CompletedJobStatus store to aid fast responses to job.getStatus. We need the equivalent for MR2, an option is to add the jobStatus to the end of the JobHistory file to which the JHS can easily jump ahead to and serve the query, it should also cache this for a fair number of recently completed jobs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-3191) docs for map output compression incorrectly reference SequenceFile
[ https://issues.apache.org/jira/browse/MAPREDUCE-3191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13971612#comment-13971612 ] Jonathan Eagles commented on MAPREDUCE-3191: +1 LGTM. With a documentation change like this, there is little to no risk of putting this into 0.23.x. Glad to see you take up this issue. It is customary to request taking over a ticket so as to not step on anyone's toes, however. docs for map output compression incorrectly reference SequenceFile -- Key: MAPREDUCE-3191 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3191 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Chen He Priority: Trivial Labels: documentation, noob Attachments: MAPREDUCE-3191-v2.patch, MAPREDUCE-3191.patch The documentation currently says that map output compression uses SequenceFile compression. This hasn't been true in several years, since we use IFile for intermediate data now. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-3191) docs for map output compression incorrectly reference SequenceFile
[ https://issues.apache.org/jira/browse/MAPREDUCE-3191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3191: --- Resolution: Fixed Fix Version/s: 2.4.1 2.5.0 0.23.11 3.0.0 Status: Resolved (was: Patch Available) docs for map output compression incorrectly reference SequenceFile -- Key: MAPREDUCE-3191 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3191 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Chen He Priority: Trivial Labels: documentation, noob Fix For: 3.0.0, 0.23.11, 2.5.0, 2.4.1 Attachments: MAPREDUCE-3191-v2.patch, MAPREDUCE-3191.patch The documentation currently says that map output compression uses SequenceFile compression. This hasn't been true in several years, since we use IFile for intermediate data now. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5836) Fix typo in RandomTextWriter
[
https://issues.apache.org/jira/browse/MAPREDUCE-5836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13969622#comment-13969622
]
Jonathan Eagles commented on MAPREDUCE-5836:
Do you think we can up this comment in both places?
hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/RandomTextWriter.java
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/RandomTextWriter.java
Fix typo in RandomTextWriter
Key: MAPREDUCE-5836
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5836
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Akira AJISAKA
Assignee: Akira AJISAKA
Priority: Trivial
Labels: newbie
Attachments: MAPREDUCE-5836.patch
In RandomTextWriter.java, there is a typo.
{code}
/**
* A random list of 100 words from /usr/share/dict/words
*/
private static String[] words = {
diurnalness, Homoiousian,
{code}
The list actually contains 1000 words.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5836) Fix typo in RandomTextWriter
[
https://issues.apache.org/jira/browse/MAPREDUCE-5836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13970012#comment-13970012
]
Jonathan Eagles commented on MAPREDUCE-5836:
+1. committing this fix. Thanks, [~ajisakaa].
Fix typo in RandomTextWriter
Key: MAPREDUCE-5836
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5836
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Akira AJISAKA
Assignee: Akira AJISAKA
Priority: Trivial
Labels: newbie
Attachments: MAPREDUCE-5836.2.patch, MAPREDUCE-5836.patch
In RandomTextWriter.java, there is a typo.
{code}
/**
* A random list of 100 words from /usr/share/dict/words
*/
private static String[] words = {
diurnalness, Homoiousian,
{code}
The list actually contains 1000 words.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (MAPREDUCE-3551) web proxy returns internal server error when application invalid
[ https://issues.apache.org/jira/browse/MAPREDUCE-3551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles resolved MAPREDUCE-3551. Resolution: Fixed This is fix in some previous version of hadoop. web proxy returns internal server error when application invalid Key: MAPREDUCE-3551 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3551 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Jonathan Eagles querying for an absent app i.e. say http://RM:8088/proxy/application_1323054018624_0004/ws/v1/mapreduce/info - returns a 500 with a stack trace instead of a 404. From an api spec point of view, this is not really an error but just a not found. This occurs with web services as well as any web ui page. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5836) Fix typo in RandomTextWriter
[
https://issues.apache.org/jira/browse/MAPREDUCE-5836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Eagles updated MAPREDUCE-5836:
---
Resolution: Fixed
Fix Version/s: 2.5.0
3.0.0
Status: Resolved (was: Patch Available)
Fix typo in RandomTextWriter
Key: MAPREDUCE-5836
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5836
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Akira AJISAKA
Assignee: Akira AJISAKA
Priority: Trivial
Labels: newbie
Fix For: 3.0.0, 2.5.0
Attachments: MAPREDUCE-5836.2.patch, MAPREDUCE-5836.patch
In RandomTextWriter.java, there is a typo.
{code}
/**
* A random list of 100 words from /usr/share/dict/words
*/
private static String[] words = {
diurnalness, Homoiousian,
{code}
The list actually contains 1000 words.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (MAPREDUCE-3120) JobHistory is not providing correct count failed,killed task
[ https://issues.apache.org/jira/browse/MAPREDUCE-3120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles resolved MAPREDUCE-3120. Resolution: Fixed Target Version/s: 2.0.0-alpha, 0.23.3, 3.0.0 (was: 0.23.3, 2.0.0-alpha, 3.0.0) Duping to MAPREDUCE-3032 as we haven't heard from the reporter on this issue in some time and believe this issue is already fixed. Feel free to reopen this ticket if this is still and issue on the latest 2.4.0 build. JobHistory is not providing correct count failed,killed task Key: MAPREDUCE-3120 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3120 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Subroto Sanyal Assignee: Subroto Sanyal Fix For: 0.24.0 Attachments: JobFail.PNG Please refer the attachment JobFail.PNG. Here the Job (WordCount) Failed as all Map Attempts were killed(intensionally) but, still the Table in UI shows 0 Killed Attempts and no reason for Failure is also available. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5607) Backport MAPREDUCE-5086 - MR app master deletes staging dir when sent a reboot command from the RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-5607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-5607: --- Resolution: Won't Fix Status: Resolved (was: Patch Available) This feature change is introduces too much risk to so close to the end of 0.23.x development and the beginning of maintenance for this line. Backport MAPREDUCE-5086 - MR app master deletes staging dir when sent a reboot command from the RM -- Key: MAPREDUCE-5607 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5607 Project: Hadoop Map/Reduce Issue Type: Sub-task Affects Versions: 0.23.9 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Attachments: MAPREDUCE-5607-branch-0.23.patch If the RM is restarted when the MR job is running, then it sends a reboot command to the job. The job ends up deleting the staging dir and that causes the next attempt to fail. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-4941) Use of org.apache.hadoop.mapred.lib.CombineFileRecordReader requires casting
[
https://issues.apache.org/jira/browse/MAPREDUCE-4941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13949938#comment-13949938
]
Jonathan Eagles commented on MAPREDUCE-4941:
[~jlowe], this patch no longer applies. Can you update when you get a chance?
Use of org.apache.hadoop.mapred.lib.CombineFileRecordReader requires casting
Key: MAPREDUCE-4941
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4941
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.0.2-alpha, 0.23.5
Reporter: Jason Lowe
Assignee: Jason Lowe
Priority: Minor
Attachments: MAPREDUCE-4941.patch, MAPREDUCE-4941.patch
Unlike its counterpart in org.apache.hadoop.mapreduce.lib.input, the
CombineFileRecordReader in mapred requires a user to cast to a RecordReader
since the constructor specification says it must have the RecordReaderK,V
class as a parameter. It should use {{Class? extends RecordReaderK,V}}
like its mapreduce counterpart to make it easier to use.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5797) Elapsed time for failed tasks that never started is wrong
[ https://issues.apache.org/jira/browse/MAPREDUCE-5797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13935965#comment-13935965 ] Jonathan Eagles commented on MAPREDUCE-5797: [~shahrs87], in general I like it. Couple of minor nits. *Hadoop uses the java oracle coding standards roughly, with the exceptions that tabs are expanded and we use 2 spaces to indent. I highly recommend the indentation and white space sections for gotchas. For example keywords followed by a parenthesis should have a space in between. http://wiki.apache.org/hadoop/HowToContribute *Of course, all things have exceptions like maintaining the edited files current formatting, however wrong. *Separate out the test cases and name them appropriate to describe the feature under test. Example, testNegativeStartTime, testNegativeEndTime. This way when it fails, the feature that is broken is immediately evident. Sorry for the overly long comment. Welcome to the Apache Hadoop community. We are happy to have you onboard. Elapsed time for failed tasks that never started is wrong Key: MAPREDUCE-5797 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5797 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, webapps Affects Versions: 0.23.9 Reporter: Rushabh S Shah Assignee: Rushabh S Shah Attachments: patch-MapReduce-5797-v2.patch, patch-MapReduce-5797.patch The elapsed time for tasks in a failed job that were never started can be way off. It looks like we're marking the start time as the beginning of the epoch (i.e.: start time = -1) but the finish time is when the task was marked as failed when the whole job failed. That causes the calculated elapsed time of the task to be a ridiculous number of hours. Tasks that fail without any attempts shouldn't have start/finish/elapsed times. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5765) Update hadoop-pipes examples README
[ https://issues.apache.org/jira/browse/MAPREDUCE-5765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-5765: --- Resolution: Fixed Fix Version/s: 2.5.0 3.0.0 Status: Resolved (was: Patch Available) +1. Thanks for the fix, Mit. Update hadoop-pipes examples README --- Key: MAPREDUCE-5765 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5765 Project: Hadoop Map/Reduce Issue Type: Bug Components: pipes Affects Versions: 3.0.0, 2.5.0 Reporter: Jonathan Eagles Assignee: Mit Desai Priority: Minor Labels: documentation Fix For: 3.0.0, 2.5.0 Attachments: MAPREDUCE-5765.patch, MAPREDUCE-5765.patch wordcount-simple is in the native/examples directory -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5713) InputFormat and JobConf JavaDoc Fixes
[ https://issues.apache.org/jira/browse/MAPREDUCE-5713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-5713: --- Summary: InputFormat and JobConf JavaDoc Fixes (was: JavaDoc Fixes) InputFormat and JobConf JavaDoc Fixes - Key: MAPREDUCE-5713 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5713 Project: Hadoop Map/Reduce Issue Type: Bug Components: documentation Affects Versions: 1.2.1, 2.2.0 Reporter: Ben Robie Assignee: Chen He Priority: Trivial Attachments: MAPREDUCE-5713.patch, hadoop-10210.patch Original Estimate: 0.5h Remaining Estimate: 0.5h https://hadoop.apache.org/docs/r1.2.1/api/org/apache/hadoop/mapred/InputFormat.html Instead of record boundaries are to respected Should be record boundaries are to be respected https://hadoop.apache.org/docs/r1.2.1/api/org/apache/hadoop/mapred/JobConf.html Instead of some parameters interact subtly rest of the framework Should be some parameters interact subtly with the rest of the framework -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5713) InputFormat and JobConf JavaDoc Fixes
[ https://issues.apache.org/jira/browse/MAPREDUCE-5713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-5713: --- Resolution: Fixed Fix Version/s: 2.5.0 3.0.0 Target Version/s: (was: ) Status: Resolved (was: Patch Available) +1 lgtm. Thanks for the contributions, [~airbots] InputFormat and JobConf JavaDoc Fixes - Key: MAPREDUCE-5713 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5713 Project: Hadoop Map/Reduce Issue Type: Bug Components: documentation Affects Versions: 1.2.1, 2.2.0 Reporter: Ben Robie Assignee: Chen He Priority: Trivial Fix For: 3.0.0, 2.5.0 Attachments: MAPREDUCE-5713.patch, hadoop-10210.patch Original Estimate: 0.5h Remaining Estimate: 0.5h https://hadoop.apache.org/docs/r1.2.1/api/org/apache/hadoop/mapred/InputFormat.html Instead of record boundaries are to respected Should be record boundaries are to be respected https://hadoop.apache.org/docs/r1.2.1/api/org/apache/hadoop/mapred/JobConf.html Instead of some parameters interact subtly rest of the framework Should be some parameters interact subtly with the rest of the framework -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5456) TestFetcher.testCopyFromHostExtraBytes is missing
[ https://issues.apache.org/jira/browse/MAPREDUCE-5456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-5456: --- Resolution: Fixed Fix Version/s: 2.5.0 3.0.0 Status: Resolved (was: Patch Available) +1. lgtm. Thanks, [~jlowe]. I verified the test does fail to shuffle with the extra byte failure message. TestFetcher.testCopyFromHostExtraBytes is missing - Key: MAPREDUCE-5456 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5456 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 2.1.0-beta Reporter: Jason Lowe Assignee: Jason Lowe Priority: Minor Fix For: 3.0.0, 2.5.0 Attachments: MAPREDUCE-5456.patch, MAPREDUCE-5456.patch I noticed that the test to verify the fix from MAPREDUCE-5308 was deleted by MAPREDUCE-5194. It looks like an accidental deletion from an upmerge. We should reinstate this unit test. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (MAPREDUCE-5765) Update hadoop-pipes examples README
Jonathan Eagles created MAPREDUCE-5765: -- Summary: Update hadoop-pipes examples README Key: MAPREDUCE-5765 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5765 Project: Hadoop Map/Reduce Issue Type: Bug Components: pipes Affects Versions: 3.0.0, 2.5.0 Reporter: Jonathan Eagles wordcount-simple is in the native/examples directory -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (MAPREDUCE-5765) Update hadoop-pipes examples README
[ https://issues.apache.org/jira/browse/MAPREDUCE-5765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13910653#comment-13910653 ] Jonathan Eagles commented on MAPREDUCE-5765: hadoop-tools/hadoop-pipes/src/main/native/examples/README.txt is the file with the error Update hadoop-pipes examples README --- Key: MAPREDUCE-5765 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5765 Project: Hadoop Map/Reduce Issue Type: Bug Components: pipes Affects Versions: 3.0.0, 2.5.0 Reporter: Jonathan Eagles Assignee: Mit Desai Labels: documentation wordcount-simple is in the native/examples directory -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (MAPREDUCE-5765) Update hadoop-pipes examples README
[ https://issues.apache.org/jira/browse/MAPREDUCE-5765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-5765: --- Priority: Minor (was: Major) Update hadoop-pipes examples README --- Key: MAPREDUCE-5765 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5765 Project: Hadoop Map/Reduce Issue Type: Bug Components: pipes Affects Versions: 3.0.0, 2.5.0 Reporter: Jonathan Eagles Assignee: Mit Desai Priority: Minor Labels: documentation wordcount-simple is in the native/examples directory -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (MAPREDUCE-5671) NaN can be created by client and assign to Progress
[ https://issues.apache.org/jira/browse/MAPREDUCE-5671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-5671: --- Resolution: Fixed Fix Version/s: (was: 2.3.0) 2.5.0 3.0.0 Status: Resolved (was: Patch Available) +1. Thanks for the patch, Chen. NaN can be created by client and assign to Progress --- Key: MAPREDUCE-5671 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5671 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Chen He Assignee: Chen He Fix For: 3.0.0, 2.5.0 Attachments: MR-5671.patch MapReduce should filter illegal progress values that do not fall into (0,1) interval when the progress value is given. If it is Float.NaN, Float.NEGATIVE_INFINITY, or smaller than 0: set progress to be 0; If its is Float.POSITIVE_INFINITY or larger than 1: set progress to be 1; -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (MAPREDUCE-5688) TestStagingCleanup fails intermittently with JDK7
[
https://issues.apache.org/jira/browse/MAPREDUCE-5688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Eagles updated MAPREDUCE-5688:
---
Resolution: Fixed
Fix Version/s: 2.5.0
3.0.0
Status: Resolved (was: Patch Available)
TestStagingCleanup fails intermittently with JDK7
-
Key: MAPREDUCE-5688
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5688
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 3.0.0, 2.3.0
Reporter: Mit Desai
Assignee: Mit Desai
Labels: java7
Fix For: 3.0.0, 2.5.0
Attachments: MAPREDUCE-5688-v2.patch, MAPREDUCE-5688-v2.patch,
MAPREDUCE-5688.patch
Due to random ordering ordering in JDK7, the test
TestStagingCleanup#testDeletionofStagingOnKillLastTry is failing
{noformat}
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 4.231 sec
FAILURE!
test(org.apache.hadoop.mapreduce.v2.app.TestStagingCleanup) Time elapsed:
3882 sec ERROR!
java.lang.NullPointerException
at
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.serviceStop(JobHistoryEventHandler.java:349)
at
org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
at
org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
at
org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
at
org.apache.hadoop.service.CompositeService.stop(CompositeService.java:159)
at
org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:132)
at
org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1399)
at
org.apache.hadoop.mapreduce.v2.app.TestStagingCleanup.testDeletionofStagingOnKillLastTry(TestStagingCleanup.java:239)
at
org.apache.hadoop.mapreduce.v2.app.TestStagingCleanup.test(TestStagingCleanup.java:82)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at junit.framework.TestCase.runTest(TestCase.java:168)
at junit.framework.TestCase.runBare(TestCase.java:134)
at junit.framework.TestResult$1.protect(TestResult.java:110)
at junit.framework.TestResult.runProtected(TestResult.java:128)
at junit.framework.TestResult.run(TestResult.java:113)
at junit.framework.TestCase.run(TestCase.java:124)
at junit.framework.TestSuite.runTest(TestSuite.java:243)
at junit.framework.TestSuite.run(TestSuite.java:238)
at
org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:242)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:137)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (MAPREDUCE-5688) TestStagingCleanup fails intermittently with JDK7
[
https://issues.apache.org/jira/browse/MAPREDUCE-5688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13888045#comment-13888045
]
Jonathan Eagles commented on MAPREDUCE-5688:
Mit, thanks for the patch. Couple of minor nits.
* Re-title this jira since this is not a test problem according to the patch,
but a race condition in the MRAppMaster that is exposed most frequently via
this test.
* Add your analysis to the jira so that the actual problem is documented and
captured for future use.
* Please determine if the java7 label is still accurate based on your analysis
TestStagingCleanup fails intermittently with JDK7
-
Key: MAPREDUCE-5688
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5688
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 3.0.0, 2.3.0
Reporter: Mit Desai
Assignee: Mit Desai
Labels: java7
Attachments: MAPREDUCE-5688.patch
Due to random ordering ordering in JDK7, the test
TestStagingCleanup#testDeletionofStagingOnKillLastTry is failing
{noformat}
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 4.231 sec
FAILURE!
test(org.apache.hadoop.mapreduce.v2.app.TestStagingCleanup) Time elapsed:
3882 sec ERROR!
java.lang.NullPointerException
at
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler.serviceStop(JobHistoryEventHandler.java:349)
at
org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
at
org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
at
org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
at
org.apache.hadoop.service.CompositeService.stop(CompositeService.java:159)
at
org.apache.hadoop.service.CompositeService.serviceStop(CompositeService.java:132)
at
org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1399)
at
org.apache.hadoop.mapreduce.v2.app.TestStagingCleanup.testDeletionofStagingOnKillLastTry(TestStagingCleanup.java:239)
at
org.apache.hadoop.mapreduce.v2.app.TestStagingCleanup.test(TestStagingCleanup.java:82)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at junit.framework.TestCase.runTest(TestCase.java:168)
at junit.framework.TestCase.runBare(TestCase.java:134)
at junit.framework.TestResult$1.protect(TestResult.java:110)
at junit.framework.TestResult.runProtected(TestResult.java:128)
at junit.framework.TestResult.run(TestResult.java:113)
at junit.framework.TestCase.run(TestCase.java:124)
at junit.framework.TestSuite.runTest(TestSuite.java:243)
at junit.framework.TestSuite.run(TestSuite.java:238)
at
org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:242)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:137)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (MAPREDUCE-3310) Custom grouping comparator cannot be set for Combiners
[
https://issues.apache.org/jira/browse/MAPREDUCE-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13873905#comment-13873905
]
Jonathan Eagles commented on MAPREDUCE-3310:
It looks like the behavior before is to default combiner group comparator to
sort output key comparator. If I route Tez's getCombinerKeyGroupingComparator()
to getSortComparator(), would this have the same functionality as
pre-MAPREDUCE-3310. This would allow Tez to compile for both hadoop-2.2.0 and
hadoop-2.4.0.
Jon
Custom grouping comparator cannot be set for Combiners
--
Key: MAPREDUCE-3310
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3310
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: client
Affects Versions: 0.20.1
Environment: All
Reporter: Mathias Herberts
Assignee: Alejandro Abdelnur
Fix For: 1.3.0, 2.4.0
Attachments: MAPREDUCE-3310-branch-1.patch,
MAPREDUCE-3310-branch-1.patch, MAPREDUCE-3310-trunk.patch,
MAPREDUCE-3310-trunk.patch, MAPREDUCE-3310-trunk.patch,
MAPREDUCE-3310-trunk.patch
Combiners are often described as 'Reducers running on the Map side'.
As Reducers, Combiners are fed K,{V}, where {V} is built by grouping values
associated with the 'same' key.
For Reducers, the comparator used for grouping values can be set
independently of that used to sort the keys (using
Job.setGroupingComparatorClass).
Such a configuration is not possible for Combiners, meaning some things done
in Reducers cannot be done in Combiners (such as secondary sort).
It would be handy to have a Job.setCombinerGroupingComparatorClass method
that would allow the setting of the grouping comparator used when applying a
Combiner.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (MAPREDUCE-3310) Custom grouping comparator cannot be set for Combiners
[
https://issues.apache.org/jira/browse/MAPREDUCE-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13872737#comment-13872737
]
Jonathan Eagles commented on MAPREDUCE-3310:
I noticed TEZ-722 is filed as a result of this check-in. I don't know if we
need to reconsider this change as part of the 2.x line or not.
Custom grouping comparator cannot be set for Combiners
--
Key: MAPREDUCE-3310
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3310
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: client
Affects Versions: 0.20.1
Environment: All
Reporter: Mathias Herberts
Assignee: Alejandro Abdelnur
Fix For: 1.3.0, 2.4.0
Attachments: MAPREDUCE-3310-branch-1.patch,
MAPREDUCE-3310-branch-1.patch, MAPREDUCE-3310-trunk.patch,
MAPREDUCE-3310-trunk.patch, MAPREDUCE-3310-trunk.patch,
MAPREDUCE-3310-trunk.patch
Combiners are often described as 'Reducers running on the Map side'.
As Reducers, Combiners are fed K,{V}, where {V} is built by grouping values
associated with the 'same' key.
For Reducers, the comparator used for grouping values can be set
independently of that used to sort the keys (using
Job.setGroupingComparatorClass).
Such a configuration is not possible for Combiners, meaning some things done
in Reducers cannot be done in Combiners (such as secondary sort).
It would be handy to have a Job.setCombinerGroupingComparatorClass method
that would allow the setting of the grouping comparator used when applying a
Combiner.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (MAPREDUCE-5681) TestJHSSecurity fails on trunk
[
https://issues.apache.org/jira/browse/MAPREDUCE-5681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13848077#comment-13848077
]
Jonathan Eagles commented on MAPREDUCE-5681:
Can you confirm this isn't the same as YARN-1463?
TestJHSSecurity fails on trunk
--
Key: MAPREDUCE-5681
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5681
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Zhijie Shen
{code}
---
T E S T S
---
Running org.apache.hadoop.mapreduce.security.TestJHSSecurity
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.763 sec
FAILURE! - in org.apache.hadoop.mapreduce.security.TestJHSSecurity
testDelegationToken(org.apache.hadoop.mapreduce.security.TestJHSSecurity)
Time elapsed: 1.56 sec ERROR!
java.lang.NullPointerException: null
at java.util.Hashtable.get(Hashtable.java:334)
at java.util.Properties.getProperty(Properties.java:932)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:874)
at org.apache.hadoop.http.HttpServer.initSpnego(HttpServer.java:892)
at org.apache.hadoop.http.HttpServer.access$100(HttpServer.java:101)
at org.apache.hadoop.http.HttpServer$Builder.build(HttpServer.java:323)
at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:232)
at
org.apache.hadoop.mapreduce.v2.hs.HistoryClientService.initializeWebApp(HistoryClientService.java:149)
at
org.apache.hadoop.mapreduce.v2.hs.HistoryClientService.serviceStart(HistoryClientService.java:118)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)
at
org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.serviceStart(JobHistoryServer.java:175)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.mapreduce.security.TestJHSSecurity.testDelegationToken(TestJHSSecurity.java:100)
Results :
Tests in error:
TestJHSSecurity.testDelegationToken:100 ? NullPointer
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0
{code}
Did some preliminary investigation, in HistoryClientService:
{code}
.withHttpSpnegoPrincipalKey(
JHAdminConfig.MR_WEBAPP_SPNEGO_USER_NAME_KEY)
{code}
MR_WEBAPP_SPNEGO_USER_NAME_KEY seems not to be in the configuration.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (MAPREDUCE-5052) Job History UI and web services confusing job start time and job submit time
[ https://issues.apache.org/jira/browse/MAPREDUCE-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-5052: --- Resolution: Fixed Fix Version/s: 3.0.0 Target Version/s: 3.0.0, 2.4.0 (was: 2.4.0) Status: Resolved (was: Patch Available) +1. Thanks for this patch, Chen. Job History UI and web services confusing job start time and job submit time Key: MAPREDUCE-5052 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5052 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, webapps Affects Versions: 0.23.6 Reporter: Kendall Thrapp Assignee: Chen He Priority: Critical Fix For: 3.0.0, 2.4.0 Attachments: MAPREDUCE-5052.patch The Start Time column shown on Job History server's main webpage (http://host:port/jobhistory) is actually showing the *submit* time for jobs. However, when you drill down to an individual job's page, there the Start Time really does refer to when the job actually started. This also true for the web services REST API, where the Jobs listing returns the submit times as startTime, but the single Job API returns the start time as startTime. The two different times being referred to by the same name is confusing. However, it is useful to have both times, as the difference between the submit time and start time can show how long a job was stuck waiting in a queue. The column on the main job history page should be changed to Submit Time and the individual job's page should show both the submit time and start time. The web services REST API should be updated with these changes as well. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (MAPREDUCE-5427) TestRMContainerAllocator.testUpdatedNodes fails on jdk7
[
https://issues.apache.org/jira/browse/MAPREDUCE-5427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13839404#comment-13839404
]
Jonathan Eagles commented on MAPREDUCE-5427:
Now that MAPREDUCE-5632 is in, the tests pass in jdk7. It looks like this jira
is more that just fixing this test for jdk7, though. Can this jira either be
retitled or duplicated to MAPREDUCE-5632?
TestRMContainerAllocator.testUpdatedNodes fails on jdk7
---
Key: MAPREDUCE-5427
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5427
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 2.1.0-beta, 2.0.5-alpha
Reporter: Nemon Lou
Assignee: Sandy Ryza
Labels: java7
Attachments: MAPREDUCE-5427.patch, MAPREDUCE-5427.patch
{code}
---
Test set: org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator
---
Tests run: 14, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 42.777 sec
FAILURE!
testUpdatedNodes(org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator)
Time elapsed: 0.14 sec FAILURE!
junit.framework.AssertionFailedError: null
at junit.framework.Assert.fail(Assert.java:47)
at junit.framework.Assert.assertTrue(Assert.java:20)
at junit.framework.Assert.assertTrue(Assert.java:27)
at
org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator.testUpdatedNodes(TestRMContainerAllocator.java:747)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
at
org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)
{code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (MAPREDUCE-5632) TestRMContainerAllocator#testUpdatedNodes fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-5632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Eagles updated MAPREDUCE-5632:
---
Resolution: Fixed
Fix Version/s: 2.4.0
3.0.0
Status: Resolved (was: Patch Available)
Thanks everybody for helping to get this in.
TestRMContainerAllocator#testUpdatedNodes fails
---
Key: MAPREDUCE-5632
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5632
Project: Hadoop Map/Reduce
Issue Type: Test
Reporter: Ted Yu
Assignee: Jonathan Eagles
Fix For: 3.0.0, 2.4.0
Attachments: YARN-1420.patch
From https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1607/console :
{code}
Running org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator
Tests run: 14, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 65.78 sec
FAILURE! - in org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator
testUpdatedNodes(org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator)
Time elapsed: 3.125 sec FAILURE!
junit.framework.AssertionFailedError: null
at junit.framework.Assert.fail(Assert.java:48)
at junit.framework.Assert.assertTrue(Assert.java:20)
at junit.framework.Assert.assertTrue(Assert.java:27)
at
org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator.testUpdatedNodes(TestRMContainerAllocator.java:779)
{code}
This assertion fails:
{code}
Assert.assertTrue(allocator.getJobUpdatedNodeEvents().isEmpty());
{code}
The List returned by allocator.getJobUpdatedNodeEvents() is:
[EventType: JOB_UPDATED_NODES]
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (MAPREDUCE-5632) TestRMContainerAllocator#testUpdatedNodes fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-5632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13837740#comment-13837740
]
Jonathan Eagles commented on MAPREDUCE-5632:
[~tucu00] or [~bikassaha] could you do a double check to make sure the tests
are correct behavior and not just matching the current incorrect behavior.
TestRMContainerAllocator#testUpdatedNodes fails
---
Key: MAPREDUCE-5632
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5632
Project: Hadoop Map/Reduce
Issue Type: Test
Reporter: Ted Yu
Assignee: Jonathan Eagles
Attachments: YARN-1420.patch
From https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1607/console :
{code}
Running org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator
Tests run: 14, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 65.78 sec
FAILURE! - in org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator
testUpdatedNodes(org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator)
Time elapsed: 3.125 sec FAILURE!
junit.framework.AssertionFailedError: null
at junit.framework.Assert.fail(Assert.java:48)
at junit.framework.Assert.assertTrue(Assert.java:20)
at junit.framework.Assert.assertTrue(Assert.java:27)
at
org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator.testUpdatedNodes(TestRMContainerAllocator.java:779)
{code}
This assertion fails:
{code}
Assert.assertTrue(allocator.getJobUpdatedNodeEvents().isEmpty());
{code}
The List returned by allocator.getJobUpdatedNodeEvents() is:
[EventType: JOB_UPDATED_NODES]
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (MAPREDUCE-5664) java.lang.RuntimeException: javax.xml.parsers.ParserConfigurationException:
[ https://issues.apache.org/jira/browse/MAPREDUCE-5664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13837754#comment-13837754 ] Jonathan Eagles commented on MAPREDUCE-5664: Could you please provide sample xml files for processing as well as job configuration and relevant part of the MapReduce class so this could be reproduced? Also, please set the hadoop version in the Affects Version of the JIRA. This will greatly help someone reproduce this issue quickly. java.lang.RuntimeException: javax.xml.parsers.ParserConfigurationException: --- Key: MAPREDUCE-5664 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5664 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: ranjini Hi, When processing xml file as input via map reduce, the error occurred is conf.Configuration: error parsing conf file: javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized. Exception in thread main java.lang.RuntimeException: javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized. at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1171) at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1030) at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:980) at org.apache.hadoop.conf.Configuration.get(Configuration.java:382) at org.apache.hadoop.util.RunJar.main(RunJar.java:109) Caused by: javax.xml.parsers.ParserConfigurationException: Feature 'http://apache.org/xml/features/xinclude' is not recognized. at org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown Source) at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1061) ... 4 more Please help to fix the issue -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5645) TestFixedLengthInputFormat fails with native libs
[ https://issues.apache.org/jira/browse/MAPREDUCE-5645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13838273#comment-13838273 ] Jonathan Eagles commented on MAPREDUCE-5645: +1. Looks good to me. TestFixedLengthInputFormat fails with native libs - Key: MAPREDUCE-5645 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5645 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 3.0.0, 2.4.0 Reporter: Jonathan Eagles Assignee: Mit Desai Labels: native Attachments: MAPREDUCE-5645.patch mvn clean install -Pnative -DskipTests hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient mvn clean test -Dtest=TestFixedLengthInputFormat Running org.apache.hadoop.mapreduce.lib.input.TestFixedLengthInputFormat Tests run: 8, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 39.957 sec FAILURE! - in org.apache.hadoop.mapreduce.lib.input.TestFixedLengthInputFormat testGzipWithTwoInputs(org.apache.hadoop.mapreduce.lib.input.TestFixedLengthInputFormat) Time elapsed: 0.029 sec ERROR! java.lang.NullPointerException: null at org.apache.hadoop.io.compress.zlib.ZlibFactory.isNativeZlibLoaded(ZlibFactory.java:65) at org.apache.hadoop.io.compress.GzipCodec.createOutputStream(GzipCodec.java:162) at org.apache.hadoop.mapreduce.lib.input.TestFixedLengthInputFormat.writeFile(TestFixedLengthInputFormat.java:397) at org.apache.hadoop.mapreduce.lib.input.TestFixedLengthInputFormat.testGzipWithTwoInputs(TestFixedLengthInputFormat.java:229) testFormatCompressedIn(org.apache.hadoop.mapreduce.lib.input.TestFixedLengthInputFormat) Time elapsed: 0.01 sec ERROR! java.lang.NullPointerException: null at org.apache.hadoop.io.compress.zlib.ZlibFactory.isNativeZlibLoaded(ZlibFactory.java:65) at org.apache.hadoop.io.compress.GzipCodec.createOutputStream(GzipCodec.java:162) at org.apache.hadoop.mapreduce.lib.input.TestFixedLengthInputFormat.createFile(TestFixedLengthInputFormat.java:261) at org.apache.hadoop.mapreduce.lib.input.TestFixedLengthInputFormat.runRandomTests(TestFixedLengthInputFormat.java:314) at org.apache.hadoop.mapreduce.lib.input.TestFixedLengthInputFormat.testFormatCompressedIn(TestFixedLengthInputFormat.java:96) Running org.apache.hadoop.mapred.TestFixedLengthInputFormat Tests run: 8, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 46.151 sec FAILURE! - in org.apache.hadoop.mapred.TestFixedLengthInputFormat testPartialRecordCompressedIn(org.apache.hadoop.mapred.TestFixedLengthInputFormat) Time elapsed: 0.031 sec ERROR! java.lang.NullPointerException: null at org.apache.hadoop.io.compress.zlib.ZlibFactory.isNativeZlibLoaded(ZlibFactory.java:65) at org.apache.hadoop.io.compress.GzipCodec.createOutputStream(GzipCodec.java:162) at org.apache.hadoop.mapred.TestFixedLengthInputFormat.writeFile(TestFixedLengthInputFormat.java:357) at org.apache.hadoop.mapred.TestFixedLengthInputFormat.runPartialRecordTest(TestFixedLengthInputFormat.java:386) at org.apache.hadoop.mapred.TestFixedLengthInputFormat.testPartialRecordCompressedIn(TestFixedLengthInputFormat.java:182) testGzipWithTwoInputs(org.apache.hadoop.mapred.TestFixedLengthInputFormat) Time elapsed: 0.009 sec ERROR! java.lang.NullPointerException: null at org.apache.hadoop.io.compress.zlib.ZlibFactory.isNativeZlibLoaded(ZlibFactory.java:65) at org.apache.hadoop.io.compress.GzipCodec.createOutputStream(GzipCodec.java:162) at org.apache.hadoop.mapred.TestFixedLengthInputFormat.writeFile(TestFixedLengthInputFormat.java:357) at org.apache.hadoop.mapred.TestFixedLengthInputFormat.testGzipWithTwoInputs(TestFixedLengthInputFormat.java:201) testFormatCompressedIn(org.apache.hadoop.mapred.TestFixedLengthInputFormat) Time elapsed: 0.017 sec ERROR! java.lang.NullPointerException: null at org.apache.hadoop.io.compress.zlib.ZlibFactory.isNativeZlibLoaded(ZlibFactory.java:65) at org.apache.hadoop.io.compress.GzipCodec.createOutputStream(GzipCodec.java:162) at org.apache.hadoop.mapred.TestFixedLengthInputFormat.createFile(TestFixedLengthInputFormat.java:234) at org.apache.hadoop.mapred.TestFixedLengthInputFormat.runRandomTests(TestFixedLengthInputFormat.java:287) at org.apache.hadoop.mapred.TestFixedLengthInputFormat.testFormatCompressedIn(TestFixedLengthInputFormat.java:90) Results : Tests in error: TestFixedLengthInputFormat.testGzipWithTwoInputs:229-writeFile:397 » NullPointer TestFixedLengthInputFormat.testFormatCompressedIn:96-runRandomTests:314-createFile:261 » NullPointer
[jira] [Updated] (MAPREDUCE-5645) TestFixedLengthInputFormat fails with native libs
[ https://issues.apache.org/jira/browse/MAPREDUCE-5645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-5645: --- Resolution: Fixed Fix Version/s: 2.4.0 3.0.0 Status: Resolved (was: Patch Available) TestFixedLengthInputFormat fails with native libs - Key: MAPREDUCE-5645 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5645 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 3.0.0, 2.4.0 Reporter: Jonathan Eagles Assignee: Mit Desai Labels: native Fix For: 3.0.0, 2.4.0 Attachments: MAPREDUCE-5645.patch mvn clean install -Pnative -DskipTests hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient mvn clean test -Dtest=TestFixedLengthInputFormat Running org.apache.hadoop.mapreduce.lib.input.TestFixedLengthInputFormat Tests run: 8, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 39.957 sec FAILURE! - in org.apache.hadoop.mapreduce.lib.input.TestFixedLengthInputFormat testGzipWithTwoInputs(org.apache.hadoop.mapreduce.lib.input.TestFixedLengthInputFormat) Time elapsed: 0.029 sec ERROR! java.lang.NullPointerException: null at org.apache.hadoop.io.compress.zlib.ZlibFactory.isNativeZlibLoaded(ZlibFactory.java:65) at org.apache.hadoop.io.compress.GzipCodec.createOutputStream(GzipCodec.java:162) at org.apache.hadoop.mapreduce.lib.input.TestFixedLengthInputFormat.writeFile(TestFixedLengthInputFormat.java:397) at org.apache.hadoop.mapreduce.lib.input.TestFixedLengthInputFormat.testGzipWithTwoInputs(TestFixedLengthInputFormat.java:229) testFormatCompressedIn(org.apache.hadoop.mapreduce.lib.input.TestFixedLengthInputFormat) Time elapsed: 0.01 sec ERROR! java.lang.NullPointerException: null at org.apache.hadoop.io.compress.zlib.ZlibFactory.isNativeZlibLoaded(ZlibFactory.java:65) at org.apache.hadoop.io.compress.GzipCodec.createOutputStream(GzipCodec.java:162) at org.apache.hadoop.mapreduce.lib.input.TestFixedLengthInputFormat.createFile(TestFixedLengthInputFormat.java:261) at org.apache.hadoop.mapreduce.lib.input.TestFixedLengthInputFormat.runRandomTests(TestFixedLengthInputFormat.java:314) at org.apache.hadoop.mapreduce.lib.input.TestFixedLengthInputFormat.testFormatCompressedIn(TestFixedLengthInputFormat.java:96) Running org.apache.hadoop.mapred.TestFixedLengthInputFormat Tests run: 8, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 46.151 sec FAILURE! - in org.apache.hadoop.mapred.TestFixedLengthInputFormat testPartialRecordCompressedIn(org.apache.hadoop.mapred.TestFixedLengthInputFormat) Time elapsed: 0.031 sec ERROR! java.lang.NullPointerException: null at org.apache.hadoop.io.compress.zlib.ZlibFactory.isNativeZlibLoaded(ZlibFactory.java:65) at org.apache.hadoop.io.compress.GzipCodec.createOutputStream(GzipCodec.java:162) at org.apache.hadoop.mapred.TestFixedLengthInputFormat.writeFile(TestFixedLengthInputFormat.java:357) at org.apache.hadoop.mapred.TestFixedLengthInputFormat.runPartialRecordTest(TestFixedLengthInputFormat.java:386) at org.apache.hadoop.mapred.TestFixedLengthInputFormat.testPartialRecordCompressedIn(TestFixedLengthInputFormat.java:182) testGzipWithTwoInputs(org.apache.hadoop.mapred.TestFixedLengthInputFormat) Time elapsed: 0.009 sec ERROR! java.lang.NullPointerException: null at org.apache.hadoop.io.compress.zlib.ZlibFactory.isNativeZlibLoaded(ZlibFactory.java:65) at org.apache.hadoop.io.compress.GzipCodec.createOutputStream(GzipCodec.java:162) at org.apache.hadoop.mapred.TestFixedLengthInputFormat.writeFile(TestFixedLengthInputFormat.java:357) at org.apache.hadoop.mapred.TestFixedLengthInputFormat.testGzipWithTwoInputs(TestFixedLengthInputFormat.java:201) testFormatCompressedIn(org.apache.hadoop.mapred.TestFixedLengthInputFormat) Time elapsed: 0.017 sec ERROR! java.lang.NullPointerException: null at org.apache.hadoop.io.compress.zlib.ZlibFactory.isNativeZlibLoaded(ZlibFactory.java:65) at org.apache.hadoop.io.compress.GzipCodec.createOutputStream(GzipCodec.java:162) at org.apache.hadoop.mapred.TestFixedLengthInputFormat.createFile(TestFixedLengthInputFormat.java:234) at org.apache.hadoop.mapred.TestFixedLengthInputFormat.runRandomTests(TestFixedLengthInputFormat.java:287) at org.apache.hadoop.mapred.TestFixedLengthInputFormat.testFormatCompressedIn(TestFixedLengthInputFormat.java:90) Results : Tests in error: TestFixedLengthInputFormat.testGzipWithTwoInputs:229-writeFile:397 » NullPointer TestFixedLengthInputFormat.testFormatCompressedIn:96-runRandomTests:314-createFile:261 »
[jira] [Commented] (MAPREDUCE-5640) Rename TestLineRecordReader in jobclient module
[ https://issues.apache.org/jira/browse/MAPREDUCE-5640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13836800#comment-13836800 ] Jonathan Eagles commented on MAPREDUCE-5640: +1. Changes look good, Jason. Checking this in shortly. Rename TestLineRecordReader in jobclient module --- Key: MAPREDUCE-5640 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5640 Project: Hadoop Map/Reduce Issue Type: Improvement Components: test Affects Versions: 0.23.9, 2.2.0 Reporter: Jason Lowe Assignee: Jason Lowe Priority: Trivial Attachments: MAPREDUCE-5640.patch HADOOP-9622 proposes to add new unit tests for LineRecordReader in the mapreduce-client-core module alongside the code. The existing LineRecordReader tests in the mapreduce-client-jobclient module should be renamed to something like TestLineRecordReaderJobs to avoid a name conflict and to better indicate these are integration tests using full jobs rather than unit tests. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5640) Rename TestLineRecordReader in jobclient module
[ https://issues.apache.org/jira/browse/MAPREDUCE-5640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-5640: --- Resolution: Fixed Fix Version/s: 0.23.10 2.4.0 3.0.0 Status: Resolved (was: Patch Available) Rename TestLineRecordReader in jobclient module --- Key: MAPREDUCE-5640 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5640 Project: Hadoop Map/Reduce Issue Type: Improvement Components: test Affects Versions: 0.23.9, 2.2.0 Reporter: Jason Lowe Assignee: Jason Lowe Priority: Trivial Fix For: 3.0.0, 2.4.0, 0.23.10 Attachments: MAPREDUCE-5640.patch HADOOP-9622 proposes to add new unit tests for LineRecordReader in the mapreduce-client-core module alongside the code. The existing LineRecordReader tests in the mapreduce-client-jobclient module should be renamed to something like TestLineRecordReaderJobs to avoid a name conflict and to better indicate these are integration tests using full jobs rather than unit tests. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (MAPREDUCE-5645) TestFixedLengthInputFormat fails with native libs
Jonathan Eagles created MAPREDUCE-5645: -- Summary: TestFixedLengthInputFormat fails with native libs Key: MAPREDUCE-5645 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5645 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 3.0.0, 2.3.0 Reporter: Jonathan Eagles Assignee: Mit Desai mvn clean install -Pnative -DskipTests hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient mvn clean test -Dtest=TestFixedLengthInputFormat Running org.apache.hadoop.mapreduce.lib.input.TestFixedLengthInputFormat Tests run: 8, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 39.957 sec FAILURE! - in org.apache.hadoop.mapreduce.lib.input.TestFixedLengthInputFormat testGzipWithTwoInputs(org.apache.hadoop.mapreduce.lib.input.TestFixedLengthInputFormat) Time elapsed: 0.029 sec ERROR! java.lang.NullPointerException: null at org.apache.hadoop.io.compress.zlib.ZlibFactory.isNativeZlibLoaded(ZlibFactory.java:65) at org.apache.hadoop.io.compress.GzipCodec.createOutputStream(GzipCodec.java:162) at org.apache.hadoop.mapreduce.lib.input.TestFixedLengthInputFormat.writeFile(TestFixedLengthInputFormat.java:397) at org.apache.hadoop.mapreduce.lib.input.TestFixedLengthInputFormat.testGzipWithTwoInputs(TestFixedLengthInputFormat.java:229) testFormatCompressedIn(org.apache.hadoop.mapreduce.lib.input.TestFixedLengthInputFormat) Time elapsed: 0.01 sec ERROR! java.lang.NullPointerException: null at org.apache.hadoop.io.compress.zlib.ZlibFactory.isNativeZlibLoaded(ZlibFactory.java:65) at org.apache.hadoop.io.compress.GzipCodec.createOutputStream(GzipCodec.java:162) at org.apache.hadoop.mapreduce.lib.input.TestFixedLengthInputFormat.createFile(TestFixedLengthInputFormat.java:261) at org.apache.hadoop.mapreduce.lib.input.TestFixedLengthInputFormat.runRandomTests(TestFixedLengthInputFormat.java:314) at org.apache.hadoop.mapreduce.lib.input.TestFixedLengthInputFormat.testFormatCompressedIn(TestFixedLengthInputFormat.java:96) Running org.apache.hadoop.mapred.TestFixedLengthInputFormat Tests run: 8, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 46.151 sec FAILURE! - in org.apache.hadoop.mapred.TestFixedLengthInputFormat testPartialRecordCompressedIn(org.apache.hadoop.mapred.TestFixedLengthInputFormat) Time elapsed: 0.031 sec ERROR! java.lang.NullPointerException: null at org.apache.hadoop.io.compress.zlib.ZlibFactory.isNativeZlibLoaded(ZlibFactory.java:65) at org.apache.hadoop.io.compress.GzipCodec.createOutputStream(GzipCodec.java:162) at org.apache.hadoop.mapred.TestFixedLengthInputFormat.writeFile(TestFixedLengthInputFormat.java:357) at org.apache.hadoop.mapred.TestFixedLengthInputFormat.runPartialRecordTest(TestFixedLengthInputFormat.java:386) at org.apache.hadoop.mapred.TestFixedLengthInputFormat.testPartialRecordCompressedIn(TestFixedLengthInputFormat.java:182) testGzipWithTwoInputs(org.apache.hadoop.mapred.TestFixedLengthInputFormat) Time elapsed: 0.009 sec ERROR! java.lang.NullPointerException: null at org.apache.hadoop.io.compress.zlib.ZlibFactory.isNativeZlibLoaded(ZlibFactory.java:65) at org.apache.hadoop.io.compress.GzipCodec.createOutputStream(GzipCodec.java:162) at org.apache.hadoop.mapred.TestFixedLengthInputFormat.writeFile(TestFixedLengthInputFormat.java:357) at org.apache.hadoop.mapred.TestFixedLengthInputFormat.testGzipWithTwoInputs(TestFixedLengthInputFormat.java:201) testFormatCompressedIn(org.apache.hadoop.mapred.TestFixedLengthInputFormat) Time elapsed: 0.017 sec ERROR! java.lang.NullPointerException: null at org.apache.hadoop.io.compress.zlib.ZlibFactory.isNativeZlibLoaded(ZlibFactory.java:65) at org.apache.hadoop.io.compress.GzipCodec.createOutputStream(GzipCodec.java:162) at org.apache.hadoop.mapred.TestFixedLengthInputFormat.createFile(TestFixedLengthInputFormat.java:234) at org.apache.hadoop.mapred.TestFixedLengthInputFormat.runRandomTests(TestFixedLengthInputFormat.java:287) at org.apache.hadoop.mapred.TestFixedLengthInputFormat.testFormatCompressedIn(TestFixedLengthInputFormat.java:90) Results : Tests in error: TestFixedLengthInputFormat.testGzipWithTwoInputs:229-writeFile:397 » NullPointer TestFixedLengthInputFormat.testFormatCompressedIn:96-runRandomTests:314-createFile:261 » NullPointer TestFixedLengthInputFormat.testPartialRecordCompressedIn:182-runPartialRecordTest:386-writeFile:357 » NullPointer TestFixedLengthInputFormat.testGzipWithTwoInputs:201-writeFile:357 » NullPointer TestFixedLengthInputFormat.testFormatCompressedIn:90-runRandomTests:287-createFile:234 » NullPointer Tests run: 16, Failures: 0, Errors: 5, Skipped: 0 -- This message was sent by Atlassian JIRA
[jira] [Moved] (MAPREDUCE-5632) TestRMContainerAllocator#testUpdatedNodes fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-5632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Eagles moved YARN-1420 to MAPREDUCE-5632:
--
Key: MAPREDUCE-5632 (was: YARN-1420)
Project: Hadoop Map/Reduce (was: Hadoop YARN)
TestRMContainerAllocator#testUpdatedNodes fails
---
Key: MAPREDUCE-5632
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5632
Project: Hadoop Map/Reduce
Issue Type: Test
Reporter: Ted Yu
Assignee: Jonathan Eagles
Attachments: YARN-1420.patch
From https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1607/console :
{code}
Running org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator
Tests run: 14, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 65.78 sec
FAILURE! - in org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator
testUpdatedNodes(org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator)
Time elapsed: 3.125 sec FAILURE!
junit.framework.AssertionFailedError: null
at junit.framework.Assert.fail(Assert.java:48)
at junit.framework.Assert.assertTrue(Assert.java:20)
at junit.framework.Assert.assertTrue(Assert.java:27)
at
org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator.testUpdatedNodes(TestRMContainerAllocator.java:779)
{code}
This assertion fails:
{code}
Assert.assertTrue(allocator.getJobUpdatedNodeEvents().isEmpty());
{code}
The List returned by allocator.getJobUpdatedNodeEvents() is:
[EventType: JOB_UPDATED_NODES]
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (MAPREDUCE-5631) TestJobEndNotifier.testNotifyRetries fails with Should have taken more than 5 seconds in jdk7
[ https://issues.apache.org/jira/browse/MAPREDUCE-5631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13827850#comment-13827850 ] Jonathan Eagles commented on MAPREDUCE-5631: Test failures not related to this patch org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator is handled by MAPREDUCE-5632 TestJobEndNotifier.testNotifyRetries fails with Should have taken more than 5 seconds in jdk7 - Key: MAPREDUCE-5631 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5631 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 3.0.0, 2.3.0, 0.23.10 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Labels: java7 Attachments: MAPREDUCE-5631.patch, MAPREDUCE-5631.patch Configuration settings are bleeding over from test to test in jdk7 environment since tests are run in random order. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5052) Job History UI and web services confusing job start time and job submit time
[ https://issues.apache.org/jira/browse/MAPREDUCE-5052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13828157#comment-13828157 ] Jonathan Eagles commented on MAPREDUCE-5052: Hi Chen. Thanks for putting up a patch for this item. Will be very useful to many. Couple of minor things based on my first pass of reading. Will add more details as I dive deeper into this. - Please update the documentation for the REST api to reflect code changes. Quick scan shows possible locations of updates -- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/HistoryServerRest.apt.vm -- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/MapredAppMasterRest.apt.vm -- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ResourceManagerRest.apt.vm -- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/WebServicesIntro.apt.vm - Still feel there is confusion about using the term launch time vs start time. Seems there is already a precedence in RM and AM for the term start time. - Assert description mismatch in _Assert.assertEquals(Queue name different after encoding and decoding, info.getLaunchTime(), parsedInfo.getLaunchTime());_ Job History UI and web services confusing job start time and job submit time Key: MAPREDUCE-5052 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5052 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, webapps Affects Versions: 0.23.6 Reporter: Kendall Thrapp Assignee: Chen He Priority: Critical Attachments: MAPREDUCE-5052-updated.patch The Start Time column shown on Job History server's main webpage (http://host:port/jobhistory) is actually showing the *submit* time for jobs. However, when you drill down to an individual job's page, there the Start Time really does refer to when the job actually started. This also true for the web services REST API, where the Jobs listing returns the submit times as startTime, but the single Job API returns the start time as startTime. The two different times being referred to by the same name is confusing. However, it is useful to have both times, as the difference between the submit time and start time can show how long a job was stuck waiting in a queue. The column on the main job history page should be changed to Submit Time and the individual job's page should show both the submit time and start time. The web services REST API should be updated with these changes as well. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5631) TestJobEndNotifier.testNotifyRetries fails with Should have taken more than 5 seconds in jdk7
[
https://issues.apache.org/jira/browse/MAPREDUCE-5631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13826960#comment-13826960
]
Jonathan Eagles commented on MAPREDUCE-5631:
Good thought, Jason. After further investigation, the failing test is using a
Configuration object. If it runs first, it passes since none of the
{mapred,yarn}-{default,site}.xml and assumes the source code default retry
interval of 5 seconds. If it runs after any of the test that run a job, it
fails since the first use of JobConf will add the default and site files to the
default resources of the configuration class. The assumption of falling back to
the source code default of 5 seconds is no longer valid, since it is picking of
the mapred-default.xml value of 1 second.
I will change this test class to use the jobconf object to avoid any test order
dependencies and add the config settings explicit to avoid changes to the
default.xml causing the test to fail again.
TestJobEndNotifier.testNotifyRetries fails with Should have taken more than 5
seconds in jdk7
-
Key: MAPREDUCE-5631
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5631
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 3.0.0, 2.3.0, 0.23.10
Reporter: Jonathan Eagles
Assignee: Jonathan Eagles
Labels: java7
Attachments: MAPREDUCE-5631.patch
Configuration settings are bleeding over from test to test in jdk7
environment since tests are run in random order.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (MAPREDUCE-5631) TestJobEndNotifier.testNotifyRetries fails with Should have taken more than 5 seconds in jdk7
[ https://issues.apache.org/jira/browse/MAPREDUCE-5631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-5631: --- Attachment: MAPREDUCE-5631.patch TestJobEndNotifier.testNotifyRetries fails with Should have taken more than 5 seconds in jdk7 - Key: MAPREDUCE-5631 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5631 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 3.0.0, 2.3.0, 0.23.10 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Labels: java7 Attachments: MAPREDUCE-5631.patch, MAPREDUCE-5631.patch Configuration settings are bleeding over from test to test in jdk7 environment since tests are run in random order. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5625) TestFixedLengthInputFormat fails in jdk7 environment
[ https://issues.apache.org/jira/browse/MAPREDUCE-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825584#comment-13825584 ] Jonathan Eagles commented on MAPREDUCE-5625: In jdk7, I believe the way that junit does reflection on a class to get the test methods now returns the tests in random order (not file order. changed in jdk7). In jdk6, the tests were always return in file order(top to bottom.) TestFixedLengthInputFormat fails in jdk7 environment Key: MAPREDUCE-5625 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5625 Project: Hadoop Map/Reduce Issue Type: Test Affects Versions: 3.0.0, 2.3.0 Reporter: Jonathan Eagles Assignee: Mariappan Asokan Labels: java7 Fix For: 3.0.0, 2.3.0 Attachments: mapreduce-5625_v1.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5631) TestJobEndNotifier.testNotifyRetries fails with Should have taken more than 5 seconds in jdk7
[ https://issues.apache.org/jira/browse/MAPREDUCE-5631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-5631: --- Attachment: MAPREDUCE-5631.patch TestJobEndNotifier.testNotifyRetries fails with Should have taken more than 5 seconds in jdk7 - Key: MAPREDUCE-5631 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5631 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 3.0.0, 2.3.0, 0.23.10 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Labels: java7 Attachments: MAPREDUCE-5631.patch Configuration settings are bleeding over from test to test in jdk7 environment since tests are run in random order. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5631) TestJobEndNotifier.testNotifyRetries fails with Should have taken more than 5 seconds in jdk7
[ https://issues.apache.org/jira/browse/MAPREDUCE-5631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-5631: --- Status: Patch Available (was: Open) TestJobEndNotifier.testNotifyRetries fails with Should have taken more than 5 seconds in jdk7 - Key: MAPREDUCE-5631 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5631 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 3.0.0, 2.3.0, 0.23.10 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Labels: java7 Attachments: MAPREDUCE-5631.patch Configuration settings are bleeding over from test to test in jdk7 environment since tests are run in random order. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5631) TestJobEndNotifier.testNotifyRetries fails with Should have taken more than 5 seconds in jdk7
[ https://issues.apache.org/jira/browse/MAPREDUCE-5631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13825770#comment-13825770 ] Jonathan Eagles commented on MAPREDUCE-5631: TestRMContainerAllocator fails due to MAPREDUCE-5427 TestJobEndNotifier.testNotifyRetries fails with Should have taken more than 5 seconds in jdk7 - Key: MAPREDUCE-5631 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5631 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 3.0.0, 2.3.0, 0.23.10 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Labels: java7 Attachments: MAPREDUCE-5631.patch Configuration settings are bleeding over from test to test in jdk7 environment since tests are run in random order. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5625) TestFixedLengthInputFormat fails in jdk7 environment
[ https://issues.apache.org/jira/browse/MAPREDUCE-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13824858#comment-13824858 ] Jonathan Eagles commented on MAPREDUCE-5625: +1. lgtm. Thanks for fixing this so quickly, [~masokan]. TestFixedLengthInputFormat fails in jdk7 environment Key: MAPREDUCE-5625 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5625 Project: Hadoop Map/Reduce Issue Type: Test Affects Versions: 3.0.0, 2.3.0 Reporter: Jonathan Eagles Assignee: Mariappan Asokan Labels: java7 Attachments: mapreduce-5625_v1.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5625) TestFixedLengthInputFormat fails in jdk7 environment
[ https://issues.apache.org/jira/browse/MAPREDUCE-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-5625: --- Resolution: Fixed Fix Version/s: 2.3.0 3.0.0 Status: Resolved (was: Patch Available) I verified this fix works successfully in my jdk7 environment. Until apache builds start running with jdk7, I feel we will continue to see issue like this. TestFixedLengthInputFormat fails in jdk7 environment Key: MAPREDUCE-5625 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5625 Project: Hadoop Map/Reduce Issue Type: Test Affects Versions: 3.0.0, 2.3.0 Reporter: Jonathan Eagles Assignee: Mariappan Asokan Labels: java7 Fix For: 3.0.0, 2.3.0 Attachments: mapreduce-5625_v1.patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (MAPREDUCE-5631) TestJobEndNotifier.testNotifyRetries fails with Should have taken more than 5 seconds in jdk7
Jonathan Eagles created MAPREDUCE-5631: -- Summary: TestJobEndNotifier.testNotifyRetries fails with Should have taken more than 5 seconds in jdk7 Key: MAPREDUCE-5631 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5631 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 3.0.0, 2.3.0, 0.23.10 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Configuration settings are bleeding over from test to test in jdk7 environment since tests are run in random order. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Assigned] (MAPREDUCE-5373) TestFetchFailure.testFetchFailureMultipleReduces could fail intermittently
[
https://issues.apache.org/jira/browse/MAPREDUCE-5373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Eagles reassigned MAPREDUCE-5373:
--
Assignee: Jonathan Eagles
TestFetchFailure.testFetchFailureMultipleReduces could fail intermittently
--
Key: MAPREDUCE-5373
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5373
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Chuan Liu
Assignee: Jonathan Eagles
The unit test case could fail intermittently on both Linux and Windows in my
testing. The error message seems suggesting the task status was wrong during
testing.
An example Linux failure:
{noformat}
---
Test set: org.apache.hadoop.mapreduce.v2.app.TestFetchFailure
---
Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 12.235 sec
FAILURE!
testFetchFailureMultipleReduces(org.apache.hadoop.mapreduce.v2.app.TestFetchFailure)
Time elapsed: 1261 sec FAILURE!
java.lang.AssertionError: expected:SUCCEEDED but was:SCHEDULED
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:147)
at
org.apache.hadoop.mapreduce.v2.app.TestFetchFailure.testFetchFailureMultipleReduces(TestFetchFailure.java:332)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
{noformat}
An example Windows failure:
{noformat}
---
Test set: org.apache.hadoop.mapreduce.v2.app.TestFetchFailure
---
Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 50.342 sec
FAILURE!
testFetchFailureMultipleReduces(org.apache.hadoop.mapreduce.v2.app.TestFetchFailure)
Time elapsed: 36175 sec FAILURE!
java.lang.AssertionError: expected:SUCCEEDED but was:RUNNING
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:147)
at
org.apache.hadoop.mapreduce.v2.app.TestFetchFailure.testFetchFailureMultipleReduces(TestFetchFailure.java:332)
[jira] [Updated] (MAPREDUCE-5373) TestFetchFailure.testFetchFailureMultipleReduces could fail intermittently
[
https://issues.apache.org/jira/browse/MAPREDUCE-5373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Eagles updated MAPREDUCE-5373:
---
Attachment: MAPREDUCE-5373.patch
TestFetchFailure.testFetchFailureMultipleReduces could fail intermittently
--
Key: MAPREDUCE-5373
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5373
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Chuan Liu
Assignee: Jonathan Eagles
Attachments: MAPREDUCE-5373.patch
The unit test case could fail intermittently on both Linux and Windows in my
testing. The error message seems suggesting the task status was wrong during
testing.
An example Linux failure:
{noformat}
---
Test set: org.apache.hadoop.mapreduce.v2.app.TestFetchFailure
---
Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 12.235 sec
FAILURE!
testFetchFailureMultipleReduces(org.apache.hadoop.mapreduce.v2.app.TestFetchFailure)
Time elapsed: 1261 sec FAILURE!
java.lang.AssertionError: expected:SUCCEEDED but was:SCHEDULED
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:147)
at
org.apache.hadoop.mapreduce.v2.app.TestFetchFailure.testFetchFailureMultipleReduces(TestFetchFailure.java:332)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
{noformat}
An example Windows failure:
{noformat}
---
Test set: org.apache.hadoop.mapreduce.v2.app.TestFetchFailure
---
Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 50.342 sec
FAILURE!
testFetchFailureMultipleReduces(org.apache.hadoop.mapreduce.v2.app.TestFetchFailure)
Time elapsed: 36175 sec FAILURE!
java.lang.AssertionError: expected:SUCCEEDED but was:RUNNING
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:147)
at
[jira] [Commented] (MAPREDUCE-5373) TestFetchFailure.testFetchFailureMultipleReduces could fail intermittently
[
https://issues.apache.org/jira/browse/MAPREDUCE-5373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13824081#comment-13824081
]
Jonathan Eagles commented on MAPREDUCE-5373:
In the test setup, 3 fetch failures for the same maptask will trigger a too
many fetch failures event and cause the map task to relaunch. This test
accidentally sends 4 fetch failures triggering the maptask to relaunch to
early, causing a race condition in the assert for the current state of the
maptask. This patch removes the unnecessary 4th fetch failure, eliminating the
race condition.
TestFetchFailure.testFetchFailureMultipleReduces could fail intermittently
--
Key: MAPREDUCE-5373
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5373
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Chuan Liu
Assignee: Jonathan Eagles
Attachments: MAPREDUCE-5373.patch
The unit test case could fail intermittently on both Linux and Windows in my
testing. The error message seems suggesting the task status was wrong during
testing.
An example Linux failure:
{noformat}
---
Test set: org.apache.hadoop.mapreduce.v2.app.TestFetchFailure
---
Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 12.235 sec
FAILURE!
testFetchFailureMultipleReduces(org.apache.hadoop.mapreduce.v2.app.TestFetchFailure)
Time elapsed: 1261 sec FAILURE!
java.lang.AssertionError: expected:SUCCEEDED but was:SCHEDULED
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:147)
at
org.apache.hadoop.mapreduce.v2.app.TestFetchFailure.testFetchFailureMultipleReduces(TestFetchFailure.java:332)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
{noformat}
An example Windows failure:
{noformat}
---
Test set: org.apache.hadoop.mapreduce.v2.app.TestFetchFailure
---
Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 50.342 sec
FAILURE!
