[jira] [Commented] (MAPREDUCE-6278) Multithreaded maven build breaks in hadoop-mapreduce-client-core
[
https://issues.apache.org/jira/browse/MAPREDUCE-6278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361865#comment-16361865
]
Chris Douglas commented on MAPREDUCE-6278:
--
bq. the patch just works
Well... the patch may work. Without a principled reason to believe it prevents
the race, we know only that some executions didn't lose it. Moreover, if some
accident in the build makes a dependency on
{{hadoop-yarn-applications-distributedshell}} sufficient today, then future
changes may accidentally break it.
bq. I think ideally we need to put every leaf modules as dependencies of the
root submodule
This we could explain, at least. I'm not sure if it's necessary or if a better
pattern exists. Please add a comment to the pom to explain why the dependencies
are listed explicitly.
> Multithreaded maven build breaks in hadoop-mapreduce-client-core
>
>
> Key: MAPREDUCE-6278
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6278
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.9.0
> Environment: Linux (Fedora 21)
>Reporter: Ewan Higgs
>Assignee: Duo Xu
>Priority: Major
> Attachments: MAPREDUCE-6278.01.patch, MAPREDUCE-6278.02.patch
>
>
> [As reported on the mailing
> list|http://comments.gmane.org/gmane.comp.jakarta.lucene.hadoop.user/52231].
> The following breaks:
> {{mvn -e package -DskipTests -Dmaven.javadoc.skip -Dtar -Pdist,native -T5}}
> ...
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-assembly-plugin:2.4:single (package-mapreduce)
> on project hadoop-mapreduce: Failed to create assembly: Artifact:
> org.apache.hadoop:hadoop-mapreduce-client-core:jar:3.0.0-SNAPSHOT (included
> by module) does not have an artifact with a file. Please ensure the package
> phase is run before the assembly is generated. -> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
> goal org.apache.maven.plugins:maven-assembly-plugin:2.4:single
> (package-mapreduce) on project hadoop-mapreduce: Failed to create assembly:
> Artifact: org.apache.hadoop:hadoop-mapreduce-client-core:jar:3.0.0-SNAPSHOT
> (included by module) does not have an artifact with a file. Please ensure the
> package phase is run before the assembly is generated.
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:216)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
> at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
> at
> org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:188)
> at
> org.apache.maven.lifecycle.internal.builder.multithreaded.MultiThreadedBuilder$1.call(MultiThreadedBuilder.java:184)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.maven.plugin.MojoExecutionException: Failed to create
> assembly: Artifact:
> org.apache.hadoop:hadoop-mapreduce-client-core:jar:3.0.0-SNAPSHOT (included
> by module) does not have an artifact with a file. Please ensure the package
> phase is run before the assembly is generated.
> at
> org.apache.maven.plugin.assembly.mojos.AbstractAssemblyMojo.execute(AbstractAssemblyMojo.java:495)
> at
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:132)
> at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
> ... 11 more
> Caused by: org.apache.maven.plugin.assembly.archive.ArchiveCreationException:
> Artifact: org.apache.hadoop:hadoop-mapreduce-client-core:jar:3.0.0-SNAPSHOT
> (included by module) does not have an artifact with a file. Please ensure the
> package phase is run before the assembly is generated.
> at
> org.apache.maven.plugin.assembly.archive.phase.ModuleSetAssemblyPhase.addModuleArtifact(ModuleSetAssemblyPhase.java:318)
> at
> org.apache.maven.plugin.assembly.archive.phase.ModuleSetAssemblyPhase.addModuleBinaries(ModuleSetAssemblyPhase.java:228)
> at
> org.apache.maven.plugin.assembly.archive.phase.ModuleSetAssemblyPhase.execute(ModuleSetAssemblyPhase.java:111)
> at
>
[jira] [Commented] (MAPREDUCE-6315) Implement retrieval of logs for crashed MR-AM via jhist in the staging directory
[ https://issues.apache.org/jira/browse/MAPREDUCE-6315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361777#comment-16361777 ] Wangda Tan commented on MAPREDUCE-6315: --- Thanks [~jira.shegalov], I temporarily moved it to 3.2.0, if you have cycle to finish, please feel free to put it to 3.1.0. > Implement retrieval of logs for crashed MR-AM via jhist in the staging > directory > > > Key: MAPREDUCE-6315 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6315 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client, mr-am >Affects Versions: 2.7.0 >Reporter: Gera Shegalov >Priority: Critical > Labels: BB2015-05-TBR > Attachments: MAPREDUCE-6315.001.patch, MAPREDUCE-6315.002.patch, > MAPREDUCE-6315.003.patch > > > When all AM attempts crash, there is no record of them in JHS. Thus no easy > way to get the logs. This JIRA automates the procedure by utilizing the jhist > file in the staging directory. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6315) Implement retrieval of logs for crashed MR-AM via jhist in the staging directory
[ https://issues.apache.org/jira/browse/MAPREDUCE-6315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated MAPREDUCE-6315: -- Target Version/s: 3.2.0 (was: 3.1.0) > Implement retrieval of logs for crashed MR-AM via jhist in the staging > directory > > > Key: MAPREDUCE-6315 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6315 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client, mr-am >Affects Versions: 2.7.0 >Reporter: Gera Shegalov >Priority: Critical > Labels: BB2015-05-TBR > Attachments: MAPREDUCE-6315.001.patch, MAPREDUCE-6315.002.patch, > MAPREDUCE-6315.003.patch > > > When all AM attempts crash, there is no record of them in JHS. Thus no easy > way to get the logs. This JIRA automates the procedure by utilizing the jhist > file in the staging directory. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7051) Fix typo in MultipleOutputFormat
[ https://issues.apache.org/jira/browse/MAPREDUCE-7051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361704#comment-16361704 ] ASF GitHub Bot commented on MAPREDUCE-7051: --- Github user ywheel commented on the issue: https://github.com/apache/hadoop/pull/338 https://issues.apache.org/jira/browse/MAPREDUCE-7051 > Fix typo in MultipleOutputFormat > > > Key: MAPREDUCE-7051 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7051 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: ywheel >Priority: Trivial > Labels: newbie > Attachments: MAPREDUCE-7051.patch > > > In org.apache.hadoop.mapred.lib.MultipleOutputFormat, there is a typo for the > java doc of getInputFileBasedOutputFileName method. > "the outfile name based on a given anme and the input file name" should be > "the outfile name based on a given name and the input file name" -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6315) Implement retrieval of logs for crashed MR-AM via jhist in the staging directory
[ https://issues.apache.org/jira/browse/MAPREDUCE-6315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361590#comment-16361590 ] Gera Shegalov commented on MAPREDUCE-6315: -- [~leftnoteasy], [~vinodkv] the Jira is not assigned, anyone is welcome to pick it up and continue with suggested additional features. If needed I can spend some cycles on rebasing/fixing the original minimum value patch but I won't have time to do additional enhancements. > Implement retrieval of logs for crashed MR-AM via jhist in the staging > directory > > > Key: MAPREDUCE-6315 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6315 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client, mr-am >Affects Versions: 2.7.0 >Reporter: Gera Shegalov >Priority: Critical > Labels: BB2015-05-TBR > Attachments: MAPREDUCE-6315.001.patch, MAPREDUCE-6315.002.patch, > MAPREDUCE-6315.003.patch > > > When all AM attempts crash, there is no record of them in JHS. Thus no easy > way to get the logs. This JIRA automates the procedure by utilizing the jhist > file in the staging directory. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7048) Uber AM can crash due to unknown task in statusUpdate
[
https://issues.apache.org/jira/browse/MAPREDUCE-7048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated MAPREDUCE-7048:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 2.7.6
2.8.4
2.9.1
2.10.0
3.0.1
3.1.0
Status: Resolved (was: Patch Available)
Thanks, [~pbacsko]! I committed this to trunk, branch-3.1, branch-3.0,
branch-3.0.1, branch-2, branch-2.9, branch-2.8, and branch-2.7.
> Uber AM can crash due to unknown task in statusUpdate
> -
>
> Key: MAPREDUCE-7048
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7048
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Affects Versions: 3.1.0, 3.0.1, 2.10.0, 2.9.1, 2.8.4, 2.7.6
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Fix For: 3.1.0, 3.0.1, 2.10.0, 2.9.1, 2.8.4, 2.7.6
>
> Attachments: MAPREDUCE-7048-001.patch, MAPREDUCE-7048-002.patch,
> MAPREDUCE-7048-003.patch, MAPREDUCE-7048-branch-2.01.patch,
> MAPREDUCE-7048-branch-2.7.01.patch, MAPREDUCE-7048-branch-2.7.01.patch,
> MAPREDUCE-7048-branch-2.8.01.patch, MAPREDUCE-7048-branch-2.9.01.patch
>
>
> The testcase TestUberAM#testThreadDumpOnTaskTimeout was supposed to be fixed
> by MAPREDUCE-7020. However, it still fails, see:
> https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7325/testReport/junit/org.apache.hadoop.mapreduce.v2/TestMRJobs/testThreadDumpOnTaskTimeout/
> (note: other tests failed as well, but those look unrelated).
> When I tried to reproduce it locally, it failed again, although with a
> slightly different error message (it was actually the same as before):
> {noformat}
> [INFO] ---
> [INFO] T E S T S
> [INFO] ---
> [INFO] Running org.apache.hadoop.mapreduce.v2.TestUberAM
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 128.192 s <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.TestUberAM
> [ERROR]
> testThreadDumpOnTaskTimeout(org.apache.hadoop.mapreduce.v2.TestUberAM) Time
> elapsed: 79.539 s <<< FAILURE!
> java.lang.AssertionError: No AppMaster log found! expected:<1> but was:<2>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.mapreduce.v2.TestMRJobs.testThreadDumpOnTaskTimeout(TestMRJobs.java:1228)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
> *Root cause:* {{System.exit()}} is still invoked at {{Task.statusUpdate()}}
> {noformat}
> public void statusUpdate(TaskUmbilicalProtocol umbilical)
> throws IOException {
> int retries = MAX_RETRIES;
> while (true) {
> try {
> if (!umbilical.statusUpdate(getTaskID(), taskStatus).getTaskFound()) {
> LOG.warn("Parent died. Exiting "+taskId);
> System.exit(66);
> }
> taskStatus.clearStatus();
> return;
> ...
> {noformat}
> At this point, the task was not found and return value of
> {{umbilical.statusUpdate()}} is false. Checking whether we run in uber mode
> seems to solve the problem.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7048) Uber AM can crash due to unknown task in statusUpdate
[
https://issues.apache.org/jira/browse/MAPREDUCE-7048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361348#comment-16361348
]
Hudson commented on MAPREDUCE-7048:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13647 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/13647/])
MAPREDUCE-7048. Uber AM can crash due to unknown task in statusUpdate. (jlowe:
rev 87e2570a1419d3616de2de3b553108ad1a8af425)
* (edit)
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Task.java
* (add)
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestTask.java
> Uber AM can crash due to unknown task in statusUpdate
> -
>
> Key: MAPREDUCE-7048
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7048
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Affects Versions: 3.1.0, 3.0.1, 2.10.0, 2.9.1, 2.8.4, 2.7.6
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: MAPREDUCE-7048-001.patch, MAPREDUCE-7048-002.patch,
> MAPREDUCE-7048-003.patch, MAPREDUCE-7048-branch-2.01.patch,
> MAPREDUCE-7048-branch-2.7.01.patch, MAPREDUCE-7048-branch-2.7.01.patch,
> MAPREDUCE-7048-branch-2.8.01.patch, MAPREDUCE-7048-branch-2.9.01.patch
>
>
> The testcase TestUberAM#testThreadDumpOnTaskTimeout was supposed to be fixed
> by MAPREDUCE-7020. However, it still fails, see:
> https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7325/testReport/junit/org.apache.hadoop.mapreduce.v2/TestMRJobs/testThreadDumpOnTaskTimeout/
> (note: other tests failed as well, but those look unrelated).
> When I tried to reproduce it locally, it failed again, although with a
> slightly different error message (it was actually the same as before):
> {noformat}
> [INFO] ---
> [INFO] T E S T S
> [INFO] ---
> [INFO] Running org.apache.hadoop.mapreduce.v2.TestUberAM
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 128.192 s <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.TestUberAM
> [ERROR]
> testThreadDumpOnTaskTimeout(org.apache.hadoop.mapreduce.v2.TestUberAM) Time
> elapsed: 79.539 s <<< FAILURE!
> java.lang.AssertionError: No AppMaster log found! expected:<1> but was:<2>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.mapreduce.v2.TestMRJobs.testThreadDumpOnTaskTimeout(TestMRJobs.java:1228)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
> *Root cause:* {{System.exit()}} is still invoked at {{Task.statusUpdate()}}
> {noformat}
> public void statusUpdate(TaskUmbilicalProtocol umbilical)
> throws IOException {
> int retries = MAX_RETRIES;
> while (true) {
> try {
> if (!umbilical.statusUpdate(getTaskID(), taskStatus).getTaskFound()) {
> LOG.warn("Parent died. Exiting "+taskId);
> System.exit(66);
> }
> taskStatus.clearStatus();
> return;
> ...
> {noformat}
> At this point, the task was not found and return value of
> {{umbilical.statusUpdate()}} is false. Checking whether we run in uber mode
> seems to solve the problem.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7048) Uber AM can crash due to unknown task in statusUpdate
[
https://issues.apache.org/jira/browse/MAPREDUCE-7048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated MAPREDUCE-7048:
--
Summary: Uber AM can crash due to unknown task in statusUpdate (was: AM
can still crash after MAPREDUCE-7020)
> Uber AM can crash due to unknown task in statusUpdate
> -
>
> Key: MAPREDUCE-7048
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7048
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Affects Versions: 3.1.0, 3.0.1, 2.10.0, 2.9.1, 2.8.4, 2.7.6
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: MAPREDUCE-7048-001.patch, MAPREDUCE-7048-002.patch,
> MAPREDUCE-7048-003.patch, MAPREDUCE-7048-branch-2.01.patch,
> MAPREDUCE-7048-branch-2.7.01.patch, MAPREDUCE-7048-branch-2.7.01.patch,
> MAPREDUCE-7048-branch-2.8.01.patch, MAPREDUCE-7048-branch-2.9.01.patch
>
>
> The testcase TestUberAM#testThreadDumpOnTaskTimeout was supposed to be fixed
> by MAPREDUCE-7020. However, it still fails, see:
> https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7325/testReport/junit/org.apache.hadoop.mapreduce.v2/TestMRJobs/testThreadDumpOnTaskTimeout/
> (note: other tests failed as well, but those look unrelated).
> When I tried to reproduce it locally, it failed again, although with a
> slightly different error message (it was actually the same as before):
> {noformat}
> [INFO] ---
> [INFO] T E S T S
> [INFO] ---
> [INFO] Running org.apache.hadoop.mapreduce.v2.TestUberAM
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 128.192 s <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.TestUberAM
> [ERROR]
> testThreadDumpOnTaskTimeout(org.apache.hadoop.mapreduce.v2.TestUberAM) Time
> elapsed: 79.539 s <<< FAILURE!
> java.lang.AssertionError: No AppMaster log found! expected:<1> but was:<2>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.mapreduce.v2.TestMRJobs.testThreadDumpOnTaskTimeout(TestMRJobs.java:1228)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
> *Root cause:* {{System.exit()}} is still invoked at {{Task.statusUpdate()}}
> {noformat}
> public void statusUpdate(TaskUmbilicalProtocol umbilical)
> throws IOException {
> int retries = MAX_RETRIES;
> while (true) {
> try {
> if (!umbilical.statusUpdate(getTaskID(), taskStatus).getTaskFound()) {
> LOG.warn("Parent died. Exiting "+taskId);
> System.exit(66);
> }
> taskStatus.clearStatus();
> return;
> ...
> {noformat}
> At this point, the task was not found and return value of
> {{umbilical.statusUpdate()}} is false. Checking whether we run in uber mode
> seems to solve the problem.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7048) AM can still crash after MAPREDUCE-7020
[
https://issues.apache.org/jira/browse/MAPREDUCE-7048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361312#comment-16361312
]
Jason Lowe commented on MAPREDUCE-7048:
---
+1 for the branch-2.7 patch as well. The ASF, whitespace, and findbug warnings
are unrelated.
Committing this.
> AM can still crash after MAPREDUCE-7020
> ---
>
> Key: MAPREDUCE-7048
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7048
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Affects Versions: 3.1.0, 3.0.1, 2.10.0, 2.9.1, 2.8.4, 2.7.6
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: MAPREDUCE-7048-001.patch, MAPREDUCE-7048-002.patch,
> MAPREDUCE-7048-003.patch, MAPREDUCE-7048-branch-2.01.patch,
> MAPREDUCE-7048-branch-2.7.01.patch, MAPREDUCE-7048-branch-2.7.01.patch,
> MAPREDUCE-7048-branch-2.8.01.patch, MAPREDUCE-7048-branch-2.9.01.patch
>
>
> The testcase TestUberAM#testThreadDumpOnTaskTimeout was supposed to be fixed
> by MAPREDUCE-7020. However, it still fails, see:
> https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7325/testReport/junit/org.apache.hadoop.mapreduce.v2/TestMRJobs/testThreadDumpOnTaskTimeout/
> (note: other tests failed as well, but those look unrelated).
> When I tried to reproduce it locally, it failed again, although with a
> slightly different error message (it was actually the same as before):
> {noformat}
> [INFO] ---
> [INFO] T E S T S
> [INFO] ---
> [INFO] Running org.apache.hadoop.mapreduce.v2.TestUberAM
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 128.192 s <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.TestUberAM
> [ERROR]
> testThreadDumpOnTaskTimeout(org.apache.hadoop.mapreduce.v2.TestUberAM) Time
> elapsed: 79.539 s <<< FAILURE!
> java.lang.AssertionError: No AppMaster log found! expected:<1> but was:<2>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.mapreduce.v2.TestMRJobs.testThreadDumpOnTaskTimeout(TestMRJobs.java:1228)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
> *Root cause:* {{System.exit()}} is still invoked at {{Task.statusUpdate()}}
> {noformat}
> public void statusUpdate(TaskUmbilicalProtocol umbilical)
> throws IOException {
> int retries = MAX_RETRIES;
> while (true) {
> try {
> if (!umbilical.statusUpdate(getTaskID(), taskStatus).getTaskFound()) {
> LOG.warn("Parent died. Exiting "+taskId);
> System.exit(66);
> }
> taskStatus.clearStatus();
> return;
> ...
> {noformat}
> At this point, the task was not found and return value of
> {{umbilical.statusUpdate()}} is false. Checking whether we run in uber mode
> seems to solve the problem.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7048) AM can still crash after MAPREDUCE-7020
[
https://issues.apache.org/jira/browse/MAPREDUCE-7048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361124#comment-16361124
]
Hadoop QA commented on MAPREDUCE-7048:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m
40s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.7 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
37s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} branch-2.7 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} branch-2.7 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
3s{color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
in branch-2.7 has 2 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} branch-2.7 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 28s{color} | {color:orange}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core:
The patch generated 2 new + 853 unchanged - 3 fixed = 855 total (was 856)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 129 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
58s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
19s{color} | {color:red} The patch generated 4 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ea57d10 |
| JIRA Issue | MAPREDUCE-7048 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910253/MAPREDUCE-7048-branch-2.7.01.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 5c4d2789aaf6 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-2.7 / 02f6030 |
| maven | version: Apache Maven 3.0.5 |
| Default Java | 1.7.0_151 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7335/artifact/out/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7335/artifact/out/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7335/artifact/out/whitespace-eol.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7335/testReport/ |
|
[jira] [Commented] (MAPREDUCE-7048) AM can still crash after MAPREDUCE-7020
[
https://issues.apache.org/jira/browse/MAPREDUCE-7048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361064#comment-16361064
]
Jason Lowe commented on MAPREDUCE-7048:
---
Thanks for the patches! +1 for the branch-2 patch, branch-2.9, and branch-2.8
patches. They're all essentially the same. I posted the same branch-2.7 patch
again to get a Jenkins run on it since it's slightly different than the others.
> AM can still crash after MAPREDUCE-7020
> ---
>
> Key: MAPREDUCE-7048
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7048
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Affects Versions: 3.1.0, 3.0.1, 2.10.0, 2.9.1, 2.8.4, 2.7.6
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: MAPREDUCE-7048-001.patch, MAPREDUCE-7048-002.patch,
> MAPREDUCE-7048-003.patch, MAPREDUCE-7048-branch-2.01.patch,
> MAPREDUCE-7048-branch-2.7.01.patch, MAPREDUCE-7048-branch-2.7.01.patch,
> MAPREDUCE-7048-branch-2.8.01.patch, MAPREDUCE-7048-branch-2.9.01.patch
>
>
> The testcase TestUberAM#testThreadDumpOnTaskTimeout was supposed to be fixed
> by MAPREDUCE-7020. However, it still fails, see:
> https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7325/testReport/junit/org.apache.hadoop.mapreduce.v2/TestMRJobs/testThreadDumpOnTaskTimeout/
> (note: other tests failed as well, but those look unrelated).
> When I tried to reproduce it locally, it failed again, although with a
> slightly different error message (it was actually the same as before):
> {noformat}
> [INFO] ---
> [INFO] T E S T S
> [INFO] ---
> [INFO] Running org.apache.hadoop.mapreduce.v2.TestUberAM
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 128.192 s <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.TestUberAM
> [ERROR]
> testThreadDumpOnTaskTimeout(org.apache.hadoop.mapreduce.v2.TestUberAM) Time
> elapsed: 79.539 s <<< FAILURE!
> java.lang.AssertionError: No AppMaster log found! expected:<1> but was:<2>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.mapreduce.v2.TestMRJobs.testThreadDumpOnTaskTimeout(TestMRJobs.java:1228)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
> *Root cause:* {{System.exit()}} is still invoked at {{Task.statusUpdate()}}
> {noformat}
> public void statusUpdate(TaskUmbilicalProtocol umbilical)
> throws IOException {
> int retries = MAX_RETRIES;
> while (true) {
> try {
> if (!umbilical.statusUpdate(getTaskID(), taskStatus).getTaskFound()) {
> LOG.warn("Parent died. Exiting "+taskId);
> System.exit(66);
> }
> taskStatus.clearStatus();
> return;
> ...
> {noformat}
> At this point, the task was not found and return value of
> {{umbilical.statusUpdate()}} is false. Checking whether we run in uber mode
> seems to solve the problem.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7048) AM can still crash after MAPREDUCE-7020
[
https://issues.apache.org/jira/browse/MAPREDUCE-7048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated MAPREDUCE-7048:
--
Attachment: MAPREDUCE-7048-branch-2.7.01.patch
> AM can still crash after MAPREDUCE-7020
> ---
>
> Key: MAPREDUCE-7048
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7048
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Affects Versions: 3.1.0, 3.0.1, 2.10.0, 2.9.1, 2.8.4, 2.7.6
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: MAPREDUCE-7048-001.patch, MAPREDUCE-7048-002.patch,
> MAPREDUCE-7048-003.patch, MAPREDUCE-7048-branch-2.01.patch,
> MAPREDUCE-7048-branch-2.7.01.patch, MAPREDUCE-7048-branch-2.7.01.patch,
> MAPREDUCE-7048-branch-2.8.01.patch, MAPREDUCE-7048-branch-2.9.01.patch
>
>
> The testcase TestUberAM#testThreadDumpOnTaskTimeout was supposed to be fixed
> by MAPREDUCE-7020. However, it still fails, see:
> https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7325/testReport/junit/org.apache.hadoop.mapreduce.v2/TestMRJobs/testThreadDumpOnTaskTimeout/
> (note: other tests failed as well, but those look unrelated).
> When I tried to reproduce it locally, it failed again, although with a
> slightly different error message (it was actually the same as before):
> {noformat}
> [INFO] ---
> [INFO] T E S T S
> [INFO] ---
> [INFO] Running org.apache.hadoop.mapreduce.v2.TestUberAM
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 128.192 s <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.TestUberAM
> [ERROR]
> testThreadDumpOnTaskTimeout(org.apache.hadoop.mapreduce.v2.TestUberAM) Time
> elapsed: 79.539 s <<< FAILURE!
> java.lang.AssertionError: No AppMaster log found! expected:<1> but was:<2>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.mapreduce.v2.TestMRJobs.testThreadDumpOnTaskTimeout(TestMRJobs.java:1228)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
> *Root cause:* {{System.exit()}} is still invoked at {{Task.statusUpdate()}}
> {noformat}
> public void statusUpdate(TaskUmbilicalProtocol umbilical)
> throws IOException {
> int retries = MAX_RETRIES;
> while (true) {
> try {
> if (!umbilical.statusUpdate(getTaskID(), taskStatus).getTaskFound()) {
> LOG.warn("Parent died. Exiting "+taskId);
> System.exit(66);
> }
> taskStatus.clearStatus();
> return;
> ...
> {noformat}
> At this point, the task was not found and return value of
> {{umbilical.statusUpdate()}} is false. Checking whether we run in uber mode
> seems to solve the problem.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7048) AM can still crash after MAPREDUCE-7020
[
https://issues.apache.org/jira/browse/MAPREDUCE-7048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16361047#comment-16361047
]
Hadoop QA commented on MAPREDUCE-7048:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 21m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-2.9 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
9s{color} | {color:green} branch-2.9 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} branch-2.9 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} branch-2.9 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
38s{color} | {color:green} branch-2.9 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
6s{color} | {color:green} branch-2.9 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} branch-2.9 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 18s{color} | {color:orange}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core:
The patch generated 3 new + 197 unchanged - 0 fixed = 200 total (was 197)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
49s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 46m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:17213a0 |
| JIRA Issue | MAPREDUCE-7048 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12910244/MAPREDUCE-7048-branch-2.9.01.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux b231981dd693 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-2.9 / 41d434d |
| maven | version: Apache Maven 3.3.9
(bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T16:41:47+00:00) |
| Default Java | 1.7.0_151 |
| findbugs | v3.0.0 |
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7334/artifact/out/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7334/testReport/ |
| Max. process+thread count | 215 (vs. ulimit of 5500) |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
U:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7334/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was
[jira] [Commented] (MAPREDUCE-7048) AM can still crash after MAPREDUCE-7020
[
https://issues.apache.org/jira/browse/MAPREDUCE-7048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360930#comment-16360930
]
Peter Bacsko commented on MAPREDUCE-7048:
-
I uploaded the patches for branch-2, branch-2.9, branch-2.8 and branch-2.7.
> AM can still crash after MAPREDUCE-7020
> ---
>
> Key: MAPREDUCE-7048
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7048
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Affects Versions: 3.1.0, 3.0.1, 2.10.0, 2.9.1, 2.8.4, 2.7.6
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: MAPREDUCE-7048-001.patch, MAPREDUCE-7048-002.patch,
> MAPREDUCE-7048-003.patch, MAPREDUCE-7048-branch-2.01.patch,
> MAPREDUCE-7048-branch-2.7.01.patch, MAPREDUCE-7048-branch-2.8.01.patch,
> MAPREDUCE-7048-branch-2.9.01.patch
>
>
> The testcase TestUberAM#testThreadDumpOnTaskTimeout was supposed to be fixed
> by MAPREDUCE-7020. However, it still fails, see:
> https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7325/testReport/junit/org.apache.hadoop.mapreduce.v2/TestMRJobs/testThreadDumpOnTaskTimeout/
> (note: other tests failed as well, but those look unrelated).
> When I tried to reproduce it locally, it failed again, although with a
> slightly different error message (it was actually the same as before):
> {noformat}
> [INFO] ---
> [INFO] T E S T S
> [INFO] ---
> [INFO] Running org.apache.hadoop.mapreduce.v2.TestUberAM
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 128.192 s <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.TestUberAM
> [ERROR]
> testThreadDumpOnTaskTimeout(org.apache.hadoop.mapreduce.v2.TestUberAM) Time
> elapsed: 79.539 s <<< FAILURE!
> java.lang.AssertionError: No AppMaster log found! expected:<1> but was:<2>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.mapreduce.v2.TestMRJobs.testThreadDumpOnTaskTimeout(TestMRJobs.java:1228)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
> *Root cause:* {{System.exit()}} is still invoked at {{Task.statusUpdate()}}
> {noformat}
> public void statusUpdate(TaskUmbilicalProtocol umbilical)
> throws IOException {
> int retries = MAX_RETRIES;
> while (true) {
> try {
> if (!umbilical.statusUpdate(getTaskID(), taskStatus).getTaskFound()) {
> LOG.warn("Parent died. Exiting "+taskId);
> System.exit(66);
> }
> taskStatus.clearStatus();
> return;
> ...
> {noformat}
> At this point, the task was not found and return value of
> {{umbilical.statusUpdate()}} is false. Checking whether we run in uber mode
> seems to solve the problem.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7048) AM can still crash after MAPREDUCE-7020
[
https://issues.apache.org/jira/browse/MAPREDUCE-7048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated MAPREDUCE-7048:
Attachment: MAPREDUCE-7048-branch-2.9.01.patch
MAPREDUCE-7048-branch-2.8.01.patch
MAPREDUCE-7048-branch-2.7.01.patch
MAPREDUCE-7048-branch-2.01.patch
> AM can still crash after MAPREDUCE-7020
> ---
>
> Key: MAPREDUCE-7048
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7048
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am
>Affects Versions: 3.1.0, 3.0.1, 2.10.0, 2.9.1, 2.8.4, 2.7.6
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: MAPREDUCE-7048-001.patch, MAPREDUCE-7048-002.patch,
> MAPREDUCE-7048-003.patch, MAPREDUCE-7048-branch-2.01.patch,
> MAPREDUCE-7048-branch-2.7.01.patch, MAPREDUCE-7048-branch-2.8.01.patch,
> MAPREDUCE-7048-branch-2.9.01.patch
>
>
> The testcase TestUberAM#testThreadDumpOnTaskTimeout was supposed to be fixed
> by MAPREDUCE-7020. However, it still fails, see:
> https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7325/testReport/junit/org.apache.hadoop.mapreduce.v2/TestMRJobs/testThreadDumpOnTaskTimeout/
> (note: other tests failed as well, but those look unrelated).
> When I tried to reproduce it locally, it failed again, although with a
> slightly different error message (it was actually the same as before):
> {noformat}
> [INFO] ---
> [INFO] T E S T S
> [INFO] ---
> [INFO] Running org.apache.hadoop.mapreduce.v2.TestUberAM
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 128.192 s <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.TestUberAM
> [ERROR]
> testThreadDumpOnTaskTimeout(org.apache.hadoop.mapreduce.v2.TestUberAM) Time
> elapsed: 79.539 s <<< FAILURE!
> java.lang.AssertionError: No AppMaster log found! expected:<1> but was:<2>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.mapreduce.v2.TestMRJobs.testThreadDumpOnTaskTimeout(TestMRJobs.java:1228)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
> *Root cause:* {{System.exit()}} is still invoked at {{Task.statusUpdate()}}
> {noformat}
> public void statusUpdate(TaskUmbilicalProtocol umbilical)
> throws IOException {
> int retries = MAX_RETRIES;
> while (true) {
> try {
> if (!umbilical.statusUpdate(getTaskID(), taskStatus).getTaskFound()) {
> LOG.warn("Parent died. Exiting "+taskId);
> System.exit(66);
> }
> taskStatus.clearStatus();
> return;
> ...
> {noformat}
> At this point, the task was not found and return value of
> {{umbilical.statusUpdate()}} is false. Checking whether we run in uber mode
> seems to solve the problem.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-5907) Improve getSplits() performance for fs implementations that can utilize performance gains from recursive listing
[
https://issues.apache.org/jira/browse/MAPREDUCE-5907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360699#comment-16360699
]
Hadoop QA commented on MAPREDUCE-5907:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s{color}
| {color:red} MAPREDUCE-5907 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | MAPREDUCE-5907 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12648040/MAPREDUCE-5907-3.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7333/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Improve getSplits() performance for fs implementations that can utilize
> performance gains from recursive listing
>
>
> Key: MAPREDUCE-5907
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5907
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client
>Affects Versions: 2.4.0
>Reporter: Sumit Kumar
>Assignee: Sumit Kumar
>Priority: Major
> Attachments: MAPREDUCE-5907-2.patch, MAPREDUCE-5907-3.patch,
> MAPREDUCE-5907.patch
>
>
> FileInputFormat (both mapreduce and mapred implementations) use recursive
> listing while calculating splits. They however do this by doing listing level
> by level. That means to discover files in /foo/bar means they do listing at
> /foo/bar first to get the immediate children, then make the same call on all
> immediate children for /foo/bar to discover their immediate children and so
> on. This doesn't scale well for object store based fs implementations like s3
> and swift because every listStatus call ends up being a webservice call to
> backend. In cases where large number of files are considered for input, this
> makes getSplits() call slow.
> This patch adds a new set of recursive list apis that gives opportunity to
> the fs implementations to optimize. The behavior remains the same for other
> implementations (that is a default implementation is provided for other fs so
> they don't have to implement anything new). However for objectstore based fs
> implementations it provides a simple change to include recursive flag as true
> (as shown in the patch) to improve listing performance.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-5907) Improve getSplits() performance for fs implementations that can utilize performance gains from recursive listing
[ https://issues.apache.org/jira/browse/MAPREDUCE-5907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated MAPREDUCE-5907: -- Labels: (was: BB2015-05-TBR) > Improve getSplits() performance for fs implementations that can utilize > performance gains from recursive listing > > > Key: MAPREDUCE-5907 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5907 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Affects Versions: 2.4.0 >Reporter: Sumit Kumar >Assignee: Sumit Kumar >Priority: Major > Attachments: MAPREDUCE-5907-2.patch, MAPREDUCE-5907-3.patch, > MAPREDUCE-5907.patch > > > FileInputFormat (both mapreduce and mapred implementations) use recursive > listing while calculating splits. They however do this by doing listing level > by level. That means to discover files in /foo/bar means they do listing at > /foo/bar first to get the immediate children, then make the same call on all > immediate children for /foo/bar to discover their immediate children and so > on. This doesn't scale well for object store based fs implementations like s3 > and swift because every listStatus call ends up being a webservice call to > backend. In cases where large number of files are considered for input, this > makes getSplits() call slow. > This patch adds a new set of recursive list apis that gives opportunity to > the fs implementations to optimize. The behavior remains the same for other > implementations (that is a default implementation is provided for other fs so > they don't have to implement anything new). However for objectstore based fs > implementations it provides a simple change to include recursive flag as true > (as shown in the patch) to improve listing performance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-5907) Improve getSplits() performance for fs implementations that can utilize performance gains from recursive listing
[ https://issues.apache.org/jira/browse/MAPREDUCE-5907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360698#comment-16360698 ] Steve Loughran commented on MAPREDUCE-5907: --- Catching up on this after spending many weeks in HADOOP-13786 * Performance boosts with s3 are moot because both the v1 and v2 algorithms are unreliable with raw s3, and slow even with a consistency layer; it's obsolete. * none of the other stores *currently* do anything with the listFiles accordingly, we don't see any immediate benefit in moving to the API, But if/when someone does implement the listFiles call in a store which also offers consistent metadata and O(1) renames, then this will help, especially in v1, where the listing is done in the serialized phase of job commit > Improve getSplits() performance for fs implementations that can utilize > performance gains from recursive listing > > > Key: MAPREDUCE-5907 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5907 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Affects Versions: 2.4.0 >Reporter: Sumit Kumar >Assignee: Sumit Kumar >Priority: Major > Labels: BB2015-05-TBR > Attachments: MAPREDUCE-5907-2.patch, MAPREDUCE-5907-3.patch, > MAPREDUCE-5907.patch > > > FileInputFormat (both mapreduce and mapred implementations) use recursive > listing while calculating splits. They however do this by doing listing level > by level. That means to discover files in /foo/bar means they do listing at > /foo/bar first to get the immediate children, then make the same call on all > immediate children for /foo/bar to discover their immediate children and so > on. This doesn't scale well for object store based fs implementations like s3 > and swift because every listStatus call ends up being a webservice call to > backend. In cases where large number of files are considered for input, this > makes getSplits() call slow. > This patch adds a new set of recursive list apis that gives opportunity to > the fs implementations to optimize. The behavior remains the same for other > implementations (that is a default implementation is provided for other fs so > they don't have to implement anything new). However for objectstore based fs > implementations it provides a simple change to include recursive flag as true > (as shown in the patch) to improve listing performance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7051) Fix typo in MultipleOutputFormat
[ https://issues.apache.org/jira/browse/MAPREDUCE-7051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ywheel updated MAPREDUCE-7051: -- Attachment: MAPREDUCE-7051.patch > Fix typo in MultipleOutputFormat > > > Key: MAPREDUCE-7051 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7051 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: ywheel >Priority: Trivial > Labels: newbie > Attachments: MAPREDUCE-7051.patch > > > In org.apache.hadoop.mapred.lib.MultipleOutputFormat, there is a typo for the > java doc of getInputFileBasedOutputFileName method. > "the outfile name based on a given anme and the input file name" should be > "the outfile name based on a given name and the input file name" -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7051) Fix typo in MultipleOutputFormat
[ https://issues.apache.org/jira/browse/MAPREDUCE-7051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ywheel updated MAPREDUCE-7051: -- Labels: newbie (was: ) > Fix typo in MultipleOutputFormat > > > Key: MAPREDUCE-7051 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7051 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: ywheel >Priority: Trivial > Labels: newbie > > In org.apache.hadoop.mapred.lib.MultipleOutputFormat, there is a typo for the > java doc of getInputFileBasedOutputFileName method. > "the outfile name based on a given anme and the input file name" should be > "the outfile name based on a given name and the input file name" -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7051) Fix typo in MultipleOutputFormat
ywheel created MAPREDUCE-7051: - Summary: Fix typo in MultipleOutputFormat Key: MAPREDUCE-7051 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7051 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: ywheel In org.apache.hadoop.mapred.lib.MultipleOutputFormat, there is a typo for the java doc of getInputFileBasedOutputFileName method. "the outfile name based on a given anme and the input file name" should be "the outfile name based on a given name and the input file name" -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
