[jira] [Updated] (MAPREDUCE-7237) Supports config the shuffle's path cache related parameters
[ https://issues.apache.org/jira/browse/MAPREDUCE-7237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wanqiang Ji updated MAPREDUCE-7237: --- Status: Patch Available (was: Open) > Supports config the shuffle's path cache related parameters > --- > > Key: MAPREDUCE-7237 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7237 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv2 >Reporter: Wanqiang Ji >Assignee: Wanqiang Ji >Priority: Major > > Nowadays the ShuffleHandler#Shuffle#pathCache's related parameters is hard > coding. We should support config these. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7235) NPE in HadoopArchiveLogs#filterAppsByAggregatedStatus
[
https://issues.apache.org/jira/browse/MAPREDUCE-7235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16921760#comment-16921760
]
Hadoop QA commented on MAPREDUCE-7235:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 16m
40s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
10s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} branch-2 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} branch-2 passed with JDK v1.8.0_222 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
27s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
30s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} branch-2 passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} branch-2 passed with JDK v1.8.0_222 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed with JDK v1.8.0_222 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} hadoop-tools/hadoop-archive-logs: The patch
generated 0 new + 5 unchanged - 3 fixed = 5 total (was 8) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed with JDK v1.7.0_95 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} the patch passed with JDK v1.8.0_222 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
41s{color} | {color:green} hadoop-archive-logs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 36m 46s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:da675796017 |
| JIRA Issue | MAPREDUCE-7235 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12979179/MAPREDUCE-7235.branch-2.01.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 75c7efb8b6e4 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | branch-2 / 493b362 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| Multi-JDK versions |
[jira] [Updated] (MAPREDUCE-7238) TestMRJobs.testJobClassloader fails intermittent
[
https://issues.apache.org/jira/browse/MAPREDUCE-7238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prabhu Joseph updated MAPREDUCE-7238:
-
Description:
*TestMRJobs.testJobClassloader fails intermittent* observed in
{code}
ERROR] testJobClassloader(org.apache.hadoop.mapreduce.v2.TestMRJobs) Time
elapsed: 29.77 s <<< FAILURE!
java.lang.AssertionError:
Job status: Application application_1567255842834_0009 failed 2 times due to AM
Container for appattempt_1567255842834_0009_02 exited with exitCode: 1
Failing this attempt.Diagnostics: [2019-08-31 12:54:14.542]Exception from
container-launch.
Container id: container_1567255842834_0009_02_01
Exit code: 1
[2019-08-31 12:54:14.546]Container exited with a non-zero exit code 1. Error
file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
[2019-08-31 12:54:14.547]Container exited with a non-zero exit code 1. Error
file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
For more detailed output, check the application tracking page:
http://6437fb7eb209:32931/cluster/app/application_1567255842834_0009 Then click
on links to logs of each attempt.
. Failing the application.
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.mapreduce.v2.TestMRJobs.testJobClassloader(TestMRJobs.java:531)
at
org.apache.hadoop.mapreduce.v2.TestMRJobs.testJobClassloader(TestMRJobs.java:473)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
{code}
was:
TestMRJobs.testThreadDumpOnTaskTimeout fails
{code}
[ERROR] testThreadDumpOnTaskTimeout(org.apache.hadoop.mapreduce.v2.TestMRJobs)
Time elapsed: 43.282 s <<< FAILURE!
java.lang.AssertionError: No thread dump
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at
org.apache.hadoop.mapreduce.v2.TestMRJobs.testThreadDumpOnTaskTimeout(TestMRJobs.java:1222)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:745)
{code}
> TestMRJobs.testJobClassloader fails intermittent
>
>
> Key: MAPREDUCE-7238
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7238
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: stdout
>
>
> *TestMRJobs.testJobClassloader fails intermittent* observed in
> {code}
> ERROR] testJobClassloader(org.apache.hadoop.mapreduce.v2.TestMRJobs) Time
> elapsed: 29.77 s <<< FAILURE!
> java.lang.AssertionError:
> Job status: Application application_1567255842834_0009 failed 2 times due to
> AM Container for appattempt_1567255842834_0009_02 exited with exitCode: 1
> Failing this attempt.Diagnostics:
[jira] [Updated] (MAPREDUCE-7238) TestMRJobs.testJobClassloader fails intermittent
[
https://issues.apache.org/jira/browse/MAPREDUCE-7238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prabhu Joseph updated MAPREDUCE-7238:
-
Attachment: stdout
> TestMRJobs.testJobClassloader fails intermittent
>
>
> Key: MAPREDUCE-7238
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7238
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
> Attachments: stdout
>
>
> *TestMRJobs.testJobClassloader fails intermittent* observed in
> {code}
> ERROR] testJobClassloader(org.apache.hadoop.mapreduce.v2.TestMRJobs) Time
> elapsed: 29.77 s <<< FAILURE!
> java.lang.AssertionError:
> Job status: Application application_1567255842834_0009 failed 2 times due to
> AM Container for appattempt_1567255842834_0009_02 exited with exitCode: 1
> Failing this attempt.Diagnostics: [2019-08-31 12:54:14.542]Exception from
> container-launch.
> Container id: container_1567255842834_0009_02_01
> Exit code: 1
> [2019-08-31 12:54:14.546]Container exited with a non-zero exit code 1. Error
> file: prelaunch.err.
> Last 4096 bytes of prelaunch.err :
> Last 4096 bytes of stderr :
> [2019-08-31 12:54:14.547]Container exited with a non-zero exit code 1. Error
> file: prelaunch.err.
> Last 4096 bytes of prelaunch.err :
> Last 4096 bytes of stderr :
> For more detailed output, check the application tracking page:
> http://6437fb7eb209:32931/cluster/app/application_1567255842834_0009 Then
> click on links to logs of each attempt.
> . Failing the application.
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.mapreduce.v2.TestMRJobs.testJobClassloader(TestMRJobs.java:531)
> at
> org.apache.hadoop.mapreduce.v2.TestMRJobs.testJobClassloader(TestMRJobs.java:473)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7238) TestMRJobs.testJobClassloader fails intermittent
[
https://issues.apache.org/jira/browse/MAPREDUCE-7238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prabhu Joseph updated MAPREDUCE-7238:
-
Summary: TestMRJobs.testJobClassloader fails intermittent (was:
TestMRJobs.testThreadDumpOnTaskTimeout fails)
> TestMRJobs.testJobClassloader fails intermittent
>
>
> Key: MAPREDUCE-7238
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7238
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 3.3.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
>
> TestMRJobs.testThreadDumpOnTaskTimeout fails
> {code}
> [ERROR]
> testThreadDumpOnTaskTimeout(org.apache.hadoop.mapreduce.v2.TestMRJobs) Time
> elapsed: 43.282 s <<< FAILURE!
> java.lang.AssertionError: No thread dump
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.mapreduce.v2.TestMRJobs.testThreadDumpOnTaskTimeout(TestMRJobs.java:1222)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (MAPREDUCE-7227) Fix job staging directory residual problem in a big yarn cluster composed of multiple independent hdfs clusters
[ https://issues.apache.org/jira/browse/MAPREDUCE-7227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16921426#comment-16921426 ] Yuan LUO edited comment on MAPREDUCE-7227 at 9/3/19 1:34 PM: - [~eepayne] Can you reproduce this issue, in our clusters this problem is very easy to happen . Image attachment [Process_Analysis.png] may help you understand what I mean. was (Author: luoyuan): !Process_Analysis.png! [~eepayne] Can you reproduce this issue, in our clusters this problem is very easy to happen . > Fix job staging directory residual problem in a big yarn cluster composed of > multiple independent hdfs clusters > --- > > Key: MAPREDUCE-7227 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7227 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster, mrv2 >Affects Versions: 2.6.0, 2.7.0, 3.1.2 >Reporter: Yuan LUO >Assignee: Yuan LUO >Priority: Major > Attachments: 1.png, 2.png, HADOOP-MAPREDUCE-7227.001.patch, > HADOOP-MAPREDUCE-7227.002.patch, HADOOP-MAPREDUCE-7227.003.patch, > HADOOP-MAPREDUCE-7227.004.patch, HADOOP-MAPREDUCE-7227.005.patch, > Process_Analysis.png > > > Our yarn cluster is made up of some independent hdfs cluster, the > 'default.FS' in every hdfs cluster is different.when user submit job to yarn > cluster, if the 'default.FS' between client and nodemanager is > inconsistent, then the job staging dir can't be cleanup by AppMaster. Because > it will produce two job staging dirs in our conditions by client and > appmaster. So we can modify AppMaster through client's ‘default.FS’ to > create job staging dir. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7227) Fix job staging directory residual problem in a big yarn cluster composed of multiple independent hdfs clusters
[ https://issues.apache.org/jira/browse/MAPREDUCE-7227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16921426#comment-16921426 ] Yuan LUO commented on MAPREDUCE-7227: - !Process_Analysis.png! [~eepayne] Can you reproduce this issue, in our clusters this problem is very easy to happen . > Fix job staging directory residual problem in a big yarn cluster composed of > multiple independent hdfs clusters > --- > > Key: MAPREDUCE-7227 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7227 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster, mrv2 >Affects Versions: 2.6.0, 2.7.0, 3.1.2 >Reporter: Yuan LUO >Assignee: Yuan LUO >Priority: Major > Attachments: 1.png, 2.png, HADOOP-MAPREDUCE-7227.001.patch, > HADOOP-MAPREDUCE-7227.002.patch, HADOOP-MAPREDUCE-7227.003.patch, > HADOOP-MAPREDUCE-7227.004.patch, HADOOP-MAPREDUCE-7227.005.patch, > Process_Analysis.png > > > Our yarn cluster is made up of some independent hdfs cluster, the > 'default.FS' in every hdfs cluster is different.when user submit job to yarn > cluster, if the 'default.FS' between client and nodemanager is > inconsistent, then the job staging dir can't be cleanup by AppMaster. Because > it will produce two job staging dirs in our conditions by client and > appmaster. So we can modify AppMaster through client's ‘default.FS’ to > create job staging dir. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7227) Fix job staging directory residual problem in a big yarn cluster composed of multiple independent hdfs clusters
[ https://issues.apache.org/jira/browse/MAPREDUCE-7227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan LUO updated MAPREDUCE-7227: Attachment: Process_Analysis.png > Fix job staging directory residual problem in a big yarn cluster composed of > multiple independent hdfs clusters > --- > > Key: MAPREDUCE-7227 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7227 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster, mrv2 >Affects Versions: 2.6.0, 2.7.0, 3.1.2 >Reporter: Yuan LUO >Assignee: Yuan LUO >Priority: Major > Attachments: 1.png, 2.png, HADOOP-MAPREDUCE-7227.001.patch, > HADOOP-MAPREDUCE-7227.002.patch, HADOOP-MAPREDUCE-7227.003.patch, > HADOOP-MAPREDUCE-7227.004.patch, HADOOP-MAPREDUCE-7227.005.patch, > Process_Analysis.png > > > Our yarn cluster is made up of some independent hdfs cluster, the > 'default.FS' in every hdfs cluster is different.when user submit job to yarn > cluster, if the 'default.FS' between client and nodemanager is > inconsistent, then the job staging dir can't be cleanup by AppMaster. Because > it will produce two job staging dirs in our conditions by client and > appmaster. So we can modify AppMaster through client's ‘default.FS’ to > create job staging dir. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7227) Fix job staging directory residual problem in a big yarn cluster composed of multiple independent hdfs clusters
[ https://issues.apache.org/jira/browse/MAPREDUCE-7227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16921365#comment-16921365 ] Yuan LUO commented on MAPREDUCE-7227: - HADOOP-MAPREDUCE-7227.005.patch for 3.3.0 > Fix job staging directory residual problem in a big yarn cluster composed of > multiple independent hdfs clusters > --- > > Key: MAPREDUCE-7227 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7227 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster, mrv2 >Affects Versions: 2.6.0, 2.7.0, 3.1.2 >Reporter: Yuan LUO >Assignee: Yuan LUO >Priority: Major > Attachments: 1.png, 2.png, HADOOP-MAPREDUCE-7227.001.patch, > HADOOP-MAPREDUCE-7227.002.patch, HADOOP-MAPREDUCE-7227.003.patch, > HADOOP-MAPREDUCE-7227.004.patch, HADOOP-MAPREDUCE-7227.005.patch > > > Our yarn cluster is made up of some independent hdfs cluster, the > 'default.FS' in every hdfs cluster is different.when user submit job to yarn > cluster, if the 'default.FS' between client and nodemanager is > inconsistent, then the job staging dir can't be cleanup by AppMaster. Because > it will produce two job staging dirs in our conditions by client and > appmaster. So we can modify AppMaster through client's ‘default.FS’ to > create job staging dir. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7227) Fix job staging directory residual problem in a big yarn cluster composed of multiple independent hdfs clusters
[ https://issues.apache.org/jira/browse/MAPREDUCE-7227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan LUO updated MAPREDUCE-7227: Attachment: HADOOP-MAPREDUCE-7227.005.patch > Fix job staging directory residual problem in a big yarn cluster composed of > multiple independent hdfs clusters > --- > > Key: MAPREDUCE-7227 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7227 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster, mrv2 >Affects Versions: 2.6.0, 2.7.0, 3.1.2 >Reporter: Yuan LUO >Assignee: Yuan LUO >Priority: Major > Attachments: 1.png, 2.png, HADOOP-MAPREDUCE-7227.001.patch, > HADOOP-MAPREDUCE-7227.002.patch, HADOOP-MAPREDUCE-7227.003.patch, > HADOOP-MAPREDUCE-7227.004.patch, HADOOP-MAPREDUCE-7227.005.patch > > > Our yarn cluster is made up of some independent hdfs cluster, the > 'default.FS' in every hdfs cluster is different.when user submit job to yarn > cluster, if the 'default.FS' between client and nodemanager is > inconsistent, then the job staging dir can't be cleanup by AppMaster. Because > it will produce two job staging dirs in our conditions by client and > appmaster. So we can modify AppMaster through client's ‘default.FS’ to > create job staging dir. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6752) Bad logging practices in mapreduce
[
https://issues.apache.org/jira/browse/MAPREDUCE-6752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16921187#comment-16921187
]
Hadoop QA commented on MAPREDUCE-6752:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
43s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 55s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
54s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
53s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 15s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m
2s{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 63m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 base:
https://builds.apache.org/job/hadoop-multibranch/job/PR-285/9/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/285 |
| JIRA Issue | MAPREDUCE-6752 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 812f2d9b6ef3 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / 915cbc9 |
| Default Java | 1.8.0_222 |
| Test Results |
https://builds.apache.org/job/hadoop-multibranch/job/PR-285/9/testReport/ |
| Max. process+thread count | 596 (vs. ulimit of 5500) |
| modules | C:
