[jira] [Assigned] (MAPREDUCE-7022) Fast fail rogue jobs based on task scratch dir size
[ https://issues.apache.org/jira/browse/MAPREDUCE-7022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johan Gustavsson reassigned MAPREDUCE-7022: --- Assignee: Jason Lowe (was: Johan Gustavsson) > Fast fail rogue jobs based on task scratch dir size > --- > > Key: MAPREDUCE-7022 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7022 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: task >Affects Versions: 2.7.0, 2.8.0, 2.9.0 >Reporter: Johan Gustavsson >Assignee: Jason Lowe >Priority: Major > Attachments: MAPREDUCE-7022.001.patch, MAPREDUCE-7022.002.patch, > MAPREDUCE-7022.003.patch, MAPREDUCE-7022.004.patch, MAPREDUCE-7022.005.patch, > MAPREDUCE-7022.006.patch, MAPREDUCE-7022.007.patch, MAPREDUCE-7022.008.patch > > > With the introduction of MAPREDUCE-6489 there are some options to kill rogue > tasks based on writes to local disk writes. In our environment are we mainly > run Hive based jobs we noticed that this counter and the size of the local > scratch dirs were very different. We had tasks where BYTES_WRITTEN counter > were at 300Gb and where it was at 10Tb both producing around 200Gb on local > disk, so it didn't help us much. So to extend this feature tasks should > monitor local scratchdir size and fail if they pass the limit. In these cases > the tasks should not be retried either but instead the job should fast fail. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7022) Fast fail rogue jobs based on task scratch dir size
[ https://issues.apache.org/jira/browse/MAPREDUCE-7022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16326703#comment-16326703 ] Johan Gustavsson commented on MAPREDUCE-7022: - Thanks for the detailed feedback [~jlowe] I believe I have dealt with all the points you brought up by: # Merging fatalError and fatalErrorFailFast into fatalError with an added boolean parameter determining fail fast. # Changed all instances of job to task except for in mapred-default.xml # Updated TestTaskImpl to send proper type event for all T_ATTEMPT_FAILED # Rooted all configs in mapreduce.job.local-fs.single-disk-limit # Removed strange comments and added fixed logging around disk monitor. Since we are using slf4j though there is no fatal so it's set to error. # Additionally I added the missing apache license notes on the 2 new classes in this patch As for the failed tests below: * org.apache.hadoop.mapreduce.v2.TestUberAM is failing on trunk too * org.apache.hadoop.mapred.TestTaskProgressReporter runs without issue when I rerun it locally against latest trunk > Fast fail rogue jobs based on task scratch dir size > --- > > Key: MAPREDUCE-7022 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7022 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: task >Affects Versions: 2.7.0, 2.8.0, 2.9.0 >Reporter: Johan Gustavsson >Assignee: Johan Gustavsson >Priority: Major > Attachments: MAPREDUCE-7022.001.patch, MAPREDUCE-7022.002.patch, > MAPREDUCE-7022.003.patch, MAPREDUCE-7022.004.patch, MAPREDUCE-7022.005.patch, > MAPREDUCE-7022.006.patch, MAPREDUCE-7022.007.patch, MAPREDUCE-7022.008.patch > > > With the introduction of MAPREDUCE-6489 there are some options to kill rogue > tasks based on writes to local disk writes. In our environment are we mainly > run Hive based jobs we noticed that this counter and the size of the local > scratch dirs were very different. We had tasks where BYTES_WRITTEN counter > were at 300Gb and where it was at 10Tb both producing around 200Gb on local > disk, so it didn't help us much. So to extend this feature tasks should > monitor local scratchdir size and fail if they pass the limit. In these cases > the tasks should not be retried either but instead the job should fast fail. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7020) Task timeout in uber mode can crash AM
[
https://issues.apache.org/jira/browse/MAPREDUCE-7020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16326290#comment-16326290
]
Peter Bacsko commented on MAPREDUCE-7020:
-
[~jlowe] yes you're right, there's still a tiny time-window which allows things
to go wrong.
I have an alternative idea: what about catching the exception that we throw? It
could be a specialized exception instead of {{IllegalStateException}} so we
know that the attempt has been removed from the map. If that happens, we just
exit from the loop and stop reporting.
> Task timeout in uber mode can crash AM
> --
>
> Key: MAPREDUCE-7020
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7020
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am
>Affects Versions: 3.1.0, 3.0.1, 2.10.0, 2.9.1, 2.8.4, 2.7.6
>Reporter: Akira Ajisaka
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: MAPREDUCE-7020-001.patch
>
>
> TestUberAM is failing
> {noformat}
> java.lang.AssertionError: No AppMaster log found! expected:<1> but was:<2>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at
> org.apache.hadoop.mapreduce.v2.TestMRJobs.testThreadDumpOnTaskTimeout(TestMRJobs.java:1228)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/614/testReport/junit/org.apache.hadoop.mapreduce.v2/TestUberAM/testThreadDumpOnTaskTimeout/
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7022) Fast fail rogue jobs based on task scratch dir size
[
https://issues.apache.org/jira/browse/MAPREDUCE-7022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16326182#comment-16326182
]
Hadoop QA commented on MAPREDUCE-7022:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 17s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 37s{color} | {color:orange}
hadoop-mapreduce-project/hadoop-mapreduce-client: The patch generated 12 new +
1144 unchanged - 3 fixed = 1156 total (was 1147) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 52s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 3m 19s{color}
| {color:red} hadoop-mapreduce-client-core in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
50s{color} | {color:green} hadoop-mapreduce-client-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
9s{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}134m 56s{color}
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
31s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}202m 7s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.mapred.TestTaskProgressReporter |
| | hadoop.mapreduce.v2.TestUberAM |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | MAPREDUCE-7022 |
[jira] [Commented] (MAPREDUCE-7021) Remove logger dependencies other than org.slf4j in hadoop-mapreduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-7021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16326134#comment-16326134 ] Takanobu Asanuma commented on MAPREDUCE-7021: - I filed the license error in MAPREDUCE-7036 and it is not related to this patch. +1 (non-binding) > Remove logger dependencies other than org.slf4j in hadoop-mapreduce > --- > > Key: MAPREDUCE-7021 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7021 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Attachments: MAPREDUCE-7021.001.patch > > > We moved logging APIs over to slf4j, however, the dependency is left. We need > to cut the dependency. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Moved] (MAPREDUCE-7036) ASF License warning in hadoop-mapreduce-client
[
https://issues.apache.org/jira/browse/MAPREDUCE-7036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takanobu Asanuma moved HADOOP-15174 to MAPREDUCE-7036:
--
Target Version/s: 3.1.0 (was: 3.1.0)
Component/s: (was: test)
test
Key: MAPREDUCE-7036 (was: HADOOP-15174)
Project: Hadoop Map/Reduce (was: Hadoop Common)
> ASF License warning in hadoop-mapreduce-client
> --
>
> Key: MAPREDUCE-7036
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7036
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Components: test
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Minor
>
> it occurred in MAPREDUCE-7021 and MAPREDUCE-7034.
> {noformat}
> Lines that start with ? in the ASF License report indicate files that do
> not have an Apache license header: !?
> /testptch/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/jobTokenPassword
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7035) Fix javadoc issues in hadoop-mapreduce-client
[
https://issues.apache.org/jira/browse/MAPREDUCE-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16326099#comment-16326099
]
Hadoop QA commented on MAPREDUCE-7035:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
28m 22s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 24s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green}
hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core
generated 0 new + 0 unchanged - 19 fixed = 0 total (was 19) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} hadoop-mapreduce-client-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} hadoop-mapreduce-client-shuffle in the patch passed.
{color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
15s{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
51s{color} | {color:green} hadoop-mapreduce-client-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
25s{color} | {color:green} hadoop-mapreduce-client-shuffle in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 61m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | MAPREDUCE-7035 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12906082/MAPREDUCE-7035.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient xml |
| uname | Linux 8c3ad0fc8096 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x8
[jira] [Commented] (MAPREDUCE-7021) Remove logger dependencies other than org.slf4j in hadoop-mapreduce
[
https://issues.apache.org/jira/browse/MAPREDUCE-7021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16326071#comment-16326071
]
Hadoop QA commented on MAPREDUCE-7021:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
10s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
1s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
41s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
28m 15s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 29s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}150m 22s{color}
| {color:red} hadoop-mapreduce-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
40s{color} | {color:green} hadoop-mapreduce-examples in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
25s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}198m 52s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.mapreduce.v2.TestUberAM |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | MAPREDUCE-7021 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12903170/MAPREDUCE-7021.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient xml |
| uname | Linux 51526a57c977 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 2e1e901 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| unit |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7298/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7298/testReport/ |
| asflicense |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7298/artifact/out/patch-asflicense-probl
[jira] [Updated] (MAPREDUCE-7035) Fix javadoc issues in hadoop-mapreduce-client
[
https://issues.apache.org/jira/browse/MAPREDUCE-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh updated MAPREDUCE-7035:
-
Status: Patch Available (was: Open)
> Fix javadoc issues in hadoop-mapreduce-client
> -
>
> Key: MAPREDUCE-7035
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7035
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
>Reporter: Mukul Kumar Singh
>Priority: Minor
> Attachments: MAPREDUCE-7035.001.patch
>
>
> Auto-generated code in map-reduce client throws many javadoc warnings.
> The warnings are thrown for the following auto-generated sources
> {code:java}
> hadoop-mapreduce-client-core/pom.xml
> target/generated-sources/avro/
> hadoop-mapreduce-client-common/pom.xml
> org.apache.hadoop.yarn.proto
> org.apache.hadoop.mapreduce.v2.proto
> org.apache.hadoop.mapreduce.v2.hs.proto
> hadoop-mapreduce-client-shuffle/pom.xml
> org.apache.hadoop.mapred.proto{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7035) Fix javadoc issues in hadoop-mapreduce-client
[
https://issues.apache.org/jira/browse/MAPREDUCE-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mukul Kumar Singh updated MAPREDUCE-7035:
-
Attachment: MAPREDUCE-7035.001.patch
> Fix javadoc issues in hadoop-mapreduce-client
> -
>
> Key: MAPREDUCE-7035
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7035
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
>Reporter: Mukul Kumar Singh
>Priority: Minor
> Attachments: MAPREDUCE-7035.001.patch
>
>
> Auto-generated code in map-reduce client throws many javadoc warnings.
> The warnings are thrown for the following auto-generated sources
> {code:java}
> hadoop-mapreduce-client-core/pom.xml
> target/generated-sources/avro/
> hadoop-mapreduce-client-common/pom.xml
> org.apache.hadoop.yarn.proto
> org.apache.hadoop.mapreduce.v2.proto
> org.apache.hadoop.mapreduce.v2.hs.proto
> hadoop-mapreduce-client-shuffle/pom.xml
> org.apache.hadoop.mapred.proto{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7035) Fix javadoc issues in hadoop-mapreduce-client
Mukul Kumar Singh created MAPREDUCE-7035:
Summary: Fix javadoc issues in hadoop-mapreduce-client
Key: MAPREDUCE-7035
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7035
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Reporter: Mukul Kumar Singh
Auto-generated code in map-reduce client throws many javadoc warnings.
The warnings are thrown for the following auto-generated sources
{code:java}
hadoop-mapreduce-client-core/pom.xml
target/generated-sources/avro/
hadoop-mapreduce-client-common/pom.xml
org.apache.hadoop.yarn.proto
org.apache.hadoop.mapreduce.v2.proto
org.apache.hadoop.mapreduce.v2.hs.proto
hadoop-mapreduce-client-shuffle/pom.xml
org.apache.hadoop.mapred.proto{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7022) Fast fail rogue jobs based on task scratch dir size
[ https://issues.apache.org/jira/browse/MAPREDUCE-7022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johan Gustavsson updated MAPREDUCE-7022: Attachment: MAPREDUCE-7022.008.patch > Fast fail rogue jobs based on task scratch dir size > --- > > Key: MAPREDUCE-7022 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7022 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: task >Affects Versions: 2.7.0, 2.8.0, 2.9.0 >Reporter: Johan Gustavsson >Assignee: Johan Gustavsson >Priority: Major > Attachments: MAPREDUCE-7022.001.patch, MAPREDUCE-7022.002.patch, > MAPREDUCE-7022.003.patch, MAPREDUCE-7022.004.patch, MAPREDUCE-7022.005.patch, > MAPREDUCE-7022.006.patch, MAPREDUCE-7022.007.patch, MAPREDUCE-7022.008.patch > > > With the introduction of MAPREDUCE-6489 there are some options to kill rogue > tasks based on writes to local disk writes. In our environment are we mainly > run Hive based jobs we noticed that this counter and the size of the local > scratch dirs were very different. We had tasks where BYTES_WRITTEN counter > were at 300Gb and where it was at 10Tb both producing around 200Gb on local > disk, so it didn't help us much. So to extend this feature tasks should > monitor local scratchdir size and fail if they pass the limit. In these cases > the tasks should not be retried either but instead the job should fast fail. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7022) Fast fail rogue jobs based on task scratch dir size
[
https://issues.apache.org/jira/browse/MAPREDUCE-7022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16326002#comment-16326002
]
Hadoop QA commented on MAPREDUCE-7022:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m
33s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
44s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 54s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
27s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 43s{color} | {color:orange}
hadoop-mapreduce-project/hadoop-mapreduce-client: The patch generated 12 new +
1144 unchanged - 3 fixed = 1156 total (was 1147) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 38s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 3m 19s{color}
| {color:red} hadoop-mapreduce-client-core in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
51s{color} | {color:green} hadoop-mapreduce-client-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
23s{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}130m 13s{color}
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
25s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}211m 47s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.mapred.TestTaskProgressReporter |
| | hadoop.mapreduce.v2.TestUberAM |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | MAPREDUCE-7022 |
