[jira] [Updated] (MAPREDUCE-6773) Implement RM Container Reuse Requestor to handle the reuse containers for resource requests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6773: - Attachment: MAPREDUCE-6773-v2.patch Updated with checkstyle and javac warning fix. > Implement RM Container Reuse Requestor to handle the reuse containers for > resource requests > --- > > Key: MAPREDUCE-6773 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6773 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6773-v0.patch, MAPREDUCE-6773-v1.patch, > MAPREDUCE-6773-v2.patch > > > Add RM Container Reuse Requestor which handles the reuse containers against > the Job reource requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6773) Implement RM Container Reuse Requestor to handle the reuse containers for resource requests
[
https://issues.apache.org/jira/browse/MAPREDUCE-6773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15550885#comment-15550885
]
Hadoop QA commented on MAPREDUCE-6773:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 51s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
47s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 37s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 56s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
21s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
42s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 31s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 31s {color}
| {color:red} hadoop-mapreduce-project_hadoop-mapreduce-client generated 3 new
+ 362 unchanged - 2 fixed = 365 total (was 364) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 34s
{color} | {color:red} hadoop-mapreduce-project/hadoop-mapreduce-client: The
patch generated 2 new + 841 unchanged - 4 fixed = 843 total (was 845) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 11s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 51s {color}
| {color:red} hadoop-mapreduce-client-app in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 50s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.mapreduce.v2.app.webapp.TestAMWebApp |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12831886/MAPREDUCE-6773-v1.patch
|
| JIRA Issue | MAPREDUCE-6773 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux b762d19d5908 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / c5ca216 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| javac |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6753/artifact/patchprocess/diff-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client.txt
|
| checkstyle |
[jira] [Commented] (MAPREDUCE-6749) MR AM should reuse containers for Map/Reduce Tasks
[
https://issues.apache.org/jira/browse/MAPREDUCE-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15550874#comment-15550874
]
Devaraj K commented on MAPREDUCE-6749:
--
Thanks [~Naganarasimha] for looking into this and looking forward to work on
this with you.
{quote}
Is it required to have number of total Maps(/reducers) to be run using
container reuse? i felt we can just keep whether to reuse the containers are
not and one assumption that map containers will be reused only for map and
reduce only for reduce
{quote}
I think the limit configuration for no of map/reduce reuse containers would
allow other applications to start running without waiting for the Job to be
finished when reuse is enabled. If there is a big Job running which could
occupy the entire cluster, and then any high priority application gets
submitted this limit for maps/reduce container would probably give a room for
high priority application to start running without preempting the containers of
the previous Job. By default there is no limit for number of containers to be
reused and if any user/Job wanted to have this constraint they can configure
it.
{quote}
did not understand the need of RMContainerReuseRequestor.java, may be i will go
through MAPREDUCE-6773 but was expecting more like just TaskAttemptIimpl gets
TA_DONE event was expecting it do have a new ReuseContainerTransition where in
if required it will reassign a new task to it else will continue to
CLEANUP_CONTAINER_TRANSITION. may be we can discuss more on this in
MAPREDUCE-6773 ?
{quote}
RMContainerReuseRequestor has the functionality for checking the container
reuse constraints and adding to reuse containers if it is eligible. This class
abstracts the reuse functionality from regular RMContainerRequestor and
RMContainerReuseRequestor will be used only when the RMContainer reuse is
enabled. If you want to have a try this feature, you can apply MAPREDUCE-6773,
MAPREDUCE-6781, MAPREDUCE-6784, MAPREDUCE-6785, MAPREDUCE-6786 and then try
this feature. I would be happy to get your feedback or suggestions if any.
{quote}
If we have some conclusion on issue 2, then we can think of whether only
TaskAttemptImpl state is sufficient or we need to modify others too.
{quote}
Sure, We can discuss it further if required.
{quote} Logs generated by the tasks in a container will be appended in the same
container log file we need to ensure we are able to display trhough JHS and
other places where ever required to display
{quote}
TaskAttempt is linked to the container and the corresponding container log
would be displayed for TaskAttempt through the JHS. Here we should note that
the whole container log which is displaying for TaskAttempt is not applicable
to the TaskAttempt and the log can be identified easily which part applicable
to it.
> MR AM should reuse containers for Map/Reduce Tasks
> --
>
> Key: MAPREDUCE-6749
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6749
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: applicationmaster, mrv2
>Reporter: Devaraj K
>Assignee: Devaraj K
> Attachments: MAPREDUCE-6749-Container Reuse-v0.pdf
>
>
> It is with the continuation of MAPREDUCE-3902, MR AM should reuse containers
> for Map/Reduce Tasks similar to the JVM Reuse feature we had in MRv1.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6773) Implement RM Container Reuse Requestor to handle the reuse containers for resource requests
[ https://issues.apache.org/jira/browse/MAPREDUCE-6773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-6773: - Attachment: MAPREDUCE-6773-v1.patch Rebased the stale patch. > Implement RM Container Reuse Requestor to handle the reuse containers for > resource requests > --- > > Key: MAPREDUCE-6773 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6773 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6773-v0.patch, MAPREDUCE-6773-v1.patch > > > Add RM Container Reuse Requestor which handles the reuse containers against > the Job reource requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6789) TestAMWebApp fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-6789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15550055#comment-15550055
]
Hadoop QA commented on MAPREDUCE-6789:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
31s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 34s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
48s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 1s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
32s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 13s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 13s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
34s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 46s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 23s
{color} | {color:green} hadoop-yarn-server-web-proxy in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 48s
{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 43m 26s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:9560f25 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12831808/MAPREDUCE-6789.001.patch
|
| JIRA Issue | MAPREDUCE-6789 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 034880c0008b 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 00160f7 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6752/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U:
. |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6752/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> TestAMWebApp fails
> --
>
>
[jira] [Updated] (MAPREDUCE-6789) TestAMWebApp fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-6789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated MAPREDUCE-6789:
Status: Patch Available (was: Open)
> TestAMWebApp fails
> --
>
> Key: MAPREDUCE-6789
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6789
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Akira Ajisaka
>Assignee: Daniel Templeton
> Attachments: MAPREDUCE-6789.001.patch
>
>
> TestAMWebApp.testMRWebAppRedirection fails.
> {noformat}
> Running org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebApp
> Tests run: 12, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 27.337 sec
> <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebApp
> testMRWebAppRedirection(org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebApp)
> Time elapsed: 0.854 sec <<< FAILURE!
> org.junit.ComparisonFailure:
> expected:<...ttp://9.9.9.9/proxy/[]application_0_/m...> but
> was:<...ttp://9.9.9.9/proxy/[redirect/]application_0_/m...>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebApp.testMRWebAppRedirection(TestAMWebApp.java:253)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6789) TestAMWebApp fails
[
https://issues.apache.org/jira/browse/MAPREDUCE-6789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Templeton updated MAPREDUCE-6789:
Attachment: MAPREDUCE-6789.001.patch
This should resolve the issue.
> TestAMWebApp fails
> --
>
> Key: MAPREDUCE-6789
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6789
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Reporter: Akira Ajisaka
>Assignee: Daniel Templeton
> Attachments: MAPREDUCE-6789.001.patch
>
>
> TestAMWebApp.testMRWebAppRedirection fails.
> {noformat}
> Running org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebApp
> Tests run: 12, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 27.337 sec
> <<< FAILURE! - in org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebApp
> testMRWebAppRedirection(org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebApp)
> Time elapsed: 0.854 sec <<< FAILURE!
> org.junit.ComparisonFailure:
> expected:<...ttp://9.9.9.9/proxy/[]application_0_/m...> but
> was:<...ttp://9.9.9.9/proxy/[redirect/]application_0_/m...>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebApp.testMRWebAppRedirection(TestAMWebApp.java:253)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6740) Enforce mapreduce.task.timeout to be at least mapreduce.task.progress-report.interval
[ https://issues.apache.org/jira/browse/MAPREDUCE-6740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15549747#comment-15549747 ] Haibo Chen commented on MAPREDUCE-6740: --- The unit test failures are unrelated to this patch. > Enforce mapreduce.task.timeout to be at least > mapreduce.task.progress-report.interval > - > > Key: MAPREDUCE-6740 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6740 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.8.0 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Minor > Fix For: 3.0.0-alpha2 > > Attachments: mapreduce6740.001.patch, mapreduce6740.002.patch, > mapreduce6740.003.patch, mapreduce6740.004.patch, mapreduce6740.005.patch, > mapreduce6740.006.patch, mapreduce6740.007.patch, mapreduce6740.008.patch, > mapreduce6740.branch-2.8.patch > > > MAPREDUCE-6242 makes task status update interval configurable to ease the > pressure on MR AM to process status updates, but it did not ensure that > mapreduce.task.timeout is no smaller than the configured value of task report > interval. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6749) MR AM should reuse containers for Map/Reduce Tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-6749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15549700#comment-15549700 ] Naganarasimha G R commented on MAPREDUCE-6749: -- Thanks [~devaraj.k] for the document and raising these jira's, it has been always discussing but no progress on this topic lets try to make some progress for this feature here . Few queries: # Is it required to have number of total Maps(/reducers) to be run using container reuse? i felt we can just keep whether to reuse the containers are not and one assumption that map containers will be reused only for map and reduce only for reduce # did not understand the need of *RMContainerReuseRequestor.java*, may be i will go through MAPREDUCE-6773 but was expecting more like just TaskAttemptIimpl gets TA_DONE event was expecting it do have a new ReuseContainerTransition where in if required it will reassign a new task to it else will continue to *CLEANUP_CONTAINER_TRANSITION*. may be we can discuss more on this in MAPREDUCE-6773 ? # If we have some conclusion on issue 2, then we can think of whether only *TaskAttemptImpl* state is sufficient or we need to modify others too. # *Logs generated by the tasks in a container will be appended in the same container log file* we need to ensure we are able to display trhough JHS and other places where ever required to display > MR AM should reuse containers for Map/Reduce Tasks > -- > > Key: MAPREDUCE-6749 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6749 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster, mrv2 >Reporter: Devaraj K >Assignee: Devaraj K > Attachments: MAPREDUCE-6749-Container Reuse-v0.pdf > > > It is with the continuation of MAPREDUCE-3902, MR AM should reuse containers > for Map/Reduce Tasks similar to the JVM Reuse feature we had in MRv1. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6740) Enforce mapreduce.task.timeout to be at least mapreduce.task.progress-report.interval
[
https://issues.apache.org/jira/browse/MAPREDUCE-6740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15549696#comment-15549696
]
Hadoop QA commented on MAPREDUCE-6740:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
37s {color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 2s
{color} | {color:green} branch-2.8 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 14s
{color} | {color:green} branch-2.8 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s {color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 59s
{color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s {color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s {color} | {color:green} branch-2.8 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s
{color} | {color:green} branch-2.8 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s
{color} | {color:green} branch-2.8 passed with JDK v1.7.0_111 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 57s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 57s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 13s
{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 13s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s {color} | {color:green} hadoop-mapreduce-project/hadoop-mapreduce-client:
The patch generated 0 new + 724 unchanged - 3 fixed = 724 total (was 727)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 54s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s
{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 57s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with
JDK v1.8.0_101. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 6s {color} |
{color:red} hadoop-mapreduce-client-app in the patch failed with JDK
v1.8.0_101. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 19s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with
JDK v1.7.0_111. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m 45s
{color} | {color:green} hadoop-mapreduce-client-app in the patch passed with
JDK v1.7.0_111. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} |
[jira] [Updated] (MAPREDUCE-6740) Enforce mapreduce.task.timeout to be at least mapreduce.task.progress-report.interval
[ https://issues.apache.org/jira/browse/MAPREDUCE-6740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated MAPREDUCE-6740: -- Attachment: mapreduce6740.branch-2.8.patch Uploading a patch for branch-2.8 only > Enforce mapreduce.task.timeout to be at least > mapreduce.task.progress-report.interval > - > > Key: MAPREDUCE-6740 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6740 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.8.0 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Minor > Fix For: 3.0.0-alpha2 > > Attachments: mapreduce6740.001.patch, mapreduce6740.002.patch, > mapreduce6740.003.patch, mapreduce6740.004.patch, mapreduce6740.005.patch, > mapreduce6740.006.patch, mapreduce6740.007.patch, mapreduce6740.008.patch, > mapreduce6740.branch-2.8.patch > > > MAPREDUCE-6242 makes task status update interval configurable to ease the > pressure on MR AM to process status updates, but it did not ensure that > mapreduce.task.timeout is no smaller than the configured value of task report > interval. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6740) Enforce mapreduce.task.timeout to be at least mapreduce.task.progress-report.interval
[ https://issues.apache.org/jira/browse/MAPREDUCE-6740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15549359#comment-15549359 ] Haibo Chen commented on MAPREDUCE-6740: --- [~kasha], Jason has committed the branch-2.8 patch in MR-6741. You should be able to commit this patch cleanly to 2.8 now. > Enforce mapreduce.task.timeout to be at least > mapreduce.task.progress-report.interval > - > > Key: MAPREDUCE-6740 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6740 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.8.0 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Minor > Fix For: 3.0.0-alpha2 > > Attachments: mapreduce6740.001.patch, mapreduce6740.002.patch, > mapreduce6740.003.patch, mapreduce6740.004.patch, mapreduce6740.005.patch, > mapreduce6740.006.patch, mapreduce6740.007.patch, mapreduce6740.008.patch > > > MAPREDUCE-6242 makes task status update interval configurable to ease the > pressure on MR AM to process status updates, but it did not ensure that > mapreduce.task.timeout is no smaller than the configured value of task report > interval. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6741) add MR support to redact job conf properties
[ https://issues.apache.org/jira/browse/MAPREDUCE-6741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-6741: -- Resolution: Fixed Status: Resolved (was: Patch Available) > add MR support to redact job conf properties > > > Key: MAPREDUCE-6741 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6741 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 2.7.2 >Reporter: Haibo Chen >Assignee: Haibo Chen > Fix For: 2.8.0, 2.7.4, 3.0.0-alpha1 > > Attachments: mapreduce6741.001.patch, mapreduce6741.002.patch, > mapreduce6741.003.patch, mapreduce6741.004.patch, mapreduce6741.005.patch, > mapreduce6741.006.patch, mapreduce6741.branch-2.8.patch > > > JHS today displays all Job conf properties in Web UI directly. Users may have > some credentials or any sensitive information they added to the job conf but > do not want to be shown in Web UI. It'd be nice if we can allow users to > specify a set of properties which JHS will filter out when Job conf is > displayed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6741) add MR support to redact job conf properties
[ https://issues.apache.org/jira/browse/MAPREDUCE-6741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-6741: -- Fix Version/s: (was: 2.9.0) 2.7.4 2.8.0 Committed to branch-2.8 and branch-2.7. > add MR support to redact job conf properties > > > Key: MAPREDUCE-6741 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6741 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 2.7.2 >Reporter: Haibo Chen >Assignee: Haibo Chen > Fix For: 2.8.0, 2.7.4, 3.0.0-alpha1 > > Attachments: mapreduce6741.001.patch, mapreduce6741.002.patch, > mapreduce6741.003.patch, mapreduce6741.004.patch, mapreduce6741.005.patch, > mapreduce6741.006.patch, mapreduce6741.branch-2.8.patch > > > JHS today displays all Job conf properties in Web UI directly. Users may have > some credentials or any sensitive information they added to the job conf but > do not want to be shown in Web UI. It'd be nice if we can allow users to > specify a set of properties which JHS will filter out when Job conf is > displayed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6741) add MR support to redact job conf properties
[ https://issues.apache.org/jira/browse/MAPREDUCE-6741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15549058#comment-15549058 ] Jason Lowe commented on MAPREDUCE-6741: --- +1 for the branch-2.8 patch. The TestJobImpl failure appears to be unrelated, and the test passes for me locally with the patch applied. > add MR support to redact job conf properties > > > Key: MAPREDUCE-6741 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6741 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 2.7.2 >Reporter: Haibo Chen >Assignee: Haibo Chen > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: mapreduce6741.001.patch, mapreduce6741.002.patch, > mapreduce6741.003.patch, mapreduce6741.004.patch, mapreduce6741.005.patch, > mapreduce6741.006.patch, mapreduce6741.branch-2.8.patch > > > JHS today displays all Job conf properties in Web UI directly. Users may have > some credentials or any sensitive information they added to the job conf but > do not want to be shown in Web UI. It'd be nice if we can allow users to > specify a set of properties which JHS will filter out when Job conf is > displayed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
