[jira] [Commented] (MAPREDUCE-6434) Add support for PartialFileOutputCommiter when checkpointing is an option during preemption
[ https://issues.apache.org/jira/browse/MAPREDUCE-6434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14723956#comment-14723956 ] Augusto Souza commented on MAPREDUCE-6434: -- [~chris.douglas], what do you think about the correction now? I tried to implement what you suggested me. > Add support for PartialFileOutputCommiter when checkpointing is an option > during preemption > --- > > Key: MAPREDUCE-6434 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6434 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Augusto Souza >Assignee: Augusto Souza > Attachments: MAPREDUCE-6434.001.patch, MAPREDUCE-6434.002.patch, > MAPREDUCE-6434.003.patch, MAPREDUCE-6434.004.patch, MAPREDUCE-6434.005.patch, > MAPREDUCE-6434.006.patch > > > Finish up some renaming work related to the annotation @Preemptable (it > should be @Checkpointable now) and help in the splitting of patch in > MAPREDUCE-5269 that is too large for being reviewed or accepted by Jenkins CI > scripts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6434) Add support for PartialFileOutputCommiter when checkpointing is an option during preemption
[ https://issues.apache.org/jira/browse/MAPREDUCE-6434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14717617#comment-14717617 ] Augusto Souza commented on MAPREDUCE-6434: -- What is the next step in order for this patch to be accepted? Add support for PartialFileOutputCommiter when checkpointing is an option during preemption --- Key: MAPREDUCE-6434 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6434 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Augusto Souza Assignee: Augusto Souza Attachments: MAPREDUCE-6434.001.patch, MAPREDUCE-6434.002.patch, MAPREDUCE-6434.003.patch, MAPREDUCE-6434.004.patch, MAPREDUCE-6434.005.patch, MAPREDUCE-6434.006.patch Finish up some renaming work related to the annotation @Preemptable (it should be @Checkpointable now) and help in the splitting of patch in MAPREDUCE-5269 that is too large for being reviewed or accepted by Jenkins CI scripts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6434) Add support for PartialFileOutputCommiter when checkpointing is an option during preemption
[ https://issues.apache.org/jira/browse/MAPREDUCE-6434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14716496#comment-14716496 ] Augusto Souza commented on MAPREDUCE-6434: -- [~chris.douglas] agreed and submitted a new version (006), but the Jenkins script didn't run to validate it, I don't know why. Also, performed tests with the map-only job I described above (with zero reducers). Is this patch acceptable now? Add support for PartialFileOutputCommiter when checkpointing is an option during preemption --- Key: MAPREDUCE-6434 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6434 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Augusto Souza Assignee: Augusto Souza Attachments: MAPREDUCE-6434.001.patch, MAPREDUCE-6434.002.patch, MAPREDUCE-6434.003.patch, MAPREDUCE-6434.004.patch, MAPREDUCE-6434.005.patch, MAPREDUCE-6434.006.patch Finish up some renaming work related to the annotation @Preemptable (it should be @Checkpointable now) and help in the splitting of patch in MAPREDUCE-5269 that is too large for being reviewed or accepted by Jenkins CI scripts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6434) Add support for PartialFileOutputCommiter when checkpointing is an option during preemption
[
https://issues.apache.org/jira/browse/MAPREDUCE-6434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14716552#comment-14716552
]
Hadoop QA commented on MAPREDUCE-6434:
--
\\
\\
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 16m 38s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 7m 58s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 4s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 47s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 27s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 35s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 26s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | mapreduce tests | 1m 45s | Tests passed in
hadoop-mapreduce-client-core. |
| | | 41m 8s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12751789/MAPREDUCE-6434.006.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 0bf2854 |
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5957/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5957/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5957/console |
This message was automatically generated.
Add support for PartialFileOutputCommiter when checkpointing is an option
during preemption
---
Key: MAPREDUCE-6434
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6434
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Augusto Souza
Assignee: Augusto Souza
Attachments: MAPREDUCE-6434.001.patch, MAPREDUCE-6434.002.patch,
MAPREDUCE-6434.003.patch, MAPREDUCE-6434.004.patch, MAPREDUCE-6434.005.patch,
MAPREDUCE-6434.006.patch
Finish up some renaming work related to the annotation @Preemptable (it
should be @Checkpointable now) and help in the splitting of patch in
MAPREDUCE-5269 that is too large for being reviewed or accepted by Jenkins CI
scripts.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6434) Add support for PartialFileOutputCommiter when checkpointing is an option during preemption
[
https://issues.apache.org/jira/browse/MAPREDUCE-6434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14707130#comment-14707130
]
Chris Douglas commented on MAPREDUCE-6434:
--
Offhand, I'd guess adding
{{TaskType.REDUCE.equals(context.getTaskAttemptID().getTaskType())}} to the
expression would prevent it from affecting more than reducers, but I haven't
looked into it. Could you test with a map-only job, where
{{context.getReducerClass()}} is undefined or not on the classpath?
Add support for PartialFileOutputCommiter when checkpointing is an option
during preemption
---
Key: MAPREDUCE-6434
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6434
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Augusto Souza
Assignee: Augusto Souza
Attachments: MAPREDUCE-6434.001.patch, MAPREDUCE-6434.002.patch,
MAPREDUCE-6434.003.patch, MAPREDUCE-6434.004.patch, MAPREDUCE-6434.005.patch
Finish up some renaming work related to the annotation @Preemptable (it
should be @Checkpointable now) and help in the splitting of patch in
MAPREDUCE-5269 that is too large for being reviewed or accepted by Jenkins CI
scripts.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6434) Add support for PartialFileOutputCommiter when checkpointing is an option during preemption
[
https://issues.apache.org/jira/browse/MAPREDUCE-6434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14707248#comment-14707248
]
Augusto Souza commented on MAPREDUCE-6434:
--
Thank you very much [~chris.douglas]!
I tested the version I submitted before with a map-only job, and I think the
IdentityReducer is used in cases with num reduce tasks setted to zero and no
setting for the reducer class, so the previous patch doesn't crash. Am I right
in this assumption? Is there another way of defining jobs in which I force the
{{context.getReducerClass()}} to get undefined?
But, I think your feedback is valid, so I am adding a another statement to the
expression to make sure only the PartialFileOutputCommiter can only be
instantiated for reduce tasks. If there is a way of making
{{context.getReducerClass()}} undefined, I can try to make better tests for the
patch too.
Add support for PartialFileOutputCommiter when checkpointing is an option
during preemption
---
Key: MAPREDUCE-6434
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6434
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Augusto Souza
Assignee: Augusto Souza
Attachments: MAPREDUCE-6434.001.patch, MAPREDUCE-6434.002.patch,
MAPREDUCE-6434.003.patch, MAPREDUCE-6434.004.patch, MAPREDUCE-6434.005.patch
Finish up some renaming work related to the annotation @Preemptable (it
should be @Checkpointable now) and help in the splitting of patch in
MAPREDUCE-5269 that is too large for being reviewed or accepted by Jenkins CI
scripts.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6434) Add support for PartialFileOutputCommiter when checkpointing is an option during preemption
[
https://issues.apache.org/jira/browse/MAPREDUCE-6434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14707498#comment-14707498
]
Chris Douglas commented on MAPREDUCE-6434:
--
Agreed, the NPE is usually not a problem since the default should be defined in
mapred-defaults, though {{JobContextImpl::getReducerClass}} can return null. At
least two cases shouldn't cause a problem for map-only jobs:
# The base {{mapreduce.Reducer}} is {{\@Checkpointable}}, so it would
instantiate a {{PartialFileOutputCommitter}}
# A {{Reducer}} in the config shouldn't cause a map-only job to fail if it's
not on the classpath (this may not be true in the current code, but this
shouldn't add another case)
We also don't want to do anything surprising for setup/cleanup tasks.
Add support for PartialFileOutputCommiter when checkpointing is an option
during preemption
---
Key: MAPREDUCE-6434
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6434
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Augusto Souza
Assignee: Augusto Souza
Attachments: MAPREDUCE-6434.001.patch, MAPREDUCE-6434.002.patch,
MAPREDUCE-6434.003.patch, MAPREDUCE-6434.004.patch, MAPREDUCE-6434.005.patch,
MAPREDUCE-6434.006.patch
Finish up some renaming work related to the annotation @Preemptable (it
should be @Checkpointable now) and help in the splitting of patch in
MAPREDUCE-5269 that is too large for being reviewed or accepted by Jenkins CI
scripts.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6434) Add support for PartialFileOutputCommiter when checkpointing is an option during preemption
[ https://issues.apache.org/jira/browse/MAPREDUCE-6434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706057#comment-14706057 ] Augusto Souza commented on MAPREDUCE-6434: -- [~chris.douglas], how do you suggest the treatment for when the reduce class is undefined? Should I catch the error and build a commiter with FileOutputCommiter? I am asking because I am new to hadoop, sorry. Thank you for the review :) Add support for PartialFileOutputCommiter when checkpointing is an option during preemption --- Key: MAPREDUCE-6434 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6434 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Augusto Souza Assignee: Augusto Souza Attachments: MAPREDUCE-6434.001.patch, MAPREDUCE-6434.002.patch, MAPREDUCE-6434.003.patch, MAPREDUCE-6434.004.patch, MAPREDUCE-6434.005.patch Finish up some renaming work related to the annotation @Preemptable (it should be @Checkpointable now) and help in the splitting of patch in MAPREDUCE-5269 that is too large for being reviewed or accepted by Jenkins CI scripts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6434) Add support for PartialFileOutputCommiter when checkpointing is an option during preemption
[
https://issues.apache.org/jira/browse/MAPREDUCE-6434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14706018#comment-14706018
]
Chris Douglas commented on MAPREDUCE-6434:
--
Thanks for updating the patch, [~augustorsouza]. Could you check this change?
{noformat}
- committer = new FileOutputCommitter(output, context);
+ try {
+if (context.getConfiguration().getBoolean(MRJobConfig.TASK_PREEMPTION,
+false)
+ context.getReducerClass()
+.isAnnotationPresent(Checkpointable.class)) {
+ committer = new PartialFileOutputCommitter(output, context);
+} else {
+ committer = new FileOutputCommitter(output, context);
+}
+ } catch (ClassNotFoundException c) {
+throw new RuntimeException(
+Internal error: reducer class is not defined , c);
+ }
{noformat}
Since preemption in MAPREDUCE-5269 only supports reduce tasks, even if
preemption is enabled for map-only jobs, the reduce class can be undefined.
Add support for PartialFileOutputCommiter when checkpointing is an option
during preemption
---
Key: MAPREDUCE-6434
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6434
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Augusto Souza
Assignee: Augusto Souza
Attachments: MAPREDUCE-6434.001.patch, MAPREDUCE-6434.002.patch,
MAPREDUCE-6434.003.patch, MAPREDUCE-6434.004.patch, MAPREDUCE-6434.005.patch
Finish up some renaming work related to the annotation @Preemptable (it
should be @Checkpointable now) and help in the splitting of patch in
MAPREDUCE-5269 that is too large for being reviewed or accepted by Jenkins CI
scripts.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6434) Add support for PartialFileOutputCommiter when checkpointing is an option during preemption
[ https://issues.apache.org/jira/browse/MAPREDUCE-6434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14698400#comment-14698400 ] Augusto Souza commented on MAPREDUCE-6434: -- Could anyone review the patch please? I need these first patch to be accept to be able to continue with future work. Add support for PartialFileOutputCommiter when checkpointing is an option during preemption --- Key: MAPREDUCE-6434 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6434 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Augusto Souza Assignee: Augusto Souza Attachments: MAPREDUCE-6434.001.patch, MAPREDUCE-6434.002.patch, MAPREDUCE-6434.003.patch, MAPREDUCE-6434.004.patch, MAPREDUCE-6434.005.patch Finish up some renaming work related to the annotation @Preemptable (it should be @Checkpointable now) and help in the splitting of patch in MAPREDUCE-5269 that is too large for being reviewed or accepted by Jenkins CI scripts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6434) Add support for PartialFileOutputCommiter when checkpointing is an option during preemption
[
https://issues.apache.org/jira/browse/MAPREDUCE-6434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14650429#comment-14650429
]
Hadoop QA commented on MAPREDUCE-6434:
--
\\
\\
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 16m 29s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 7m 46s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 44s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 46s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 23s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 26s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | mapreduce tests | 1m 46s | Tests passed in
hadoop-mapreduce-client-core. |
| | | 40m 20s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12748314/MAPREDUCE-6434.005.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 90b5104 |
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5925/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5925/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5925/console |
This message was automatically generated.
Add support for PartialFileOutputCommiter when checkpointing is an option
during preemption
---
Key: MAPREDUCE-6434
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6434
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Augusto Souza
Assignee: Augusto Souza
Priority: Trivial
Attachments: MAPREDUCE-6434.001.patch, MAPREDUCE-6434.002.patch,
MAPREDUCE-6434.003.patch, MAPREDUCE-6434.004.patch, MAPREDUCE-6434.005.patch
Finish up some renaming work related to the annotation @Preemptable (it
should be @Checkpointable now) and help in the splitting of patch in
MAPREDUCE-5269 that is too large for being reviewed or accepted by Jenkins CI
scripts.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6434) Add support for PartialFileOutputCommiter when checkpointing is an option during preemption
[ https://issues.apache.org/jira/browse/MAPREDUCE-6434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14648396#comment-14648396 ] Augusto Souza commented on MAPREDUCE-6434: -- Just to be clear, I think those issues are unrelated to the changes I performed, by my tests they are from trunk and not from my patch, is that ok to submit the patch this way? Anyone could review the issue for me please? Add support for PartialFileOutputCommiter when checkpointing is an option during preemption --- Key: MAPREDUCE-6434 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6434 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Augusto Souza Assignee: Augusto Souza Priority: Trivial Attachments: MAPREDUCE-6434.001.patch, MAPREDUCE-6434.002.patch Finish up some renaming work related to the annotation @Preemptable (it should be @Checkpointable now) and help in the splitting of patch in MAPREDUCE-5269 that is too large for being reviewed or accepted by Jenkins CI scripts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6434) Add support for PartialFileOutputCommiter when checkpointing is an option during preemption
[
https://issues.apache.org/jira/browse/MAPREDUCE-6434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14648625#comment-14648625
]
Hadoop QA commented on MAPREDUCE-6434:
--
\\
\\
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 10s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 8m 9s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 17s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 21s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 49s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 30s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 31s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | mapreduce tests | 1m 50s | Tests passed in
hadoop-mapreduce-client-core. |
| | | 42m 14s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12748103/MAPREDUCE-6434.004.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 88d8736 |
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5919/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5919/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5919/console |
This message was automatically generated.
Add support for PartialFileOutputCommiter when checkpointing is an option
during preemption
---
Key: MAPREDUCE-6434
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6434
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Augusto Souza
Assignee: Augusto Souza
Priority: Trivial
Attachments: MAPREDUCE-6434.001.patch, MAPREDUCE-6434.002.patch,
MAPREDUCE-6434.003.patch, MAPREDUCE-6434.004.patch
Finish up some renaming work related to the annotation @Preemptable (it
should be @Checkpointable now) and help in the splitting of patch in
MAPREDUCE-5269 that is too large for being reviewed or accepted by Jenkins CI
scripts.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6434) Add support for PartialFileOutputCommiter when checkpointing is an option during preemption
[
https://issues.apache.org/jira/browse/MAPREDUCE-6434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14648442#comment-14648442
]
Hadoop QA commented on MAPREDUCE-6434:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12748084/MAPREDUCE-6434.003.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 88d8736 |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5918/console |
This message was automatically generated.
Add support for PartialFileOutputCommiter when checkpointing is an option
during preemption
---
Key: MAPREDUCE-6434
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6434
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Augusto Souza
Assignee: Augusto Souza
Priority: Trivial
Attachments: MAPREDUCE-6434.001.patch, MAPREDUCE-6434.002.patch,
MAPREDUCE-6434.003.patch
Finish up some renaming work related to the annotation @Preemptable (it
should be @Checkpointable now) and help in the splitting of patch in
MAPREDUCE-5269 that is too large for being reviewed or accepted by Jenkins CI
scripts.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6434) Add support for PartialFileOutputCommiter when checkpointing is an option during preemption
[ https://issues.apache.org/jira/browse/MAPREDUCE-6434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14645107#comment-14645107 ] Augusto Souza commented on MAPREDUCE-6434: -- Can anyone review this patch for me, please? Add support for PartialFileOutputCommiter when checkpointing is an option during preemption --- Key: MAPREDUCE-6434 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6434 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Augusto Souza Assignee: Augusto Souza Priority: Trivial Attachments: MAPREDUCE-6434.001.patch, MAPREDUCE-6434.002.patch Finish up some renaming work related to the annotation @Preemptable (it should be @Checkpointable now) and help in the splitting of patch in MAPREDUCE-5269 that is too large for being reviewed or accepted by Jenkins CI scripts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6434) Add support for PartialFileOutputCommiter when checkpointing is an option during preemption
[
https://issues.apache.org/jira/browse/MAPREDUCE-6434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14630749#comment-14630749
]
Hadoop QA commented on MAPREDUCE-6434:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 24m 29s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 3 new or modified test files. |
| {color:green}+1{color} | javac | 7m 48s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 51s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | site | 2m 59s | Site still builds. |
| {color:green}+1{color} | checkstyle | 3m 47s | There were no new checkstyle
issues. |
| {color:red}-1{color} | whitespace | 0m 4s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 20s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 6m 49s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | common tests | 22m 31s | Tests passed in
hadoop-common. |
| {color:green}+1{color} | mapreduce tests | 9m 2s | Tests passed in
hadoop-mapreduce-client-app. |
| {color:green}+1{color} | mapreduce tests | 1m 47s | Tests passed in
hadoop-mapreduce-client-core. |
| {color:red}-1{color} | hdfs tests | 99m 47s | Tests failed in hadoop-hdfs. |
| | | 191m 15s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.shortcircuit.TestShortCircuitCache |
| | hadoop.hdfs.TestDistributedFileSystem |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12745728/MAPREDUCE-6434.002.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle site |
| git revision | trunk / ee36f4f |
| whitespace |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5893/artifact/patchprocess/whitespace.txt
|
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5893/artifact/patchprocess/testrun_hadoop-common.txt
|
| hadoop-mapreduce-client-app test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5893/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
|
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5893/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5893/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5893/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5893/console |
This message was automatically generated.
Add support for PartialFileOutputCommiter when checkpointing is an option
during preemption
---
Key: MAPREDUCE-6434
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6434
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Augusto Souza
Assignee: Augusto Souza
Priority: Trivial
Attachments: MAPREDUCE-6434.001.patch, MAPREDUCE-6434.002.patch
Finish up some renaming work related to the annotation @Preemptable (it
should be @Checkpointable now) and help in the splitting of patch in
MAPREDUCE-5269 that is too large for being reviewed or accepted by Jenkins CI
scripts.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6434) Add support for PartialFileOutputCommiter when checkpointing is an option during preemption
[
https://issues.apache.org/jira/browse/MAPREDUCE-6434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14629073#comment-14629073
]
Hadoop QA commented on MAPREDUCE-6434:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 16m 0s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 7m 39s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 36s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 46s | The applied patch generated 2
new checkstyle issues (total was 40, now 9). |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 24s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 25s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | mapreduce tests | 1m 45s | Tests passed in
hadoop-mapreduce-client-core. |
| | | 39m 36s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12745554/MAPREDUCE-6434.001.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 3ec0a04 |
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5890/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
|
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5890/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5890/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5890/console |
This message was automatically generated.
Add support for PartialFileOutputCommiter when checkpointing is an option
during preemption
---
Key: MAPREDUCE-6434
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6434
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Augusto Souza
Assignee: Augusto Souza
Priority: Trivial
Attachments: MAPREDUCE-6434.001.patch
Finish up some renaming work related to the annotation @Preemptable (it
should be @Checkpointable now) and help in the splitting of patch in
MAPREDUCE-5269 that is too large for being reviewed or accepted by Jenkins CI
scripts.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6434) Add support for PartialFileOutputCommiter when checkpointing is an option during preemption
[ https://issues.apache.org/jira/browse/MAPREDUCE-6434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14628935#comment-14628935 ] Augusto Souza commented on MAPREDUCE-6434: -- This JIRA will finish up some renaming work from of annotation @Preemptable to @Checkpointable as decided in MAPREDUCE-5176 Add support for PartialFileOutputCommiter when checkpointing is an option during preemption --- Key: MAPREDUCE-6434 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6434 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Augusto Souza Assignee: Augusto Souza Priority: Trivial -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6434) Add support for PartialFileOutputCommiter when checkpointing is an option during preemption
[ https://issues.apache.org/jira/browse/MAPREDUCE-6434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14628931#comment-14628931 ] Augusto Souza commented on MAPREDUCE-6434: -- In the comments of MAPREDUCE-5269 we decided to split it into other JIRAs in order to make review easier and make the patch for MAPREDUCE-5269 more granular. Add support for PartialFileOutputCommiter when checkpointing is an option during preemption --- Key: MAPREDUCE-6434 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6434 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Augusto Souza Assignee: Augusto Souza Priority: Trivial -- This message was sent by Atlassian JIRA (v6.3.4#6332)
