[
https://issues.apache.org/jira/browse/MAPREDUCE-7282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17199388#comment-17199388
]
Hadoop QA commented on MAPREDUCE-7282:
--------------------------------------
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m
1s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 34m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 13s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
28s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
24s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
38s{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 38s{color}
| {color:red}
hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 generated 1 new + 150
unchanged - 0 fixed = 151 total (was 150) {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 33s{color}
| {color:red}
hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01
with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 generated 1 new
+ 147 unchanged - 0 fixed = 148 total (was 147) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 23s{color} | {color:orange}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core:
The patch generated 5 new + 19 unchanged - 5 fixed = 24 total (was 24) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} xml {color} | {color:red} 0m 2s{color} |
{color:red} The patch has 1 ill-formed XML file(s). {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 17m
5s{color} | {color:red} patch has errors when building and testing our client
artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed with JDK Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 12s{color}
| {color:red} hadoop-mapreduce-client-core in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 86m 12s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| XML | Parsing Error(s): |
| |
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
|
| Failed junit tests | hadoop.mapreduce.TestJobSubmissionFiles |
| | hadoop.mapred.lib.TestCombineFileRecordReader |
| | hadoop.mapreduce.lib.output.TestFileOutputFormat |
| | hadoop.mapred.TestJobAclsManager |
| | hadoop.mapreduce.lib.input.TestFileInputFormat |
| | hadoop.mapreduce.TestJobResourceUploaderWithSharedCache |
| | hadoop.mapred.TestCounters |
| | hadoop.mapreduce.lib.input.TestCombineFileRecordReader |
| | hadoop.mapreduce.lib.jobcontrol.TestJobControl |
| | hadoop.mapred.lib.db.TestDBInputFormat |
| | hadoop.mapreduce.TestShufflePlugin |
| | hadoop.mapreduce.lib.input.TestLineRecordReader |
| | hadoop.mapreduce.task.reduce.TestMergeManager |
| | hadoop.mapred.TestTaskProgressReporter |
| | hadoop.mapreduce.TestContextFactory |
| | hadoop.mapreduce.lib.output.TestPathOutputCommitterFactory |
| | hadoop.mapreduce.tools.TestCLI |
| | hadoop.mapred.TestTaskLog |
| | hadoop.mapreduce.lib.output.TestFileOutputCommitter |
| | hadoop.mapreduce.task.reduce.TestFetcher |
| | hadoop.mapreduce.task.reduce.TestMerger |
| | hadoop.mapred.TestMapTask |
| | hadoop.mapred.TestFileOutputCommitter |
| | hadoop.mapreduce.security.TestTokenCache |
| | hadoop.mapreduce.TestJobResourceUploader |
| | hadoop.mapred.TestIndexCache |
| | hadoop.mapreduce.split.TestJobSplitWriterWithEC |
| | hadoop.mapreduce.split.TestJobSplitWriter |
| | hadoop.mapred.TestLineRecordReader |
| | hadoop.mapreduce.task.reduce.TestShuffleScheduler |
| | hadoop.mapreduce.lib.db.TestDbClasses |
| | hadoop.mapred.TestFileInputFormat |
| | hadoop.mapred.TestJobEndNotifier |
| | hadoop.mapreduce.filecache.TestClientDistributedCacheManager |
| | hadoop.mapred.TestJobConf |
| | hadoop.mapreduce.TestJobMonitorAndPrint |
| | hadoop.mapred.TestTask |
| | hadoop.mapred.TestJobInfo |
| | hadoop.mapred.TestSkipBadRecords |
| | hadoop.mapreduce.TestCluster |
| | hadoop.mapreduce.filecache.TestDistributedCache |
| | hadoop.mapreduce.TestJob |
| | hadoop.mapred.TestQueue |
| | hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2320/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2320 |
| JIRA Issue | MAPREDUCE-7282 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle xml |
| uname | Linux e88d81d7096d 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 7a6265ac425 |
| Default Java | Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 |
| javac |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2320/1/artifact/out/diff-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdkUbuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1.txt
|
| javac |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2320/1/artifact/out/diff-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdkPrivateBuild-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01.txt
|
| checkstyle |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2320/1/artifact/out/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
|
| xml |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2320/1/artifact/out/xml.txt
|
| unit |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2320/1/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
|
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2320/1/testReport/ |
| Max. process+thread count | 307 (vs. ulimit of 5500) |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
U:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2320/1/console |
| versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
> MR v2 commit algorithm is dangerous, should be deprecated and not the default
> -----------------------------------------------------------------------------
>
> Key: MAPREDUCE-7282
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7282
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 3.3.0, 3.2.1, 3.1.3, 3.3.1
> Reporter: Steve Loughran
> Priority: Major
>
> The v2 MR commit algorithm moves files from the task attempt dir into the
> dest dir on task commit -one by one
> It is therefore not atomic
> # if a task commit fails partway through and another task attempt commits
> -unless exactly the same filenames are used, output of the first attempt may
> be included in the final result
> # if a worker partitions partway through task commit, and then continues
> after another attempt has committed, it may partially overwrite the output
> -even when the filenames are the same
> Both MR and spark assume that task commits are atomic. Either they need to
> consider that this is not the case, we add a way to probe for a committer
> supporting atomic task commit, and the engines both add handling for task
> commit failures (probably fail job)
> Better: we remove this as the default, maybe also warn when it is being used
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
