[
https://issues.apache.org/jira/browse/MAPREDUCE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17316437#comment-17316437
]
Hadoop QA commented on MAPREDUCE-7329:
--------------------------------------
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
49s{color} | | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | | {color:green} No case conflicting files found. {color} |
| {color:blue}0{color} | {color:blue} codespell {color} | {color:blue} 0m
1s{color} | | {color:blue} codespell was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | | {color:green} The patch does not contain any @author tags.
{color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | | {color:green} The patch appears to include 1 new or modified
test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 13m
48s{color} | | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
0s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
59s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
28s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
18s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 2m
19s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 46s{color} | | {color:green} branch has no errors when building and
testing our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
2s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
33s{color} | | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
33s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
4s{color} | | {color:green} the patch passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
4s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} blanks {color} | {color:green} 0m
0s{color} | | {color:green} The patch has no blanks issues. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 52s{color} |
[/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt|https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/6/artifact/out/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt]
| {color:orange} hadoop-mapreduce-project/hadoop-mapreduce-client: The patch
generated 11 new + 7 unchanged - 0 fixed = 18 total (was 7) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
1s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | | {color:green} the patch passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 2m
13s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 10s{color} | | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} || ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 39s{color}
|
[/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt|https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/6/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt]
| {color:red} hadoop-mapreduce-client-core in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}130m
25s{color} | | {color:green} hadoop-mapreduce-client-jobclient in the patch
passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
35s{color} |
[/results-asflicense.txt|https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/6/artifact/out/results-asflicense.txt]
| {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}236m 43s{color} |
| {color:black}{color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.mapred.TestJobEndNotifier |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/2775 |
| JIRA Issue | MAPREDUCE-7329 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient spotbugs checkstyle codespell |
| uname | Linux cb7456b0eb4c 4.15.0-126-generic #129-Ubuntu SMP Mon Nov 23
18:53:38 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2012f3038f5ef44bb1db62cd239927503b1c4bf1 |
| Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/6/testReport/ |
| Max. process+thread count | 1176 (vs. ulimit of 5500) |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
U: hadoop-mapreduce-project/hadoop-mapreduce-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/6/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
> HadoopPipes task failed as a result of ping timeout exception
> -------------------------------------------------------------
>
> Key: MAPREDUCE-7329
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7329
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 2.6.0
> Reporter: chaoli
> Priority: Major
> Labels: patch, pull-request-available
> Fix For: 2.6.0, 3.0.0
>
> Attachments:
> 0001-MAPREDUCE-7329-HadoopPipes-task-may-fail-when-linux-.patch,
> image-2021-03-15-14-29-49-475.png, image-2021-03-15-14-37-32-184.png
>
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> {color:#FF0000}*Hadoop Pipes Ping implement has a bug*{color}. Recently, we
> upgrade linux kernel version from 3.x to 4.x. And we find hadoop pipe task
> exit with connect timeout which is implemented by PingThread in
> HadoopPipes.cc.
> !image-2021-03-15-14-37-32-184.png!
> After a deep research, we finally find that current ping server won't accept
> ping client created socket, which may cause critical problem:
> * it will cause tcp accept queue full(default 50)
> * when client close socket, server socket won't call close method, which
> will leave too many CLOSE_WAIT socket fd existed(default 2h), and accept
> queue never cleared.
> * Even worse, in 4.x linux kernel version, it will cause tcp drop packet
> directly which makes ping client connect time out. While In 3.x linux kernel
> version, when accept queue full, client can also make half connection till
> sync queue full (default 2048), so from client side, ping will aslo work till
> sync queue full. And after 3 hours, task will also exit with connect timeout
> exception.
> To fix this bug, we introduced a PingSocketCleaner thread, which will
> continuously accept ping socket connect from ping client. When socket close
> from client, cleaner thread will detecte closed inputStream reading, then
> finally close socket from sever side.
> Refrenced by linux kernel patch:
> [https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5ea8ea2cb7]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
