[jira] [Updated] (MAPREDUCE-7334) TestJobEndNotifier fails

2021-04-07 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated MAPREDUCE-7334:
--
Labels: pull-request-available  (was: )

> TestJobEndNotifier fails
> 
>
> Key: MAPREDUCE-7334
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7334
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: test
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
>  Labels: pull-request-available
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> [https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/8/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt]
> {quote}
> [INFO] Running org.apache.hadoop.mapred.TestJobEndNotifier
> [ERROR] Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 47.81 
> s <<< FAILURE! - in org.apache.hadoop.mapred.TestJobEndNotifier
> [ERROR] testNotificationTimeout(org.apache.hadoop.mapred.TestJobEndNotifier)  
> Time elapsed: 47.322 s  <<< ERROR!
> java.util.concurrent.TimeoutException
>   at org.eclipse.jetty.util.FutureCallback.get(FutureCallback.java:130)
>   at org.eclipse.jetty.util.FutureCallback.get(FutureCallback.java:30)
>   at 
> org.eclipse.jetty.server.handler.AbstractHandlerContainer.doShutdown(AbstractHandlerContainer.java:175)
>   at org.eclipse.jetty.server.Server.doStop(Server.java:447)
>   at 
> org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:94)
>   at org.apache.hadoop.http.HttpServer2.stop(HttpServer2.java:1499)
>   at 
> org.apache.hadoop.mapred.TestJobEndNotifier.tearDown(TestJobEndNotifier.java:127)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
>   at 
> org.junit.internal.runners.statements.RunAfters.invokeMethod(RunAfters.java:46)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
>   at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
>   at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
>   at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
>   at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {quote}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org



[jira] [Work logged] (MAPREDUCE-7334) TestJobEndNotifier fails

2021-04-07 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7334?focusedWorklogId=578922=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-578922
 ]

ASF GitHub Bot logged work on MAPREDUCE-7334:
-

Author: ASF GitHub Bot
Created on: 08/Apr/21 05:55
Start Date: 08/Apr/21 05:55
Worklog Time Spent: 10m 
  Work Description: aajisaka opened a new pull request #2877:
URL: https://github.com/apache/hadoop/pull/2877


   Reduced the sleep time from 1000 to 3 seconds to shut down HttpServer2 
gracefully.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 578922)
Remaining Estimate: 0h
Time Spent: 10m

> TestJobEndNotifier fails
> 
>
> Key: MAPREDUCE-7334
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7334
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: test
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> [https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/8/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt]
> {quote}
> [INFO] Running org.apache.hadoop.mapred.TestJobEndNotifier
> [ERROR] Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 47.81 
> s <<< FAILURE! - in org.apache.hadoop.mapred.TestJobEndNotifier
> [ERROR] testNotificationTimeout(org.apache.hadoop.mapred.TestJobEndNotifier)  
> Time elapsed: 47.322 s  <<< ERROR!
> java.util.concurrent.TimeoutException
>   at org.eclipse.jetty.util.FutureCallback.get(FutureCallback.java:130)
>   at org.eclipse.jetty.util.FutureCallback.get(FutureCallback.java:30)
>   at 
> org.eclipse.jetty.server.handler.AbstractHandlerContainer.doShutdown(AbstractHandlerContainer.java:175)
>   at org.eclipse.jetty.server.Server.doStop(Server.java:447)
>   at 
> org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:94)
>   at org.apache.hadoop.http.HttpServer2.stop(HttpServer2.java:1499)
>   at 
> org.apache.hadoop.mapred.TestJobEndNotifier.tearDown(TestJobEndNotifier.java:127)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
>   at 
> org.junit.internal.runners.statements.RunAfters.invokeMethod(RunAfters.java:46)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
>   at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
>   at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
>   at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
>   at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
>   at 
> 

[jira] [Commented] (MAPREDUCE-7334) TestJobEndNotifier fails

2021-04-07 Thread Akira Ajisaka (Jira)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-7334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17316874#comment-17316874
 ] 

Akira Ajisaka commented on MAPREDUCE-7334:
--

After this test, it shutdowns the HttpServer2. HADOOP-17133 added 
StatisticsHandler, and it shutdowns gracefully because the handler implements 
org.eclipse.jetty.util.component.Graceful interface. In this test, there is a 
remaining request on DelayServlet, and it does not allow StatisticsHandler to 
shut down.

To fix this problem, I want to reduce the sleep time in DelayServlet from 1000 
to 3 seconds.

> TestJobEndNotifier fails
> 
>
> Key: MAPREDUCE-7334
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7334
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: test
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
>
> [https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/8/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt]
> {quote}
> [INFO] Running org.apache.hadoop.mapred.TestJobEndNotifier
> [ERROR] Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 47.81 
> s <<< FAILURE! - in org.apache.hadoop.mapred.TestJobEndNotifier
> [ERROR] testNotificationTimeout(org.apache.hadoop.mapred.TestJobEndNotifier)  
> Time elapsed: 47.322 s  <<< ERROR!
> java.util.concurrent.TimeoutException
>   at org.eclipse.jetty.util.FutureCallback.get(FutureCallback.java:130)
>   at org.eclipse.jetty.util.FutureCallback.get(FutureCallback.java:30)
>   at 
> org.eclipse.jetty.server.handler.AbstractHandlerContainer.doShutdown(AbstractHandlerContainer.java:175)
>   at org.eclipse.jetty.server.Server.doStop(Server.java:447)
>   at 
> org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:94)
>   at org.apache.hadoop.http.HttpServer2.stop(HttpServer2.java:1499)
>   at 
> org.apache.hadoop.mapred.TestJobEndNotifier.tearDown(TestJobEndNotifier.java:127)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
>   at 
> org.junit.internal.runners.statements.RunAfters.invokeMethod(RunAfters.java:46)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
>   at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
>   at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
>   at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
>   at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {quote}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: 

[jira] [Assigned] (MAPREDUCE-7334) TestJobEndNotifier fails

2021-04-07 Thread Akira Ajisaka (Jira)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akira Ajisaka reassigned MAPREDUCE-7334:


Assignee: Akira Ajisaka

> TestJobEndNotifier fails
> 
>
> Key: MAPREDUCE-7334
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7334
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: test
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
>
> [https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/8/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt]
> {quote}
> [INFO] Running org.apache.hadoop.mapred.TestJobEndNotifier
> [ERROR] Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 47.81 
> s <<< FAILURE! - in org.apache.hadoop.mapred.TestJobEndNotifier
> [ERROR] testNotificationTimeout(org.apache.hadoop.mapred.TestJobEndNotifier)  
> Time elapsed: 47.322 s  <<< ERROR!
> java.util.concurrent.TimeoutException
>   at org.eclipse.jetty.util.FutureCallback.get(FutureCallback.java:130)
>   at org.eclipse.jetty.util.FutureCallback.get(FutureCallback.java:30)
>   at 
> org.eclipse.jetty.server.handler.AbstractHandlerContainer.doShutdown(AbstractHandlerContainer.java:175)
>   at org.eclipse.jetty.server.Server.doStop(Server.java:447)
>   at 
> org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:94)
>   at org.apache.hadoop.http.HttpServer2.stop(HttpServer2.java:1499)
>   at 
> org.apache.hadoop.mapred.TestJobEndNotifier.tearDown(TestJobEndNotifier.java:127)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
>   at 
> org.junit.internal.runners.statements.RunAfters.invokeMethod(RunAfters.java:46)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
>   at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
>   at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
>   at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
>   at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
>   at 
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {quote}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org



[jira] [Created] (MAPREDUCE-7334) TestJobEndNotifier fails

2021-04-07 Thread Akira Ajisaka (Jira)
Akira Ajisaka created MAPREDUCE-7334:


 Summary: TestJobEndNotifier fails
 Key: MAPREDUCE-7334
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7334
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: test
Reporter: Akira Ajisaka


[https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/8/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt]
{quote}
[INFO] Running org.apache.hadoop.mapred.TestJobEndNotifier
[ERROR] Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 47.81 s 
<<< FAILURE! - in org.apache.hadoop.mapred.TestJobEndNotifier
[ERROR] testNotificationTimeout(org.apache.hadoop.mapred.TestJobEndNotifier)  
Time elapsed: 47.322 s  <<< ERROR!
java.util.concurrent.TimeoutException
at org.eclipse.jetty.util.FutureCallback.get(FutureCallback.java:130)
at org.eclipse.jetty.util.FutureCallback.get(FutureCallback.java:30)
at 
org.eclipse.jetty.server.handler.AbstractHandlerContainer.doShutdown(AbstractHandlerContainer.java:175)
at org.eclipse.jetty.server.Server.doStop(Server.java:447)
at 
org.eclipse.jetty.util.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:94)
at org.apache.hadoop.http.HttpServer2.stop(HttpServer2.java:1499)
at 
org.apache.hadoop.mapred.TestJobEndNotifier.tearDown(TestJobEndNotifier.java:127)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at 
org.junit.internal.runners.statements.RunAfters.invokeMethod(RunAfters.java:46)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at 
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at 
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at 
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
at 
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
at 
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
at 
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
{quote}
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org



[jira] [Work logged] (MAPREDUCE-7270) TestHistoryViewerPrinter could be failed when the locale isn't English.

2021-04-07 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7270?focusedWorklogId=578848=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-578848
 ]

ASF GitHub Bot logged work on MAPREDUCE-7270:
-

Author: ASF GitHub Bot
Created on: 08/Apr/21 03:09
Start Date: 08/Apr/21 03:09
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #1942:
URL: https://github.com/apache/hadoop/pull/1942#issuecomment-815412413


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 36s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  1s |  |  codespell was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to 
include 1 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |  35m 53s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |   0m 41s |  |  trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  compile  |   0m 36s |  |  trunk passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08  |
   | +1 :green_heart: |  checkstyle  |   0m 38s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   0m 43s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   0m 25s |  |  trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javadoc  |   0m 23s |  |  trunk passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08  |
   | +1 :green_heart: |  spotbugs  |   1m 23s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  16m 32s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +1 :green_heart: |  mvninstall  |   0m 33s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 34s |  |  the patch passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javac  |   0m 34s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   0m 30s |  |  the patch passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08  |
   | +1 :green_heart: |  javac  |   0m 30s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks 
issues.  |
   | +1 :green_heart: |  checkstyle  |   0m 29s |  |  the patch passed  |
   | +1 :green_heart: |  mvnsite  |   0m 33s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   0m 16s |  |  the patch passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javadoc  |   0m 15s |  |  the patch passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08  |
   | +1 :green_heart: |  spotbugs  |   1m 24s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  16m  4s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |
   | -1 :x: |  unit  |   8m  2s | 
[/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1942/11/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt)
 |  hadoop-mapreduce-client-core in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   0m 33s |  |  The patch does not 
generate ASF License warnings.  |
   |  |   |  87m 40s |  |  |
   
   
   | Reason | Tests |
   |---:|:--|
   | Failed junit tests | hadoop.mapred.TestJobEndNotifier |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1942/11/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/1942 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall 
mvnsite unit shadedclient spotbugs checkstyle codespell |
   | uname | Linux 5340e704bc40 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 
16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
   | Build tool | maven |
   | Personality | dev-support/bin/hadoop.sh |
   | git revision | trunk / 93cc19586f294fc7e1886010394ada5287b83285 |
   | Default Java | Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
   | Multi-JDK versions | 
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 
/usr/lib/jvm/java-8-openjdk-amd64:Private 
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 |
   |  Test Results | 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-1942/11/testReport/ 

[jira] [Work logged] (MAPREDUCE-7270) TestHistoryViewerPrinter could be failed when the locale isn't English.

2021-04-07 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7270?focusedWorklogId=578829=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-578829
 ]

ASF GitHub Bot logged work on MAPREDUCE-7270:
-

Author: ASF GitHub Bot
Created on: 08/Apr/21 01:39
Start Date: 08/Apr/21 01:39
Worklog Time Spent: 10m 
  Work Description: sungpeo commented on pull request #1942:
URL: https://github.com/apache/hadoop/pull/1942#issuecomment-815383855


   @liuml07 Okay, I rebased trunk and fixed the warning.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 578829)
Time Spent: 1.5h  (was: 1h 20m)

> TestHistoryViewerPrinter could be failed when the locale isn't English.
> ---
>
> Key: MAPREDUCE-7270
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7270
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>  Components: test
>Reporter: Sungpeo Kook
>Assignee: Sungpeo Kook
>Priority: Minor
>  Labels: pull-request-available
> Attachments: MAPREDUCE-7270.patch
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Both of testHumanPrinter and testHumanPrinterAll have expected string for 
> assertion.
> But the actual result includes the Dateformat which can be different depends 
> on Locale.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org



[jira] [Commented] (MAPREDUCE-7333) SecureShuffleUtils.toHex(byte[]) creates malformed hex string

2021-04-07 Thread Ahmed Hussein (Jira)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-7333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17316680#comment-17316680
 ] 

Ahmed Hussein commented on MAPREDUCE-7333:
--

[~Marcono1234] Thanks for spotting this bug.
I suggest to delete {{toHex()}} implementation because it is not being used 
anywhere in the Hadoop trunk.
If you find some code redundancy that could be improved by calling {{toHex()}}, 
then please go ahead with:
* fixing the implementation
* replace redundant code blocks with calls to {{toHex()}}.

> SecureShuffleUtils.toHex(byte[]) creates malformed hex string
> -
>
> Key: MAPREDUCE-7333
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7333
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Affects Versions: 3.2.2
>Reporter: Marcono1234
>Assignee: Peter Bacsko
>Priority: Major
>
> {{org.apache.hadoop.mapreduce.security.SecureShuffleUtils.toHex(byte[])}} 
> creates malformed hex strings:
> {code}
> for (byte b : ba) {
> ps.printf("%x", b);
> }
> {code}
> The pattern {{"%x"}} would for bytes < 16 only have on hex char and for 
> example both {{1, 0}} and {{16}} would have the result {{"10"}}.
> A correct (and more efficient) implementation would be:
> {code}
> public static String toHex(byte[] ba) {
> StringBuilder sb = new StringBuilder(ba.length * 2);
> for (byte b : ba) {
> int unsignedB = b & 0xFF;
> if (unsignedB < 16) {
> sb.append('0');
> }
> sb.append(Integer.toHexString(unsignedB));
> }
> return sb.toString();
> }
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org



[jira] [Commented] (MAPREDUCE-7329) HadoopPipes task failed as a result of ping timeout exception

2021-04-07 Thread Hadoop QA (Jira)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17316635#comment-17316635
 ] 

Hadoop QA commented on MAPREDUCE-7329:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime ||  Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
50s{color} |  | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green}  0m  
0s{color} |  | {color:green} No case conflicting files found. {color} |
| {color:blue}0{color} | {color:blue} codespell {color} | {color:blue}  0m  
0s{color} |  | {color:blue} codespell was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} |  | {color:green} The patch does not contain any @author tags. 
{color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} |  | {color:green} The patch appears to include 1 new or modified 
test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 14m  
5s{color} |  | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m 
31s{color} |  | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
49s{color} |  | {color:green} trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m  
5s{color} |  | {color:green} trunk passed with JDK Private 
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
55s{color} |  | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
10s{color} |  | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
42s{color} |  | {color:green} trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
39s{color} |  | {color:green} trunk passed with JDK Private 
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green}  1m 
57s{color} |  | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
16m  8s{color} |  | {color:green} branch has no errors when building and 
testing our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
20s{color} |  | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
59s{color} |  | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
25s{color} |  | {color:green} the patch passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  2m 
25s{color} |  | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m  
3s{color} |  | {color:green} the patch passed with JDK Private 
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  2m  
3s{color} |  | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} blanks {color} | {color:green}  0m  
0s{color} |  | {color:green} The patch has no blanks issues. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 51s{color} | 
[/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt|https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/8/artifact/out/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt]
 | {color:orange} hadoop-mapreduce-project/hadoop-mapreduce-client: The patch 
generated 9 new + 39 unchanged - 0 fixed = 48 total (was 39) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
58s{color} |  | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
31s{color} |  | {color:green} the patch passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
29s{color} |  | {color:green} the patch passed with JDK Private 
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| 

[jira] [Work logged] (MAPREDUCE-7329) HadoopPipes task failed as a result of ping timeout exception

2021-04-07 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7329?focusedWorklogId=578643=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-578643
 ]

ASF GitHub Bot logged work on MAPREDUCE-7329:
-

Author: ASF GitHub Bot
Created on: 07/Apr/21 20:19
Start Date: 07/Apr/21 20:19
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#issuecomment-815232495


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 50s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to 
include 1 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  14m  5s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  22m 31s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |   2m 49s |  |  trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  compile  |   2m  5s |  |  trunk passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08  |
   | +1 :green_heart: |  checkstyle  |   0m 55s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   1m 10s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   0m 42s |  |  trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javadoc  |   0m 39s |  |  trunk passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08  |
   | +1 :green_heart: |  spotbugs  |   1m 57s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  16m  8s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 20s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   0m 59s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   2m 25s |  |  the patch passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javac  |   2m 25s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   2m  3s |  |  the patch passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08  |
   | +1 :green_heart: |  javac  |   2m  3s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks 
issues.  |
   | -0 :warning: |  checkstyle  |   0m 51s | 
[/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/8/artifact/out/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt)
 |  hadoop-mapreduce-project/hadoop-mapreduce-client: The patch generated 9 new 
+ 39 unchanged - 0 fixed = 48 total (was 39)  |
   | +1 :green_heart: |  mvnsite  |   0m 58s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   0m 31s |  |  the patch passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javadoc  |   0m 29s |  |  the patch passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08  |
   | +1 :green_heart: |  spotbugs  |   2m  9s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  16m 11s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |
   | -1 :x: |  unit  |   7m 40s | 
[/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/8/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt)
 |  hadoop-mapreduce-client-core in the patch passed.  |
   | +1 :green_heart: |  unit  | 130m 30s |  |  
hadoop-mapreduce-client-jobclient in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   0m 35s |  |  The patch does not 
generate ASF License warnings.  |
   |  |   | 231m 14s |  |  |
   
   
   | Reason | Tests |
   |---:|:--|
   | Failed junit tests | hadoop.mapred.TestJobEndNotifier |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/8/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/2775 |
   | JIRA Issue | MAPREDUCE-7329 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall 
mvnsite unit shadedclient spotbugs checkstyle codespell |
   | 

[jira] [Work logged] (MAPREDUCE-7329) HadoopPipes task failed as a result of ping timeout exception

2021-04-07 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7329?focusedWorklogId=578641=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-578641
 ]

ASF GitHub Bot logged work on MAPREDUCE-7329:
-

Author: ASF GitHub Bot
Created on: 07/Apr/21 20:14
Start Date: 07/Apr/21 20:14
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#issuecomment-815225279


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 37s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  0s |  |  codespell was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to 
include 1 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  14m 36s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  21m  4s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |   2m 30s |  |  trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  compile  |   2m  8s |  |  trunk passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08  |
   | +1 :green_heart: |  checkstyle  |   0m 59s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   1m 15s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   0m 49s |  |  trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javadoc  |   0m 42s |  |  trunk passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08  |
   | +1 :green_heart: |  spotbugs  |   2m  4s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  14m  7s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 30s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   1m 21s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   2m 28s |  |  the patch passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javac  |   2m 28s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   2m  0s |  |  the patch passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08  |
   | +1 :green_heart: |  javac  |   2m  0s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks 
issues.  |
   | -0 :warning: |  checkstyle  |   0m 50s | 
[/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/7/artifact/out/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt)
 |  hadoop-mapreduce-project/hadoop-mapreduce-client: The patch generated 9 new 
+ 39 unchanged - 0 fixed = 48 total (was 39)  |
   | +1 :green_heart: |  mvnsite  |   1m  3s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   0m 36s |  |  the patch passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javadoc  |   0m 30s |  |  the patch passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08  |
   | +1 :green_heart: |  spotbugs  |   2m 12s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  13m 47s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |
   | -1 :x: |  unit  |   7m 43s | 
[/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/7/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt)
 |  hadoop-mapreduce-client-core in the patch passed.  |
   | +1 :green_heart: |  unit  | 130m 50s |  |  
hadoop-mapreduce-client-jobclient in the patch passed.  |
   | +1 :green_heart: |  asflicense  |   0m 38s |  |  The patch does not 
generate ASF License warnings.  |
   |  |   | 227m 21s |  |  |
   
   
   | Reason | Tests |
   |---:|:--|
   | Failed junit tests | hadoop.mapred.TestJobEndNotifier |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/7/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/2775 |
   | JIRA Issue | MAPREDUCE-7329 |
   | Optional Tests | dupname asflicense compile javac javadoc mvninstall 
mvnsite unit shadedclient spotbugs checkstyle codespell |
   | 

[jira] [Commented] (MAPREDUCE-7329) HadoopPipes task failed as a result of ping timeout exception

2021-04-07 Thread Hadoop QA (Jira)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17316633#comment-17316633
 ] 

Hadoop QA commented on MAPREDUCE-7329:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime ||  Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
37s{color} |  | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green}  0m  
0s{color} |  | {color:green} No case conflicting files found. {color} |
| {color:blue}0{color} | {color:blue} codespell {color} | {color:blue}  0m  
0s{color} |  | {color:blue} codespell was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} |  | {color:green} The patch does not contain any @author tags. 
{color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} |  | {color:green} The patch appears to include 1 new or modified 
test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 14m 
36s{color} |  | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m 
 4s{color} |  | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
30s{color} |  | {color:green} trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m  
8s{color} |  | {color:green} trunk passed with JDK Private 
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
59s{color} |  | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
15s{color} |  | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
49s{color} |  | {color:green} trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
42s{color} |  | {color:green} trunk passed with JDK Private 
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green}  2m  
4s{color} |  | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
14m  7s{color} |  | {color:green} branch has no errors when building and 
testing our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
30s{color} |  | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
21s{color} |  | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
28s{color} |  | {color:green} the patch passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  2m 
28s{color} |  | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m  
0s{color} |  | {color:green} the patch passed with JDK Private 
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  2m  
0s{color} |  | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} blanks {color} | {color:green}  0m  
0s{color} |  | {color:green} The patch has no blanks issues. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 50s{color} | 
[/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt|https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/7/artifact/out/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt]
 | {color:orange} hadoop-mapreduce-project/hadoop-mapreduce-client: The patch 
generated 9 new + 39 unchanged - 0 fixed = 48 total (was 39) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
3s{color} |  | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
36s{color} |  | {color:green} the patch passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
30s{color} |  | {color:green} the patch passed with JDK Private 
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| 

[jira] [Work logged] (MAPREDUCE-7329) HadoopPipes task failed as a result of ping timeout exception

2021-04-07 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7329?focusedWorklogId=578429=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-578429
 ]

ASF GitHub Bot logged work on MAPREDUCE-7329:
-

Author: ASF GitHub Bot
Created on: 07/Apr/21 15:46
Start Date: 07/Apr/21 15:46
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#issuecomment-815020952


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 49s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  1s |  |  codespell was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to 
include 1 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  13m 48s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  26m  0s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |   2m 59s |  |  trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  compile  |   2m 28s |  |  trunk passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08  |
   | +1 :green_heart: |  checkstyle  |   1m  1s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   1m 18s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   0m 46s |  |  trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javadoc  |   0m 42s |  |  trunk passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08  |
   | +1 :green_heart: |  spotbugs  |   2m 19s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  16m 46s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 20s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   1m  2s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   2m 33s |  |  the patch passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javac  |   2m 33s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   2m  4s |  |  the patch passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08  |
   | +1 :green_heart: |  javac  |   2m  4s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks 
issues.  |
   | -0 :warning: |  checkstyle  |   0m 52s | 
[/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/6/artifact/out/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt)
 |  hadoop-mapreduce-project/hadoop-mapreduce-client: The patch generated 11 
new + 7 unchanged - 0 fixed = 18 total (was 7)  |
   | +1 :green_heart: |  mvnsite  |   1m  1s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   0m 32s |  |  the patch passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javadoc  |   0m 30s |  |  the patch passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08  |
   | +1 :green_heart: |  spotbugs  |   2m 13s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  16m 10s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |
   | -1 :x: |  unit  |   7m 39s | 
[/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/6/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt)
 |  hadoop-mapreduce-client-core in the patch passed.  |
   | +1 :green_heart: |  unit  | 130m 25s |  |  
hadoop-mapreduce-client-jobclient in the patch passed.  |
   | -1 :x: |  asflicense  |   0m 35s | 
[/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/6/artifact/out/results-asflicense.txt)
 |  The patch generated 1 ASF License warnings.  |
   |  |   | 236m 43s |  |  |
   
   
   | Reason | Tests |
   |---:|:--|
   | Failed junit tests | hadoop.mapred.TestJobEndNotifier |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/6/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/2775 |
   | JIRA Issue | MAPREDUCE-7329 |
   | Optional Tests | dupname 

[jira] [Commented] (MAPREDUCE-7329) HadoopPipes task failed as a result of ping timeout exception

2021-04-07 Thread Hadoop QA (Jira)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17316437#comment-17316437
 ] 

Hadoop QA commented on MAPREDUCE-7329:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime ||  Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
49s{color} |  | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green}  0m  
0s{color} |  | {color:green} No case conflicting files found. {color} |
| {color:blue}0{color} | {color:blue} codespell {color} | {color:blue}  0m  
1s{color} |  | {color:blue} codespell was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} |  | {color:green} The patch does not contain any @author tags. 
{color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} |  | {color:green} The patch appears to include 1 new or modified 
test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 13m 
48s{color} |  | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m 
 0s{color} |  | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
59s{color} |  | {color:green} trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
28s{color} |  | {color:green} trunk passed with JDK Private 
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 1s{color} |  | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
18s{color} |  | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
46s{color} |  | {color:green} trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
42s{color} |  | {color:green} trunk passed with JDK Private 
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green}  2m 
19s{color} |  | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
16m 46s{color} |  | {color:green} branch has no errors when building and 
testing our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
20s{color} |  | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
 2s{color} |  | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
33s{color} |  | {color:green} the patch passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  2m 
33s{color} |  | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m  
4s{color} |  | {color:green} the patch passed with JDK Private 
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  2m  
4s{color} |  | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} blanks {color} | {color:green}  0m  
0s{color} |  | {color:green} The patch has no blanks issues. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 52s{color} | 
[/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt|https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/6/artifact/out/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt]
 | {color:orange} hadoop-mapreduce-project/hadoop-mapreduce-client: The patch 
generated 11 new + 7 unchanged - 0 fixed = 18 total (was 7) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
1s{color} |  | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
32s{color} |  | {color:green} the patch passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
30s{color} |  | {color:green} the patch passed with JDK Private 
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| 

[jira] [Work logged] (MAPREDUCE-7329) HadoopPipes task failed as a result of ping timeout exception

2021-04-07 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7329?focusedWorklogId=578419=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-578419
 ]

ASF GitHub Bot logged work on MAPREDUCE-7329:
-

Author: ASF GitHub Bot
Created on: 07/Apr/21 15:28
Start Date: 07/Apr/21 15:28
Worklog Time Spent: 10m 
  Work Description: hadoop-yetus commented on pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#issuecomment-815007568


   :broken_heart: **-1 overall**
   
   
   
   
   
   
   | Vote | Subsystem | Runtime |  Logfile | Comment |
   |::|--:|:|::|:---:|
   | +0 :ok: |  reexec  |   0m 36s |  |  Docker mode activated.  |
    _ Prechecks _ |
   | +1 :green_heart: |  dupname  |   0m  0s |  |  No case conflicting files 
found.  |
   | +0 :ok: |  codespell  |   0m  1s |  |  codespell was not available.  |
   | +1 :green_heart: |  @author  |   0m  0s |  |  The patch does not contain 
any @author tags.  |
   | +1 :green_heart: |  test4tests  |   0m  0s |  |  The patch appears to 
include 1 new or modified test files.  |
    _ trunk Compile Tests _ |
   | +0 :ok: |  mvndep  |  14m  2s |  |  Maven dependency ordering for branch  |
   | +1 :green_heart: |  mvninstall  |  22m 43s |  |  trunk passed  |
   | +1 :green_heart: |  compile  |   2m 36s |  |  trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  compile  |   2m 11s |  |  trunk passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08  |
   | +1 :green_heart: |  checkstyle  |   1m  3s |  |  trunk passed  |
   | +1 :green_heart: |  mvnsite  |   1m 17s |  |  trunk passed  |
   | +1 :green_heart: |  javadoc  |   0m 48s |  |  trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javadoc  |   0m 42s |  |  trunk passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08  |
   | +1 :green_heart: |  spotbugs  |   2m 12s |  |  trunk passed  |
   | +1 :green_heart: |  shadedclient  |  16m  5s |  |  branch has no errors 
when building and testing our client artifacts.  |
    _ Patch Compile Tests _ |
   | +0 :ok: |  mvndep  |   0m 25s |  |  Maven dependency ordering for patch  |
   | +1 :green_heart: |  mvninstall  |   1m  3s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   2m 26s |  |  the patch passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javac  |   2m 26s |  |  the patch passed  |
   | +1 :green_heart: |  compile  |   2m  4s |  |  the patch passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08  |
   | +1 :green_heart: |  javac  |   2m  4s |  |  the patch passed  |
   | +1 :green_heart: |  blanks  |   0m  0s |  |  The patch has no blanks 
issues.  |
   | -0 :warning: |  checkstyle  |   0m 54s | 
[/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/5/artifact/out/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt)
 |  hadoop-mapreduce-project/hadoop-mapreduce-client: The patch generated 6 new 
+ 7 unchanged - 0 fixed = 13 total (was 7)  |
   | +1 :green_heart: |  mvnsite  |   1m  6s |  |  the patch passed  |
   | +1 :green_heart: |  javadoc  |   0m 34s |  |  the patch passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04  |
   | +1 :green_heart: |  javadoc  |   0m 33s |  |  the patch passed with JDK 
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08  |
   | +1 :green_heart: |  spotbugs  |   2m 20s |  |  the patch passed  |
   | +1 :green_heart: |  shadedclient  |  16m 15s |  |  patch has no errors 
when building and testing our client artifacts.  |
    _ Other Tests _ |
   | -1 :x: |  unit  |   7m 50s | 
[/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/5/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt)
 |  hadoop-mapreduce-client-core in the patch passed.  |
   | +1 :green_heart: |  unit  | 134m 28s |  |  
hadoop-mapreduce-client-jobclient in the patch passed.  |
   | -1 :x: |  asflicense  |   0m 38s | 
[/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/5/artifact/out/results-asflicense.txt)
 |  The patch generated 1 ASF License warnings.  |
   |  |   | 236m 48s |  |  |
   
   
   | Reason | Tests |
   |---:|:--|
   | Failed junit tests | hadoop.mapred.TestJobEndNotifier |
   
   
   | Subsystem | Report/Notes |
   |--:|:-|
   | Docker | ClientAPI=1.41 ServerAPI=1.41 base: 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/5/artifact/out/Dockerfile
 |
   | GITHUB PR | https://github.com/apache/hadoop/pull/2775 |
   | JIRA Issue | MAPREDUCE-7329 |
   | Optional Tests | dupname 

[jira] [Commented] (MAPREDUCE-7329) HadoopPipes task failed as a result of ping timeout exception

2021-04-07 Thread Hadoop QA (Jira)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-7329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17316426#comment-17316426
 ] 

Hadoop QA commented on MAPREDUCE-7329:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime ||  Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
36s{color} |  | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green}  0m  
0s{color} |  | {color:green} No case conflicting files found. {color} |
| {color:blue}0{color} | {color:blue} codespell {color} | {color:blue}  0m  
1s{color} |  | {color:blue} codespell was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} |  | {color:green} The patch does not contain any @author tags. 
{color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} |  | {color:green} The patch appears to include 1 new or modified 
test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 14m  
2s{color} |  | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m 
43s{color} |  | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
36s{color} |  | {color:green} trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
11s{color} |  | {color:green} trunk passed with JDK Private 
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 3s{color} |  | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
17s{color} |  | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
48s{color} |  | {color:green} trunk passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
42s{color} |  | {color:green} trunk passed with JDK Private 
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green}  2m 
12s{color} |  | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
16m  5s{color} |  | {color:green} branch has no errors when building and 
testing our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
25s{color} |  | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
 3s{color} |  | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
26s{color} |  | {color:green} the patch passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  2m 
26s{color} |  | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m  
4s{color} |  | {color:green} the patch passed with JDK Private 
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  2m  
4s{color} |  | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} blanks {color} | {color:green}  0m  
0s{color} |  | {color:green} The patch has no blanks issues. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 54s{color} | 
[/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt|https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2775/5/artifact/out/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt]
 | {color:orange} hadoop-mapreduce-project/hadoop-mapreduce-client: The patch 
generated 6 new + 7 unchanged - 0 fixed = 13 total (was 7) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
6s{color} |  | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
34s{color} |  | {color:green} the patch passed with JDK 
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
33s{color} |  | {color:green} the patch passed with JDK Private 
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| 

[jira] [Commented] (MAPREDUCE-7333) SecureShuffleUtils.toHex(byte[]) creates malformed hex string

2021-04-07 Thread Ahmed Hussein (Jira)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-7333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17316390#comment-17316390
 ] 

Ahmed Hussein commented on MAPREDUCE-7333:
--

Thanks [~pbacsko] for the changes.
Sorry for the delay. I was away on a vacation.
I am going to take a look at the changes.

> SecureShuffleUtils.toHex(byte[]) creates malformed hex string
> -
>
> Key: MAPREDUCE-7333
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7333
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Affects Versions: 3.2.2
>Reporter: Marcono1234
>Assignee: Peter Bacsko
>Priority: Major
>
> {{org.apache.hadoop.mapreduce.security.SecureShuffleUtils.toHex(byte[])}} 
> creates malformed hex strings:
> {code}
> for (byte b : ba) {
> ps.printf("%x", b);
> }
> {code}
> The pattern {{"%x"}} would for bytes < 16 only have on hex char and for 
> example both {{1, 0}} and {{16}} would have the result {{"10"}}.
> A correct (and more efficient) implementation would be:
> {code}
> public static String toHex(byte[] ba) {
> StringBuilder sb = new StringBuilder(ba.length * 2);
> for (byte b : ba) {
> int unsignedB = b & 0xFF;
> if (unsignedB < 16) {
> sb.append('0');
> }
> sb.append(Integer.toHexString(unsignedB));
> }
> return sb.toString();
> }
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org



[jira] [Work logged] (MAPREDUCE-7329) HadoopPipes task failed as a result of ping timeout exception

2021-04-07 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7329?focusedWorklogId=578257=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-578257
 ]

ASF GitHub Bot logged work on MAPREDUCE-7329:
-

Author: ASF GitHub Bot
Created on: 07/Apr/21 10:08
Start Date: 07/Apr/21 10:08
Worklog Time Spent: 10m 
  Work Description: lichaojacobs commented on pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#issuecomment-814788086


   > > And we should do like this:
   > 
   > LGTM
   > 
   > I thought only setting so_timeout fixes this problem. However, it is not 
true. The problem is that the Application does not call `serverSocket.accept()` 
for ping requests to drain the tcp accept queue. Agreed to implement the ping 
socket cleaner. Thanks.
   
   Yes, this is exactly what i am trying to fix. I will try to update with a 
regression test soon. Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 578257)
Time Spent: 4h 40m  (was: 4.5h)

> HadoopPipes task failed as a result of ping timeout exception
> -
>
> Key: MAPREDUCE-7329
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7329
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: chaoli
>Priority: Major
>  Labels: patch, pull-request-available
> Fix For: 2.6.0, 3.0.0
>
> Attachments: 
> 0001-MAPREDUCE-7329-HadoopPipes-task-may-fail-when-linux-.patch, 
> image-2021-03-15-14-29-49-475.png, image-2021-03-15-14-37-32-184.png
>
>  Time Spent: 4h 40m
>  Remaining Estimate: 0h
>
> {color:#FF}*Hadoop Pipes Ping implement has a bug*{color}. Recently, we 
> upgrade linux kernel version from 3.x to 4.x. And we find hadoop pipe task 
> exit with connect timeout which is implemented by PingThread in 
> HadoopPipes.cc.
> !image-2021-03-15-14-37-32-184.png!
> After a deep research, we finally find that current ping server won't accept 
> ping client created socket, which may cause critical problem: 
>  *  it will cause tcp accept queue full(default 50)
>  *  when client close socket, server socket won't call close method, which 
> will leave too many CLOSE_WAIT socket fd existed(default 2h), and accept 
> queue never cleared.
>  * Even worse, in 4.x linux kernel version, it will cause tcp drop packet 
> directly which makes ping client connect time out. While In 3.x linux kernel 
> version, when accept queue full, client can also make half connection till 
> sync queue full (default 2048), so from client side, ping will aslo work till 
> sync queue full. And after 3 hours, task will also exit with connect timeout 
> exception.
> To fix this bug, we introduced a PingSocketCleaner thread, which will 
> continuously accept ping socket connect from ping client. When socket close 
> from client,  cleaner thread will detecte closed inputStream reading, then 
> finally close socket from sever side.
> Refrenced by linux kernel patch: 
> [https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5ea8ea2cb7]
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org



[jira] [Work logged] (MAPREDUCE-7329) HadoopPipes task failed as a result of ping timeout exception

2021-04-07 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7329?focusedWorklogId=578250=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-578250
 ]

ASF GitHub Bot logged work on MAPREDUCE-7329:
-

Author: ASF GitHub Bot
Created on: 07/Apr/21 09:49
Start Date: 07/Apr/21 09:49
Worklog Time Spent: 10m 
  Work Description: aajisaka commented on pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#issuecomment-814776572


   > And we should do like this:
   
   LGTM
   
   I thought only setting so_timeout fixes this problem. However, it is not 
true. The problem is that the Application does not call `serverSocket.accept()` 
for ping requests to drain the tcp accept queue. Agreed to implement the ping 
socket cleaner. Thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 578250)
Time Spent: 4.5h  (was: 4h 20m)

> HadoopPipes task failed as a result of ping timeout exception
> -
>
> Key: MAPREDUCE-7329
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7329
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: chaoli
>Priority: Major
>  Labels: patch, pull-request-available
> Fix For: 2.6.0, 3.0.0
>
> Attachments: 
> 0001-MAPREDUCE-7329-HadoopPipes-task-may-fail-when-linux-.patch, 
> image-2021-03-15-14-29-49-475.png, image-2021-03-15-14-37-32-184.png
>
>  Time Spent: 4.5h
>  Remaining Estimate: 0h
>
> {color:#FF}*Hadoop Pipes Ping implement has a bug*{color}. Recently, we 
> upgrade linux kernel version from 3.x to 4.x. And we find hadoop pipe task 
> exit with connect timeout which is implemented by PingThread in 
> HadoopPipes.cc.
> !image-2021-03-15-14-37-32-184.png!
> After a deep research, we finally find that current ping server won't accept 
> ping client created socket, which may cause critical problem: 
>  *  it will cause tcp accept queue full(default 50)
>  *  when client close socket, server socket won't call close method, which 
> will leave too many CLOSE_WAIT socket fd existed(default 2h), and accept 
> queue never cleared.
>  * Even worse, in 4.x linux kernel version, it will cause tcp drop packet 
> directly which makes ping client connect time out. While In 3.x linux kernel 
> version, when accept queue full, client can also make half connection till 
> sync queue full (default 2048), so from client side, ping will aslo work till 
> sync queue full. And after 3 hours, task will also exit with connect timeout 
> exception.
> To fix this bug, we introduced a PingSocketCleaner thread, which will 
> continuously accept ping socket connect from ping client. When socket close 
> from client,  cleaner thread will detecte closed inputStream reading, then 
> finally close socket from sever side.
> Refrenced by linux kernel patch: 
> [https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5ea8ea2cb7]
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org



[jira] [Work logged] (MAPREDUCE-7329) HadoopPipes task failed as a result of ping timeout exception

2021-04-07 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7329?focusedWorklogId=578211=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-578211
 ]

ASF GitHub Bot logged work on MAPREDUCE-7329:
-

Author: ASF GitHub Bot
Created on: 07/Apr/21 08:06
Start Date: 07/Apr/21 08:06
Worklog Time Spent: 10m 
  Work Description: lichaojacobs commented on pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#issuecomment-814699686


   > > Of course, but need to accept client socket first?
   > 
   > No, this method must be called before accepting client sockets.
   > 
   > > if we set so timeout, then which value is suitable?
   > 
   > The default value is 60 seconds in Hadoop IPC client (ipc.ping.interval). 
Is it too long for Hadoop Pipes application?
   
   The value sounds reasonable. And I think we should set so timeout on client 
socket?  cause `ServerSocket#setSoTimeout(int)` just affect `accept` 
   
![image](https://user-images.githubusercontent.com/7459785/113832328-157b1b00-97bb-11eb-95b5-6eac9db40ef6.png)
   
   And we should do like this:
   
![image](https://user-images.githubusercontent.com/7459785/113831991-c208cd00-97ba-11eb-85d0-b3c7ec27b58b.png)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 578211)
Time Spent: 4h 20m  (was: 4h 10m)

> HadoopPipes task failed as a result of ping timeout exception
> -
>
> Key: MAPREDUCE-7329
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7329
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: chaoli
>Priority: Major
>  Labels: patch, pull-request-available
> Fix For: 2.6.0, 3.0.0
>
> Attachments: 
> 0001-MAPREDUCE-7329-HadoopPipes-task-may-fail-when-linux-.patch, 
> image-2021-03-15-14-29-49-475.png, image-2021-03-15-14-37-32-184.png
>
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>
> {color:#FF}*Hadoop Pipes Ping implement has a bug*{color}. Recently, we 
> upgrade linux kernel version from 3.x to 4.x. And we find hadoop pipe task 
> exit with connect timeout which is implemented by PingThread in 
> HadoopPipes.cc.
> !image-2021-03-15-14-37-32-184.png!
> After a deep research, we finally find that current ping server won't accept 
> ping client created socket, which may cause critical problem: 
>  *  it will cause tcp accept queue full(default 50)
>  *  when client close socket, server socket won't call close method, which 
> will leave too many CLOSE_WAIT socket fd existed(default 2h), and accept 
> queue never cleared.
>  * Even worse, in 4.x linux kernel version, it will cause tcp drop packet 
> directly which makes ping client connect time out. While In 3.x linux kernel 
> version, when accept queue full, client can also make half connection till 
> sync queue full (default 2048), so from client side, ping will aslo work till 
> sync queue full. And after 3 hours, task will also exit with connect timeout 
> exception.
> To fix this bug, we introduced a PingSocketCleaner thread, which will 
> continuously accept ping socket connect from ping client. When socket close 
> from client,  cleaner thread will detecte closed inputStream reading, then 
> finally close socket from sever side.
> Refrenced by linux kernel patch: 
> [https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5ea8ea2cb7]
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org



[jira] [Work logged] (MAPREDUCE-7329) HadoopPipes task failed as a result of ping timeout exception

2021-04-07 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7329?focusedWorklogId=578194=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-578194
 ]

ASF GitHub Bot logged work on MAPREDUCE-7329:
-

Author: ASF GitHub Bot
Created on: 07/Apr/21 07:43
Start Date: 07/Apr/21 07:43
Worklog Time Spent: 10m 
  Work Description: aajisaka commented on pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#issuecomment-814684423


   > Of course, but need to accept client socket first?
   
   No, this method must be called before accepting client sockets.
   
   > if we set so timeout, then which value is suitable?
   
   The default value is 60 seconds in Hadoop IPC client (ipc.ping.interval). Is 
it too long for Hadoop Pipes application?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 578194)
Time Spent: 4h 10m  (was: 4h)

> HadoopPipes task failed as a result of ping timeout exception
> -
>
> Key: MAPREDUCE-7329
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7329
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: chaoli
>Priority: Major
>  Labels: patch, pull-request-available
> Fix For: 2.6.0, 3.0.0
>
> Attachments: 
> 0001-MAPREDUCE-7329-HadoopPipes-task-may-fail-when-linux-.patch, 
> image-2021-03-15-14-29-49-475.png, image-2021-03-15-14-37-32-184.png
>
>  Time Spent: 4h 10m
>  Remaining Estimate: 0h
>
> {color:#FF}*Hadoop Pipes Ping implement has a bug*{color}. Recently, we 
> upgrade linux kernel version from 3.x to 4.x. And we find hadoop pipe task 
> exit with connect timeout which is implemented by PingThread in 
> HadoopPipes.cc.
> !image-2021-03-15-14-37-32-184.png!
> After a deep research, we finally find that current ping server won't accept 
> ping client created socket, which may cause critical problem: 
>  *  it will cause tcp accept queue full(default 50)
>  *  when client close socket, server socket won't call close method, which 
> will leave too many CLOSE_WAIT socket fd existed(default 2h), and accept 
> queue never cleared.
>  * Even worse, in 4.x linux kernel version, it will cause tcp drop packet 
> directly which makes ping client connect time out. While In 3.x linux kernel 
> version, when accept queue full, client can also make half connection till 
> sync queue full (default 2048), so from client side, ping will aslo work till 
> sync queue full. And after 3 hours, task will also exit with connect timeout 
> exception.
> To fix this bug, we introduced a PingSocketCleaner thread, which will 
> continuously accept ping socket connect from ping client. When socket close 
> from client,  cleaner thread will detecte closed inputStream reading, then 
> finally close socket from sever side.
> Refrenced by linux kernel patch: 
> [https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5ea8ea2cb7]
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org



[jira] [Work logged] (MAPREDUCE-7329) HadoopPipes task failed as a result of ping timeout exception

2021-04-07 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7329?focusedWorklogId=578176=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-578176
 ]

ASF GitHub Bot logged work on MAPREDUCE-7329:
-

Author: ASF GitHub Bot
Created on: 07/Apr/21 07:07
Start Date: 07/Apr/21 07:07
Worklog Time Spent: 10m 
  Work Description: lichaojacobs commented on pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#issuecomment-814661823


   > Can `ServerSocket#setSoTimeout(int)` be used instead of modifying the 
kernel parameter?
   
   Of course, but need to accept client socket first? In my cleaner implement, 
it will close socket correctly, and thus accept queue will also be cleared. 
   And there is another problem,  if we set so timeout, then which value is 
suitable? If too short, data stream may block due to gc or other reasons which 
will cause task failed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 578176)
Time Spent: 4h  (was: 3h 50m)

> HadoopPipes task failed as a result of ping timeout exception
> -
>
> Key: MAPREDUCE-7329
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7329
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: chaoli
>Priority: Major
>  Labels: patch, pull-request-available
> Fix For: 2.6.0, 3.0.0
>
> Attachments: 
> 0001-MAPREDUCE-7329-HadoopPipes-task-may-fail-when-linux-.patch, 
> image-2021-03-15-14-29-49-475.png, image-2021-03-15-14-37-32-184.png
>
>  Time Spent: 4h
>  Remaining Estimate: 0h
>
> {color:#FF}*Hadoop Pipes Ping implement has a bug*{color}. Recently, we 
> upgrade linux kernel version from 3.x to 4.x. And we find hadoop pipe task 
> exit with connect timeout which is implemented by PingThread in 
> HadoopPipes.cc.
> !image-2021-03-15-14-37-32-184.png!
> After a deep research, we finally find that current ping server won't accept 
> ping client created socket, which may cause critical problem: 
>  *  it will cause tcp accept queue full(default 50)
>  *  when client close socket, server socket won't call close method, which 
> will leave too many CLOSE_WAIT socket fd existed(default 2h), and accept 
> queue never cleared.
>  * Even worse, in 4.x linux kernel version, it will cause tcp drop packet 
> directly which makes ping client connect time out. While In 3.x linux kernel 
> version, when accept queue full, client can also make half connection till 
> sync queue full (default 2048), so from client side, ping will aslo work till 
> sync queue full. And after 3 hours, task will also exit with connect timeout 
> exception.
> To fix this bug, we introduced a PingSocketCleaner thread, which will 
> continuously accept ping socket connect from ping client. When socket close 
> from client,  cleaner thread will detecte closed inputStream reading, then 
> finally close socket from sever side.
> Refrenced by linux kernel patch: 
> [https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5ea8ea2cb7]
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org



[jira] [Work logged] (MAPREDUCE-7329) HadoopPipes task failed as a result of ping timeout exception

2021-04-07 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7329?focusedWorklogId=578171=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-578171
 ]

ASF GitHub Bot logged work on MAPREDUCE-7329:
-

Author: ASF GitHub Bot
Created on: 07/Apr/21 06:54
Start Date: 07/Apr/21 06:54
Worklog Time Spent: 10m 
  Work Description: aajisaka commented on pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#issuecomment-814654347


   Can `ServerSocket#setSoTimeout(int)` be used instead of modifying the kernel 
parameter?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 578171)
Time Spent: 3h 50m  (was: 3h 40m)

> HadoopPipes task failed as a result of ping timeout exception
> -
>
> Key: MAPREDUCE-7329
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7329
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: chaoli
>Priority: Major
>  Labels: patch, pull-request-available
> Fix For: 2.6.0, 3.0.0
>
> Attachments: 
> 0001-MAPREDUCE-7329-HadoopPipes-task-may-fail-when-linux-.patch, 
> image-2021-03-15-14-29-49-475.png, image-2021-03-15-14-37-32-184.png
>
>  Time Spent: 3h 50m
>  Remaining Estimate: 0h
>
> {color:#FF}*Hadoop Pipes Ping implement has a bug*{color}. Recently, we 
> upgrade linux kernel version from 3.x to 4.x. And we find hadoop pipe task 
> exit with connect timeout which is implemented by PingThread in 
> HadoopPipes.cc.
> !image-2021-03-15-14-37-32-184.png!
> After a deep research, we finally find that current ping server won't accept 
> ping client created socket, which may cause critical problem: 
>  *  it will cause tcp accept queue full(default 50)
>  *  when client close socket, server socket won't call close method, which 
> will leave too many CLOSE_WAIT socket fd existed(default 2h), and accept 
> queue never cleared.
>  * Even worse, in 4.x linux kernel version, it will cause tcp drop packet 
> directly which makes ping client connect time out. While In 3.x linux kernel 
> version, when accept queue full, client can also make half connection till 
> sync queue full (default 2048), so from client side, ping will aslo work till 
> sync queue full. And after 3 hours, task will also exit with connect timeout 
> exception.
> To fix this bug, we introduced a PingSocketCleaner thread, which will 
> continuously accept ping socket connect from ping client. When socket close 
> from client,  cleaner thread will detecte closed inputStream reading, then 
> finally close socket from sever side.
> Refrenced by linux kernel patch: 
> [https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5ea8ea2cb7]
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org



[jira] [Work logged] (MAPREDUCE-7329) HadoopPipes task failed as a result of ping timeout exception

2021-04-07 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7329?focusedWorklogId=578170=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-578170
 ]

ASF GitHub Bot logged work on MAPREDUCE-7329:
-

Author: ASF GitHub Bot
Created on: 07/Apr/21 06:47
Start Date: 07/Apr/21 06:47
Worklog Time Spent: 10m 
  Work Description: lichaojacobs commented on a change in pull request 
#2775:
URL: https://github.com/apache/hadoop/pull/2775#discussion_r608382114



##
File path: 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/pipes/Application.java
##
@@ -133,6 +135,10 @@
 
 process = runClient(cmd, env);
 clientSocket = serverSocket.accept();
+// start ping socket cleaner
+socketCleaner = new PingSocketCleaner("ping-socket-cleaner");
+socketCleaner.setDaemon(true);
+socketCleaner.start();

Review comment:
   ok, i will do that in cleanup method




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 578170)
Time Spent: 3h 40m  (was: 3.5h)

> HadoopPipes task failed as a result of ping timeout exception
> -
>
> Key: MAPREDUCE-7329
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7329
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: chaoli
>Priority: Major
>  Labels: patch, pull-request-available
> Fix For: 2.6.0, 3.0.0
>
> Attachments: 
> 0001-MAPREDUCE-7329-HadoopPipes-task-may-fail-when-linux-.patch, 
> image-2021-03-15-14-29-49-475.png, image-2021-03-15-14-37-32-184.png
>
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>
> {color:#FF}*Hadoop Pipes Ping implement has a bug*{color}. Recently, we 
> upgrade linux kernel version from 3.x to 4.x. And we find hadoop pipe task 
> exit with connect timeout which is implemented by PingThread in 
> HadoopPipes.cc.
> !image-2021-03-15-14-37-32-184.png!
> After a deep research, we finally find that current ping server won't accept 
> ping client created socket, which may cause critical problem: 
>  *  it will cause tcp accept queue full(default 50)
>  *  when client close socket, server socket won't call close method, which 
> will leave too many CLOSE_WAIT socket fd existed(default 2h), and accept 
> queue never cleared.
>  * Even worse, in 4.x linux kernel version, it will cause tcp drop packet 
> directly which makes ping client connect time out. While In 3.x linux kernel 
> version, when accept queue full, client can also make half connection till 
> sync queue full (default 2048), so from client side, ping will aslo work till 
> sync queue full. And after 3 hours, task will also exit with connect timeout 
> exception.
> To fix this bug, we introduced a PingSocketCleaner thread, which will 
> continuously accept ping socket connect from ping client. When socket close 
> from client,  cleaner thread will detecte closed inputStream reading, then 
> finally close socket from sever side.
> Refrenced by linux kernel patch: 
> [https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5ea8ea2cb7]
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org



[jira] [Work logged] (MAPREDUCE-7329) HadoopPipes task failed as a result of ping timeout exception

2021-04-07 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7329?focusedWorklogId=578169=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-578169
 ]

ASF GitHub Bot logged work on MAPREDUCE-7329:
-

Author: ASF GitHub Bot
Created on: 07/Apr/21 06:45
Start Date: 07/Apr/21 06:45
Worklog Time Spent: 10m 
  Work Description: lichaojacobs commented on pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#issuecomment-814649507


   > Can we set a shorter timeout than the default (2h) in the server socket to 
clear the queue?
   
   if do this, we need to modify kernel parameter `net.ipv4.tcp_keepalive_time` 
which i think is out of application scope and also dangerous


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 578169)
Time Spent: 3.5h  (was: 3h 20m)

> HadoopPipes task failed as a result of ping timeout exception
> -
>
> Key: MAPREDUCE-7329
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7329
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: chaoli
>Priority: Major
>  Labels: patch, pull-request-available
> Fix For: 2.6.0, 3.0.0
>
> Attachments: 
> 0001-MAPREDUCE-7329-HadoopPipes-task-may-fail-when-linux-.patch, 
> image-2021-03-15-14-29-49-475.png, image-2021-03-15-14-37-32-184.png
>
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> {color:#FF}*Hadoop Pipes Ping implement has a bug*{color}. Recently, we 
> upgrade linux kernel version from 3.x to 4.x. And we find hadoop pipe task 
> exit with connect timeout which is implemented by PingThread in 
> HadoopPipes.cc.
> !image-2021-03-15-14-37-32-184.png!
> After a deep research, we finally find that current ping server won't accept 
> ping client created socket, which may cause critical problem: 
>  *  it will cause tcp accept queue full(default 50)
>  *  when client close socket, server socket won't call close method, which 
> will leave too many CLOSE_WAIT socket fd existed(default 2h), and accept 
> queue never cleared.
>  * Even worse, in 4.x linux kernel version, it will cause tcp drop packet 
> directly which makes ping client connect time out. While In 3.x linux kernel 
> version, when accept queue full, client can also make half connection till 
> sync queue full (default 2048), so from client side, ping will aslo work till 
> sync queue full. And after 3 hours, task will also exit with connect timeout 
> exception.
> To fix this bug, we introduced a PingSocketCleaner thread, which will 
> continuously accept ping socket connect from ping client. When socket close 
> from client,  cleaner thread will detecte closed inputStream reading, then 
> finally close socket from sever side.
> Refrenced by linux kernel patch: 
> [https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5ea8ea2cb7]
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org



[jira] [Work logged] (MAPREDUCE-7329) HadoopPipes task failed as a result of ping timeout exception

2021-04-07 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7329?focusedWorklogId=578167=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-578167
 ]

ASF GitHub Bot logged work on MAPREDUCE-7329:
-

Author: ASF GitHub Bot
Created on: 07/Apr/21 06:37
Start Date: 07/Apr/21 06:37
Worklog Time Spent: 10m 
  Work Description: aajisaka commented on pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#issuecomment-814645217


   Can we set a shorter timeout than the default (2h) in the server socket to 
clear the queue?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 578167)
Time Spent: 3h 20m  (was: 3h 10m)

> HadoopPipes task failed as a result of ping timeout exception
> -
>
> Key: MAPREDUCE-7329
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7329
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: chaoli
>Priority: Major
>  Labels: patch, pull-request-available
> Fix For: 2.6.0, 3.0.0
>
> Attachments: 
> 0001-MAPREDUCE-7329-HadoopPipes-task-may-fail-when-linux-.patch, 
> image-2021-03-15-14-29-49-475.png, image-2021-03-15-14-37-32-184.png
>
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> {color:#FF}*Hadoop Pipes Ping implement has a bug*{color}. Recently, we 
> upgrade linux kernel version from 3.x to 4.x. And we find hadoop pipe task 
> exit with connect timeout which is implemented by PingThread in 
> HadoopPipes.cc.
> !image-2021-03-15-14-37-32-184.png!
> After a deep research, we finally find that current ping server won't accept 
> ping client created socket, which may cause critical problem: 
>  *  it will cause tcp accept queue full(default 50)
>  *  when client close socket, server socket won't call close method, which 
> will leave too many CLOSE_WAIT socket fd existed(default 2h), and accept 
> queue never cleared.
>  * Even worse, in 4.x linux kernel version, it will cause tcp drop packet 
> directly which makes ping client connect time out. While In 3.x linux kernel 
> version, when accept queue full, client can also make half connection till 
> sync queue full (default 2048), so from client side, ping will aslo work till 
> sync queue full. And after 3 hours, task will also exit with connect timeout 
> exception.
> To fix this bug, we introduced a PingSocketCleaner thread, which will 
> continuously accept ping socket connect from ping client. When socket close 
> from client,  cleaner thread will detecte closed inputStream reading, then 
> finally close socket from sever side.
> Refrenced by linux kernel patch: 
> [https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5ea8ea2cb7]
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org



[jira] [Work logged] (MAPREDUCE-7329) HadoopPipes task failed as a result of ping timeout exception

2021-04-07 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7329?focusedWorklogId=578157=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-578157
 ]

ASF GitHub Bot logged work on MAPREDUCE-7329:
-

Author: ASF GitHub Bot
Created on: 07/Apr/21 06:14
Start Date: 07/Apr/21 06:14
Worklog Time Spent: 10m 
  Work Description: aajisaka commented on a change in pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#discussion_r608365538



##
File path: 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/pipes/Application.java
##
@@ -133,6 +135,10 @@
 
 process = runClient(cmd, env);
 clientSocket = serverSocket.accept();
+// start ping socket cleaner
+socketCleaner = new PingSocketCleaner("ping-socket-cleaner");
+socketCleaner.setDaemon(true);
+socketCleaner.start();

Review comment:
   The cleaner thread is never stopped. Would you clean up the thread in 
the cleanup method?

##
File path: 
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/pipes/Application.java
##
@@ -266,4 +272,32 @@ public static String createDigest(byte[] password, String 
data)
 return SecureShuffleUtils.hashFromString(data, key);
   }
 
+  private class PingSocketCleaner extends Thread {
+PingSocketCleaner(String name) {
+  super(name);
+}
+
+@Override
+public void run() {
+  LOG.info("PingSocketCleaner started...");
+  while (true) {
+Socket clientSocket = null;
+try {
+  clientSocket = serverSocket.accept();
+  LOG.debug("Connection received from {}", 
clientSocket.getInetAddress());
+  int readData = 0;
+  while (readData != -1) {
+readData = clientSocket.getInputStream().read();
+  }
+  LOG.debug("close socket cause client has closed.");
+  clientSocket.close();
+} catch (IOException exception) {
+  LOG.error("PingSocketCleaner exception", exception);
+  IOUtils.closeSocket(clientSocket);

Review comment:
   Do not need this line. `IOUtils.closeSocket` in the finally clause is 
always executed.
   ```suggestion
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 578157)
Time Spent: 3h 10m  (was: 3h)

> HadoopPipes task failed as a result of ping timeout exception
> -
>
> Key: MAPREDUCE-7329
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7329
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: chaoli
>Priority: Major
>  Labels: patch, pull-request-available
> Fix For: 2.6.0, 3.0.0
>
> Attachments: 
> 0001-MAPREDUCE-7329-HadoopPipes-task-may-fail-when-linux-.patch, 
> image-2021-03-15-14-29-49-475.png, image-2021-03-15-14-37-32-184.png
>
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> {color:#FF}*Hadoop Pipes Ping implement has a bug*{color}. Recently, we 
> upgrade linux kernel version from 3.x to 4.x. And we find hadoop pipe task 
> exit with connect timeout which is implemented by PingThread in 
> HadoopPipes.cc.
> !image-2021-03-15-14-37-32-184.png!
> After a deep research, we finally find that current ping server won't accept 
> ping client created socket, which may cause critical problem: 
>  *  it will cause tcp accept queue full(default 50)
>  *  when client close socket, server socket won't call close method, which 
> will leave too many CLOSE_WAIT socket fd existed(default 2h), and accept 
> queue never cleared.
>  * Even worse, in 4.x linux kernel version, it will cause tcp drop packet 
> directly which makes ping client connect time out. While In 3.x linux kernel 
> version, when accept queue full, client can also make half connection till 
> sync queue full (default 2048), so from client side, ping will aslo work till 
> sync queue full. And after 3 hours, task will also exit with connect timeout 
> exception.
> To fix this bug, we introduced a PingSocketCleaner thread, which will 
> continuously accept ping socket connect from ping client. When socket close 
> from client,  cleaner thread will detecte closed inputStream reading, then 
> finally close socket from sever side.
> Refrenced by linux kernel patch: 
> [https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5ea8ea2cb7]
>  



--
This message was sent by Atlassian Jira

[jira] [Work logged] (MAPREDUCE-7329) HadoopPipes task failed as a result of ping timeout exception

2021-04-07 Thread ASF GitHub Bot (Jira)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-7329?focusedWorklogId=578148=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-578148
 ]

ASF GitHub Bot logged work on MAPREDUCE-7329:
-

Author: ASF GitHub Bot
Created on: 07/Apr/21 06:07
Start Date: 07/Apr/21 06:07
Worklog Time Spent: 10m 
  Work Description: aajisaka commented on pull request #2775:
URL: https://github.com/apache/hadoop/pull/2775#issuecomment-814629478


   Can we create a regression test for this issue?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 578148)
Time Spent: 3h  (was: 2h 50m)

> HadoopPipes task failed as a result of ping timeout exception
> -
>
> Key: MAPREDUCE-7329
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7329
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: chaoli
>Priority: Major
>  Labels: patch, pull-request-available
> Fix For: 2.6.0, 3.0.0
>
> Attachments: 
> 0001-MAPREDUCE-7329-HadoopPipes-task-may-fail-when-linux-.patch, 
> image-2021-03-15-14-29-49-475.png, image-2021-03-15-14-37-32-184.png
>
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> {color:#FF}*Hadoop Pipes Ping implement has a bug*{color}. Recently, we 
> upgrade linux kernel version from 3.x to 4.x. And we find hadoop pipe task 
> exit with connect timeout which is implemented by PingThread in 
> HadoopPipes.cc.
> !image-2021-03-15-14-37-32-184.png!
> After a deep research, we finally find that current ping server won't accept 
> ping client created socket, which may cause critical problem: 
>  *  it will cause tcp accept queue full(default 50)
>  *  when client close socket, server socket won't call close method, which 
> will leave too many CLOSE_WAIT socket fd existed(default 2h), and accept 
> queue never cleared.
>  * Even worse, in 4.x linux kernel version, it will cause tcp drop packet 
> directly which makes ping client connect time out. While In 3.x linux kernel 
> version, when accept queue full, client can also make half connection till 
> sync queue full (default 2048), so from client side, ping will aslo work till 
> sync queue full. And after 3 hours, task will also exit with connect timeout 
> exception.
> To fix this bug, we introduced a PingSocketCleaner thread, which will 
> continuously accept ping socket connect from ping client. When socket close 
> from client,  cleaner thread will detecte closed inputStream reading, then 
> finally close socket from sever side.
> Refrenced by linux kernel patch: 
> [https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=5ea8ea2cb7]
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org