[
https://issues.apache.org/jira/browse/YARN-11530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17763374#comment-17763374
]
ASF GitHub Bot commented on YARN-11530:
---------------------------------------
hadoop-yetus commented on PR #6037:
URL: https://github.com/apache/hadoop/pull/6037#issuecomment-1712647741
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 38s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 22s | | trunk passed |
| +1 :green_heart: | compile | 17m 14s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 16m 23s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 1m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 48s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 21s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 0s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 2m 36s | | trunk passed |
| +1 :green_heart: | shadedclient | 36m 15s | | branch has no errors
when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 58s | | the patch passed |
| +1 :green_heart: | compile | 16m 24s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 16m 24s | | the patch passed |
| +1 :green_heart: | compile | 16m 11s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 16m 11s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 21s |
[/results-checkstyle-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6037/2/artifact/out/results-checkstyle-hadoop-common-project_hadoop-common.txt)
| hadoop-common-project/hadoop-common: The patch generated 2 new + 160
unchanged - 0 fixed = 162 total (was 160) |
| +1 :green_heart: | mvnsite | 1m 46s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 19s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 59s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 2m 46s | | the patch passed |
| +1 :green_heart: | shadedclient | 36m 12s | | patch has no errors
when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 19m 14s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 13s | | The patch does not
generate ASF License warnings. |
| | | 225m 26s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6037/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6037 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 600535f8e75c 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 27c5fb08bf74ce0ab4c1842472cf65c67bc055e0 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6037/2/testReport/ |
| Max. process+thread count | 1302 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6037/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Server$Listener stating too many open files when setting
> ipc.server.read.threadpool.size big enough
> ---------------------------------------------------------------------------------------------------
>
> Key: YARN-11530
> URL: https://issues.apache.org/jira/browse/YARN-11530
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: ConfX
> Priority: Critical
> Labels: pull-request-available
> Attachments: reproduce.sh
>
>
> h2. What happened?
> Got an IOException stating "Too many open files" when running
> org.apache.hadoop.yarn.TestRPCFactories#test
> h2. Where's the bug?
> In the constructor of org.apache.hadoop.ipc.Server$Listener, the listener
> opens a bunch of readers:
> {code:java}
> readers = new Reader[readThreads];
> for (int i = 0; i < readThreads; i++) {
> Reader reader = new Reader(
> "Socket Reader #" + (i + 1) + " for port " + port);
> readers[i] = reader;
> reader.start();
> }
> {code}
> without checking on the value readThreads. When the parameter
> ipc.server.read.threadpool.size is set big enough, the system would run out
> of new readers to open. The listener should try to catch exceptions thrown
> during the creation of the reader.
> h3. Stacktrace
> {code}
> java.lang.ExceptionInInitializerError
> ...
> Caused by: java.io.IOException: Too many open files
> at java.base/sun.nio.ch.FileDispatcherImpl.init(Native Method)
> at
> java.base/sun.nio.ch.FileDispatcherImpl.<clinit>(FileDispatcherImpl.java:38)
> ...
> {code}
> h2. How to reproduce?
> (1) set ipc.server.read.threadpool.size to 50000
> (2) run org.apache.hadoop.yarn.TestRPCFactories#test
> You can use the reproduce.sh in the attachment to easily reproduce the bug:
> We have tested this bug on both Ubuntu and MacOS. *The bug is volatile and
> appears in different forms on the two OS we have tested*. On MacOS it outputs
> the too many open files error in stderr. On Ubuntu the JVM crashes directly:
> {code}
> [WARNING] Corrupted STDOUT by directly writing to native stream in forked JVM
> 1.
> ...
> ExecutionException The forked VM terminated without properly saying goodbye.
> VM crash or System.exit called?
> ...
> Error occurred in starting fork, check output in log
>
>
> Process Exit Code: 1
>
>
> Crashed tests:
>
>
> org.apache.hadoop.yarn.TestRPCFactories
> ...
> Caused by: org.apache.maven.surefire.booter.SurefireBooterForkException: The
> forked VM terminated without properly saying goodbye. VM crash or System.exit
> called?
> {code}
> We are happy to provide a patch after this issue is confirmed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
