[
https://issues.apache.org/jira/browse/HDFS-16668?focusedWorklogId=792229&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-792229
]
ASF GitHub Bot logged work on HDFS-16668:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 18/Jul/22 16:25
Start Date: 18/Jul/22 16:25
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on PR #4577:
URL: https://github.com/apache/hadoop/pull/4577#issuecomment-1187707282
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 30s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 45m 47s | | trunk passed |
| +1 :green_heart: | compile | 2m 12s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 1m 55s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 33s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 8s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 39s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 11s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 47s | | trunk passed |
| +1 :green_heart: | shadedclient | 28m 57s | | branch has no errors
when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 26s | | the patch passed |
| +1 :green_heart: | compile | 1m 37s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 1m 37s | | the patch passed |
| +1 :green_heart: | compile | 1m 22s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 22s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 59s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 28s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 1s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 31s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 41s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 5s | | patch has no errors
when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 383m 39s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 2s | | The patch does not
generate ASF License warnings. |
| | | 512m 59s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4577/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4577 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 61f6abf37940 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24
17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c597a80cef6c085b2ba258b77db44c93e7cde717 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4577/1/testReport/ |
| Max. process+thread count | 2012 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4577/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
Issue Time Tracking
-------------------
Worklog Id: (was: 792229)
Remaining Estimate: 0h
Time Spent: 10m
> Clean up MoverExecutor after each iteration to avoid potential thread leak
> --------------------------------------------------------------------------
>
> Key: HDFS-16668
> URL: https://issues.apache.org/jira/browse/HDFS-16668
> Project: Hadoop HDFS
> Issue Type: Improvement
> Affects Versions: 3.3.3
> Reporter: Tai Zhou
> Priority: Major
> Attachments: screenshot-1.png, screenshot-2.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Hi,
> I am working on a HDFS smart storage management project recently. It is based
> on the Mover in Hadoop-hdfs project. I noticed that most code in Mover is
> similar to Balancer. However, Mover doesn't clean up MoverExecutor as
> Balancer does.
> If we have multiple NameSystem for Namenode Connectors or have a large number
> of datanodes, Mover will result in threads leaking because there might be
> numerous iterations to process these namespaces. Like our project, we
> modified some source code so that we can use mover.run() once we found the
> blocks did not match the expected storage policies. So our application will
> initialize Namenode Connector and Mover continually. It turns out we have
> thousands of threads or threads pools for MoverExecutor.
> here is what it looks like. We can see here are 9000+ threads like this in
> WAIT condition.
> !screenshot-2.png|width=558,height=209!
> I know generally users may not use Mover like us. They might use it by CLI.
> But more and more users are planing to apply RBF or multiple NameSystems, or
> with a large cluster of datanodes. Mover CLI have to keep more than thousands
> of thread after pressing the enter key.
> I have pulled a quick fix code, if you guys are interested, plz take a look
> at it.
> thx.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
