[jira] [Commented] (HDFS-12163) Ozone: MiniOzoneCluster uses 400+ threads
[
https://issues.apache.org/jira/browse/HDFS-12163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16110619#comment-16110619
]
Hadoop QA commented on HDFS-12163:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
51s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
57s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 72m 33s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}100m 27s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.web.client.TestKeys |
| | hadoop.cblock.TestCBlockReadWrite |
| | hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy |
| | hadoop.ozone.ksm.TestKSMMetrcis |
| | hadoop.cblock.TestBufferManager |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.server.namenode.ha.TestInitializeSharedEdits |
| Timed out junit tests | org.apache.hadoop.ozone.web.client.TestKeysRatis |
| | org.apache.hadoop.ozone.container.ozoneimpl.TestOzoneContainerRatis |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12163 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12879976/HDFS-12163-HDFS-7240.002.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 17e48af37438 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 92945d0 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/20529/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/20529/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/20529/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This
[jira] [Commented] (HDFS-12163) Ozone: MiniOzoneCluster uses 400+ threads
[ https://issues.apache.org/jira/browse/HDFS-12163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16110235#comment-16110235 ] Weiwei Yang commented on HDFS-12163: Hi [~anu] +1 to use the builder pattern, I will revise the patch to do so. Do you want me to remove rest of changes if you think those are non-trivial and not that necessary? Thanks > Ozone: MiniOzoneCluster uses 400+ threads > - > > Key: HDFS-12163 > URL: https://issues.apache.org/jira/browse/HDFS-12163 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone, test >Reporter: Tsz Wo Nicholas Sze >Assignee: Weiwei Yang > Attachments: HDFS-12163-HDFS-7240.001.patch, most_used_threads.png, > TestOzoneThreadCount20170719.patch, thread_dump.png > > > Checked the number of active threads used in MiniOzoneCluster with various > settings: > - Local handlers > - Distributed handlers > - Ratis-Netty > - Ratis-gRPC > The results are similar for all the settings. It uses 400+ threads for an > 1-datanode MiniOzoneCluster. > Moreover, there is a thread leak -- a number of the threads do not shutdown > after the test is finished. Therefore, when tests run consecutively, the > later tests use more threads. > Will post the details in comments. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12163) Ozone: MiniOzoneCluster uses 400+ threads
[ https://issues.apache.org/jira/browse/HDFS-12163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16109349#comment-16109349 ] Anu Engineer commented on HDFS-12163: - Not sure if we need all the other changes too for the thread count to come down, or only setting handlers to 20 is needed. I suggest that we make this change via the builder pattern we have instead of the Debugging patch you have. So some tests are able to change the miniOzoneCluster KSM handler count. Or you can check if the thread count is set to a default value and if so, set them to 20. Either one works. Conceptually, I am +1 on this change. Thanks for finding and fixing it. > Ozone: MiniOzoneCluster uses 400+ threads > - > > Key: HDFS-12163 > URL: https://issues.apache.org/jira/browse/HDFS-12163 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone, test >Reporter: Tsz Wo Nicholas Sze >Assignee: Weiwei Yang > Attachments: HDFS-12163-HDFS-7240.001.patch, most_used_threads.png, > TestOzoneThreadCount20170719.patch, thread_dump.png > > > Checked the number of active threads used in MiniOzoneCluster with various > settings: > - Local handlers > - Distributed handlers > - Ratis-Netty > - Ratis-gRPC > The results are similar for all the settings. It uses 400+ threads for an > 1-datanode MiniOzoneCluster. > Moreover, there is a thread leak -- a number of the threads do not shutdown > after the test is finished. Therefore, when tests run consecutively, the > later tests use more threads. > Will post the details in comments. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12163) Ozone: MiniOzoneCluster uses 400+ threads
[ https://issues.apache.org/jira/browse/HDFS-12163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16108610#comment-16108610 ] Weiwei Yang commented on HDFS-12163: Thanks [~linyiqun] for the review, I will address your comment in next patch. > Ozone: MiniOzoneCluster uses 400+ threads > - > > Key: HDFS-12163 > URL: https://issues.apache.org/jira/browse/HDFS-12163 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone, test >Reporter: Tsz Wo Nicholas Sze >Assignee: Weiwei Yang > Attachments: HDFS-12163-HDFS-7240.001.patch, most_used_threads.png, > TestOzoneThreadCount20170719.patch, thread_dump.png > > > Checked the number of active threads used in MiniOzoneCluster with various > settings: > - Local handlers > - Distributed handlers > - Ratis-Netty > - Ratis-gRPC > The results are similar for all the settings. It uses 400+ threads for an > 1-datanode MiniOzoneCluster. > Moreover, there is a thread leak -- a number of the threads do not shutdown > after the test is finished. Therefore, when tests run consecutively, the > later tests use more threads. > Will post the details in comments. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12163) Ozone: MiniOzoneCluster uses 400+ threads
[
https://issues.apache.org/jira/browse/HDFS-12163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16108559#comment-16108559
]
Hadoop QA commented on HDFS-12163:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
38s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
41s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
36s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
36s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
40s{color} | {color:green} HDFS-7240 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
16s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 67m 30s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}107m 10s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.web.client.TestKeys |
| | hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy |
| | hadoop.ozone.ksm.TestKSMMetrcis |
| | hadoop.ozone.scm.TestXceiverClientManager |
| Timed out junit tests | org.apache.hadoop.ozone.web.client.TestKeysRatis |
| | org.apache.hadoop.ozone.container.ozoneimpl.TestOzoneContainerRatis |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | HDFS-12163 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12879778/HDFS-12163-HDFS-7240.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux aee6fd7fb78c 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-7240 / 1fd2790 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/20513/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
[jira] [Commented] (HDFS-12163) Ozone: MiniOzoneCluster uses 400+ threads
[
https://issues.apache.org/jira/browse/HDFS-12163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16108483#comment-16108483
]
Yiqun Lin commented on HDFS-12163:
--
Thanks [~cheersyang], very nice working!
I just take a quick look of your patch. i have one comment:
{code}
+ // Overwrite handler number to 20, 200 is not necessary for testing.
+ conf.setInt(KSMConfigKeys.OZONE_KSM_HANDLER_COUNT_KEY, 20);
{code}
Can you define a new var like following instead of the number {{20}}? It will
be easy to find and be adjusted once we want to change the value later.
{code}
private static final int KSM_TEST_HANDLER_COUNT = 20;
{code}
Thanks.
> Ozone: MiniOzoneCluster uses 400+ threads
> -
>
> Key: HDFS-12163
> URL: https://issues.apache.org/jira/browse/HDFS-12163
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone, test
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Weiwei Yang
> Attachments: HDFS-12163-HDFS-7240.001.patch, most_used_threads.png,
> TestOzoneThreadCount20170719.patch, thread_dump.png
>
>
> Checked the number of active threads used in MiniOzoneCluster with various
> settings:
> - Local handlers
> - Distributed handlers
> - Ratis-Netty
> - Ratis-gRPC
> The results are similar for all the settings. It uses 400+ threads for an
> 1-datanode MiniOzoneCluster.
> Moreover, there is a thread leak -- a number of the threads do not shutdown
> after the test is finished. Therefore, when tests run consecutively, the
> later tests use more threads.
> Will post the details in comments.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12163) Ozone: MiniOzoneCluster uses 400+ threads
[ https://issues.apache.org/jira/browse/HDFS-12163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16108441#comment-16108441 ] Weiwei Yang commented on HDFS-12163: Submitted v1 patch to fix 2 issues 1) set KSM handler to 20 instead of 200 for mini ozone cluster; 2) fix some potential leaks. Following tables are the comparison result before and after applying the patch, I have only tested distributed/local handler test cases, Distributed handlers with 1 datanode || Mode, numOfDn || Step || NumOfThreads || Change || | (distributed,1) | init | 6 | 0 | | (distributed,1) | MiniOzoneCluster | 222 | *-180* | | (distributed,1) | shutdown | 79 | *-3* | | (distributed,1) | sleep | 12 | *-1* | Local handlers with 1 datanode || Mode, numOfDn || Step || NumOfThreads || Change || | (local,1) | init | 6 | *-11* | | (local,1) | MiniOzoneCluster | 222 | *-183* | | (local,1) | shutdown | 79 | *-6* | | (local,1) | sleep | 12 | *-4* | Distributed handlers with 5 datanode || Mode, numOfDn || Step || NumOfThreads || Change || | (distributed,5) | init | 6 | 0 | | (distributed,5) | MiniOzoneCluster | 407 | *-180* | | (distributed,5) | shutdown | 336 | *-11* | | (distributed,5) | sleep | 13 | *-2* | Local handlers with 5 datanode || Mode, numOfDn || Step || NumOfThreads || Change || | (local,5) | init | 16 | *-4* | | (local,5) | MiniOzoneCluster | 408 | *-184* | | (local,5) | shutdown | 337 | *-15* | | (local,5) | sleep | 14 | *-5* | > Ozone: MiniOzoneCluster uses 400+ threads > - > > Key: HDFS-12163 > URL: https://issues.apache.org/jira/browse/HDFS-12163 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone, test >Reporter: Tsz Wo Nicholas Sze >Assignee: Weiwei Yang > Attachments: most_used_threads.png, > TestOzoneThreadCount20170719.patch, thread_dump.png > > > Checked the number of active threads used in MiniOzoneCluster with various > settings: > - Local handlers > - Distributed handlers > - Ratis-Netty > - Ratis-gRPC > The results are similar for all the settings. It uses 400+ threads for an > 1-datanode MiniOzoneCluster. > Moreover, there is a thread leak -- a number of the threads do not shutdown > after the test is finished. Therefore, when tests run consecutively, the > later tests use more threads. > Will post the details in comments. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12163) Ozone: MiniOzoneCluster uses 400+ threads
[
https://issues.apache.org/jira/browse/HDFS-12163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16108392#comment-16108392
]
Weiwei Yang commented on HDFS-12163:
Hi [~anu]
Here is the result after set 20 handler for KSM,
| (distributed,1) | init | 6 |
| (distributed,1) | MiniOzoneCluster | 222 |
| (distributed,1) | shutdown | 80 |
| (distributed,1) | sleep | 13 |
I will overwrite this value to 20 in {{MiniOzoneCluster}} since 200 is not
necessary for testing.
> Ozone: MiniOzoneCluster uses 400+ threads
> -
>
> Key: HDFS-12163
> URL: https://issues.apache.org/jira/browse/HDFS-12163
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone, test
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Weiwei Yang
> Attachments: most_used_threads.png,
> TestOzoneThreadCount20170719.patch, thread_dump.png
>
>
> Checked the number of active threads used in MiniOzoneCluster with various
> settings:
> - Local handlers
> - Distributed handlers
> - Ratis-Netty
> - Ratis-gRPC
> The results are similar for all the settings. It uses 400+ threads for an
> 1-datanode MiniOzoneCluster.
> Moreover, there is a thread leak -- a number of the threads do not shutdown
> after the test is finished. Therefore, when tests run consecutively, the
> later tests use more threads.
> Will post the details in comments.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12163) Ozone: MiniOzoneCluster uses 400+ threads
[ https://issues.apache.org/jira/browse/HDFS-12163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16108378#comment-16108378 ] Anu Engineer commented on HDFS-12163: - that is a nice find, can you please post the thread count with something like 20 KSM handler count? Also, we need to file a JIRA to track that issue. We should not launch that many threads unless we need them, but I am guessing this is part of RPC layer, so we might not want to change it now. > Ozone: MiniOzoneCluster uses 400+ threads > - > > Key: HDFS-12163 > URL: https://issues.apache.org/jira/browse/HDFS-12163 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone, test >Reporter: Tsz Wo Nicholas Sze >Assignee: Weiwei Yang > Attachments: most_used_threads.png, > TestOzoneThreadCount20170719.patch, thread_dump.png > > > Checked the number of active threads used in MiniOzoneCluster with various > settings: > - Local handlers > - Distributed handlers > - Ratis-Netty > - Ratis-gRPC > The results are similar for all the settings. It uses 400+ threads for an > 1-datanode MiniOzoneCluster. > Moreover, there is a thread leak -- a number of the threads do not shutdown > after the test is finished. Therefore, when tests run consecutively, the > later tests use more threads. > Will post the details in comments. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12163) Ozone: MiniOzoneCluster uses 400+ threads
[ https://issues.apache.org/jira/browse/HDFS-12163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16108379#comment-16108379 ] Anu Engineer commented on HDFS-12163: - bq. I will check the leak issue a bit later. [~xyao] has fixed a bunch of leak issues, they were due to us not closing the miniOzoneCluster. > Ozone: MiniOzoneCluster uses 400+ threads > - > > Key: HDFS-12163 > URL: https://issues.apache.org/jira/browse/HDFS-12163 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone, test >Reporter: Tsz Wo Nicholas Sze >Assignee: Weiwei Yang > Attachments: most_used_threads.png, > TestOzoneThreadCount20170719.patch, thread_dump.png > > > Checked the number of active threads used in MiniOzoneCluster with various > settings: > - Local handlers > - Distributed handlers > - Ratis-Netty > - Ratis-gRPC > The results are similar for all the settings. It uses 400+ threads for an > 1-datanode MiniOzoneCluster. > Moreover, there is a thread leak -- a number of the threads do not shutdown > after the test is finished. Therefore, when tests run consecutively, the > later tests use more threads. > Will post the details in comments. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12163) Ozone: MiniOzoneCluster uses 400+ threads
[
https://issues.apache.org/jira/browse/HDFS-12163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16108359#comment-16108359
]
Weiwei Yang commented on HDFS-12163:
Run 1 DN in mini ozone cluster, there were *402* threads once the cluster is
up, checked the thread dump, there are *280* RPC handler threads, with
following trace
{code}
IPC Server handler 0 on 51641@2330 - priority:5 - threadId:0x45 - nativeId:NA -
state:WAITING
stackTrace:
java.lang.Thread.State: WAITING
at sun.misc.Unsafe.park(Unsafe.java:-1)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:287)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2661)
{code}
this is because
{{dfs.storage.service.handler.count}} = {{10}}
{{ozone.ksm.handler.count.key}} = {{200}}
we are setting a pretty big value for KSM handlers, this is not necessary for a
mini cluster, we can overwrite this config in mini cluster to reduce the number
of threads. I will check the leak issue a bit later.
> Ozone: MiniOzoneCluster uses 400+ threads
> -
>
> Key: HDFS-12163
> URL: https://issues.apache.org/jira/browse/HDFS-12163
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone, test
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Weiwei Yang
> Attachments: TestOzoneThreadCount20170719.patch
>
>
> Checked the number of active threads used in MiniOzoneCluster with various
> settings:
> - Local handlers
> - Distributed handlers
> - Ratis-Netty
> - Ratis-gRPC
> The results are similar for all the settings. It uses 400+ threads for an
> 1-datanode MiniOzoneCluster.
> Moreover, there is a thread leak -- a number of the threads do not shutdown
> after the test is finished. Therefore, when tests run consecutively, the
> later tests use more threads.
> Will post the details in comments.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12163) Ozone: MiniOzoneCluster uses 400+ threads
[ https://issues.apache.org/jira/browse/HDFS-12163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16101467#comment-16101467 ] Weiwei Yang commented on HDFS-12163: Got a bit occupied with delete key work, won't be working on this too soon :(. If anybody likes to take over, please go ahead. Otherwise I will revisit this after that part is done.:) > Ozone: MiniOzoneCluster uses 400+ threads > - > > Key: HDFS-12163 > URL: https://issues.apache.org/jira/browse/HDFS-12163 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone, test >Reporter: Tsz Wo Nicholas Sze >Assignee: Weiwei Yang > Attachments: TestOzoneThreadCount20170719.patch > > > Checked the number of active threads used in MiniOzoneCluster with various > settings: > - Local handlers > - Distributed handlers > - Ratis-Netty > - Ratis-gRPC > The results are similar for all the settings. It uses 400+ threads for an > 1-datanode MiniOzoneCluster. > Moreover, there is a thread leak -- a number of the threads do not shutdown > after the test is finished. Therefore, when tests run consecutively, the > later tests use more threads. > Will post the details in comments. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12163) Ozone: MiniOzoneCluster uses 400+ threads
[ https://issues.apache.org/jira/browse/HDFS-12163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16094984#comment-16094984 ] Tsz Wo Nicholas Sze commented on HDFS-12163: Running using mvn gives similar results. This problem is not specific to IntelliJ. - mvn test with 1-datanode cluster- | (distributed,1) | init | 5 | | (distributed,1) | MiniOzoneCluster | 401 | | (distributed,1) | shutdown | 53 | | (distributed,1) | sleep | 26 | | (GRPC,1) | init | 26 | | (GRPC,1) | MiniOzoneCluster | 421 | | (GRPC,1) | createRatisCluster | 431 | | (GRPC,1) | shutdown | 84 | | (GRPC,1) | sleep | 65 | | (local,1) | init | 65 | | (local,1) | MiniOzoneCluster | 461 | | (local,1) | shutdown | 113 | | (local,1) | sleep | 104 | | (NETTY,1) | init | 104 | | (NETTY,1) | MiniOzoneCluster | 504 | | (NETTY,1) | createRatisCluster | 526 | | (NETTY,1) | shutdown | 178 | | (NETTY,1) | sleep | 162 | - mvn test with 5-datanode clusters | (distributed,5) | init | 5 | | (distributed,5) | MiniOzoneCluster | 592 | | (distributed,5) | shutdown | 212 | | (distributed,5) | sleep | 93 | | (GRPC,5) | init | 93 | | (GRPC,5) | MiniOzoneCluster | 692 | | (GRPC,5) | createRatisCluster | 739 | | (GRPC,5) | shutdown | 359 | | (GRPC,5) | sleep | 289 | | (local,5) | init | 289 | | (local,5) | MiniOzoneCluster | 902 | | (local,5) | shutdown | 523 | | (local,5) | sleep | 501 | | (NETTY,5) | init | 501 | | (NETTY,5) | MiniOzoneCluster | 1152 | | (NETTY,5) | createRatisCluster | 1280 | | (NETTY,5) | shutdown | 903 | | (NETTY,5) | sleep | 845 | > Ozone: MiniOzoneCluster uses 400+ threads > - > > Key: HDFS-12163 > URL: https://issues.apache.org/jira/browse/HDFS-12163 > Project: Hadoop HDFS > Issue Type: Bug > Components: ozone, test >Reporter: Tsz Wo Nicholas Sze >Assignee: Weiwei Yang > Attachments: TestOzoneThreadCount20170719.patch > > > Checked the number of active threads used in MiniOzoneCluster with various > settings: > - Local handlers > - Distributed handlers > - Ratis-Netty > - Ratis-gRPC > The results are similar for all the settings. It uses 400+ threads for an > 1-datanode MiniOzoneCluster. > Moreover, there is a thread leak -- a number of the threads do not shutdown > after the test is finished. Therefore, when tests run consecutively, the > later tests use more threads. > Will post the details in comments. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12163) Ozone: MiniOzoneCluster uses 400+ threads
[ https://issues.apache.org/jira/browse/HDFS-12163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16094961#comment-16094961 ] Tsz Wo Nicholas Sze commented on HDFS-12163: [~cheersyang], thanks for taking this up. Assigning to you... > Ozone: MiniOzoneCluster uses 400+ threads > - > > Key: HDFS-12163 > URL: https://issues.apache.org/jira/browse/HDFS-12163 > Project: Hadoop HDFS > Issue Type: Bug > Components: ozone, test >Reporter: Tsz Wo Nicholas Sze >Assignee: Weiwei Yang > Attachments: TestOzoneThreadCount20170719.patch > > > Checked the number of active threads used in MiniOzoneCluster with various > settings: > - Local handlers > - Distributed handlers > - Ratis-Netty > - Ratis-gRPC > The results are similar for all the settings. It uses 400+ threads for an > 1-datanode MiniOzoneCluster. > Moreover, there is a thread leak -- a number of the threads do not shutdown > after the test is finished. Therefore, when tests run consecutively, the > later tests use more threads. > Will post the details in comments. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12163) Ozone: MiniOzoneCluster uses 400+ threads
[ https://issues.apache.org/jira/browse/HDFS-12163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16094067#comment-16094067 ] Weiwei Yang commented on HDFS-12163: Very interesting, I observed a number of leaks as well in the past. Most recent one I am working on is HDFS-12098, that problem was caused by thread leak too. [~szetszwo] are you working on a fix? If not, I can look into this problem this weekend. > Ozone: MiniOzoneCluster uses 400+ threads > - > > Key: HDFS-12163 > URL: https://issues.apache.org/jira/browse/HDFS-12163 > Project: Hadoop HDFS > Issue Type: Bug > Components: ozone, test >Reporter: Tsz Wo Nicholas Sze > Attachments: TestOzoneThreadCount20170719.patch > > > Checked the number of active threads used in MiniOzoneCluster with various > settings: > - Local handlers > - Distributed handlers > - Ratis-Netty > - Ratis-gRPC > The results are similar for all the settings. It uses 400+ threads for an > 1-datanode MiniOzoneCluster. > Moreover, there is a thread leak -- a number of the threads do not shutdown > after the test is finished. Therefore, when tests run consecutively, the > later tests use more threads. > Will post the details in comments. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12163) Ozone: MiniOzoneCluster uses 400+ threads
[ https://issues.apache.org/jira/browse/HDFS-12163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16093885#comment-16093885 ] Tsz Wo Nicholas Sze commented on HDFS-12163: All the results above are generated using IntelliJ. I will try running the tests with maven directly. > Ozone: MiniOzoneCluster uses 400+ threads > - > > Key: HDFS-12163 > URL: https://issues.apache.org/jira/browse/HDFS-12163 > Project: Hadoop HDFS > Issue Type: Bug > Components: ozone, test >Reporter: Tsz Wo Nicholas Sze > Attachments: TestOzoneThreadCount20170719.patch > > > Checked the number of active threads used in MiniOzoneCluster with various > settings: > - Local handlers > - Distributed handlers > - Ratis-Netty > - Ratis-gRPC > The results are similar for all the settings. It uses 400+ threads for an > 1-datanode MiniOzoneCluster. > Moreover, there is a thread leak -- a number of the threads do not shutdown > after the test is finished. Therefore, when tests run consecutively, the > later tests use more threads. > Will post the details in comments. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12163) Ozone: MiniOzoneCluster uses 400+ threads
[ https://issues.apache.org/jira/browse/HDFS-12163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16093852#comment-16093852 ] Tsz Wo Nicholas Sze commented on HDFS-12163: The thread leak is serious in the 5-datanode consecutive run. There are 838 threads remaining at the end. > Ozone: MiniOzoneCluster uses 400+ threads > - > > Key: HDFS-12163 > URL: https://issues.apache.org/jira/browse/HDFS-12163 > Project: Hadoop HDFS > Issue Type: Bug > Components: ozone, test >Reporter: Tsz Wo Nicholas Sze > Attachments: TestOzoneThreadCount20170719.patch > > > Checked the number of active threads used in MiniOzoneCluster with various > settings: > - Local handlers > - Distributed handlers > - Ratis-Netty > - Ratis-gRPC > The results are similar for all the settings. It uses 400+ threads for an > 1-datanode MiniOzoneCluster. > Moreover, there is a thread leak -- a number of the threads do not shutdown > after the test is finished. Therefore, when tests run consecutively, the > later tests use more threads. > Will post the details in comments. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12163) Ozone: MiniOzoneCluster uses 400+ threads
[ https://issues.apache.org/jira/browse/HDFS-12163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16093848#comment-16093848 ] Tsz Wo Nicholas Sze commented on HDFS-12163: 5-datanode tests - Local handlers with 5 datanode | (local,5) | init | 6 | | (local,5) | MiniOzoneCluster | 589 | | (local,5) | shutdown | 209 | | (local,5) | sleep | 89 | - Distributed handlers with 5 datanode | (distributed,5) | init | 6 | | (distributed,5) | MiniOzoneCluster | 589 | | (distributed,5) | shutdown | 209 | | (distributed,5) | sleep | 89 | - Ratis-Netty handlers with 5 datanode | (NETTY,5) | init | 6 | | (NETTY,5) | MiniOzoneCluster | 600 | | (NETTY,5) | createRatisCluster | 712 | | (NETTY,5) | shutdown | 332 | | (NETTY,5) | sleep | 132 | - Ratis-gRPC handlers with 5 datanode | (GRPC,5) | init | 6 | | (GRPC,5) | MiniOzoneCluster | 596 | | (GRPC,5) | createRatisCluster | 637 | | (GRPC,5) | shutdown | 257 | | (GRPC,5) | sleep | 138 | - All 4 cases above running consecutively. | (distributed,5) | init | 6 | | (distributed,5) | MiniOzoneCluster | 589 | | (distributed,5) | shutdown | 209 | | (distributed,5) | sleep | 90 | | (GRPC,5) | init | 90 | | (GRPC,5) | MiniOzoneCluster | 691 | | (GRPC,5) | createRatisCluster | 740 | | (GRPC,5) | shutdown | 360 | | (GRPC,5) | sleep | 290 | | (local,5) | init | 290 | | (local,5) | MiniOzoneCluster | 907 | | (local,5) | shutdown | 527 | | (local,5) | sleep | 505 | | (NETTY,5) | init | 505 | | (NETTY,5) | MiniOzoneCluster | 1147 | | (NETTY,5) | createRatisCluster | 1269 | | (NETTY,5) | shutdown | 890 | | (NETTY,5) | sleep | 838 | > Ozone: MiniOzoneCluster uses 400+ threads > - > > Key: HDFS-12163 > URL: https://issues.apache.org/jira/browse/HDFS-12163 > Project: Hadoop HDFS > Issue Type: Bug > Components: ozone, test >Reporter: Tsz Wo Nicholas Sze > Attachments: TestOzoneThreadCount20170719.patch > > > Checked the number of active threads used in MiniOzoneCluster with various > settings: > - Local handlers > - Distributed handlers > - Ratis-Netty > - Ratis-gRPC > The results are similar for all the settings. It uses 400+ threads for an > 1-datanode MiniOzoneCluster. > Moreover, there is a thread leak -- a number of the threads do not shutdown > after the test is finished. Therefore, when tests run consecutively, the > later tests use more threads. > Will post the details in comments. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12163) Ozone: MiniOzoneCluster uses 400+ threads
[ https://issues.apache.org/jira/browse/HDFS-12163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16093842#comment-16093842 ] Tsz Wo Nicholas Sze commented on HDFS-12163: 1-datanode tests - Ratis-gRPC with 1 datanode | (GRPC,1) | init | 6 | | (GRPC,1) | MiniOzoneCluster | 404 | | (GRPC,1) | createRatisCluster | 413 | | (GRPC,1) | shutdown | 65 | | (GRPC,1) | sleep | 38 | - Ratis-Netty with 1 datanode | (NETTY,1) | init | 6 | | (NETTY,1) | MiniOzoneCluster | 404 | | (NETTY,1) | createRatisCluster | 424 | | (NETTY,1) | shutdown | 76 | | (NETTY,1) | sleep | 33 | - Distributed handlers with 1 datanode | (distributed,1) | init | 6 | | (distributed,1) | MiniOzoneCluster | 402 | | (distributed,1) | shutdown | 54 | | (distributed,1) | sleep | 27 | - Local handlers with 1 datanode | (local,1) | init | 6 | | (local,1) | MiniOzoneCluster | 402 | | (local,1) | shutdown | 54 | | (local,1) | sleep | 27 | All the results above are from separated runs. Below is the result running them together. - All 4 cases above running consecutively. | (distributed,1) | init | 6 | | (distributed,1) | MiniOzoneCluster | 402 | | (distributed,1) | shutdown | 54 | | (distributed,1) | sleep | 27 | | (GRPC,1) | init | 27 | | (GRPC,1) | MiniOzoneCluster | 423 | | (GRPC,1) | createRatisCluster | 434 | | (GRPC,1) | shutdown | 85 | | (GRPC,1) | sleep | 67 | | (local,1) | init | 67 | | (local,1) | MiniOzoneCluster | 463 | | (local,1) | shutdown | 115 | | (local,1) | sleep | 106 | | (NETTY,1) | init | 106 | | (NETTY,1) | MiniOzoneCluster | 507 | | (NETTY,1) | createRatisCluster | 530 | | (NETTY,1) | shutdown | 182 | | (NETTY,1) | sleep | 166 | As shown in the table, some threads are not terminated after MiniOzoneCluster shutdown and a 10-second sleep. For example, in (distributed,1), it has 6 threads initially but there are 27 threads after sleep. > Ozone: MiniOzoneCluster uses 400+ threads > - > > Key: HDFS-12163 > URL: https://issues.apache.org/jira/browse/HDFS-12163 > Project: Hadoop HDFS > Issue Type: Bug > Components: ozone, test >Reporter: Tsz Wo Nicholas Sze > Attachments: TestOzoneThreadCount20170719.patch > > > Checked the number of active threads used in MiniOzoneCluster with various > settings: > - Local handlers > - Distributed handlers > - Ratis-Netty > - Ratis-gRPC > The results are similar for all the settings. It uses 400+ threads for an > 1-datanode MiniOzoneCluster. > Moreover, there is a thread leak -- a number of the threads do not shutdown > after the test is finished. Therefore, when tests run consecutively, the > later tests use more threads. > Will post the details in comments. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
