[jira] [Commented] (HDFS-12050) Ozone: StorageHandler: Implementation of "close" for releasing resources during shutdown
[ https://issues.apache.org/jira/browse/HDFS-12050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16455126#comment-16455126 ] Hudson commented on HDFS-12050: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14070 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14070/]) HDFS-12050. Ozone: StorageHandler: Implementation of close for releasing (omalley: rev 0a8c903d3f43eb4fc7c0327e04caf12f8cf5b949) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ObjectStoreHandler.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/web/interfaces/StorageHandler.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/web/storage/DistributedStorageHandler.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/web/localstorage/LocalStorageHandler.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/scm/XceiverClientManager.java > Ozone: StorageHandler: Implementation of "close" for releasing resources > during shutdown > > > Key: HDFS-12050 > URL: https://issues.apache.org/jira/browse/HDFS-12050 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Nanda kumar >Assignee: Nanda kumar >Priority: Minor > Attachments: HDFS-12050-HDFS-7240.000.patch, > HDFS-12050-HDFS-7240.001.patch > > > When we use DistributedStorageHandler and call {{newKeyWriter}}, it will > create {{XceiverClientSpi}} and add it to clientCache which will create a > non-daemon {{nioEventLoopGroup}} thread. Since > {{XceiverClientManager#releaseClient}} doesn't invalidate the client, close > is not called on the client object. Cleanup is triggered as part of both > eviction as well as releaseClient, but the connection is closed only when > both of the conditions are satisfied. > This {{StorageHandler#close}} can be used to close the connections during > shutdown. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12050) Ozone: StorageHandler: Implementation of "close" for releasing resources during shutdown
[ https://issues.apache.org/jira/browse/HDFS-12050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16450876#comment-16450876 ] Hudson commented on HDFS-12050: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14057 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14057/]) HDFS-12050. Ozone: StorageHandler: Implementation of close for releasing (cliang: rev ba647764f3bccc318956b24104f972f4c9b0d35d) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/web/interfaces/StorageHandler.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/scm/XceiverClientManager.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ObjectStoreHandler.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/web/storage/DistributedStorageHandler.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/web/localstorage/LocalStorageHandler.java > Ozone: StorageHandler: Implementation of "close" for releasing resources > during shutdown > > > Key: HDFS-12050 > URL: https://issues.apache.org/jira/browse/HDFS-12050 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Nanda kumar >Assignee: Nanda kumar >Priority: Minor > Attachments: HDFS-12050-HDFS-7240.000.patch, > HDFS-12050-HDFS-7240.001.patch > > > When we use DistributedStorageHandler and call {{newKeyWriter}}, it will > create {{XceiverClientSpi}} and add it to clientCache which will create a > non-daemon {{nioEventLoopGroup}} thread. Since > {{XceiverClientManager#releaseClient}} doesn't invalidate the client, close > is not called on the client object. Cleanup is triggered as part of both > eviction as well as releaseClient, but the connection is closed only when > both of the conditions are satisfied. > This {{StorageHandler#close}} can be used to close the connections during > shutdown. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12050) Ozone: StorageHandler: Implementation of "close" for releasing resources during shutdown
[ https://issues.apache.org/jira/browse/HDFS-12050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16069030#comment-16069030 ] Chen Liang commented on HDFS-12050: --- The failed tests are unrelated. Committed to the feature branch, thanks [~nandakumar131] for the contribution! > Ozone: StorageHandler: Implementation of "close" for releasing resources > during shutdown > > > Key: HDFS-12050 > URL: https://issues.apache.org/jira/browse/HDFS-12050 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Nandakumar >Assignee: Nandakumar >Priority: Minor > Attachments: HDFS-12050-HDFS-7240.000.patch, > HDFS-12050-HDFS-7240.001.patch > > > When we use DistributedStorageHandler and call {{newKeyWriter}}, it will > create {{XceiverClientSpi}} and add it to clientCache which will create a > non-daemon {{nioEventLoopGroup}} thread. Since > {{XceiverClientManager#releaseClient}} doesn't invalidate the client, close > is not called on the client object. Cleanup is triggered as part of both > eviction as well as releaseClient, but the connection is closed only when > both of the conditions are satisfied. > This {{StorageHandler#close}} can be used to close the connections during > shutdown. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12050) Ozone: StorageHandler: Implementation of "close" for releasing resources during shutdown
[ https://issues.apache.org/jira/browse/HDFS-12050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16067380#comment-16067380 ] Hadoop QA commented on HDFS-12050: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 53s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 13s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 13s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 33s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m 6s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 5s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 44s{color} | {color:green} HDFS-7240 passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 43s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 52s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 35s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 17s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 76m 33s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 23s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}122m 39s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.ozone.container.ozoneimpl.TestRatisManager | | | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 | | | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting | | | hadoop.ozone.container.ozoneimpl.TestOzoneContainer | | | hadoop.ozone.container.ozoneimpl.TestOzoneContainerRatis | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 | | Timed out junit tests | org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HDFS-12050 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12874940/HDFS-12050-HDFS-7240.001.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 26b51ba7c823 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | HDFS-7240 / 4c68393 | | Default Java | 1.8.0_131 | | findbugs | v3.1.0-RC1 | | unit |
[jira] [Commented] (HDFS-12050) Ozone: StorageHandler: Implementation of "close" for releasing resources during shutdown
[ https://issues.apache.org/jira/browse/HDFS-12050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16067075#comment-16067075 ] Nandakumar commented on HDFS-12050: --- Upload patch v01 with {{clientCache.cleanup()}} in {{XceiverClientManager}} > Ozone: StorageHandler: Implementation of "close" for releasing resources > during shutdown > > > Key: HDFS-12050 > URL: https://issues.apache.org/jira/browse/HDFS-12050 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Nandakumar >Assignee: Nandakumar >Priority: Minor > Attachments: HDFS-12050-HDFS-7240.000.patch, > HDFS-12050-HDFS-7240.001.patch > > > When we use DistributedStorageHandler and call {{newKeyWriter}}, it will > create {{XceiverClientSpi}} and add it to clientCache which will create a > non-daemon {{nioEventLoopGroup}} thread. Since > {{XceiverClientManager#releaseClient}} doesn't invalidate the client, close > is not called on the client object. Cleanup is triggered as part of both > eviction as well as releaseClient, but the connection is closed only when > both of the conditions are satisfied. > This {{StorageHandler#close}} can be used to close the connections during > shutdown. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12050) Ozone: StorageHandler: Implementation of "close" for releasing resources during shutdown
[ https://issues.apache.org/jira/browse/HDFS-12050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16067054#comment-16067054 ] Nandakumar commented on HDFS-12050: --- Thanks [~vagarychen] for the review. We only have {{RemovalListener}} configured for {{clientCache}}, so I think it's better to call {{clientCache.cleanup()}} after {{clientCache.invalidateAll()}}. It will make sure that RemovalListener is invoked immediately. I will upload a new patch with the same. > Ozone: StorageHandler: Implementation of "close" for releasing resources > during shutdown > > > Key: HDFS-12050 > URL: https://issues.apache.org/jira/browse/HDFS-12050 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Nandakumar >Assignee: Nandakumar >Priority: Minor > Attachments: HDFS-12050-HDFS-7240.000.patch > > > When we use DistributedStorageHandler and call {{newKeyWriter}}, it will > create {{XceiverClientSpi}} and add it to clientCache which will create a > non-daemon {{nioEventLoopGroup}} thread. Since > {{XceiverClientManager#releaseClient}} doesn't invalidate the client, close > is not called on the client object. Cleanup is triggered as part of both > eviction as well as releaseClient, but the connection is closed only when > both of the conditions are satisfied. > This {{StorageHandler#close}} can be used to close the connections during > shutdown. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12050) Ozone: StorageHandler: Implementation of "close" for releasing resources during shutdown
[ https://issues.apache.org/jira/browse/HDFS-12050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16066961#comment-16066961 ] Hadoop QA commented on HDFS-12050: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 40s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 30s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 47s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 43s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 47s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 48s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 42s{color} | {color:green} HDFS-7240 passed {color} | | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 45s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 42s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 24s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 92m 2s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 20s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}134m 30s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 | | | hadoop.ozone.container.ozoneimpl.TestRatisManager | | | hadoop.ozone.container.ozoneimpl.TestOzoneContainerRatis | | | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting | | | hadoop.hdfs.server.balancer.TestBalancer | | | hadoop.hdfs.web.TestWebHdfsTimeouts | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HDFS-12050 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12874730/HDFS-12050-HDFS-7240.000.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 92a3d9ebeb89 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | HDFS-7240 / f98c352 | | Default Java | 1.8.0_131 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/20077/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/20077/testReport/ | |
[jira] [Commented] (HDFS-12050) Ozone: StorageHandler: Implementation of "close" for releasing resources during shutdown
[ https://issues.apache.org/jira/browse/HDFS-12050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16066943#comment-16066943 ] Chen Liang commented on HDFS-12050: --- Thanks [~nandakumar131] for the catch! The patch LGTM, one question though, I'm not entirely sure about this, but do we also need to call {{clientCache.cleanup()}} somewhere? (e.g. before clientCache.invalidateAll()). > Ozone: StorageHandler: Implementation of "close" for releasing resources > during shutdown > > > Key: HDFS-12050 > URL: https://issues.apache.org/jira/browse/HDFS-12050 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Nandakumar >Assignee: Nandakumar >Priority: Minor > Attachments: HDFS-12050-HDFS-7240.000.patch > > > When we use DistributedStorageHandler and call {{newKeyWriter}}, it will > create {{XceiverClientSpi}} and add it to clientCache which will create a > non-daemon {{nioEventLoopGroup}} thread. Since > {{XceiverClientManager#releaseClient}} doesn't invalidate the client, close > is not called on the client object. Cleanup is triggered as part of both > eviction as well as releaseClient, but the connection is closed only when > both of the conditions are satisfied. > This {{StorageHandler#close}} can be used to close the connections during > shutdown. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12050) Ozone: StorageHandler: Implementation of "close" for releasing resources during shutdown
[ https://issues.apache.org/jira/browse/HDFS-12050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065375#comment-16065375 ] Nandakumar commented on HDFS-12050: --- Without this patch eviction is done only when eviction criteria is satisfied (like timeout, cache reaches max limit etc). > Ozone: StorageHandler: Implementation of "close" for releasing resources > during shutdown > > > Key: HDFS-12050 > URL: https://issues.apache.org/jira/browse/HDFS-12050 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Nandakumar >Assignee: Nandakumar > > When we use DistributedStorageHandler and call {{newKeyWriter}}, it will > create {{XceiverClientSpi}} and add it to clientCache which will create a > non-daemon {{nioEventLoopGroup}} thread. Since > {{XceiverClientManager#releaseClient}} doesn't invalidate the client, close > is not called on the client object. Cleanup is triggered as part of both > eviction as well as releaseClient, but the connection is closed only when > both of the conditions are satisfied. > This {{StorageHandler#close}} can be used to close the connections during > shutdown. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org