[jira] [Commented] (HDFS-12163) Ozone: MiniOzoneCluster uses 400+ threads
[ https://issues.apache.org/jira/browse/HDFS-12163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108483#comment-16108483 ] Yiqun Lin commented on HDFS-12163: -- Thanks [~cheersyang], very nice working! I just take a quick look of your patch. i have one comment: {code} + // Overwrite handler number to 20, 200 is not necessary for testing. + conf.setInt(KSMConfigKeys.OZONE_KSM_HANDLER_COUNT_KEY, 20); {code} Can you define a new var like following instead of the number {{20}}? It will be easy to find and be adjusted once we want to change the value later. {code} private static final int KSM_TEST_HANDLER_COUNT = 20; {code} Thanks. > Ozone: MiniOzoneCluster uses 400+ threads > - > > Key: HDFS-12163 > URL: https://issues.apache.org/jira/browse/HDFS-12163 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone, test >Reporter: Tsz Wo Nicholas Sze >Assignee: Weiwei Yang > Attachments: HDFS-12163-HDFS-7240.001.patch, most_used_threads.png, > TestOzoneThreadCount20170719.patch, thread_dump.png > > > Checked the number of active threads used in MiniOzoneCluster with various > settings: > - Local handlers > - Distributed handlers > - Ratis-Netty > - Ratis-gRPC > The results are similar for all the settings. It uses 400+ threads for an > 1-datanode MiniOzoneCluster. > Moreover, there is a thread leak -- a number of the threads do not shutdown > after the test is finished. Therefore, when tests run consecutively, the > later tests use more threads. > Will post the details in comments. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12082) BlockInvalidateLimit value is incorrectly set after namenode heartbeat interval reconfigured
[ https://issues.apache.org/jira/browse/HDFS-12082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108469#comment-16108469 ] Weiwei Yang commented on HDFS-12082: Thanks [~arpiagariu] for the help :). > BlockInvalidateLimit value is incorrectly set after namenode heartbeat > interval reconfigured > - > > Key: HDFS-12082 > URL: https://issues.apache.org/jira/browse/HDFS-12082 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs, namenode >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: HDFS-12082.001.patch, HDFS-12082.002.patch, > HDFS-12082.003.patch, HDFS-12082.004.patch > > > HDFS-1477 provides an option to reconfigured namenode heartbeat interval > without restarting the namenode. When the heartbeat interval is reconfigured, > {{blockInvalidateLimit}} gets recounted > {code} > this.blockInvalidateLimit = Math.max(20 * (int) (intervalSeconds), > DFSConfigKeys.DFS_BLOCK_INVALIDATE_LIMIT_DEFAULT); > {code} > this doesn't honor the existing value set by {{dfs.block.invalidate.limit}}. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12235) Ozone: DeleteKey-3: KSM SCM block deletion message and ACK interactions
[ https://issues.apache.org/jira/browse/HDFS-12235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiwei Yang updated HDFS-12235: --- Description: KSM and SCM interaction for delete key operation, both KSM and SCM stores key state info in a backlog, KSM needs to scan this log and send block-deletion command to SCM, once SCM is fully aware of the message, KSM removes the key completely from namespace. See more from the design doc under HDFS-11922, this is task break down 2. (was: KSM and SCM interaction for delete key operation, both KSM and SCM stores key state info in a backlog, KSM needs to scan this log and send block-deletion command to SCM, once SCM is fully aware of the message, KSM removes the key completely from namespace.) > Ozone: DeleteKey-3: KSM SCM block deletion message and ACK interactions > --- > > Key: HDFS-12235 > URL: https://issues.apache.org/jira/browse/HDFS-12235 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Weiwei Yang >Assignee: Weiwei Yang > > KSM and SCM interaction for delete key operation, both KSM and SCM stores key > state info in a backlog, KSM needs to scan this log and send block-deletion > command to SCM, once SCM is fully aware of the message, KSM removes the key > completely from namespace. See more from the design doc under HDFS-11922, > this is task break down 2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12235) Ozone: DeleteKey-3: KSM SCM block deletion message and ACK interactions
Weiwei Yang created HDFS-12235: -- Summary: Ozone: DeleteKey-3: KSM SCM block deletion message and ACK interactions Key: HDFS-12235 URL: https://issues.apache.org/jira/browse/HDFS-12235 Project: Hadoop HDFS Issue Type: Sub-task Components: ozone Affects Versions: HDFS-7240 Reporter: Weiwei Yang Assignee: Weiwei Yang KSM and SCM interaction for delete key operation, both KSM and SCM stores key state info in a backlog, KSM needs to scan this log and send block-deletion command to SCM, once SCM is fully aware of the message, KSM removes the key completely from namespace. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12163) Ozone: MiniOzoneCluster uses 400+ threads
[ https://issues.apache.org/jira/browse/HDFS-12163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiwei Yang updated HDFS-12163: --- Attachment: HDFS-12163-HDFS-7240.001.patch > Ozone: MiniOzoneCluster uses 400+ threads > - > > Key: HDFS-12163 > URL: https://issues.apache.org/jira/browse/HDFS-12163 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone, test >Reporter: Tsz Wo Nicholas Sze >Assignee: Weiwei Yang > Attachments: HDFS-12163-HDFS-7240.001.patch, most_used_threads.png, > TestOzoneThreadCount20170719.patch, thread_dump.png > > > Checked the number of active threads used in MiniOzoneCluster with various > settings: > - Local handlers > - Distributed handlers > - Ratis-Netty > - Ratis-gRPC > The results are similar for all the settings. It uses 400+ threads for an > 1-datanode MiniOzoneCluster. > Moreover, there is a thread leak -- a number of the threads do not shutdown > after the test is finished. Therefore, when tests run consecutively, the > later tests use more threads. > Will post the details in comments. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12163) Ozone: MiniOzoneCluster uses 400+ threads
[ https://issues.apache.org/jira/browse/HDFS-12163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiwei Yang updated HDFS-12163: --- Status: Patch Available (was: Open) > Ozone: MiniOzoneCluster uses 400+ threads > - > > Key: HDFS-12163 > URL: https://issues.apache.org/jira/browse/HDFS-12163 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone, test >Reporter: Tsz Wo Nicholas Sze >Assignee: Weiwei Yang > Attachments: HDFS-12163-HDFS-7240.001.patch, most_used_threads.png, > TestOzoneThreadCount20170719.patch, thread_dump.png > > > Checked the number of active threads used in MiniOzoneCluster with various > settings: > - Local handlers > - Distributed handlers > - Ratis-Netty > - Ratis-gRPC > The results are similar for all the settings. It uses 400+ threads for an > 1-datanode MiniOzoneCluster. > Moreover, there is a thread leak -- a number of the threads do not shutdown > after the test is finished. Therefore, when tests run consecutively, the > later tests use more threads. > Will post the details in comments. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-12163) Ozone: MiniOzoneCluster uses 400+ threads
[ https://issues.apache.org/jira/browse/HDFS-12163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108441#comment-16108441 ] Weiwei Yang edited comment on HDFS-12163 at 8/1/17 6:27 AM: Submitted v1 patch to fix 2 issues 1) set KSM handler to 20 instead of 200 for mini ozone cluster; 2) fix some potential leaks. Following tables are the comparison result before and after applying the patch, I have only tested distributed/local handler test cases, Distributed handlers with 1 datanode || Mode, numOfDn || Step || NumOfThreads || Change || | (distributed,1) | init | 6 | 0 | | (distributed,1) | MiniOzoneCluster | 222 | {color:red} *-180* {color} | | (distributed,1) | shutdown | 79 | {color:red}*-3*{color} | | (distributed,1) | sleep | 12 | {color:red}*-1*{color} | Local handlers with 1 datanode || Mode, numOfDn || Step || NumOfThreads || Change || | (local,1) | init | 6 | {color:red}*-11*{color} | | (local,1) | MiniOzoneCluster | 222 | {color:red}*-183*{color} | | (local,1) | shutdown | 79 | {color:red}*-6*{color} | | (local,1) | sleep | 12 | {color:red}*-4*{color} | Distributed handlers with 5 datanode || Mode, numOfDn || Step || NumOfThreads || Change || | (distributed,5) | init | 6 | 0 | | (distributed,5) | MiniOzoneCluster | 407 | {color:red}*-180*{color} | | (distributed,5) | shutdown | 336 | {color:red}*-11*{color} | | (distributed,5) | sleep | 13 | {color:red}*-2*{color} | Local handlers with 5 datanode || Mode, numOfDn || Step || NumOfThreads || Change || | (local,5) | init | 16 | {color:red}*-4*{color} | | (local,5) | MiniOzoneCluster | 408 | {color:red}*-184*{color} | | (local,5) | shutdown | 337 | {color:red}*-15*{color} | | (local,5) | sleep | 14 | {color:red}*-5*{color} | Hope it makes sense. was (Author: cheersyang): Submitted v1 patch to fix 2 issues 1) set KSM handler to 20 instead of 200 for mini ozone cluster; 2) fix some potential leaks. Following tables are the comparison result before and after applying the patch, I have only tested distributed/local handler test cases, Distributed handlers with 1 datanode || Mode, numOfDn || Step || NumOfThreads || Change || | (distributed,1) | init | 6 | 0 | | (distributed,1) | MiniOzoneCluster | 222 | *-180* | | (distributed,1) | shutdown | 79 | *-3* | | (distributed,1) | sleep | 12 | *-1* | Local handlers with 1 datanode || Mode, numOfDn || Step || NumOfThreads || Change || | (local,1) | init | 6 | *-11* | | (local,1) | MiniOzoneCluster | 222 | *-183* | | (local,1) | shutdown | 79 | *-6* | | (local,1) | sleep | 12 | *-4* | Distributed handlers with 5 datanode || Mode, numOfDn || Step || NumOfThreads || Change || | (distributed,5) | init | 6 | 0 | | (distributed,5) | MiniOzoneCluster | 407 | *-180* | | (distributed,5) | shutdown | 336 | *-11* | | (distributed,5) | sleep | 13 | *-2* | Local handlers with 5 datanode || Mode, numOfDn || Step || NumOfThreads || Change || | (local,5) | init | 16 | *-4* | | (local,5) | MiniOzoneCluster | 408 | *-184* | | (local,5) | shutdown | 337 | *-15* | | (local,5) | sleep | 14 | *-5* | > Ozone: MiniOzoneCluster uses 400+ threads > - > > Key: HDFS-12163 > URL: https://issues.apache.org/jira/browse/HDFS-12163 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone, test >Reporter: Tsz Wo Nicholas Sze >Assignee: Weiwei Yang > Attachments: most_used_threads.png, > TestOzoneThreadCount20170719.patch, thread_dump.png > > > Checked the number of active threads used in MiniOzoneCluster with various > settings: > - Local handlers > - Distributed handlers > - Ratis-Netty > - Ratis-gRPC > The results are similar for all the settings. It uses 400+ threads for an > 1-datanode MiniOzoneCluster. > Moreover, there is a thread leak -- a number of the threads do not shutdown > after the test is finished. Therefore, when tests run consecutively, the > later tests use more threads. > Will post the details in comments. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12163) Ozone: MiniOzoneCluster uses 400+ threads
[ https://issues.apache.org/jira/browse/HDFS-12163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108441#comment-16108441 ] Weiwei Yang commented on HDFS-12163: Submitted v1 patch to fix 2 issues 1) set KSM handler to 20 instead of 200 for mini ozone cluster; 2) fix some potential leaks. Following tables are the comparison result before and after applying the patch, I have only tested distributed/local handler test cases, Distributed handlers with 1 datanode || Mode, numOfDn || Step || NumOfThreads || Change || | (distributed,1) | init | 6 | 0 | | (distributed,1) | MiniOzoneCluster | 222 | *-180* | | (distributed,1) | shutdown | 79 | *-3* | | (distributed,1) | sleep | 12 | *-1* | Local handlers with 1 datanode || Mode, numOfDn || Step || NumOfThreads || Change || | (local,1) | init | 6 | *-11* | | (local,1) | MiniOzoneCluster | 222 | *-183* | | (local,1) | shutdown | 79 | *-6* | | (local,1) | sleep | 12 | *-4* | Distributed handlers with 5 datanode || Mode, numOfDn || Step || NumOfThreads || Change || | (distributed,5) | init | 6 | 0 | | (distributed,5) | MiniOzoneCluster | 407 | *-180* | | (distributed,5) | shutdown | 336 | *-11* | | (distributed,5) | sleep | 13 | *-2* | Local handlers with 5 datanode || Mode, numOfDn || Step || NumOfThreads || Change || | (local,5) | init | 16 | *-4* | | (local,5) | MiniOzoneCluster | 408 | *-184* | | (local,5) | shutdown | 337 | *-15* | | (local,5) | sleep | 14 | *-5* | > Ozone: MiniOzoneCluster uses 400+ threads > - > > Key: HDFS-12163 > URL: https://issues.apache.org/jira/browse/HDFS-12163 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone, test >Reporter: Tsz Wo Nicholas Sze >Assignee: Weiwei Yang > Attachments: most_used_threads.png, > TestOzoneThreadCount20170719.patch, thread_dump.png > > > Checked the number of active threads used in MiniOzoneCluster with various > settings: > - Local handlers > - Distributed handlers > - Ratis-Netty > - Ratis-gRPC > The results are similar for all the settings. It uses 400+ threads for an > 1-datanode MiniOzoneCluster. > Moreover, there is a thread leak -- a number of the threads do not shutdown > after the test is finished. Therefore, when tests run consecutively, the > later tests use more threads. > Will post the details in comments. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11984) Ozone: Ensures listKey lists all required key fields
[ https://issues.apache.org/jira/browse/HDFS-11984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiwei Yang updated HDFS-11984: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: HDFS-7240 Status: Resolved (was: Patch Available) > Ozone: Ensures listKey lists all required key fields > > > Key: HDFS-11984 > URL: https://issues.apache.org/jira/browse/HDFS-11984 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Weiwei Yang >Assignee: Yiqun Lin > Fix For: HDFS-7240 > > Attachments: HDFS-11984-HDFS-7240.001.patch, > HDFS-11984-HDFS-7240.002.patch, HDFS-11984-HDFS-7240.003.patch, > HDFS-11984-HDFS-7240.004.patch > > > HDFS-11782 implements the listKey operation which only lists the basic key > fields, we need to make sure it return all required fields > # version > # md5hash > # createdOn > # size > # keyName > this task is depending on the work of HDFS-11886. See more discussion [here | > https://issues.apache.org/jira/browse/HDFS-11782?focusedCommentId=16045562&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16045562]. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11984) Ozone: Ensures listKey lists all required key fields
[ https://issues.apache.org/jira/browse/HDFS-11984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108405#comment-16108405 ] Weiwei Yang commented on HDFS-11984: Looks good, the failure tests seemed not related. I am going to commit this shortly. > Ozone: Ensures listKey lists all required key fields > > > Key: HDFS-11984 > URL: https://issues.apache.org/jira/browse/HDFS-11984 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Weiwei Yang >Assignee: Yiqun Lin > Attachments: HDFS-11984-HDFS-7240.001.patch, > HDFS-11984-HDFS-7240.002.patch, HDFS-11984-HDFS-7240.003.patch, > HDFS-11984-HDFS-7240.004.patch > > > HDFS-11782 implements the listKey operation which only lists the basic key > fields, we need to make sure it return all required fields > # version > # md5hash > # createdOn > # size > # keyName > this task is depending on the work of HDFS-11886. See more discussion [here | > https://issues.apache.org/jira/browse/HDFS-11782?focusedCommentId=16045562&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16045562]. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9213) Minicluster with Kerberos generates some stacks when checking the ports
[ https://issues.apache.org/jira/browse/HDFS-9213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108402#comment-16108402 ] Hadoop QA commented on HDFS-9213: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 59s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 42s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 7s{color} | {color:green} trunk passed {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 55s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 47s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 58s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 58s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 38s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 7 unchanged - 0 fixed = 10 total (was 7) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 5s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 46s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red}102m 9s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 16s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}132m 20s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 | | | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting | | | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 | | | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HDFS-9213 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12766796/hdfs-9213.v1.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 24833789ce4b 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / a4aa1cb | | Default Java | 1.8.0_131 | | findbugs | v3.1.0-RC1 | | findbugs | https://builds.apache.org/job/PreCommit-HDFS-Build/20512/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html | | checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/20512/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/20512/artifac
[jira] [Comment Edited] (HDFS-12163) Ozone: MiniOzoneCluster uses 400+ threads
[ https://issues.apache.org/jira/browse/HDFS-12163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108392#comment-16108392 ] Weiwei Yang edited comment on HDFS-12163 at 8/1/17 5:27 AM: Hi [~anu] Here is the result after set 20 handler for KSM, | (distributed,1) | init | 6 | | (distributed,1) | MiniOzoneCluster | 222 | | (distributed,1) | shutdown | 80 | | (distributed,1) | sleep | 13 | I will overwrite this value to 20 in {{MiniOzoneCluster}} since 200 is not necessary for testing. bq. Xiaoyu Yao has fixed a bunch of leak issues, they were due to us not closing the miniOzoneCluster. Yes, it looks like some of the leaks are already fixed, we can see the number of open threads decreased to 13 now. I will take a look if there is any other potential leak and fix them if I found any. was (Author: cheersyang): Hi [~anu] Here is the result after set 20 handler for KSM, | (distributed,1) | init | 6 | | (distributed,1) | MiniOzoneCluster | 222 | | (distributed,1) | shutdown | 80 | | (distributed,1) | sleep | 13 | I will overwrite this value to 20 in {{MiniOzoneCluster}} since 200 is not necessary for testing. > Ozone: MiniOzoneCluster uses 400+ threads > - > > Key: HDFS-12163 > URL: https://issues.apache.org/jira/browse/HDFS-12163 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone, test >Reporter: Tsz Wo Nicholas Sze >Assignee: Weiwei Yang > Attachments: most_used_threads.png, > TestOzoneThreadCount20170719.patch, thread_dump.png > > > Checked the number of active threads used in MiniOzoneCluster with various > settings: > - Local handlers > - Distributed handlers > - Ratis-Netty > - Ratis-gRPC > The results are similar for all the settings. It uses 400+ threads for an > 1-datanode MiniOzoneCluster. > Moreover, there is a thread leak -- a number of the threads do not shutdown > after the test is finished. Therefore, when tests run consecutively, the > later tests use more threads. > Will post the details in comments. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12163) Ozone: MiniOzoneCluster uses 400+ threads
[ https://issues.apache.org/jira/browse/HDFS-12163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108392#comment-16108392 ] Weiwei Yang commented on HDFS-12163: Hi [~anu] Here is the result after set 20 handler for KSM, | (distributed,1) | init | 6 | | (distributed,1) | MiniOzoneCluster | 222 | | (distributed,1) | shutdown | 80 | | (distributed,1) | sleep | 13 | I will overwrite this value to 20 in {{MiniOzoneCluster}} since 200 is not necessary for testing. > Ozone: MiniOzoneCluster uses 400+ threads > - > > Key: HDFS-12163 > URL: https://issues.apache.org/jira/browse/HDFS-12163 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone, test >Reporter: Tsz Wo Nicholas Sze >Assignee: Weiwei Yang > Attachments: most_used_threads.png, > TestOzoneThreadCount20170719.patch, thread_dump.png > > > Checked the number of active threads used in MiniOzoneCluster with various > settings: > - Local handlers > - Distributed handlers > - Ratis-Netty > - Ratis-gRPC > The results are similar for all the settings. It uses 400+ threads for an > 1-datanode MiniOzoneCluster. > Moreover, there is a thread leak -- a number of the threads do not shutdown > after the test is finished. Therefore, when tests run consecutively, the > later tests use more threads. > Will post the details in comments. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12151) Hadoop 2 clients cannot writeBlock to Hadoop 3 DataNodes
[ https://issues.apache.org/jira/browse/HDFS-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108388#comment-16108388 ] Hadoop QA commented on HDFS-12151: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 29s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 56s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 3s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 43s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 58s{color} | {color:green} trunk passed {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 40s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 47s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 45s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 50s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 46s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red}102m 3s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 21s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}131m 33s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HDFS-12151 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12879765/HDFS-12151.007.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux c587a50d23d6 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 1a78c0f | | Default Java | 1.8.0_131 | | findbugs | v3.1.0-RC1 | | findbugs | https://builds.apache.org/job/PreCommit-HDFS-Build/20511/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/20511/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/20511/testReport/ | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/20511/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Hadoop 2 clients cannot writeBlock to Hadoop 3 DataNodes > --
[jira] [Commented] (HDFS-12163) Ozone: MiniOzoneCluster uses 400+ threads
[ https://issues.apache.org/jira/browse/HDFS-12163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108378#comment-16108378 ] Anu Engineer commented on HDFS-12163: - that is a nice find, can you please post the thread count with something like 20 KSM handler count? Also, we need to file a JIRA to track that issue. We should not launch that many threads unless we need them, but I am guessing this is part of RPC layer, so we might not want to change it now. > Ozone: MiniOzoneCluster uses 400+ threads > - > > Key: HDFS-12163 > URL: https://issues.apache.org/jira/browse/HDFS-12163 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone, test >Reporter: Tsz Wo Nicholas Sze >Assignee: Weiwei Yang > Attachments: most_used_threads.png, > TestOzoneThreadCount20170719.patch, thread_dump.png > > > Checked the number of active threads used in MiniOzoneCluster with various > settings: > - Local handlers > - Distributed handlers > - Ratis-Netty > - Ratis-gRPC > The results are similar for all the settings. It uses 400+ threads for an > 1-datanode MiniOzoneCluster. > Moreover, there is a thread leak -- a number of the threads do not shutdown > after the test is finished. Therefore, when tests run consecutively, the > later tests use more threads. > Will post the details in comments. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12163) Ozone: MiniOzoneCluster uses 400+ threads
[ https://issues.apache.org/jira/browse/HDFS-12163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108379#comment-16108379 ] Anu Engineer commented on HDFS-12163: - bq. I will check the leak issue a bit later. [~xyao] has fixed a bunch of leak issues, they were due to us not closing the miniOzoneCluster. > Ozone: MiniOzoneCluster uses 400+ threads > - > > Key: HDFS-12163 > URL: https://issues.apache.org/jira/browse/HDFS-12163 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone, test >Reporter: Tsz Wo Nicholas Sze >Assignee: Weiwei Yang > Attachments: most_used_threads.png, > TestOzoneThreadCount20170719.patch, thread_dump.png > > > Checked the number of active threads used in MiniOzoneCluster with various > settings: > - Local handlers > - Distributed handlers > - Ratis-Netty > - Ratis-gRPC > The results are similar for all the settings. It uses 400+ threads for an > 1-datanode MiniOzoneCluster. > Moreover, there is a thread leak -- a number of the threads do not shutdown > after the test is finished. Therefore, when tests run consecutively, the > later tests use more threads. > Will post the details in comments. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12157) Do fsyncDirectory(..) outside of FSDataset lock
[ https://issues.apache.org/jira/browse/HDFS-12157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108367#comment-16108367 ] Hadoop QA commented on HDFS-12157: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m 41s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} branch-2.7 Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 8s{color} | {color:green} branch-2.7 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 57s{color} | {color:green} branch-2.7 passed with JDK v1.8.0_131 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s{color} | {color:green} branch-2.7 passed with JDK v1.7.0_131 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 26s{color} | {color:green} branch-2.7 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 59s{color} | {color:green} branch-2.7 passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 52s{color} | {color:green} branch-2.7 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 59s{color} | {color:green} branch-2.7 passed with JDK v1.8.0_131 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 45s{color} | {color:green} branch-2.7 passed with JDK v1.7.0_131 {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 52s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s{color} | {color:green} the patch passed with JDK v1.8.0_131 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 54s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 58s{color} | {color:green} the patch passed with JDK v1.7.0_131 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 58s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 22s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 125 unchanged - 1 fixed = 125 total (was 126) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 56s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 60 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 3s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s{color} | {color:green} the patch passed with JDK v1.8.0_131 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 43s{color} | {color:green} the patch passed with JDK v1.7.0_131 {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 43m 38s{color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_131. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 18s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}130m 13s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | JDK v1.8.0_131 Failed junit tests | hadoop.hdfs.web.TestHttpsFileSystem | | | hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots | | JDK v1.7.0_131 Failed junit tests | hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes | | | hadoop.hdfs.TestDFSShell | | | hadoop.hdfs.web.TestHttpsFileSystem | | | hadoop.hdfs.server.datanode.TestDirectoryScanner | | | hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots | \\ \\ || Subsystem || Report/Notes || | Docker | Image:y
[jira] [Commented] (HDFS-11984) Ozone: Ensures listKey lists all required key fields
[ https://issues.apache.org/jira/browse/HDFS-11984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108366#comment-16108366 ] Hadoop QA commented on HDFS-11984: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 4 new or modified test files. {color} | || || || || {color:brown} HDFS-7240 Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 52s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 8s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 48s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 13s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 21s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 0s{color} | {color:green} HDFS-7240 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 4s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 43s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 9s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 29s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 73m 38s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 18s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}107m 33s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.ozone.container.common.TestDatanodeStateMachine | | | hadoop.ozone.ksm.TestKSMMetrcis | | | hadoop.hdfs.TestEncryptionZones | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 | | | hadoop.hdfs.server.datanode.TestDirectoryScanner | | | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 | | | hadoop.ozone.web.client.TestKeys | | Timed out junit tests | org.apache.hadoop.ozone.web.client.TestKeysRatis | | | org.apache.hadoop.ozone.container.ozoneimpl.TestOzoneContainerRatis | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HDFS-11984 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12879764/HDFS-11984-HDFS-7240.004.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux c9e20e3a4fef 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | HDFS-7240 / c49297b | | Default Java | 1.8.0_131 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/20510/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/20510/testReport/ | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/20510/console
[jira] [Updated] (HDFS-12163) Ozone: MiniOzoneCluster uses 400+ threads
[ https://issues.apache.org/jira/browse/HDFS-12163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiwei Yang updated HDFS-12163: --- Attachment: most_used_threads.png > Ozone: MiniOzoneCluster uses 400+ threads > - > > Key: HDFS-12163 > URL: https://issues.apache.org/jira/browse/HDFS-12163 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone, test >Reporter: Tsz Wo Nicholas Sze >Assignee: Weiwei Yang > Attachments: most_used_threads.png, > TestOzoneThreadCount20170719.patch, thread_dump.png > > > Checked the number of active threads used in MiniOzoneCluster with various > settings: > - Local handlers > - Distributed handlers > - Ratis-Netty > - Ratis-gRPC > The results are similar for all the settings. It uses 400+ threads for an > 1-datanode MiniOzoneCluster. > Moreover, there is a thread leak -- a number of the threads do not shutdown > after the test is finished. Therefore, when tests run consecutively, the > later tests use more threads. > Will post the details in comments. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12163) Ozone: MiniOzoneCluster uses 400+ threads
[ https://issues.apache.org/jira/browse/HDFS-12163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiwei Yang updated HDFS-12163: --- Attachment: thread_dump.png > Ozone: MiniOzoneCluster uses 400+ threads > - > > Key: HDFS-12163 > URL: https://issues.apache.org/jira/browse/HDFS-12163 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone, test >Reporter: Tsz Wo Nicholas Sze >Assignee: Weiwei Yang > Attachments: TestOzoneThreadCount20170719.patch, thread_dump.png > > > Checked the number of active threads used in MiniOzoneCluster with various > settings: > - Local handlers > - Distributed handlers > - Ratis-Netty > - Ratis-gRPC > The results are similar for all the settings. It uses 400+ threads for an > 1-datanode MiniOzoneCluster. > Moreover, there is a thread leak -- a number of the threads do not shutdown > after the test is finished. Therefore, when tests run consecutively, the > later tests use more threads. > Will post the details in comments. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12163) Ozone: MiniOzoneCluster uses 400+ threads
[ https://issues.apache.org/jira/browse/HDFS-12163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108359#comment-16108359 ] Weiwei Yang commented on HDFS-12163: Run 1 DN in mini ozone cluster, there were *402* threads once the cluster is up, checked the thread dump, there are *280* RPC handler threads, with following trace {code} IPC Server handler 0 on 51641@2330 - priority:5 - threadId:0x45 - nativeId:NA - state:WAITING stackTrace: java.lang.Thread.State: WAITING at sun.misc.Unsafe.park(Unsafe.java:-1) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) at org.apache.hadoop.ipc.CallQueueManager.take(CallQueueManager.java:287) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2661) {code} this is because {{dfs.storage.service.handler.count}} = {{10}} {{ozone.ksm.handler.count.key}} = {{200}} we are setting a pretty big value for KSM handlers, this is not necessary for a mini cluster, we can overwrite this config in mini cluster to reduce the number of threads. I will check the leak issue a bit later. > Ozone: MiniOzoneCluster uses 400+ threads > - > > Key: HDFS-12163 > URL: https://issues.apache.org/jira/browse/HDFS-12163 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone, test >Reporter: Tsz Wo Nicholas Sze >Assignee: Weiwei Yang > Attachments: TestOzoneThreadCount20170719.patch > > > Checked the number of active threads used in MiniOzoneCluster with various > settings: > - Local handlers > - Distributed handlers > - Ratis-Netty > - Ratis-gRPC > The results are similar for all the settings. It uses 400+ threads for an > 1-datanode MiniOzoneCluster. > Moreover, there is a thread leak -- a number of the threads do not shutdown > after the test is finished. Therefore, when tests run consecutively, the > later tests use more threads. > Will post the details in comments. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12173) MiniDFSCluster cannot reliably use NameNode#stop
[ https://issues.apache.org/jira/browse/HDFS-12173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108346#comment-16108346 ] Ajay Yadav commented on HDFS-12173: --- [~daryn], Could you please share more information on this. Since Namenode has only one stop method does it make sense to update Stop method itself to call HAState#setState(context, HAServiceState.STOPPING)? > MiniDFSCluster cannot reliably use NameNode#stop > > > Key: HDFS-12173 > URL: https://issues.apache.org/jira/browse/HDFS-12173 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.7.0 >Reporter: Daryn Sharp > > Sporadic test failures occur because {{NameNode#stop}} used by the mini > cluster does not properly manage the HA context's state. It directly calls > {{HAState#exitState(context)}} instead of > {{HAState#setState(context,state)}}. The latter will properly lock the > namesystem and update the ha state while locked, while the former does not. > The result is that while the cluster is stopping, the lock is released and > any queued rpc calls think the NN is still active and are processed while the > NN is in an unstable half-stopped state. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9213) Minicluster with Kerberos generates some stacks when checking the ports
[ https://issues.apache.org/jira/browse/HDFS-9213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108321#comment-16108321 ] mingleizhang commented on HDFS-9213: We should keep forward working on this jira. In flink project, we can not run some tests until this issue fixed. > Minicluster with Kerberos generates some stacks when checking the ports > --- > > Key: HDFS-9213 > URL: https://issues.apache.org/jira/browse/HDFS-9213 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Affects Versions: 3.0.0-alpha1 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon >Priority: Minor > Attachments: hdfs-9213.v1.patch, hdfs-9213.v1.patch > > > When using the minicluster with kerberos the various checks in > SecureDataNodeStarter fail because the ports are not fixed. > Stacks like this one: > {quote} > java.lang.RuntimeException: Unable to bind on specified streaming port in > secure context. Needed 0, got 49670 > at > org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter.getSecureResources(SecureDataNodeStarter.java:108) > {quote} > There is already a setting to desactivate this type of check for testing, it > could be used here as well -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-9213) Minicluster with Kerberos generates some stacks when checking the ports
[ https://issues.apache.org/jira/browse/HDFS-9213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108322#comment-16108322 ] mingleizhang commented on HDFS-9213: [~linyiqun] Could you please take a look on this PR ? Thanks. :) > Minicluster with Kerberos generates some stacks when checking the ports > --- > > Key: HDFS-9213 > URL: https://issues.apache.org/jira/browse/HDFS-9213 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Affects Versions: 3.0.0-alpha1 >Reporter: Nicolas Liochon >Assignee: Nicolas Liochon >Priority: Minor > Attachments: hdfs-9213.v1.patch, hdfs-9213.v1.patch > > > When using the minicluster with kerberos the various checks in > SecureDataNodeStarter fail because the ports are not fixed. > Stacks like this one: > {quote} > java.lang.RuntimeException: Unable to bind on specified streaming port in > secure context. Needed 0, got 49670 > at > org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter.getSecureResources(SecureDataNodeStarter.java:108) > {quote} > There is already a setting to desactivate this type of check for testing, it > could be used here as well -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11826) Federation Namenode Heartbeat
[ https://issues.apache.org/jira/browse/HDFS-11826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108319#comment-16108319 ] Hadoop QA commented on HDFS-11826: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 6 new or modified test files. {color} | || || || || {color:brown} HDFS-10467 Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 24s{color} | {color:green} HDFS-10467 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s{color} | {color:green} HDFS-10467 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 45s{color} | {color:green} HDFS-10467 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 57s{color} | {color:green} HDFS-10467 passed {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 47s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in HDFS-10467 has 10 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s{color} | {color:green} HDFS-10467 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 46s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 41s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 654 unchanged - 0 fixed = 657 total (was 654) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 52s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 91m 26s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 20s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}121m 29s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090 | | | hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 | | | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HDFS-11826 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12879755/HDFS-11826-HDFS-10467-001.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml | | uname | Linux fd1c99af9284 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | HDFS-10467 / fae1d1e | | Default Java | 1.8.0_131 | | findbugs | v3.1.0-RC1 | | findbugs | https://builds.apache.org/job/PreCommit-HDFS-Build/20508/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html | | checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/20508/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/20508/artifact/patchprocess/patch-unit-hadoop-h
[jira] [Commented] (HDFS-12151) Hadoop 2 clients cannot writeBlock to Hadoop 3 DataNodes
[ https://issues.apache.org/jira/browse/HDFS-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108308#comment-16108308 ] Sean Mackrory commented on HDFS-12151: -- Ah sorry about that - I seem to be blind to the yellow checkstyle warnings... I did confirm that the test failures are flaky. They succeed locally and timeouts also occurred in the same class (often the same function) in several recent runs of the Pre-Commit jobs. Of course I just closed the editor where I noted all the URLs to said jobs, but they're there and they're recent, I promise :) Thanks for the review! Attaching a final patch with the checkstyle issues addressed. Reran the new test and the 2 that failed last time locally, and had a clean Yetus run. > Hadoop 2 clients cannot writeBlock to Hadoop 3 DataNodes > > > Key: HDFS-12151 > URL: https://issues.apache.org/jira/browse/HDFS-12151 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rolling upgrades >Affects Versions: 3.0.0-alpha4 >Reporter: Sean Mackrory >Assignee: Sean Mackrory > Attachments: HDFS-12151.001.patch, HDFS-12151.002.patch, > HDFS-12151.003.patch, HDFS-12151.004.patch, HDFS-12151.005.patch, > HDFS-12151.006.patch, HDFS-12151.007.patch > > > Trying to write to a Hadoop 3 DataNode with a Hadoop 2 client currently > fails. On the client side it looks like this: > {code} > 17/07/14 13:31:58 INFO hdfs.DFSClient: Exception in > createBlockOutputStream > java.io.EOFException: Premature EOF: no length prefix available > at > org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2280) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1318) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1237) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:449){code} > But on the DataNode side there's an ArrayOutOfBoundsException because there > aren't any targetStorageIds: > {code} > java.lang.ArrayIndexOutOfBoundsException: 0 > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:815) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:173) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:107) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) > at java.lang.Thread.run(Thread.java:745){code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12151) Hadoop 2 clients cannot writeBlock to Hadoop 3 DataNodes
[ https://issues.apache.org/jira/browse/HDFS-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Mackrory updated HDFS-12151: - Attachment: HDFS-12151.007.patch > Hadoop 2 clients cannot writeBlock to Hadoop 3 DataNodes > > > Key: HDFS-12151 > URL: https://issues.apache.org/jira/browse/HDFS-12151 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rolling upgrades >Affects Versions: 3.0.0-alpha4 >Reporter: Sean Mackrory >Assignee: Sean Mackrory > Attachments: HDFS-12151.001.patch, HDFS-12151.002.patch, > HDFS-12151.003.patch, HDFS-12151.004.patch, HDFS-12151.005.patch, > HDFS-12151.006.patch, HDFS-12151.007.patch > > > Trying to write to a Hadoop 3 DataNode with a Hadoop 2 client currently > fails. On the client side it looks like this: > {code} > 17/07/14 13:31:58 INFO hdfs.DFSClient: Exception in > createBlockOutputStream > java.io.EOFException: Premature EOF: no length prefix available > at > org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2280) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1318) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1237) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:449){code} > But on the DataNode side there's an ArrayOutOfBoundsException because there > aren't any targetStorageIds: > {code} > java.lang.ArrayIndexOutOfBoundsException: 0 > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:815) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:173) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:107) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) > at java.lang.Thread.run(Thread.java:745){code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11984) Ozone: Ensures listKey lists all required key fields
[ https://issues.apache.org/jira/browse/HDFS-11984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yiqun Lin updated HDFS-11984: - Attachment: HDFS-11984-HDFS-7240.004.patch Rebase the code and attach the new patch. After this, I'd like to add creation time in bucket/volume info as well this week. It will be a useful field to show in the response. > Ozone: Ensures listKey lists all required key fields > > > Key: HDFS-11984 > URL: https://issues.apache.org/jira/browse/HDFS-11984 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Weiwei Yang >Assignee: Yiqun Lin > Attachments: HDFS-11984-HDFS-7240.001.patch, > HDFS-11984-HDFS-7240.002.patch, HDFS-11984-HDFS-7240.003.patch, > HDFS-11984-HDFS-7240.004.patch > > > HDFS-11782 implements the listKey operation which only lists the basic key > fields, we need to make sure it return all required fields > # version > # md5hash > # createdOn > # size > # keyName > this task is depending on the work of HDFS-11886. See more discussion [here | > https://issues.apache.org/jira/browse/HDFS-11782?focusedCommentId=16045562&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16045562]. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12157) Do fsyncDirectory(..) outside of FSDataset lock
[ https://issues.apache.org/jira/browse/HDFS-12157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinayakumar B updated HDFS-12157: - Attachment: HDFS-12157-branch-2.7-01.patch Attaching branch-2.7 patch again for jenkins > Do fsyncDirectory(..) outside of FSDataset lock > --- > > Key: HDFS-12157 > URL: https://issues.apache.org/jira/browse/HDFS-12157 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Reporter: Vinayakumar B >Assignee: Vinayakumar B >Priority: Critical > Attachments: HDFS-12157-01.patch, HDFS-12157-branch-2-01.patch, > HDFS-12157-branch-2.7-01.patch, HDFS-12157-branch-2.7-01.patch > > > HDFS-5042 introduced fsyncDirectory(..) to save blocks from power failure. > Do it outside of FSDataset lock to avoid overall performance degradation if > disk takes more time to sync. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12217) HDFS snapshots doesn't capture all open files when one of the open files is deleted
[ https://issues.apache.org/jira/browse/HDFS-12217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108296#comment-16108296 ] Hadoop QA commented on HDFS-12217: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 33s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 38s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 54s{color} | {color:green} trunk passed {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 40s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 48s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 47s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 36s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 53s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 47s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 67m 48s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 17s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 93m 21s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 | | | hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HDFS-12217 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12879752/HDFS-12217.04.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 425425fe3a9a 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / ea56812 | | Default Java | 1.8.0_131 | | findbugs | v3.1.0-RC1 | | findbugs | https://builds.apache.org/job/PreCommit-HDFS-Build/20507/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/20507/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/20507/testReport/ | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/20507/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > HDFS snapshots doesn't capture all open files when one of the open files is > deleted > -
[jira] [Commented] (HDFS-11082) Erasure Coding : Provide replicated EC policy to just replicating the files
[ https://issues.apache.org/jira/browse/HDFS-11082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108275#comment-16108275 ] Andrew Wang commented on HDFS-11082: Hi [~Sammi] any progress on this one? > Erasure Coding : Provide replicated EC policy to just replicating the files > --- > > Key: HDFS-11082 > URL: https://issues.apache.org/jira/browse/HDFS-11082 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: erasure-coding >Reporter: Rakesh R >Assignee: SammiChen >Priority: Critical > Labels: hdfs-ec-3.0-must-do > > The idea of this jira is to provide a new {{replicated EC policy}} so that we > can override the EC policy on a parent directory and go back to just > replicating the files based on replication factors. > Thanks [~andrew.wang] for the > [discussions|https://issues.apache.org/jira/browse/HDFS-11072?focusedCommentId=15620743&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15620743]. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11826) Federation Namenode Heartbeat
[ https://issues.apache.org/jira/browse/HDFS-11826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Inigo Goiri updated HDFS-11826: --- Attachment: HDFS-11826-HDFS-10467-001.patch * Fixing unit tests. * Fixing check styles. * Adding message when the heartbeating is not properly configured. > Federation Namenode Heartbeat > - > > Key: HDFS-11826 > URL: https://issues.apache.org/jira/browse/HDFS-11826 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs >Reporter: Inigo Goiri >Assignee: Inigo Goiri > Fix For: HDFS-10467 > > Attachments: HDFS-11826-HDFS-10467-000.patch, > HDFS-11826-HDFS-10467-001.patch > > > Add a service to the Router to check the state of a Namenode and report it > into the State Store. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11580) Ozone: Support asynchronus client API for SCM and containers
[ https://issues.apache.org/jira/browse/HDFS-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108229#comment-16108229 ] Anu Engineer commented on HDFS-11580: - [~vagarychen], [~linyiqun], [~nanda], [~msingh] Thanks for the patches and code reviews. I think one of my old changes is creating a road block in this work. Allow me to explain. # When I first wrote this code, I wanted to offer a synchronous interface to make it easy to write clients. So I painfully wrapped the Async interface offered by Netty into a Sync one -- Probably due to a mistaken impression that it will make the life of programmers easy. # Then when we came to do the async work -- what we should have done in the first place is to make the underlying netty's async interface first class, which is currently hidden under the {{sendCommand}} function. # Instead, since the code already offers a synchronous interface, this code patch tries to create a thread pool and attempts to call into the "Sync" interface offered by the XcieverClient. I propose that we *don't* use a thread pool to make the "sync" -> "async", instead, we expose the async interface as-is. # Since I wrote the original code, I propose that I post a patch with the idea that I am talking about so that we can see what I am proposing. Please let me know if you have any concerns or questions. > Ozone: Support asynchronus client API for SCM and containers > > > Key: HDFS-11580 > URL: https://issues.apache.org/jira/browse/HDFS-11580 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Yiqun Lin > Attachments: HDFS-11580-HDFS-7240.001.patch, > HDFS-11580-HDFS-7240.002.patch, HDFS-11580-HDFS-7240.003.patch, > HDFS-11580-HDFS-7240.004.patch, HDFS-11580-HDFS-7240.005.patch, > HDFS-11580-HDFS-7240.006.patch, HDFS-11580-HDFS-7240.007.patch, > HDFS-11580-HDFS-7240.008.patch, HDFS-11580-HDFS-7240.009.patch, > HDFS-11580-HDFS-7240.010.patch, HDFS-11580-HDFS-7240.011.patch > > > This is an umbrella JIRA that needs to support a set of APIs in Asynchronous > form. > For containers -- or the datanode API currently supports a call > {{sendCommand}}. we need to build proper programming interface and support an > async interface. > There is also a set of SCM API that clients can call, it would be nice to > support Async interface for those too. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12234) [SPS] Allow setting Xattr without SPS running.
Lei (Eddy) Xu created HDFS-12234: Summary: [SPS] Allow setting Xattr without SPS running. Key: HDFS-12234 URL: https://issues.apache.org/jira/browse/HDFS-12234 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: HDFS-10285 Reporter: Lei (Eddy) Xu As discussed in HDFS-10285, if this API is widely used by downstream projects (i.e., HBase), it should allow the client to call this API without querying the running status of SPS service. It would introduce great burden for this API to be used. Given the constraints this SPS service has (i.e., can not run with Mover , and might be disabled by default), it should allow the API call success as long as related xattr being persisted. SPS can run later to catch on. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12233) [SPS] Add API to unset SPS on a path
[ https://issues.apache.org/jira/browse/HDFS-12233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lei (Eddy) Xu updated HDFS-12233: - Summary: [SPS] Add API to unset SPS on a path (was: Add API to unset SPS on a path) > [SPS] Add API to unset SPS on a path > > > Key: HDFS-12233 > URL: https://issues.apache.org/jira/browse/HDFS-12233 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Affects Versions: HDFS-10285 >Reporter: Lei (Eddy) Xu > > As discussed in HDFS-10285, we should allow to unset SPS on a path. > For example, an user might mistakenly set SPS on "/", and triggers > significant amount of data movement. Unset SPS will allow user to fix his own > mistake. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12049) Recommissioning live nodes stalls the NN
[ https://issues.apache.org/jira/browse/HDFS-12049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-12049: -- Target Version/s: 2.9.0 (was: 2.8.2) > Recommissioning live nodes stalls the NN > > > Key: HDFS-12049 > URL: https://issues.apache.org/jira/browse/HDFS-12049 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Daryn Sharp >Priority: Critical > > A node refresh will recommission included nodes that are alive and in > decommissioning or decommissioned state. The recommission will scan all > blocks on the node, find over replicated blocks, chose an excess, queue an > invalidate. > The process is expensive and worsened by overhead of storage types (even when > not in use). It can be especially devastating because the write lock is held > for the entire node refresh. _Recommissioning 67 nodes with ~500k > blocks/node stalled rpc services for over 4 mins._ -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12049) Recommissioning live nodes stalls the NN
[ https://issues.apache.org/jira/browse/HDFS-12049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108222#comment-16108222 ] Junping Du commented on HDFS-12049: --- Moved. Thanks Daryn. > Recommissioning live nodes stalls the NN > > > Key: HDFS-12049 > URL: https://issues.apache.org/jira/browse/HDFS-12049 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.6.0 >Reporter: Daryn Sharp >Priority: Critical > > A node refresh will recommission included nodes that are alive and in > decommissioning or decommissioned state. The recommission will scan all > blocks on the node, find over replicated blocks, chose an excess, queue an > invalidate. > The process is expensive and worsened by overhead of storage types (even when > not in use). It can be especially devastating because the write lock is held > for the entire node refresh. _Recommissioning 67 nodes with ~500k > blocks/node stalled rpc services for over 4 mins._ -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-12233) Add API to unset SPS on a path
Lei (Eddy) Xu created HDFS-12233: Summary: Add API to unset SPS on a path Key: HDFS-12233 URL: https://issues.apache.org/jira/browse/HDFS-12233 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, namenode Affects Versions: HDFS-10285 Reporter: Lei (Eddy) Xu As discussed in HDFS-10285, we should allow to unset SPS on a path. For example, an user might mistakenly set SPS on "/", and triggers significant amount of data movement. Unset SPS will allow user to fix his own mistake. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10285) Storage Policy Satisfier in Namenode
[ https://issues.apache.org/jira/browse/HDFS-10285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108218#comment-16108218 ] Lei (Eddy) Xu commented on HDFS-10285: -- Thanks for the reply, [~umamaheswararao]! bq. If you feel things can be done even after merge, Yea, None of the above is blocking the merge, I will file related JIRAs under HDFS-12226. > Storage Policy Satisfier in Namenode > > > Key: HDFS-10285 > URL: https://issues.apache.org/jira/browse/HDFS-10285 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: HDFS-10285 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-10285-consolidated-merge-patch-00.patch, > HDFS-10285-consolidated-merge-patch-01.patch, > HDFS-SPS-TestReport-20170708.pdf, > Storage-Policy-Satisfier-in-HDFS-June-20-2017.pdf, > Storage-Policy-Satisfier-in-HDFS-May10.pdf > > > Heterogeneous storage in HDFS introduced the concept of storage policy. These > policies can be set on directory/file to specify the user preference, where > to store the physical block. When user set the storage policy before writing > data, then the blocks could take advantage of storage policy preferences and > stores physical block accordingly. > If user set the storage policy after writing and completing the file, then > the blocks would have been written with default storage policy (nothing but > DISK). User has to run the ‘Mover tool’ explicitly by specifying all such > file names as a list. In some distributed system scenarios (ex: HBase) it > would be difficult to collect all the files and run the tool as different > nodes can write files separately and file can have different paths. > Another scenarios is, when user rename the files from one effected storage > policy file (inherited policy from parent directory) to another storage > policy effected directory, it will not copy inherited storage policy from > source. So it will take effect from destination file/dir parent storage > policy. This rename operation is just a metadata change in Namenode. The > physical blocks still remain with source storage policy. > So, Tracking all such business logic based file names could be difficult for > admins from distributed nodes(ex: region servers) and running the Mover tool. > Here the proposal is to provide an API from Namenode itself for trigger the > storage policy satisfaction. A Daemon thread inside Namenode should track > such calls and process to DN as movement commands. > Will post the detailed design thoughts document soon. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12217) HDFS snapshots doesn't capture all open files when one of the open files is deleted
[ https://issues.apache.org/jira/browse/HDFS-12217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manoj Govindassamy updated HDFS-12217: -- Attachment: HDFS-12217.04.patch Thanks for the quick review [~jojochuang]. Attached v04 patch to address the following. Can you please take a look at the latest patch. bq. LeaseManager#getINodeWithLeases() seems to be a test-only method. If so, it doesn't need public modifier (package private is sufficient) and we can also add a {{@VisibleForTesting }} annotation. Done. bq. On a separate note, and this is totally unrelated to this patch. It looks like LeaseManager#addLease assumes the inodeId is an id for an INodeFile, which makes perfect sense. Will take this up in a new patch as there are many callers that need to get refactored for this,. bq. Note that in your patch, DirectorySnapshottableFeature#addSnapshot would capture exception, but doesn't log the exception. I also think that in addition to the snapshot name, you should print the snapshot root path for the error message. Done. -- Exception is already logged at LeaseManager#getINodeWithLeases. -- Now added exception details to the SnapshotException message also. > HDFS snapshots doesn't capture all open files when one of the open files is > deleted > --- > > Key: HDFS-12217 > URL: https://issues.apache.org/jira/browse/HDFS-12217 > Project: Hadoop HDFS > Issue Type: Bug > Components: snapshots >Affects Versions: 3.0.0-alpha1 >Reporter: Manoj Govindassamy >Assignee: Manoj Govindassamy > Attachments: HDFS-12217.01.patch, HDFS-12217.02.patch, > HDFS-12217.03.patch, HDFS-12217.04.patch > > > With the fix for HDFS-11402, HDFS Snapshots can additionally capture all the > open files. Just like all other files, these open files in the snapshots will > remain immutable. But, sometimes it is found that snapshots fail to capture > all the open files in the system. > Under the following conditions, LeaseManager will fail to find INode > corresponding to an active lease > * a file is opened for writing (LeaseManager allots a lease), and > * the same file is deleted while it is still open for writing and having > active lease, and > * the same file is not referenced in any other Snapshots/Trash > {{INode[] LeaseManager#getINodesWithLease()}} can thus return null for few > leases there by causing the caller to trip over and not return all the open > files needed by the snapshot manager. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11948) Ozone: change TestRatisManager to check cluster with data
[ https://issues.apache.org/jira/browse/HDFS-11948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-11948: --- Attachment: HDFS-11948-HDFS-7240.20170731.patch > Ozone: change TestRatisManager to check cluster with data > - > > Key: HDFS-11948 > URL: https://issues.apache.org/jira/browse/HDFS-11948 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Attachments: HDFS-11948-HDFS-7240.20170614.patch, > HDFS-11948-HDFS-7240.20170731.patch > > > TestRatisManager first creates multiple Ratis clusters. Then it changes the > membership and closes some clusters. However, it does not test the clusters > with data. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12228) [SPS]: Add storage policy satisfier related metrics
[ https://issues.apache.org/jira/browse/HDFS-12228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108212#comment-16108212 ] Uma Maheswara Rao G commented on HDFS-12228: Please consider [~eddyxu] suggestions below. {quote} And since this call essentially triggers a large async background task, should we put some logs here? Similarly, it'd be nice to have related JMX stats and some indications in web UI, to be easier to integrate with other systems. {quote} I think logs can be improved even before merge. [~eddyxu], could you point us where you looking to ad more logging? > [SPS]: Add storage policy satisfier related metrics > --- > > Key: HDFS-12228 > URL: https://issues.apache.org/jira/browse/HDFS-12228 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Rakesh R > > This jira to discuss and implement metrics needed for SPS feature. > Below are few metrics: > # count of {{inprogress}} block movements > # count of {{successful}} block movements > # count of {{failed}} block movements > Need to analyse and add more. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11948) Ozone: change TestRatisManager to check cluster with data
[ https://issues.apache.org/jira/browse/HDFS-11948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-11948: --- Attachment: (was: HDFS-11948-HDFS-7240.20170731.patch) > Ozone: change TestRatisManager to check cluster with data > - > > Key: HDFS-11948 > URL: https://issues.apache.org/jira/browse/HDFS-11948 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Attachments: HDFS-11948-HDFS-7240.20170614.patch > > > TestRatisManager first creates multiple Ratis clusters. Then it changes the > membership and closes some clusters. However, it does not test the clusters > with data. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10285) Storage Policy Satisfier in Namenode
[ https://issues.apache.org/jira/browse/HDFS-10285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108210#comment-16108210 ] Uma Maheswara Rao G commented on HDFS-10285: Hi [~eddyxu], Thank you for the review. Here are my replies. {quote} Non-recursively set xattr. Please kindly re-consider to use recursive async call. If the use cases are mostly targeted to the downstream projects like HBase and etc., the chance of these projects mistakenly call satisfyStoragePolicy on wrong directory (i.e., "/") is rare, but it will make the projects to manage large / deep namespace difficult, i.e., hbase needs to iterate the namespace itself and calls the same amount of "setXattr" anyway (because the # of files to move is the same). Similar to "rm -rf /", while it is bad that "rm" allows to do it, but IMO it should not prevent users / applications to use "rm -rf" in a sensible way. {quote} Thank you for providing feedback and exposing pinpoints from user stand point of view. As this moment, seems like recursive is more helpful think to consider from the feedbacks, by Andrew and you. We will work on this item. {quote} The newly added public void removeXattr(long id, String xattrName). While its name seems very generic, it seems only allow taking sps xattr as legit parameter. Should we demote it from public API in Namesystem? {quote} This was intentional. Since Namesystem is generic interface between BM and FSNamesystem, API name can be more generic incase if thats useful for other purposes. Means any Xattrs you can pass to this API to remove it. It may not be good to add more specific APIs to it. {quote} Would it make sense to have an admin command to unset SPS on a path? For an user to undo his own mistake. {quote} Make sense to consider it. Would you mind to file a JIRA under HDFS-12226 ? {quote} FSNamesystem#satisfyStoragePolicy. Is this only setting xattr? Can we do the setting xattr part without SPS running? I was thinking the scenarios that: some downstream projects (i.e., hbase) start to routinely use this API, while for some reason (i.e., mover is running or cluster misconfiguration), SPS is not running, should we still allow these projects to successfully call the satisfyStoragePolicy(), and allow SPS to catch up later on? {quote} Interesting point. Worth filing a JIRA for more discussion on this? There could be some risk: who will clean that Xattr incase, if admin is never enabling SPS. May be we should bring, self expiry or something like that. We have created followup JIRA, which is intend improve the feature even after merging into trunk. If you feel things can be done even after merge, please file under HDFS-12226 {quote} And since this call essentially triggers a large async background task, should we put some logs here? Similarly, it'd be nice to have related JMX stats and some indications in web UI, to be easier to integrate with other systems. {quote} Good suggestions. I will add this comment under metrics JIRA. HDFS-12228 to track. Thank you helping on reviews > Storage Policy Satisfier in Namenode > > > Key: HDFS-10285 > URL: https://issues.apache.org/jira/browse/HDFS-10285 > Project: Hadoop HDFS > Issue Type: New Feature > Components: datanode, namenode >Affects Versions: HDFS-10285 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-10285-consolidated-merge-patch-00.patch, > HDFS-10285-consolidated-merge-patch-01.patch, > HDFS-SPS-TestReport-20170708.pdf, > Storage-Policy-Satisfier-in-HDFS-June-20-2017.pdf, > Storage-Policy-Satisfier-in-HDFS-May10.pdf > > > Heterogeneous storage in HDFS introduced the concept of storage policy. These > policies can be set on directory/file to specify the user preference, where > to store the physical block. When user set the storage policy before writing > data, then the blocks could take advantage of storage policy preferences and > stores physical block accordingly. > If user set the storage policy after writing and completing the file, then > the blocks would have been written with default storage policy (nothing but > DISK). User has to run the ‘Mover tool’ explicitly by specifying all such > file names as a list. In some distributed system scenarios (ex: HBase) it > would be difficult to collect all the files and run the tool as different > nodes can write files separately and file can have different paths. > Another scenarios is, when user rename the files from one effected storage > policy file (inherited policy from parent directory) to another storage > policy effected directory, it will not copy inherited storage policy from > source. So it will take effect from destination file/dir parent storage > policy. This rename operation is just a meta
[jira] [Updated] (HDFS-11948) Ozone: change TestRatisManager to check cluster with data
[ https://issues.apache.org/jira/browse/HDFS-11948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-11948: --- Attachment: HDFS-11948-HDFS-7240.20170731.patch HDFS-11948-HDFS-7240.20170731.patch: fixes test failures. Note that this patch requires RATIS-96 and RATIS-97. > Ozone: change TestRatisManager to check cluster with data > - > > Key: HDFS-11948 > URL: https://issues.apache.org/jira/browse/HDFS-11948 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Attachments: HDFS-11948-HDFS-7240.20170614.patch, > HDFS-11948-HDFS-7240.20170731.patch > > > TestRatisManager first creates multiple Ratis clusters. Then it changes the > membership and closes some clusters. However, it does not test the clusters > with data. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11984) Ozone: Ensures listKey lists all required key fields
[ https://issues.apache.org/jira/browse/HDFS-11984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108206#comment-16108206 ] Weiwei Yang commented on HDFS-11984: Oops, v3 patch doesn't apply any more, could you please submit a new patch for this? Thanks [~linyiqun]. > Ozone: Ensures listKey lists all required key fields > > > Key: HDFS-11984 > URL: https://issues.apache.org/jira/browse/HDFS-11984 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Weiwei Yang >Assignee: Yiqun Lin > Attachments: HDFS-11984-HDFS-7240.001.patch, > HDFS-11984-HDFS-7240.002.patch, HDFS-11984-HDFS-7240.003.patch > > > HDFS-11782 implements the listKey operation which only lists the basic key > fields, we need to make sure it return all required fields > # version > # md5hash > # createdOn > # size > # keyName > this task is depending on the work of HDFS-11886. See more discussion [here | > https://issues.apache.org/jira/browse/HDFS-11782?focusedCommentId=16045562&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16045562]. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12072) Provide fairness between EC and non-EC recovery tasks.
[ https://issues.apache.org/jira/browse/HDFS-12072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108203#comment-16108203 ] Hadoop QA commented on HDFS-12072: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 16s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 35s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 53s{color} | {color:green} trunk passed {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 40s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 48s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 44s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 32s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 74 unchanged - 0 fixed = 77 total (was 74) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 50s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 47s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 64m 51s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 15s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 89m 46s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.server.blockmanagement.TestHeartbeatHandling | | | hadoop.hdfs.server.balancer.TestBalancer | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 | | | hadoop.hdfs.TestDecommissionWithStriped | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure150 | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HDFS-12072 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12879737/HDFS-12072.00.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 74b6334de6ab 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 2be9412 | | Default Java | 1.8.0_131 | | findbugs | v3.1.0-RC1 | | findbugs | https://builds.apache.org/job/PreCommit-HDFS-Build/20506/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html | | checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/20506/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/20506/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/20506/testReport/ | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdf
[jira] [Issue Comment Deleted] (HDFS-12154) Incorrect javadoc description in StorageLocationChecker#check
[ https://issues.apache.org/jira/browse/HDFS-12154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiwei Yang updated HDFS-12154: --- Comment: was deleted (was: +1, pending on jenkins. Thanks [~nandakumar131] to fix this.) > Incorrect javadoc description in StorageLocationChecker#check > - > > Key: HDFS-12154 > URL: https://issues.apache.org/jira/browse/HDFS-12154 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Nandakumar >Assignee: Nandakumar > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: HDFS-12154.000.patch > > > {{StorageLocationChecker#check}} returns list of healthy volumes, but javadoc > states that it returns failed volumes. > {code} > /** >* Initiate a check of the supplied storage volumes and return >* a list of failed volumes. >* >* StorageLocations are returned in the same order as the input >* for compatibility with existing unit tests. >* >* @param conf HDFS configuration. >* @param dataDirs list of volumes to check. >* @return returns a list of failed volumes. Returns the empty list if >* there are no failed volumes. >* >* @throws InterruptedException if the check was interrupted. >* @throws IOException if the number of failed volumes exceeds the >* maximum allowed or if there are no good >* volumes. >*/ > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (HDFS-12154) Incorrect javadoc description in StorageLocationChecker#check
[ https://issues.apache.org/jira/browse/HDFS-12154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiwei Yang updated HDFS-12154: --- Comment: was deleted (was: Looks good to me, +1, committing now.) > Incorrect javadoc description in StorageLocationChecker#check > - > > Key: HDFS-12154 > URL: https://issues.apache.org/jira/browse/HDFS-12154 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Nandakumar >Assignee: Nandakumar > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: HDFS-12154.000.patch > > > {{StorageLocationChecker#check}} returns list of healthy volumes, but javadoc > states that it returns failed volumes. > {code} > /** >* Initiate a check of the supplied storage volumes and return >* a list of failed volumes. >* >* StorageLocations are returned in the same order as the input >* for compatibility with existing unit tests. >* >* @param conf HDFS configuration. >* @param dataDirs list of volumes to check. >* @return returns a list of failed volumes. Returns the empty list if >* there are no failed volumes. >* >* @throws InterruptedException if the check was interrupted. >* @throws IOException if the number of failed volumes exceeds the >* maximum allowed or if there are no good >* volumes. >*/ > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11984) Ozone: Ensures listKey lists all required key fields
[ https://issues.apache.org/jira/browse/HDFS-11984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108202#comment-16108202 ] Weiwei Yang commented on HDFS-11984: Thanks [~linyiqun] for resolving the remaining issues, looks good to me now. I will commit this shortly. > Ozone: Ensures listKey lists all required key fields > > > Key: HDFS-11984 > URL: https://issues.apache.org/jira/browse/HDFS-11984 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Weiwei Yang >Assignee: Yiqun Lin > Attachments: HDFS-11984-HDFS-7240.001.patch, > HDFS-11984-HDFS-7240.002.patch, HDFS-11984-HDFS-7240.003.patch > > > HDFS-11782 implements the listKey operation which only lists the basic key > fields, we need to make sure it return all required fields > # version > # md5hash > # createdOn > # size > # keyName > this task is depending on the work of HDFS-11886. See more discussion [here | > https://issues.apache.org/jira/browse/HDFS-11782?focusedCommentId=16045562&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16045562]. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12154) Incorrect javadoc description in StorageLocationChecker#check
[ https://issues.apache.org/jira/browse/HDFS-12154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108198#comment-16108198 ] Hudson commented on HDFS-12154: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12084 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/12084/]) HDFS-12154. Incorrect javadoc description in (arp: rev ea568123fa76e4683d355a67be01b730d0c11068) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/checker/StorageLocationChecker.java > Incorrect javadoc description in StorageLocationChecker#check > - > > Key: HDFS-12154 > URL: https://issues.apache.org/jira/browse/HDFS-12154 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Nandakumar >Assignee: Nandakumar > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: HDFS-12154.000.patch > > > {{StorageLocationChecker#check}} returns list of healthy volumes, but javadoc > states that it returns failed volumes. > {code} > /** >* Initiate a check of the supplied storage volumes and return >* a list of failed volumes. >* >* StorageLocations are returned in the same order as the input >* for compatibility with existing unit tests. >* >* @param conf HDFS configuration. >* @param dataDirs list of volumes to check. >* @return returns a list of failed volumes. Returns the empty list if >* there are no failed volumes. >* >* @throws InterruptedException if the check was interrupted. >* @throws IOException if the number of failed volumes exceeds the >* maximum allowed or if there are no good >* volumes. >*/ > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12209) VolumeScanner scan cursor not save periodic
[ https://issues.apache.org/jira/browse/HDFS-12209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108184#comment-16108184 ] Chen Liang commented on HDFS-12209: --- Thanks [~fatkun] for reporting this! Although I think it should be slightly better to make the change the other way, i.e. change the {{Time.now()}} to {{Time.monotonicNow()}} in {{FsVolumeImpl}}. > VolumeScanner scan cursor not save periodic > --- > > Key: HDFS-12209 > URL: https://issues.apache.org/jira/browse/HDFS-12209 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs >Affects Versions: 2.6.0 > Environment: cdh5.4.0 >Reporter: fatkun > Attachments: HDFS-12209.patch > > > The bug introduce from HDFS-7430 , the time is not same, one is monotonicMs > and other is clock time. It should use Time.now() both > VolumeScanner.java > {code:java} > long saveDelta = monotonicMs - curBlockIter.getLastSavedMs(); > if (saveDelta >= conf.cursorSaveMs) { > LOG.debug("{}: saving block iterator {} after {} ms.", > this, curBlockIter, saveDelta); > saveBlockIterator(curBlockIter); > } > {code} > curBlockIter.getLastSavedMs() init here > FsVolumeImpl.java > {code:java} > BlockIteratorState() { > lastSavedMs = iterStartMs = Time.now(); > curFinalizedDir = null; > curFinalizedSubDir = null; > curEntry = null; > atEnd = false; > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12154) Incorrect javadoc description in StorageLocationChecker#check
[ https://issues.apache.org/jira/browse/HDFS-12154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-12154: - Fix Version/s: (was: HDFS-7240) 3.0.0-beta1 2.9.0 Hi [~nanda], thanks for catching this. Committed to trunk and branch-2 since the same issue exists in those branches. > Incorrect javadoc description in StorageLocationChecker#check > - > > Key: HDFS-12154 > URL: https://issues.apache.org/jira/browse/HDFS-12154 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Nandakumar >Assignee: Nandakumar >Priority: Trivial > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: HDFS-12154.000.patch > > > {{StorageLocationChecker#check}} returns list of healthy volumes, but javadoc > states that it returns failed volumes. > {code} > /** >* Initiate a check of the supplied storage volumes and return >* a list of failed volumes. >* >* StorageLocations are returned in the same order as the input >* for compatibility with existing unit tests. >* >* @param conf HDFS configuration. >* @param dataDirs list of volumes to check. >* @return returns a list of failed volumes. Returns the empty list if >* there are no failed volumes. >* >* @throws InterruptedException if the check was interrupted. >* @throws IOException if the number of failed volumes exceeds the >* maximum allowed or if there are no good >* volumes. >*/ > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12154) Incorrect javadoc description in StorageLocationChecker#check
[ https://issues.apache.org/jira/browse/HDFS-12154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-12154: - Priority: Major (was: Trivial) > Incorrect javadoc description in StorageLocationChecker#check > - > > Key: HDFS-12154 > URL: https://issues.apache.org/jira/browse/HDFS-12154 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Nandakumar >Assignee: Nandakumar > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: HDFS-12154.000.patch > > > {{StorageLocationChecker#check}} returns list of healthy volumes, but javadoc > states that it returns failed volumes. > {code} > /** >* Initiate a check of the supplied storage volumes and return >* a list of failed volumes. >* >* StorageLocations are returned in the same order as the input >* for compatibility with existing unit tests. >* >* @param conf HDFS configuration. >* @param dataDirs list of volumes to check. >* @return returns a list of failed volumes. Returns the empty list if >* there are no failed volumes. >* >* @throws InterruptedException if the check was interrupted. >* @throws IOException if the number of failed volumes exceeds the >* maximum allowed or if there are no good >* volumes. >*/ > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10285) Storage Policy Satisfier in Namenode
[ https://issues.apache.org/jira/browse/HDFS-10285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108182#comment-16108182 ] Uma Maheswara Rao G commented on HDFS-10285: Hi [~andrew.wang], thank you so much for the thorough review. Please find my replies below. {quote} For the automatic usecase, I agree that metrics are probably the best we can do. However, the API exposed here is for interactive usecases (e.g. a user calling the shell command and polling until it's done). I think we need to do more here to expose the status. Even for the HBase usecase, it'd still want to know about satisfier status so it can bubble it up to an HBase admin. {quote} We have filed JIRA for this already HDFS-12228. Sure, we will think more about status reporting part. Anyway I will file a ticket for this as well to track. Now quick question on your example above “a user calling the shell command and polling until it's done” , you mean command should blocked by polling internally? or user will call status check periodically? How much time server should hold the status? {quote} Can this be addressed by throttling? I think the SPS operations aren't too different from decommissioning, since they're both doing block placement and tracking data movement, and the decom throttles work okay. We've also encountered directories with millions of files before, so there's a need for throttles anyway. Maybe we can do something generic here that can be shared with HDFS-10899. Re-encryption will be faster than SPS, but it's not fast since it needs to talk to the KMS. Xiao's benchmarks indicate that a re-encrypt operation will likely run for hours. On the upside, the benchmarks also show that scanning through an already-re-encrypted zone is quite fast (seconds). I expect it'll be similarly fast for SPS if a user submits subdir or duplicate requests. Would be good to benchmark this. I also don't understand the aversion to FIFO execution. It reduces code complexity and is easy for admins to reason about. If we want to do something more fancy, there should be a broader question around the API for resource management. Is it fair share, priorities, limits, some combination? What are these applied to (users, files, directories, queues with ACLs)? {quote} Throttling is one of the task we have filed HDFS-12227 already. But thats focussing on DN level throttling level and will add to track it to consider NN throttling as well. I think as of now, FIFO model is one way to go ahead, each dir root can be main element to pick first and sub dir will get eventually next priority if user calls on sub directory while higher directory already in progress. {quote} What's the total SPS work timeout in minutes? The node is declared dead after 10.5 minutes, but if the network partition is shorter than that, it won't need to re-register. 5 mins also seems kind of long for an IN_PROGRESS update, since it should take a few seconds for each block movement. Also, we can't depend on re-registration with NN for fencing the old C-DN, since there could be a network partition that is just between the NN and old C-DN, and the old C-DN can still talk to other DNs. I don't know how this affects correctness, but having multiple C-DNs makes debugging harder. {quote} Even though old C-DN working with other DN to transfer blocks(scenario could be rare), DNs will allow only one block. Whoever transfer block first that DN will win, other wll get Block already exist exception. Since NN is tracking that file associated block, it has to remove its tracking element. Example: In worst case, old c-DN completed all movement successfully. New C-DN attempts will fail. Then NN will get result as failure from new C-DN. Now NN will retry, this time blocks would have been satisfied, since old C-DN already did. So, NN will simply ignore that and remove xattr as finished. IN-PROGRESS we send to indicate NN that DN is working on it. This should be very rare condition, as DN will transfer blocks faster than that. This is to make sure DN is running. Right now a file element will be retried after self retry timeout. This case only for failure case where C-DN is not reported anything at all (Dead, or out of network). Also it will happen to that files which are assigned to that C-DN. Right now, self retry timeout was configured 20mins, can be tuned to >10mins. [ We have just made this configurations like PendingReplicationMonitor, where it will reassign to LowReconstructionBlocks list approximately with in 10mins]. {quote} Even assuming we do the xattr optimization, I believe the NN still has a queue of pending work items so they can be retried if the C-DNs fail. How many items might be in this queue, for a large SPS request? Is it throttled? {quote} Pending queue will be depending on number of files which were failed to move by C-DN. Queue contains I
[jira] [Commented] (HDFS-12072) Provide fairness between EC and non-EC recovery tasks.
[ https://issues.apache.org/jira/browse/HDFS-12072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108165#comment-16108165 ] Andrew Wang commented on HDFS-12072: Thanks for working on this Eddy! A few comments from looking at the patch: * Prefer if we rename instances of "ReplicateTask" to "ReplicationTask" * One idea to avoid polling twice, since we know the size of each queue (getNumberOfBlocksToBeReplicated and getNumberOfBlocksToBeErasureCoded), we can compute a ratio and then multiply. * Adding a slightly longer explanatory comment would also be good * Nit: There's also an unused import in the test which will probably be flagged in checkstyle. > Provide fairness between EC and non-EC recovery tasks. > -- > > Key: HDFS-12072 > URL: https://issues.apache.org/jira/browse/HDFS-12072 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Affects Versions: 3.0.0-alpha3 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-12072.00.patch > > > In {{DatanodeManager#handleHeartbeat}}, it takes up to {{maxTransfer}} > reconstruction tasks for non-EC, then if the request can not be full filled, > it takes more tasks from EC reconstruction tasks. > {code} > List pendingList = nodeinfo.getReplicationCommand( > maxTransfers); > if (pendingList != null) { > cmds.add(new BlockCommand(DatanodeProtocol.DNA_TRANSFER, blockPoolId, > pendingList)); > maxTransfers -= pendingList.size(); > } > // check pending erasure coding tasks > List pendingECList = nodeinfo > .getErasureCodeCommand(maxTransfers); > if (pendingECList != null) { > cmds.add(new BlockECReconstructionCommand( > DNA_ERASURE_CODING_RECONSTRUCTION, pendingECList)); > } > {code} > So on a large cluster, if there are large number of constantly non-EC > reconstruction tasks, EC reconstruction tasks do not have a chance to run. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12217) HDFS snapshots doesn't capture all open files when one of the open files is deleted
[ https://issues.apache.org/jira/browse/HDFS-12217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108147#comment-16108147 ] Wei-Chiu Chuang commented on HDFS-12217: Thanks [~manojg] [~yzhangal]. One nit I found is that {{LeaseManager#getINodeWithLeases()}} seems to be a test-only method. If so, it doesn't need public modifier (package private is sufficient) and we can also add a {{@VisibleForTesting }} annotation. On a separate note, and this is totally unrelated to this patch. It looks like {{LeaseManager#addLease}} assumes the inodeId is an id for an INodeFile, which makes perfect sense. The member variable {{leasesById}} also implicitly makes the assumption that the id is for an INodeFile. However, there's no assertion to ensure it is the case in the future. I wonder if it makes sense to add a new {{addLease(String, INodeFile)}} method to make sure only INodeFile is passed in. So it would go like: {code} Lease addLease(String holder, INodeFile inodeFile) { addLease(holder, inodeFile.getId()); } private synchronized Lease addLease(String holder, long inodeId) { ... } {code} Note that in your patch, {{DirectorySnapshottableFeature#addSnapshot}} would capture exception, but doesn't log the exception. I also think that in addition to the snapshot name, you should print the snapshot root path for the error message. > HDFS snapshots doesn't capture all open files when one of the open files is > deleted > --- > > Key: HDFS-12217 > URL: https://issues.apache.org/jira/browse/HDFS-12217 > Project: Hadoop HDFS > Issue Type: Bug > Components: snapshots >Affects Versions: 3.0.0-alpha1 >Reporter: Manoj Govindassamy >Assignee: Manoj Govindassamy > Attachments: HDFS-12217.01.patch, HDFS-12217.02.patch, > HDFS-12217.03.patch > > > With the fix for HDFS-11402, HDFS Snapshots can additionally capture all the > open files. Just like all other files, these open files in the snapshots will > remain immutable. But, sometimes it is found that snapshots fail to capture > all the open files in the system. > Under the following conditions, LeaseManager will fail to find INode > corresponding to an active lease > * a file is opened for writing (LeaseManager allots a lease), and > * the same file is deleted while it is still open for writing and having > active lease, and > * the same file is not referenced in any other Snapshots/Trash > {{INode[] LeaseManager#getINodesWithLease()}} can thus return null for few > leases there by causing the caller to trip over and not return all the open > files needed by the snapshot manager. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Deleted] (HDFS-11318) HDFS FSNamesystem LeaseManager.findPath BLOCK ALL FSNamesystem Ops
[ https://issues.apache.org/jira/browse/HDFS-11318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal deleted HDFS-11318: - > HDFS FSNamesystem LeaseManager.findPath BLOCK ALL FSNamesystem Ops > -- > > Key: HDFS-11318 > URL: https://issues.apache.org/jira/browse/HDFS-11318 > Project: Hadoop HDFS > Issue Type: Bug > Environment: CentOS6.5 Hadoop-2.2.0 >Reporter: zhangyubiao >Priority: Critical > > ``` > "IPC Server handler 69 on 8021" daemon prio=10 tid=0x7f0714c59000 > nid=0x17a23 runnable [0x7eee3ec2f000] >java.lang.Thread.State: RUNNABLE > at org.apache.hadoop.hdfs.server.namenode.INode.compareTo(INode.java:641) > at org.apache.hadoop.hdfs.server.namenode.INode.compareTo(INode.java:52) > at > org.apache.hadoop.hdfs.util.ReadOnlyList$Util.binarySearch(ReadOnlyList.java:73) > at > org.apache.hadoop.hdfs.server.namenode.INodeDirectory.getChild(INodeDirectory.java:323) > at > org.apache.hadoop.hdfs.server.namenode.INodesInPath.resolve(INodesInPath.java:216) > at > org.apache.hadoop.hdfs.server.namenode.INodeDirectory.getLastINodeInPath(INodeDirectory.java:330) > at > org.apache.hadoop.hdfs.server.namenode.FSDirectory.getLastINodeInPath(FSDirectory.java:1655) > at > org.apache.hadoop.hdfs.server.namenode.FSDirectory.getINode(FSDirectory.java:1645) > at > org.apache.hadoop.hdfs.server.namenode.LeaseManager$Lease.findPath(LeaseManager.java:259) > at > org.apache.hadoop.hdfs.server.namenode.LeaseManager$Lease.access$300(LeaseManager.java:228) > at > org.apache.hadoop.hdfs.server.namenode.LeaseManager.findPath(LeaseManager.java:189) > - locked <0x7ef67f8fe698> (a > org.apache.hadoop.hdfs.server.namenode.LeaseManager) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.closeFileCommitBlocks(FSNamesystem.java:4020) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.commitBlockSynchronization(FSNamesystem.java:3989) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.commitBlockSynchronization(NameNodeRpcServer.java:647) > at > org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.commitBlockSynchronization(DatanodeProtocolServerSideTranslatorPB.java:241) > at > org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:24093) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042) > ``` -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11324) FSNamesystem LeaseManager findPath block FSNamesystem Ops
[ https://issues.apache.org/jira/browse/HDFS-11324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108145#comment-16108145 ] Arpit Agarwal commented on HDFS-11324: -- Deleted a bunch of unresolvable duplicates of this issue. [~piaoyu zhang] please try to avoid spamming Jira with multiple duplicates of each bug. > FSNamesystem LeaseManager findPath block FSNamesystem Ops > - > > Key: HDFS-11324 > URL: https://issues.apache.org/jira/browse/HDFS-11324 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.2.0 > Environment: CentOS6.5 Hadoop-2.2.0 >Reporter: zhangyubiao >Priority: Critical > Labels: HDFS > > LeaseManager findPath Cause all FSNamesystem Ops wait -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Deleted] (HDFS-11322) FSNamesystem LeaseManager findPath BLOCK ALL FSNamesystem Ops
[ https://issues.apache.org/jira/browse/HDFS-11322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal deleted HDFS-11322: - > FSNamesystem LeaseManager findPath BLOCK ALL FSNamesystem Ops > - > > Key: HDFS-11322 > URL: https://issues.apache.org/jira/browse/HDFS-11322 > Project: Hadoop HDFS > Issue Type: Bug > Environment: CentOS6.5 Hadoop-2.2.0 >Reporter: zhangyubiao >Priority: Critical > > LeaseManager findPath Cause all FSNamesystem Ops wait -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Deleted] (HDFS-11321) FSNamesystem LeaseManager findPath BLOCK ALL FSNamesystem Ops
[ https://issues.apache.org/jira/browse/HDFS-11321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal deleted HDFS-11321: - > FSNamesystem LeaseManager findPath BLOCK ALL FSNamesystem Ops > - > > Key: HDFS-11321 > URL: https://issues.apache.org/jira/browse/HDFS-11321 > Project: Hadoop HDFS > Issue Type: Bug > Environment: CentOS6.5 Hadoop-2.2.0 >Reporter: zhangyubiao >Priority: Critical > > "IPC Server handler 69 on 8021" daemon prio=10 tid=0x7f0714c59000 > nid=0x17a23 runnable [0x7eee3ec2f000] >java.lang.Thread.State: RUNNABLE > at org.apache.hadoop.hdfs.server.namenode.INode.compareTo(INode.java:641) > at org.apache.hadoop.hdfs.server.namenode.INode.compareTo(INode.java:52) > at > org.apache.hadoop.hdfs.util.ReadOnlyList$Util.binarySearch(ReadOnlyList.java:73) > at > org.apache.hadoop.hdfs.server.namenode.INodeDirectory.getChild(INodeDirectory.java:323) > at > org.apache.hadoop.hdfs.server.namenode.INodesInPath.resolve(INodesInPath.java:216) > at > org.apache.hadoop.hdfs.server.namenode.INodeDirectory.getLastINodeInPath(INodeDirectory.java:330) > at > org.apache.hadoop.hdfs.server.namenode.FSDirectory.getLastINodeInPath(FSDirectory.java:1655) > at > org.apache.hadoop.hdfs.server.namenode.FSDirectory.getINode(FSDirectory.java:1645) > at > org.apache.hadoop.hdfs.server.namenode.LeaseManager$Lease.findPath(LeaseManager.java:259) > at > org.apache.hadoop.hdfs.server.namenode.LeaseManager$Lease.access$300(LeaseManager.java:228) > at > org.apache.hadoop.hdfs.server.namenode.LeaseManager.findPath(LeaseManager.java:189) > - locked <0x7ef67f8fe698> (a > org.apache.hadoop.hdfs.server.namenode.LeaseManager) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.closeFileCommitBlocks(FSNamesystem.java:4020) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.commitBlockSynchronization(FSNamesystem.java:3989) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.commitBlockSynchronization(NameNodeRpcServer.java:647) > at > org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.commitBlockSynchronization(DatanodeProtocolServerSideTranslatorPB.java:241) > at > org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:24093) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Deleted] (HDFS-11323) FSNamesystem LeaseManager findPath block FSNamesystem Ops
[ https://issues.apache.org/jira/browse/HDFS-11323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal deleted HDFS-11323: - > FSNamesystem LeaseManager findPath block FSNamesystem Ops > - > > Key: HDFS-11323 > URL: https://issues.apache.org/jira/browse/HDFS-11323 > Project: Hadoop HDFS > Issue Type: Bug > Environment: CentOS6.5 Hadoop-2.2.0 >Reporter: zhangyubiao >Priority: Critical > > LeaseManager findPath Cause all FSNamesystem Ops wait -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Deleted] (HDFS-11320) FSNamesystem LeaseManager.findPath BLOCK ALL FSNamesystem Ops
[ https://issues.apache.org/jira/browse/HDFS-11320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal deleted HDFS-11320: - > FSNamesystem LeaseManager.findPath BLOCK ALL FSNamesystem Ops > - > > Key: HDFS-11320 > URL: https://issues.apache.org/jira/browse/HDFS-11320 > Project: Hadoop HDFS > Issue Type: Bug > Environment: CentOS6.5 Hadoop-2.2.0 >Reporter: zhangyubiao >Priority: Critical > > "IPC Server handler 69 on 8021" daemon prio=10 tid=0x7f0714c59000 > nid=0x17a23 runnable [0x7eee3ec2f000] >java.lang.Thread.State: RUNNABLE > at org.apache.hadoop.hdfs.server.namenode.INode.compareTo(INode.java:641) > at org.apache.hadoop.hdfs.server.namenode.INode.compareTo(INode.java:52) > at > org.apache.hadoop.hdfs.util.ReadOnlyList$Util.binarySearch(ReadOnlyList.java:73) > at > org.apache.hadoop.hdfs.server.namenode.INodeDirectory.getChild(INodeDirectory.java:323) > at > org.apache.hadoop.hdfs.server.namenode.INodesInPath.resolve(INodesInPath.java:216) > at > org.apache.hadoop.hdfs.server.namenode.INodeDirectory.getLastINodeInPath(INodeDirectory.java:330) > at > org.apache.hadoop.hdfs.server.namenode.FSDirectory.getLastINodeInPath(FSDirectory.java:1655) > at > org.apache.hadoop.hdfs.server.namenode.FSDirectory.getINode(FSDirectory.java:1645) > at > org.apache.hadoop.hdfs.server.namenode.LeaseManager$Lease.findPath(LeaseManager.java:259) > at > org.apache.hadoop.hdfs.server.namenode.LeaseManager$Lease.access$300(LeaseManager.java:228) > at > org.apache.hadoop.hdfs.server.namenode.LeaseManager.findPath(LeaseManager.java:189) > - locked <0x7ef67f8fe698> (a > org.apache.hadoop.hdfs.server.namenode.LeaseManager) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.closeFileCommitBlocks(FSNamesystem.java:4020) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.commitBlockSynchronization(FSNamesystem.java:3989) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.commitBlockSynchronization(NameNodeRpcServer.java:647) > at > org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.commitBlockSynchronization(DatanodeProtocolServerSideTranslatorPB.java:241) > at > org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:24093) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Deleted] (HDFS-11319) HDFS FSNamesystem LeaseManager.findPath BLOCK ALL FSNamesystem Ops
[ https://issues.apache.org/jira/browse/HDFS-11319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal deleted HDFS-11319: - > HDFS FSNamesystem LeaseManager.findPath BLOCK ALL FSNamesystem Ops > -- > > Key: HDFS-11319 > URL: https://issues.apache.org/jira/browse/HDFS-11319 > Project: Hadoop HDFS > Issue Type: Sub-task > Environment: CentOS6.5 Hadoop-2.2.0 >Reporter: zhangyubiao >Priority: Critical > > "IPC Server handler 69 on 8021" daemon prio=10 tid=0x7f0714c59000 > nid=0x17a23 runnable [0x7eee3ec2f000] >java.lang.Thread.State: RUNNABLE > at org.apache.hadoop.hdfs.server.namenode.INode.compareTo(INode.java:641) > at org.apache.hadoop.hdfs.server.namenode.INode.compareTo(INode.java:52) > at > org.apache.hadoop.hdfs.util.ReadOnlyList$Util.binarySearch(ReadOnlyList.java:73) > at > org.apache.hadoop.hdfs.server.namenode.INodeDirectory.getChild(INodeDirectory.java:323) > at > org.apache.hadoop.hdfs.server.namenode.INodesInPath.resolve(INodesInPath.java:216) > at > org.apache.hadoop.hdfs.server.namenode.INodeDirectory.getLastINodeInPath(INodeDirectory.java:330) > at > org.apache.hadoop.hdfs.server.namenode.FSDirectory.getLastINodeInPath(FSDirectory.java:1655) > at > org.apache.hadoop.hdfs.server.namenode.FSDirectory.getINode(FSDirectory.java:1645) > at > org.apache.hadoop.hdfs.server.namenode.LeaseManager$Lease.findPath(LeaseManager.java:259) > at > org.apache.hadoop.hdfs.server.namenode.LeaseManager$Lease.access$300(LeaseManager.java:228) > at > org.apache.hadoop.hdfs.server.namenode.LeaseManager.findPath(LeaseManager.java:189) > - locked <0x7ef67f8fe698> (a > org.apache.hadoop.hdfs.server.namenode.LeaseManager) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.closeFileCommitBlocks(FSNamesystem.java:4020) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.commitBlockSynchronization(FSNamesystem.java:3989) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.commitBlockSynchronization(NameNodeRpcServer.java:647) > at > org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolServerSideTranslatorPB.commitBlockSynchronization(DatanodeProtocolServerSideTranslatorPB.java:241) > at > org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$DatanodeProtocolService$2.callBlockingMethod(DatanodeProtocolProtos.java:24093) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:928) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2048) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2042) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11326) FSNamesystem closeFileCommitBlocks block FSNamesystem Ops
[ https://issues.apache.org/jira/browse/HDFS-11326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108140#comment-16108140 ] Arpit Agarwal commented on HDFS-11326: -- Deleted unresolveable duplicates of this Jira HDFS-11325, 11329, 11330, 11331 and 11332. > FSNamesystem closeFileCommitBlocks block FSNamesystem Ops > - > > Key: HDFS-11326 > URL: https://issues.apache.org/jira/browse/HDFS-11326 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.2.0 > Environment: CentOS 6.5 Hadoop-2.2.0 >Reporter: zhangyubiao >Priority: Critical > > Seems like String src = leaseManager.findPath(pendingFile); cause to much > time hold the write lock. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Deleted] (HDFS-11331) FSNamesystem closeFileCommitBlocks block all FSNamesystem Ops
[ https://issues.apache.org/jira/browse/HDFS-11331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal deleted HDFS-11331: - > FSNamesystem closeFileCommitBlocks block all FSNamesystem Ops > - > > Key: HDFS-11331 > URL: https://issues.apache.org/jira/browse/HDFS-11331 > Project: Hadoop HDFS > Issue Type: Bug > Environment: CentOS6.5 Hadoop-2.2.0 >Reporter: zhangyubiao > > FSNamesystem closeFileCommitBlocks block all FSNamesystem Ops > Seems like leaseManager.findPath(pendingFile) hold the write lock for long > time -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12196) Ozone: DeleteKey-2: Implement container recycling service to delete stale blocks at background
[ https://issues.apache.org/jira/browse/HDFS-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108138#comment-16108138 ] Anu Engineer commented on HDFS-12196: - bq. That absolutely makes sense. I have already created an abstract class AbstractRecyclingService which is a base class for all Recycling Services I was suggesting something little more ambitious. I was suggesting that we need to create an interface called Background tasks for KSM, SCM and Datanode. All tasks like Recycling task or volume scanner or garbage collection tasks for incomplete objects etc, Should run in that mode. I do see that having a Recycling Interface is nice, so it is easy to see how KSM Recycling, SCM Recycling, and Datanode stuff all relate to each other. I would really love if they are all part of Background Task interface for Ozone. > Ozone: DeleteKey-2: Implement container recycling service to delete stale > blocks at background > -- > > Key: HDFS-12196 > URL: https://issues.apache.org/jira/browse/HDFS-12196 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Attachments: HDFS-12196-HDFS-7240.001.patch > > > Implement a recycling service running on datanode to delete stale blocks. > The recycling service scans staled blocks for each container and delete > chunks and references periodically. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Deleted] (HDFS-11325) FSNamesystem closeFileCommitBlocks block FSNamesystem Ops
[ https://issues.apache.org/jira/browse/HDFS-11325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal deleted HDFS-11325: - > FSNamesystem closeFileCommitBlocks block FSNamesystem Ops > - > > Key: HDFS-11325 > URL: https://issues.apache.org/jira/browse/HDFS-11325 > Project: Hadoop HDFS > Issue Type: Bug > Environment: CentOS 6.5 Hadoop-2.2.0 >Reporter: zhangyubiao >Priority: Critical > > Seems like String src = leaseManager.findPath(pendingFile); cause to much > time hold the write lock. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Deleted] (HDFS-11329) FSNamesystem closeFileCommitBlocks block all FSNamesystem Ops
[ https://issues.apache.org/jira/browse/HDFS-11329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal deleted HDFS-11329: - > FSNamesystem closeFileCommitBlocks block all FSNamesystem Ops > - > > Key: HDFS-11329 > URL: https://issues.apache.org/jira/browse/HDFS-11329 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: zhangyubiao >Priority: Critical > > FSNamesystem closeFileCommitBlocks block all FSNamesystem Ops -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Deleted] (HDFS-11330) FSNamesystem closeFileCommitBlocks block all FSNamesystem Ops
[ https://issues.apache.org/jira/browse/HDFS-11330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal deleted HDFS-11330: - > FSNamesystem closeFileCommitBlocks block all FSNamesystem Ops > - > > Key: HDFS-11330 > URL: https://issues.apache.org/jira/browse/HDFS-11330 > Project: Hadoop HDFS > Issue Type: Bug > Environment: CentOS 6.5 Hadoop-2.2.0 >Reporter: zhangyubiao >Priority: Critical > > FSNamesystem closeFileCommitBlocks block all FSNamesystem Ops -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Deleted] (HDFS-11332) FSNamesystem closeFileCommitBlocks block all FSNamesystem Ops
[ https://issues.apache.org/jira/browse/HDFS-11332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal deleted HDFS-11332: - > FSNamesystem closeFileCommitBlocks block all FSNamesystem Ops > - > > Key: HDFS-11332 > URL: https://issues.apache.org/jira/browse/HDFS-11332 > Project: Hadoop HDFS > Issue Type: Bug > Environment: CentOS6.5 Hadoop-2.2.0 >Reporter: zhangyubiao > > FSNamesystem closeFileCommitBlocks block all FSNamesystem Ops > Seems like leaseManager.findPath(pendingFile) hold the write lock for long > time -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12034) Ozone: Web interface for KSM
[ https://issues.apache.org/jira/browse/HDFS-12034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDFS-12034: Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) [~elek] Thank you for the contribution. [~aw] Thanks for the help with rat-check and very valuable comments on the License issue. I have committed this to the feature branch > Ozone: Web interface for KSM > > > Key: HDFS-12034 > URL: https://issues.apache.org/jira/browse/HDFS-12034 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Elek, Marton >Assignee: Elek, Marton > Attachments: HDFS-12034-HDFS-7240.001.patch, > HDFS-12034-HDFS-7240.002.patch, HDFS-12034-HDFS-7240.003.patch > > > This is the pair of the HDFS-12005 but it's about the web interface of the > Ozone KSM server. I created a seperated issue to collect the required > data/mxbean separated and handle the two web interface independent one by one. > Required data (Work in progress): > * KSMMetrics data (numVolumeCreates, numVolumeModifes) > * Available volumes (similar to the file browser of the namenode web ui) > * Available buckets (per volumes) > * Available keys (per buckets) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11920) Ozone : add key partition
[ https://issues.apache.org/jira/browse/HDFS-11920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108125#comment-16108125 ] Chen Liang commented on HDFS-11920: --- Thanks [~xyao] for the comments! I will commit this shortly. > Ozone : add key partition > - > > Key: HDFS-11920 > URL: https://issues.apache.org/jira/browse/HDFS-11920 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-11920-HDFS-7240.001.patch, > HDFS-11920-HDFS-7240.002.patch, HDFS-11920-HDFS-7240.003.patch, > HDFS-11920-HDFS-7240.004.patch, HDFS-11920-HDFS-7240.005.patch, > HDFS-11920-HDFS-7240.006.patch, HDFS-11920-HDFS-7240.007.patch, > HDFS-11920-HDFS-7240.008.patch, HDFS-11920-HDFS-7240.009.patch, > HDFS-11920-HDFS-7240.010.patch > > > Currently, each key corresponds to one single SCM block, and putKey/getKey > writes/reads to this single SCM block. This works fine for keys with > reasonably small data size. However if the data is too huge, (e.g. not even > fits into a single container), then we need to be able to partition the key > data into multiple blocks, each in one container. This JIRA changes the > key-related classes to support this. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11948) Ozone: change TestRatisManager to check cluster with data
[ https://issues.apache.org/jira/browse/HDFS-11948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-11948: --- Status: Open (was: Patch Available) > Ozone: change TestRatisManager to check cluster with data > - > > Key: HDFS-11948 > URL: https://issues.apache.org/jira/browse/HDFS-11948 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Attachments: HDFS-11948-HDFS-7240.20170614.patch > > > TestRatisManager first creates multiple Ratis clusters. Then it changes the > membership and closes some clusters. However, it does not test the clusters > with data. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11326) FSNamesystem closeFileCommitBlocks block FSNamesystem Ops
[ https://issues.apache.org/jira/browse/HDFS-11326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-11326: - Hadoop Flags: (was: Incompatible change) > FSNamesystem closeFileCommitBlocks block FSNamesystem Ops > - > > Key: HDFS-11326 > URL: https://issues.apache.org/jira/browse/HDFS-11326 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 2.2.0 > Environment: CentOS 6.5 Hadoop-2.2.0 >Reporter: zhangyubiao >Priority: Critical > > Seems like String src = leaseManager.findPath(pendingFile); cause to much > time hold the write lock. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-11920) Ozone : add key partition
[ https://issues.apache.org/jira/browse/HDFS-11920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chen Liang updated HDFS-11920: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Ozone : add key partition > - > > Key: HDFS-11920 > URL: https://issues.apache.org/jira/browse/HDFS-11920 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-11920-HDFS-7240.001.patch, > HDFS-11920-HDFS-7240.002.patch, HDFS-11920-HDFS-7240.003.patch, > HDFS-11920-HDFS-7240.004.patch, HDFS-11920-HDFS-7240.005.patch, > HDFS-11920-HDFS-7240.006.patch, HDFS-11920-HDFS-7240.007.patch, > HDFS-11920-HDFS-7240.008.patch, HDFS-11920-HDFS-7240.009.patch, > HDFS-11920-HDFS-7240.010.patch > > > Currently, each key corresponds to one single SCM block, and putKey/getKey > writes/reads to this single SCM block. This works fine for keys with > reasonably small data size. However if the data is too huge, (e.g. not even > fits into a single container), then we need to be able to partition the key > data into multiple blocks, each in one container. This JIRA changes the > key-related classes to support this. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12072) Provide fairness between EC and non-EC recovery tasks.
[ https://issues.apache.org/jira/browse/HDFS-12072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lei (Eddy) Xu updated HDFS-12072: - Status: Patch Available (was: Open) > Provide fairness between EC and non-EC recovery tasks. > -- > > Key: HDFS-12072 > URL: https://issues.apache.org/jira/browse/HDFS-12072 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Affects Versions: 3.0.0-alpha3 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-12072.00.patch > > > In {{DatanodeManager#handleHeartbeat}}, it takes up to {{maxTransfer}} > reconstruction tasks for non-EC, then if the request can not be full filled, > it takes more tasks from EC reconstruction tasks. > {code} > List pendingList = nodeinfo.getReplicationCommand( > maxTransfers); > if (pendingList != null) { > cmds.add(new BlockCommand(DatanodeProtocol.DNA_TRANSFER, blockPoolId, > pendingList)); > maxTransfers -= pendingList.size(); > } > // check pending erasure coding tasks > List pendingECList = nodeinfo > .getErasureCodeCommand(maxTransfers); > if (pendingECList != null) { > cmds.add(new BlockECReconstructionCommand( > DNA_ERASURE_CODING_RECONSTRUCTION, pendingECList)); > } > {code} > So on a large cluster, if there are large number of constantly non-EC > reconstruction tasks, EC reconstruction tasks do not have a chance to run. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12072) Provide fairness between EC and non-EC recovery tasks.
[ https://issues.apache.org/jira/browse/HDFS-12072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lei (Eddy) Xu updated HDFS-12072: - Attachment: HDFS-12072.00.patch Update the patch to get half of {{maxTransfers}} tasks from each of 3x and EC pending recovery queue. > Provide fairness between EC and non-EC recovery tasks. > -- > > Key: HDFS-12072 > URL: https://issues.apache.org/jira/browse/HDFS-12072 > Project: Hadoop HDFS > Issue Type: Improvement > Components: erasure-coding >Affects Versions: 3.0.0-alpha3 >Reporter: Lei (Eddy) Xu >Assignee: Lei (Eddy) Xu > Labels: hdfs-ec-3.0-nice-to-have > Attachments: HDFS-12072.00.patch > > > In {{DatanodeManager#handleHeartbeat}}, it takes up to {{maxTransfer}} > reconstruction tasks for non-EC, then if the request can not be full filled, > it takes more tasks from EC reconstruction tasks. > {code} > List pendingList = nodeinfo.getReplicationCommand( > maxTransfers); > if (pendingList != null) { > cmds.add(new BlockCommand(DatanodeProtocol.DNA_TRANSFER, blockPoolId, > pendingList)); > maxTransfers -= pendingList.size(); > } > // check pending erasure coding tasks > List pendingECList = nodeinfo > .getErasureCodeCommand(maxTransfers); > if (pendingECList != null) { > cmds.add(new BlockECReconstructionCommand( > DNA_ERASURE_CODING_RECONSTRUCTION, pendingECList)); > } > {code} > So on a large cluster, if there are large number of constantly non-EC > reconstruction tasks, EC reconstruction tasks do not have a chance to run. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11920) Ozone : add key partition
[ https://issues.apache.org/jira/browse/HDFS-11920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108115#comment-16108115 ] Xiaoyu Yao commented on HDFS-11920: --- Thanks [~vagarychen] for updating the patch. Agree that we can leave the delete part to the DeleteKey feature. +1 given the two Jenkins failures are known issues. > Ozone : add key partition > - > > Key: HDFS-11920 > URL: https://issues.apache.org/jira/browse/HDFS-11920 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-11920-HDFS-7240.001.patch, > HDFS-11920-HDFS-7240.002.patch, HDFS-11920-HDFS-7240.003.patch, > HDFS-11920-HDFS-7240.004.patch, HDFS-11920-HDFS-7240.005.patch, > HDFS-11920-HDFS-7240.006.patch, HDFS-11920-HDFS-7240.007.patch, > HDFS-11920-HDFS-7240.008.patch, HDFS-11920-HDFS-7240.009.patch, > HDFS-11920-HDFS-7240.010.patch > > > Currently, each key corresponds to one single SCM block, and putKey/getKey > writes/reads to this single SCM block. This works fine for keys with > reasonably small data size. However if the data is too huge, (e.g. not even > fits into a single container), then we need to be able to partition the key > data into multiple blocks, each in one container. This JIRA changes the > key-related classes to support this. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12034) Ozone: Web interface for KSM
[ https://issues.apache.org/jira/browse/HDFS-12034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108101#comment-16108101 ] Anu Engineer commented on HDFS-12034: - [~elek] Thanks for the update. I am glad we were able to resolve this issue. [~aw] Thank you very much for helping out on both the build issues and especially the legal stuff. Really appreciate your comments and help here. I will commit this to feature branch shortly. > Ozone: Web interface for KSM > > > Key: HDFS-12034 > URL: https://issues.apache.org/jira/browse/HDFS-12034 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Elek, Marton >Assignee: Elek, Marton > Attachments: HDFS-12034-HDFS-7240.001.patch, > HDFS-12034-HDFS-7240.002.patch, HDFS-12034-HDFS-7240.003.patch > > > This is the pair of the HDFS-12005 but it's about the web interface of the > Ozone KSM server. I created a seperated issue to collect the required > data/mxbean separated and handle the two web interface independent one by one. > Required data (Work in progress): > * KSMMetrics data (numVolumeCreates, numVolumeModifes) > * Available volumes (similar to the file browser of the namenode web ui) > * Available buckets (per volumes) > * Available keys (per buckets) -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11920) Ozone : add key partition
[ https://issues.apache.org/jira/browse/HDFS-11920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108099#comment-16108099 ] Chen Liang commented on HDFS-11920: --- Failed tests are unrelated. > Ozone : add key partition > - > > Key: HDFS-11920 > URL: https://issues.apache.org/jira/browse/HDFS-11920 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Chen Liang >Assignee: Chen Liang > Attachments: HDFS-11920-HDFS-7240.001.patch, > HDFS-11920-HDFS-7240.002.patch, HDFS-11920-HDFS-7240.003.patch, > HDFS-11920-HDFS-7240.004.patch, HDFS-11920-HDFS-7240.005.patch, > HDFS-11920-HDFS-7240.006.patch, HDFS-11920-HDFS-7240.007.patch, > HDFS-11920-HDFS-7240.008.patch, HDFS-11920-HDFS-7240.009.patch, > HDFS-11920-HDFS-7240.010.patch > > > Currently, each key corresponds to one single SCM block, and putKey/getKey > writes/reads to this single SCM block. This works fine for keys with > reasonably small data size. However if the data is too huge, (e.g. not even > fits into a single container), then we need to be able to partition the key > data into multiple blocks, each in one container. This JIRA changes the > key-related classes to support this. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12162) Update listStatus document to describe the behavior when the argument is a file
[ https://issues.apache.org/jira/browse/HDFS-12162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108081#comment-16108081 ] Ajay Yadav commented on HDFS-12162: --- Hi [~yzhangal], Are you referring to webhdfs documentation? Do you mean we should update webhdfs documentation to include an example for file path. Thanks!! > Update listStatus document to describe the behavior when the argument is a > file > --- > > Key: HDFS-12162 > URL: https://issues.apache.org/jira/browse/HDFS-12162 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, httpfs >Reporter: Yongjun Zhang >Assignee: Ajay Yadav > > The listStatus method can take in either directory path or file path as > input, however, currently both the javadoc and external document describe it > as only taking directory as input. This jira is to update the document about > the behavior when the argument is a file path. > Thanks [~xiaochen] for the review and discussion in HDFS-12139, creating this > jira is the result of our discussion there. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12162) Update listStatus document to describe the behavior when the argument is a file
[ https://issues.apache.org/jira/browse/HDFS-12162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Yadav reassigned HDFS-12162: - Assignee: Ajay Yadav > Update listStatus document to describe the behavior when the argument is a > file > --- > > Key: HDFS-12162 > URL: https://issues.apache.org/jira/browse/HDFS-12162 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, httpfs >Reporter: Yongjun Zhang >Assignee: Ajay Yadav > > The listStatus method can take in either directory path or file path as > input, however, currently both the javadoc and external document describe it > as only taking directory as input. This jira is to update the document about > the behavior when the argument is a file path. > Thanks [~xiaochen] for the review and discussion in HDFS-12139, creating this > jira is the result of our discussion there. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-11283) why should we not introduce distributed database to storage hdfs's metadata?
[ https://issues.apache.org/jira/browse/HDFS-11283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal resolved HDFS-11283. -- Resolution: Fixed Hi [~chenrongwei], the hdfs-dev mailing list is the right place for these questions. Resolving this. > why should we not introduce distributed database to storage hdfs's metadata? > - > > Key: HDFS-11283 > URL: https://issues.apache.org/jira/browse/HDFS-11283 > Project: Hadoop HDFS > Issue Type: Wish >Reporter: chenrongwei > > why should we not introduce distributed database to storage hdfs's metadata? > In my opinion,it maybe loss some performance,but it has below improvements: > 1、enhance NN's extend ability,such as NN can support much more files and > blocks. The problem of massive little files always make me headache. > 2、In most MR cluster aren't care the performance loss,but more care the > cluster's scale. > 3、NN's HA implements maybe more simpler and reasonable. > so I think maybe we should add a new work mode building on distributed > database for NN. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11764) NPE when the GroupMappingServiceProvider has no group
[ https://issues.apache.org/jira/browse/HDFS-11764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108054#comment-16108054 ] runlinzhang commented on HDFS-11764: Thank you for your attention,the version is branch-2.7.2 When the default implementation JniBasedUnixGroupsMapping GroupMappingServiceProvider, And nodemanager does not configure the group for the user, which causes the group to be null, which makes null judgment to avoid this problem > NPE when the GroupMappingServiceProvider has no group > -- > > Key: HDFS-11764 > URL: https://issues.apache.org/jira/browse/HDFS-11764 > Project: Hadoop HDFS > Issue Type: Bug > Components: security >Affects Versions: 2.7.2 >Reporter: runlinzhang >Priority: Critical > Fix For: 2.7.2 > > Attachments: image.png > > > The following code can throw NPE if GroupMappingServiceProvider.getGroups() > returns null. > public List load(String user) throws Exception { > List groups = fetchGroupList(user); > if (groups.isEmpty()) { > if (isNegativeCacheEnabled()) { > negativeCache.add(user); > } > // We throw here to prevent Cache from retaining an empty group > throw noGroupsForUser(user); > } > return groups; > } > eg:image -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11580) Ozone: Support asynchronus client API for SCM and containers
[ https://issues.apache.org/jira/browse/HDFS-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108049#comment-16108049 ] Chen Liang commented on HDFS-11580: --- Failed tests are unrelated. > Ozone: Support asynchronus client API for SCM and containers > > > Key: HDFS-11580 > URL: https://issues.apache.org/jira/browse/HDFS-11580 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Affects Versions: HDFS-7240 >Reporter: Anu Engineer >Assignee: Yiqun Lin > Attachments: HDFS-11580-HDFS-7240.001.patch, > HDFS-11580-HDFS-7240.002.patch, HDFS-11580-HDFS-7240.003.patch, > HDFS-11580-HDFS-7240.004.patch, HDFS-11580-HDFS-7240.005.patch, > HDFS-11580-HDFS-7240.006.patch, HDFS-11580-HDFS-7240.007.patch, > HDFS-11580-HDFS-7240.008.patch, HDFS-11580-HDFS-7240.009.patch, > HDFS-11580-HDFS-7240.010.patch, HDFS-11580-HDFS-7240.011.patch > > > This is an umbrella JIRA that needs to support a set of APIs in Asynchronous > form. > For containers -- or the datanode API currently supports a call > {{sendCommand}}. we need to build proper programming interface and support an > async interface. > There is also a set of SCM API that clients can call, it would be nice to > support Async interface for those too. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12232) ConfiguredFailoverProxyProvider caches DNS results indefinitely
[ https://issues.apache.org/jira/browse/HDFS-12232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-12232: --- Summary: ConfiguredFailoverProxyProvider caches DNS results indefinitely (was: ConfiguredFailoverProxyProvider caches DNS results infinitely) > ConfiguredFailoverProxyProvider caches DNS results indefinitely > --- > > Key: HDFS-12232 > URL: https://issues.apache.org/jira/browse/HDFS-12232 > Project: Hadoop HDFS > Issue Type: Bug > Components: ha, hdfs >Reporter: Haohui Mai > > {{ConfiguredFailoverProxyProvider}} resolves the IP addresses and stores them > in the internal state. As a result when migrating the standby NN to a > different machine, it requires restarting all HDFS clients. > A better approach is to follow Java's policy of caching DNS names: > http://javaeesupportpatterns.blogspot.com/2011/03/java-dns-cache-reference-guide.html -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12196) Ozone: DeleteKey-2: Implement container recycling service to delete stale blocks at background
[ https://issues.apache.org/jira/browse/HDFS-12196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108029#comment-16108029 ] Chen Liang commented on HDFS-12196: --- Thanks [~cheersyang] for the patch, looks pretty good to me overall! Some initial comments: 1. Although {{RecyclingTask}} has a {{getPriority()}} method, it seems that only the implementation class of {{AbstractRecyclingService}} will be enforcing the priority (currently, {{ContainerRecyclingService}}). So if more implementation of AbstractRecyclingService get added later, each of them will need to have this line {code} PriorityQueue tasks = new PriorityQueue<>( (task1, task2) -> task1.getPriority() - task2.getPriority()); {code} Otherwise seems the priorities will be completely ignored? Can we somehow add the priority enforcement to the abstractions also? e.g. somehow add a abstract priority queue class or something. 2. I was wondering will there be anyone that will be reading {{RecyclingResult}}? In other words, will there be cases that some logic will check the result of recycling tasks and take action accordingly? Seems unit test is the only one that does this in this patch. If not, then having this abstraction only to make unit test work seems a little bit of a overkill to me...This is also related to the result caching part. Also I felt the term "recycling" here is not that informative, because it does not seem that we recycling anything, but can't think of a better alternative... > Ozone: DeleteKey-2: Implement container recycling service to delete stale > blocks at background > -- > > Key: HDFS-12196 > URL: https://issues.apache.org/jira/browse/HDFS-12196 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ozone >Reporter: Weiwei Yang >Assignee: Weiwei Yang > Attachments: HDFS-12196-HDFS-7240.001.patch > > > Implement a recycling service running on datanode to delete stale blocks. > The recycling service scans staled blocks for each container and delete > chunks and references periodically. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-10899) Add functionality to re-encrypt EDEKs
[ https://issues.apache.org/jira/browse/HDFS-10899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108020#comment-16108020 ] Hadoop QA commented on HDFS-10899: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 28s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 3 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 9s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 35s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 56s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 35s{color} | {color:green} trunk passed {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 32s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs-client in trunk has 2 extant Findbugs warnings. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 45s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 1s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 22s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 40s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 54s{color} | {color:orange} hadoop-hdfs-project: The patch generated 65 new + 962 unchanged - 2 fixed = 1027 total (was 964) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 29s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 3s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 16s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 94m 22s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 17s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}131m 38s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting | | | hadoop.hdfs.TestReconstructStripedFile | | | hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HDFS-10899 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12879696/HDFS-10899.12.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc xml | | uname | Linux 2fa0ba68abf2 3.13.0-117-generic #164
[jira] [Commented] (HDFS-12151) Hadoop 2 clients cannot writeBlock to Hadoop 3 DataNodes
[ https://issues.apache.org/jira/browse/HDFS-12151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108012#comment-16108012 ] Andrew Wang commented on HDFS-12151: Thanks for the rev Sean, LGTM. Could you address the checkstyles and confirm the failed tests are flakes? +1 pending that. > Hadoop 2 clients cannot writeBlock to Hadoop 3 DataNodes > > > Key: HDFS-12151 > URL: https://issues.apache.org/jira/browse/HDFS-12151 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rolling upgrades >Affects Versions: 3.0.0-alpha4 >Reporter: Sean Mackrory >Assignee: Sean Mackrory > Attachments: HDFS-12151.001.patch, HDFS-12151.002.patch, > HDFS-12151.003.patch, HDFS-12151.004.patch, HDFS-12151.005.patch, > HDFS-12151.006.patch > > > Trying to write to a Hadoop 3 DataNode with a Hadoop 2 client currently > fails. On the client side it looks like this: > {code} > 17/07/14 13:31:58 INFO hdfs.DFSClient: Exception in > createBlockOutputStream > java.io.EOFException: Premature EOF: no length prefix available > at > org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2280) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1318) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1237) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:449){code} > But on the DataNode side there's an ArrayOutOfBoundsException because there > aren't any targetStorageIds: > {code} > java.lang.ArrayIndexOutOfBoundsException: 0 > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:815) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:173) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:107) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:290) > at java.lang.Thread.run(Thread.java:745){code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12034) Ozone: Web interface for KSM
[ https://issues.apache.org/jira/browse/HDFS-12034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108004#comment-16108004 ] Hadoop QA commented on HDFS-12034: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} HDFS-7240 Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 53s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 20s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 35s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 1s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 11m 6s{color} | {color:green} HDFS-7240 passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: . {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 14s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 44s{color} | {color:green} HDFS-7240 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 6s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m 41s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 0s{color} | {color:orange} root: The patch generated 3 new + 0 unchanged - 0 fixed = 3 total (was 0) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 11m 35s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch 1 line(s) with tabs. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: . {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 36s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 44s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 13m 38s{color} | {color:red} root in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 30s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}122m 24s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.ha.TestZKFailoverController | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HDFS-12034 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12879693/HDFS-12034-HDFS-7240.003.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml | | uname | Linux ade669f5fbee 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | HDFS-7240 / 1f5353d | | Default Java | 1.8.0_131 | | findbugs | v3.1.0-RC1 | | checksty
[jira] [Commented] (HDFS-11848) Enhance dfsadmin listOpenFiles command to list files under a given path
[ https://issues.apache.org/jira/browse/HDFS-11848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108000#comment-16108000 ] Wei-Chiu Chuang commented on HDFS-11848: +1 for the purposal. Other than path filter, would we be interested in other types of filters? E.g. client name, machine? > Enhance dfsadmin listOpenFiles command to list files under a given path > --- > > Key: HDFS-11848 > URL: https://issues.apache.org/jira/browse/HDFS-11848 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.0.0-alpha1 >Reporter: Manoj Govindassamy >Assignee: Yiqun Lin > > HDFS-10480 adds {{listOpenFiles}} option is to {{dfsadmin}} command to list > all the open files in the system. > One more thing that would be nice here is to filter the output on a passed > path or DataNode. Usecases: An admin might already know a stale file by path > (perhaps from fsck's -openforwrite), and wants to figure out who the lease > holder is. Proposal here is add suboptions to {{listOpenFiles}} to list files > filtered by path. > {{LeaseManager#getINodeWithLeases(INodeDirectory)}} can be used to get the > open file list for any given ancestor directory. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11580) Ozone: Support asynchronus client API for SCM and containers
[ https://issues.apache.org/jira/browse/HDFS-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108002#comment-16108002 ] Hadoop QA commented on HDFS-11580: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} HDFS-7240 Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 37s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 14s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 42s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 46s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 45s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 51s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 44s{color} | {color:green} HDFS-7240 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 35s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 35s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 41s{color} | {color:green} hadoop-hdfs-project: The patch generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 36s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 47s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 33s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 15s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 74m 34s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 18s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}115m 28s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure140 | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 | | | hadoop.ozone.container.ozoneimpl.TestRatisManager | | | hadoop.ozone.web.client.TestKeys | | Timed out junit tests | org.apache.hadoop.ozone.web.client.TestKeysRatis | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HDFS-11580 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12879691/HDFS-11580-HDFS-7240.011.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 1eab87e93add 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | HDFS-7240 / 1f5353d | | Default Java | 1.8.0_131 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/20502/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://build
[jira] [Commented] (HDFS-11826) Federation Namenode Heartbeat
[ https://issues.apache.org/jira/browse/HDFS-11826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16107990#comment-16107990 ] Chris Douglas commented on HDFS-11826: -- +1 looks straightforward. Only one minor suggestion: exit with an error if the feature is enabled, but none of the heartbeat services are valid (i.e., {{createHeartbeatServices}} returns an empty map). > Federation Namenode Heartbeat > - > > Key: HDFS-11826 > URL: https://issues.apache.org/jira/browse/HDFS-11826 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: fs >Reporter: Inigo Goiri >Assignee: Inigo Goiri > Fix For: HDFS-10467 > > Attachments: HDFS-11826-HDFS-10467-000.patch > > > Add a service to the Router to check the state of a Namenode and report it > into the State Store. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11920) Ozone : add key partition
[ https://issues.apache.org/jira/browse/HDFS-11920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16107979#comment-16107979 ] Hadoop QA commented on HDFS-11920: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} HDFS-7240 Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 28s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 12s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 36s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 43s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 37s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 40s{color} | {color:green} HDFS-7240 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 46s{color} | {color:green} HDFS-7240 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 45s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 43s{color} | {color:green} hadoop-hdfs-project: The patch generated 0 new + 2 unchanged - 1 fixed = 2 total (was 3) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 5s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 37s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 19s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 71m 55s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 18s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}112m 2s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.ozone.web.client.TestKeys | | | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks | | | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure070 | | Timed out junit tests | org.apache.hadoop.ozone.web.client.TestKeysRatis | | | org.apache.hadoop.ozone.container.ozoneimpl.TestOzoneContainerRatis | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HDFS-11920 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12879676/HDFS-11920-HDFS-7240.010.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle cc xml | | uname | Linux 11276005d74d 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patch
[jira] [Commented] (HDFS-12131) Add some of the FSNamesystem JMX values as metrics
[ https://issues.apache.org/jira/browse/HDFS-12131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16107978#comment-16107978 ] Andrew Wang commented on HDFS-12131: Sure, let's do the deprecate+removal in separate JIRAs for clarity. Some of the new metrics do not have unit test coverage. Could we add this? > Add some of the FSNamesystem JMX values as metrics > -- > > Key: HDFS-12131 > URL: https://issues.apache.org/jira/browse/HDFS-12131 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs, namenode >Reporter: Erik Krogen >Assignee: Erik Krogen >Priority: Minor > Attachments: HDFS-12131.000.patch, HDFS-12131.001.patch, > HDFS-12131.002.patch, HDFS-12131.002.patch, HDFS-12131.003.patch, > HDFS-12131.004.patch > > > A number of useful numbers are emitted via the FSNamesystem JMX, but not > through the metrics system. These would be useful to be able to track over > time, e.g. to alert on via standard metrics systems or to view trends and > rate changes: > * NumLiveDataNodes > * NumDeadDataNodes > * NumDecomLiveDataNodes > * NumDecomDeadDataNodes > * NumDecommissioningDataNodes > * NumStaleStorages > This is a simple change that just requires annotating the JMX methods with > {{@Metric}}. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-11826) Federation Namenode Heartbeat
[ https://issues.apache.org/jira/browse/HDFS-11826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16107974#comment-16107974 ] Hadoop QA commented on HDFS-11826: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 6 new or modified test files. {color} | || || || || {color:brown} HDFS-10467 Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 17s{color} | {color:green} HDFS-10467 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s{color} | {color:green} HDFS-10467 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 38s{color} | {color:green} HDFS-10467 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 48s{color} | {color:green} HDFS-10467 passed {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 42s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs in HDFS-10467 has 10 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s{color} | {color:green} HDFS-10467 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 46s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 43s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 35s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch generated 11 new + 654 unchanged - 0 fixed = 665 total (was 654) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 48s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 42s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 67m 12s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 16s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 94m 50s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts | | | hadoop.tools.TestHdfsConfigFields | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HDFS-11826 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12879683/HDFS-11826-HDFS-10467-000.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 9007297f11dc 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | HDFS-10467 / fae1d1e | | Default Java | 1.8.0_131 | | findbugs | v3.1.0-RC1 | | findbugs | https://builds.apache.org/job/PreCommit-HDFS-Build/20503/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html | | checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/20503/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/20503/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/20503/testReport/ | | modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/20503/console | | Powered by | Apache Yetus 0.6.0-
[jira] [Commented] (HDFS-12117) HttpFS does not seem to support SNAPSHOT related methods for WebHDFS REST Interface
[ https://issues.apache.org/jira/browse/HDFS-12117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16107940#comment-16107940 ] Hadoop QA commented on HDFS-12117: -- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 50s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 22s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 19s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 14s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 27s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 22s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 19s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 19s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 18s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs-httpfs: The patch generated 28 new + 409 unchanged - 0 fixed = 437 total (was 409) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 34s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 7s{color} | {color:green} hadoop-hdfs-httpfs in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 27m 24s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | HDFS-12117 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12879704/HDFS-12117.006.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 2b866646a431 3.13.0-117-generic #164-Ubuntu SMP Fri Apr 7 11:05:26 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 3e23415 | | Default Java | 1.8.0_131 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/20505/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-httpfs.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/20505/testReport/ | | modules | C: hadoop-hdfs-project/hadoop-hdfs-httpfs U: hadoop-hdfs-project/hadoop-hdfs-httpfs | | Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/20505/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > HttpFS does not seem to support SNAPSHOT related methods for WebHDFS REST > Interface > --- > > Key: HDFS-12117 > URL: https://issues.apache.org/jira/browse/HDFS-12117 > Project: Hadoop HDFS > Issue Type: New Feature > Components: httpfs >Affects Version
[jira] [Updated] (HDFS-12117) HttpFS does not seem to support SNAPSHOT related methods for WebHDFS REST Interface
[ https://issues.apache.org/jira/browse/HDFS-12117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HDFS-12117: Status: In Progress (was: Patch Available) > HttpFS does not seem to support SNAPSHOT related methods for WebHDFS REST > Interface > --- > > Key: HDFS-12117 > URL: https://issues.apache.org/jira/browse/HDFS-12117 > Project: Hadoop HDFS > Issue Type: New Feature > Components: httpfs >Affects Versions: 3.0.0-alpha3 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil > Attachments: HDFS-12117.003.patch, HDFS-12117.004.patch, > HDFS-12117.005.patch, HDFS-12117.006.patch, HDFS-12117.patch.01, > HDFS-12117.patch.02 > > > Currently, HttpFS is lacking implementation for SNAPSHOT related methods from > WebHDFS REST interface as defined by WebHDFS documentation [WebHDFS > documentation|https://archive.cloudera.com/cdh5/cdh/5/hadoop/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Snapshot_Operations] > I would like to work on this implementation, following the existing design > approach already implemented by other WebHDFS methods on current HttpFS > project, so I'll be proposing an initial patch soon for reviews. > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-12117) HttpFS does not seem to support SNAPSHOT related methods for WebHDFS REST Interface
[ https://issues.apache.org/jira/browse/HDFS-12117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil updated HDFS-12117: Attachment: HDFS-12117.006.patch Thanks for the suggestions [~jojochuang]! Attaching a new patch with the Client side tests added as suggested. While implementing these, I noticed that *LocalFileSystem* currently has no support for snapshot related methods. I'm not sure if that's intended to be that way, or should we add support for this as well? *TestHttpFSFileSystemLocalFileSystem* would fail when trying to run snapshot related methods defined on *BaseTestHttpFSWith*, so for now, these tests would only really run if *FileSystem* implementation is different from *LocalFileSystem*. For example, rename snapshot test: {noformat} private void testRenameSnapshot() throws Exception { if (!this.isLocalFS()) { Path snapshottablePath = new Path("/tmp/tmp-snap-test"); createSnapshotTestsPreconditions(snapshottablePath); //Now get the FileSystem instance that's being tested FileSystem fs = this.getHttpFSFileSystem(); fs.createSnapshot(snapshottablePath, "snap-to-rename"); fs.renameSnapshot(snapshottablePath, "snap-to-rename", "snap-new-name"); Path snapshotsDir = new Path("/tmp/tmp-snap-test/.snapshot"); FileStatus[] snapshotItems = fs.listStatus(snapshotsDir); assertTrue("Should have exactly one snapshot.", snapshotItems.length == 1); String resultingSnapName = snapshotItems[0].getPath().getName(); assertTrue("Snapshot name is not same as passed name.", "snap-new-name".equals(resultingSnapName)); cleanSnapshotTests(snapshottablePath, resultingSnapName); } } {noformat} > HttpFS does not seem to support SNAPSHOT related methods for WebHDFS REST > Interface > --- > > Key: HDFS-12117 > URL: https://issues.apache.org/jira/browse/HDFS-12117 > Project: Hadoop HDFS > Issue Type: New Feature > Components: httpfs >Affects Versions: 3.0.0-alpha3 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil > Attachments: HDFS-12117.003.patch, HDFS-12117.004.patch, > HDFS-12117.005.patch, HDFS-12117.006.patch, HDFS-12117.patch.01, > HDFS-12117.patch.02 > > > Currently, HttpFS is lacking implementation for SNAPSHOT related methods from > WebHDFS REST interface as defined by WebHDFS documentation [WebHDFS > documentation|https://archive.cloudera.com/cdh5/cdh/5/hadoop/hadoop-project-dist/hadoop-hdfs/WebHDFS.html#Snapshot_Operations] > I would like to work on this implementation, following the existing design > approach already implemented by other WebHDFS methods on current HttpFS > project, so I'll be proposing an initial patch soon for reviews. > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org