[jira] [Commented] (HDFS-13831) Make block increment deletion number configurable
[ https://issues.apache.org/jira/browse/HDFS-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17327227#comment-17327227 ] Qi Zhu commented on HDFS-13831: --- [~weichiu] [~linyiqun] [~gaofeng6] I created HDFS-15994 to improve this more usable in huge clusters with heavy deletion. Thanks. > Make block increment deletion number configurable > - > > Key: HDFS-13831 > URL: https://issues.apache.org/jira/browse/HDFS-13831 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Yiqun Lin >Assignee: Ryan Wu >Priority: Major > Fix For: 2.10.0, 3.2.0, 3.0.4, 3.1.2 > > Attachments: HDFS-13831.001.patch, HDFS-13831.002.patch, > HDFS-13831.003.patch, HDFS-13831.004.patch, HDFS-13831.branch-3.0.001.patch > > > When NN deletes a large directory, it will hold the write lock long time. For > improving this, we remove the blocks in a batch way. So that other waiters > have a chance to get the lock. But right now, the batch number is a > hard-coded value. > {code:java} > static int BLOCK_DELETION_INCREMENT = 1000; > {code} > We can make this value configurable, so that we can control the frequency of > other waiters to get the lock chance. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13831) Make block increment deletion number configurable
[ https://issues.apache.org/jira/browse/HDFS-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17259448#comment-17259448 ] GeoffreyStark commented on HDFS-13831: -- After I changed parameter “dfs.namenode.block.deletion.increment” to 50 and parameter “dfs.namenode.storageinfo.defragment.ratio” to 0.9, RPC still maintained a high delay state when the .Trash deleted 400T files. Do you have any good suggestions?[~jianliang.wu] [~linyiqun] > Make block increment deletion number configurable > - > > Key: HDFS-13831 > URL: https://issues.apache.org/jira/browse/HDFS-13831 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Yiqun Lin >Assignee: Ryan Wu >Priority: Major > Fix For: 2.10.0, 3.2.0, 3.0.4, 3.1.2 > > Attachments: HDFS-13831.001.patch, HDFS-13831.002.patch, > HDFS-13831.003.patch, HDFS-13831.004.patch, HDFS-13831.branch-3.0.001.patch > > > When NN deletes a large directory, it will hold the write lock long time. For > improving this, we remove the blocks in a batch way. So that other waiters > have a chance to get the lock. But right now, the batch number is a > hard-coded value. > {code} > static int BLOCK_DELETION_INCREMENT = 1000; > {code} > We can make this value configurable, so that we can control the frequency of > other waiters to get the lock chance. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13831) Make block increment deletion number configurable
[ https://issues.apache.org/jira/browse/HDFS-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17243126#comment-17243126 ] GeoffreyStark commented on HDFS-13831: -- I will get your feedback after benchmarking the modification of this parameter in my spare time > Make block increment deletion number configurable > - > > Key: HDFS-13831 > URL: https://issues.apache.org/jira/browse/HDFS-13831 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Yiqun Lin >Assignee: Ryan Wu >Priority: Major > Fix For: 2.10.0, 3.2.0, 3.0.4, 3.1.2 > > Attachments: HDFS-13831.001.patch, HDFS-13831.002.patch, > HDFS-13831.003.patch, HDFS-13831.004.patch, HDFS-13831.branch-3.0.001.patch > > > When NN deletes a large directory, it will hold the write lock long time. For > improving this, we remove the blocks in a batch way. So that other waiters > have a chance to get the lock. But right now, the batch number is a > hard-coded value. > {code} > static int BLOCK_DELETION_INCREMENT = 1000; > {code} > We can make this value configurable, so that we can control the frequency of > other waiters to get the lock chance. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13831) Make block increment deletion number configurable
[ https://issues.apache.org/jira/browse/HDFS-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17243125#comment-17243125 ] GeoffreyStark commented on HDFS-13831: -- [~jianliang.wu] Excuse me again:D Would reducing this parameter increase the time consumed by the overall deletion process? > Make block increment deletion number configurable > - > > Key: HDFS-13831 > URL: https://issues.apache.org/jira/browse/HDFS-13831 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Yiqun Lin >Assignee: Ryan Wu >Priority: Major > Fix For: 2.10.0, 3.2.0, 3.0.4, 3.1.2 > > Attachments: HDFS-13831.001.patch, HDFS-13831.002.patch, > HDFS-13831.003.patch, HDFS-13831.004.patch, HDFS-13831.branch-3.0.001.patch > > > When NN deletes a large directory, it will hold the write lock long time. For > improving this, we remove the blocks in a batch way. So that other waiters > have a chance to get the lock. But right now, the batch number is a > hard-coded value. > {code} > static int BLOCK_DELETION_INCREMENT = 1000; > {code} > We can make this value configurable, so that we can control the frequency of > other waiters to get the lock chance. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13831) Make block increment deletion number configurable
[ https://issues.apache.org/jira/browse/HDFS-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17242953#comment-17242953 ] GeoffreyStark commented on HDFS-13831: -- Well, thank you very much. I see. That's what I was thinking yesterday:D > Make block increment deletion number configurable > - > > Key: HDFS-13831 > URL: https://issues.apache.org/jira/browse/HDFS-13831 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Yiqun Lin >Assignee: Ryan Wu >Priority: Major > Fix For: 2.10.0, 3.2.0, 3.0.4, 3.1.2 > > Attachments: HDFS-13831.001.patch, HDFS-13831.002.patch, > HDFS-13831.003.patch, HDFS-13831.004.patch, HDFS-13831.branch-3.0.001.patch > > > When NN deletes a large directory, it will hold the write lock long time. For > improving this, we remove the blocks in a batch way. So that other waiters > have a chance to get the lock. But right now, the batch number is a > hard-coded value. > {code} > static int BLOCK_DELETION_INCREMENT = 1000; > {code} > We can make this value configurable, so that we can control the frequency of > other waiters to get the lock chance. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13831) Make block increment deletion number configurable
[ https://issues.apache.org/jira/browse/HDFS-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17242880#comment-17242880 ] Ryan Wu commented on HDFS-13831: IMO you could make the loop faster to hold the lock for a short time. > Make block increment deletion number configurable > - > > Key: HDFS-13831 > URL: https://issues.apache.org/jira/browse/HDFS-13831 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Yiqun Lin >Assignee: Ryan Wu >Priority: Major > Fix For: 2.10.0, 3.2.0, 3.0.4, 3.1.2 > > Attachments: HDFS-13831.001.patch, HDFS-13831.002.patch, > HDFS-13831.003.patch, HDFS-13831.004.patch, HDFS-13831.branch-3.0.001.patch > > > When NN deletes a large directory, it will hold the write lock long time. For > improving this, we remove the blocks in a batch way. So that other waiters > have a chance to get the lock. But right now, the batch number is a > hard-coded value. > {code} > static int BLOCK_DELETION_INCREMENT = 1000; > {code} > We can make this value configurable, so that we can control the frequency of > other waiters to get the lock chance. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13831) Make block increment deletion number configurable
[ https://issues.apache.org/jira/browse/HDFS-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17242227#comment-17242227 ] GeoffreyStark commented on HDFS-13831: -- Can I understand that the purpose of lowering "dfs.namenode.block.deletion.increment" parameter is to make the loop of "blockManager.removeBlock(iter.next())" process faster each time, hold the lock for a short time, and others obejct obtain the lock more easily even though the lock contention frequency is high? > Make block increment deletion number configurable > - > > Key: HDFS-13831 > URL: https://issues.apache.org/jira/browse/HDFS-13831 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Yiqun Lin >Assignee: Ryan Wu >Priority: Major > Fix For: 2.10.0, 3.2.0, 3.0.4, 3.1.2 > > Attachments: HDFS-13831.001.patch, HDFS-13831.002.patch, > HDFS-13831.003.patch, HDFS-13831.004.patch, HDFS-13831.branch-3.0.001.patch > > > When NN deletes a large directory, it will hold the write lock long time. For > improving this, we remove the blocks in a batch way. So that other waiters > have a chance to get the lock. But right now, the batch number is a > hard-coded value. > {code} > static int BLOCK_DELETION_INCREMENT = 1000; > {code} > We can make this value configurable, so that we can control the frequency of > other waiters to get the lock chance. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13831) Make block increment deletion number configurable
[ https://issues.apache.org/jira/browse/HDFS-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17242059#comment-17242059 ] GeoffreyStark commented on HDFS-13831: -- [~jianliang.wu] According to what you said about the HDFS-13671, I suspect that the cause of my problem is basically the same as what it said. I will monitor jstack log again when RPC has a high delay and compare it with HDFS-13671. Thank you for your help > Make block increment deletion number configurable > - > > Key: HDFS-13831 > URL: https://issues.apache.org/jira/browse/HDFS-13831 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Yiqun Lin >Assignee: Ryan Wu >Priority: Major > Fix For: 2.10.0, 3.2.0, 3.0.4, 3.1.2 > > Attachments: HDFS-13831.001.patch, HDFS-13831.002.patch, > HDFS-13831.003.patch, HDFS-13831.004.patch, HDFS-13831.branch-3.0.001.patch > > > When NN deletes a large directory, it will hold the write lock long time. For > improving this, we remove the blocks in a batch way. So that other waiters > have a chance to get the lock. But right now, the batch number is a > hard-coded value. > {code} > static int BLOCK_DELETION_INCREMENT = 1000; > {code} > We can make this value configurable, so that we can control the frequency of > other waiters to get the lock chance. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13831) Make block increment deletion number configurable
[ https://issues.apache.org/jira/browse/HDFS-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17242056#comment-17242056 ] GeoffreyStark commented on HDFS-13831: -- [~jianliang.wu] If the value of this parameter is reduced, does that mean that there will be more writeLock()? It has been shown in my screenshot that there will be a lot of high latency in the place of method writeLock() > Make block increment deletion number configurable > - > > Key: HDFS-13831 > URL: https://issues.apache.org/jira/browse/HDFS-13831 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Yiqun Lin >Assignee: Ryan Wu >Priority: Major > Fix For: 2.10.0, 3.2.0, 3.0.4, 3.1.2 > > Attachments: HDFS-13831.001.patch, HDFS-13831.002.patch, > HDFS-13831.003.patch, HDFS-13831.004.patch, HDFS-13831.branch-3.0.001.patch > > > When NN deletes a large directory, it will hold the write lock long time. For > improving this, we remove the blocks in a batch way. So that other waiters > have a chance to get the lock. But right now, the batch number is a > hard-coded value. > {code} > static int BLOCK_DELETION_INCREMENT = 1000; > {code} > We can make this value configurable, so that we can control the frequency of > other waiters to get the lock chance. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13831) Make block increment deletion number configurable
[ https://issues.apache.org/jira/browse/HDFS-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17242051#comment-17242051 ] Ryan Wu commented on HDFS-13831: [~gaofeng6] To make the parameters("dfs.namenode.block.deletion.increment") smaller would help. But the root cause is mentioned from HDFS-13671. > Make block increment deletion number configurable > - > > Key: HDFS-13831 > URL: https://issues.apache.org/jira/browse/HDFS-13831 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Yiqun Lin >Assignee: Ryan Wu >Priority: Major > Fix For: 2.10.0, 3.2.0, 3.0.4, 3.1.2 > > Attachments: HDFS-13831.001.patch, HDFS-13831.002.patch, > HDFS-13831.003.patch, HDFS-13831.004.patch, HDFS-13831.branch-3.0.001.patch > > > When NN deletes a large directory, it will hold the write lock long time. For > improving this, we remove the blocks in a batch way. So that other waiters > have a chance to get the lock. But right now, the batch number is a > hard-coded value. > {code} > static int BLOCK_DELETION_INCREMENT = 1000; > {code} > We can make this value configurable, so that we can control the frequency of > other waiters to get the lock chance. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13831) Make block increment deletion number configurable
[ https://issues.apache.org/jira/browse/HDFS-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17241420#comment-17241420 ] Frank Top Frank commented on HDFS-13831: [~linyiqun], [~jianliang.wu] [~weichiu] Hello, recently, in A production environment with more than 1000 nodes(HDP3.1.0;HADOOP-3.1.0), when A large number of files were deleted, RPC response was particularly high. I detected the phenomenon in the following figure with tool Arthas(Alibaba) !image-2020-12-01-17-58-47-313.png! The HDP code is as follows {code:java} static int BLOCK_DELETION_INCREMENT = 1000; .."... .. void removeBlocks(BlocksMapUpdateInfo blocks) {//写入editlog后,循环刚才收集到的blocks,然后调用blockManager的removeBlock来处理要删除的数据块. List toDeleteList = blocks.getToDeleteList(); Iterator iter = toDeleteList.iterator(); while (iter.hasNext()) { writeLock(); try { //循环收集到的块,这里双重限制:常量限制(BLOCK_DELETION_INCREMENT)和块数量限制(iter.hasNext()) //每次默认限制删除块的增量是BLOCK_DELETION_INCREMENT(1000). for (int i = 0; i < BLOCK_DELETION_INCREMENT && iter.hasNext(); i++) { blockManager.removeBlock(iter.next());//用blockManager的removeBlock来处理要删除的数据块. } } finally { writeUnlock("removeBlocks"); } } }{code} Should I inject your patch and make the parameters("dfs.namenode.block.deletion.increment") bigger? > Make block increment deletion number configurable > - > > Key: HDFS-13831 > URL: https://issues.apache.org/jira/browse/HDFS-13831 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Yiqun Lin >Assignee: Ryan Wu >Priority: Major > Fix For: 2.10.0, 3.2.0, 3.0.4, 3.1.2 > > Attachments: HDFS-13831.001.patch, HDFS-13831.002.patch, > HDFS-13831.003.patch, HDFS-13831.004.patch, HDFS-13831.branch-3.0.001.patch > > > When NN deletes a large directory, it will hold the write lock long time. For > improving this, we remove the blocks in a batch way. So that other waiters > have a chance to get the lock. But right now, the batch number is a > hard-coded value. > {code} > static int BLOCK_DELETION_INCREMENT = 1000; > {code} > We can make this value configurable, so that we can control the frequency of > other waiters to get the lock chance. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13831) Make block increment deletion number configurable
[ https://issues.apache.org/jira/browse/HDFS-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595551#comment-16595551 ] Wei-Chiu Chuang commented on HDFS-13831: +1 Test failures unrelated. Will commit later today. > Make block increment deletion number configurable > - > > Key: HDFS-13831 > URL: https://issues.apache.org/jira/browse/HDFS-13831 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Yiqun Lin >Assignee: Ryan Wu >Priority: Major > Fix For: 2.10.0, 3.2.0, 3.1.2 > > Attachments: HDFS-13831.001.patch, HDFS-13831.002.patch, > HDFS-13831.003.patch, HDFS-13831.004.patch, HDFS-13831.branch-3.0.001.patch > > > When NN deletes a large directory, it will hold the write lock long time. For > improving this, we remove the blocks in a batch way. So that other waiters > have a chance to get the lock. But right now, the batch number is a > hard-coded value. > {code} > static int BLOCK_DELETION_INCREMENT = 1000; > {code} > We can make this value configurable, so that we can control the frequency of > other waiters to get the lock chance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13831) Make block increment deletion number configurable
[ https://issues.apache.org/jira/browse/HDFS-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595526#comment-16595526 ] genericqa commented on HDFS-13831: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m 41s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} branch-3.0 Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m 1s{color} | {color:green} branch-3.0 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 6s{color} | {color:green} branch-3.0 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 1s{color} | {color:green} branch-3.0 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 12s{color} | {color:green} branch-3.0 passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 43s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 56s{color} | {color:green} branch-3.0 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 47s{color} | {color:green} branch-3.0 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 53s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 54s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 609 unchanged - 2 fixed = 609 total (was 611) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 28s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 57s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 46s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 77m 57s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 30s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}149m 24s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy | | | hadoop.hdfs.TestLeaseRecovery2 | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:1776208 | | JIRA Issue | HDFS-13831 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12937480/HDFS-13831.branch-3.0.001.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml | | uname | Linux 7bb41942fcde 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27 10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | branch-3.0 / d89a485 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_181 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/24906/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results |
[jira] [Commented] (HDFS-13831) Make block increment deletion number configurable
[ https://issues.apache.org/jira/browse/HDFS-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595249#comment-16595249 ] Wei-Chiu Chuang commented on HDFS-13831: +1 from me too. The description in core-default.xml can be reworded to make it more clear for administrators. But I am okay with that. Cherrypicking it into branch-3.1 and branch-3.0 > Make block increment deletion number configurable > - > > Key: HDFS-13831 > URL: https://issues.apache.org/jira/browse/HDFS-13831 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Yiqun Lin >Assignee: Ryan Wu >Priority: Major > Fix For: 2.10.0, 3.2.0 > > Attachments: HDFS-13831.001.patch, HDFS-13831.002.patch, > HDFS-13831.003.patch, HDFS-13831.004.patch > > > When NN deletes a large directory, it will hold the write lock long time. For > improving this, we remove the blocks in a batch way. So that other waiters > have a chance to get the lock. But right now, the batch number is a > hard-coded value. > {code} > static int BLOCK_DELETION_INCREMENT = 1000; > {code} > We can make this value configurable, so that we can control the frequency of > other waiters to get the lock chance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13831) Make block increment deletion number configurable
[ https://issues.apache.org/jira/browse/HDFS-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593262#comment-16593262 ] Hudson commented on HDFS-13831: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14829 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14829/]) HDFS-13831. Make block increment deletion number configurable. (yqlin: rev b9b964d25335943fb15cdfcf369d123bbd7e454a) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestLargeDirectoryDelete.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java > Make block increment deletion number configurable > - > > Key: HDFS-13831 > URL: https://issues.apache.org/jira/browse/HDFS-13831 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Yiqun Lin >Assignee: Ryan Wu >Priority: Major > Fix For: 2.10.0, 3.2.0 > > Attachments: HDFS-13831.001.patch, HDFS-13831.002.patch, > HDFS-13831.003.patch, HDFS-13831.004.patch > > > When NN deletes a large directory, it will hold the write lock long time. For > improving this, we remove the blocks in a batch way. So that other waiters > have a chance to get the lock. But right now, the batch number is a > hard-coded value. > {code} > static int BLOCK_DELETION_INCREMENT = 1000; > {code} > We can make this value configurable, so that we can control the frequency of > other waiters to get the lock chance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13831) Make block increment deletion number configurable
[ https://issues.apache.org/jira/browse/HDFS-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593246#comment-16593246 ] Yiqun Lin commented on HDFS-13831: -- The latest build PreCommit-HDFS-Build is for the latest patch and it looks good. +1. Committing this. > Make block increment deletion number configurable > - > > Key: HDFS-13831 > URL: https://issues.apache.org/jira/browse/HDFS-13831 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Yiqun Lin >Assignee: Ryan Wu >Priority: Major > Attachments: HDFS-13831.001.patch, HDFS-13831.002.patch, > HDFS-13831.003.patch, HDFS-13831.004.patch > > > When NN deletes a large directory, it will hold the write lock long time. For > improving this, we remove the blocks in a batch way. So that other waiters > have a chance to get the lock. But right now, the batch number is a > hard-coded value. > {code} > static int BLOCK_DELETION_INCREMENT = 1000; > {code} > We can make this value configurable, so that we can control the frequency of > other waiters to get the lock chance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13831) Make block increment deletion number configurable
[ https://issues.apache.org/jira/browse/HDFS-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593212#comment-16593212 ] genericqa commented on HDFS-13831: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 35s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m 53s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 12s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 14s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 19s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 15m 1s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 16s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 58s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 15s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 7s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 7s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 1m 8s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 628 unchanged - 2 fixed = 629 total (was 630) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 8s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 46s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 23s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 59s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red}109m 34s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 31s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}177m 11s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy | | | hadoop.hdfs.server.namenode.TestNameNodeMXBean | | | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure | | | hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes | | | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 | | JIRA Issue | HDFS-13831 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12937192/HDFS-13831.004.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml | | uname | Linux 04cd4feba8e8 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14 08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / a4121c7 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_181 | | findbugs |
[jira] [Commented] (HDFS-13831) Make block increment deletion number configurable
[ https://issues.apache.org/jira/browse/HDFS-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593211#comment-16593211 ] genericqa commented on HDFS-13831: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 36s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 24s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 56s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 58s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 3s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 31s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 51s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 57s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 50s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 46s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 628 unchanged - 2 fixed = 628 total (was 630) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 55s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 1s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 3s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 38s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 51s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red}102m 32s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 32s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}155m 43s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.web.TestWebHdfsTimeouts | | | hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 | | JIRA Issue | HDFS-13831 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12937200/HDFS-13831.004.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml | | uname | Linux 3407590027b7 4.4.0-133-generic #159-Ubuntu SMP Fri Aug 10 07:31:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / a4121c7 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_181 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/24886/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/24886/testReport/ | |
[jira] [Commented] (HDFS-13831) Make block increment deletion number configurable
[ https://issues.apache.org/jira/browse/HDFS-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16593148#comment-16593148 ] Ryan Wu commented on HDFS-13831: Thanks [~xiaochen] and [~linyiqun] I have updated the patch to adopt the Xiao's suggestion. > Make block increment deletion number configurable > - > > Key: HDFS-13831 > URL: https://issues.apache.org/jira/browse/HDFS-13831 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Yiqun Lin >Assignee: Ryan Wu >Priority: Major > Attachments: HDFS-13831.001.patch, HDFS-13831.002.patch, > HDFS-13831.003.patch, HDFS-13831.004.patch > > > When NN deletes a large directory, it will hold the write lock long time. For > improving this, we remove the blocks in a batch way. So that other waiters > have a chance to get the lock. But right now, the batch number is a > hard-coded value. > {code} > static int BLOCK_DELETION_INCREMENT = 1000; > {code} > We can make this value configurable, so that we can control the frequency of > other waiters to get the lock chance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13831) Make block increment deletion number configurable
[ https://issues.apache.org/jira/browse/HDFS-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16591375#comment-16591375 ] Yiqun Lin commented on HDFS-13831: -- Thanks [~xiaochen] for taking a look. I will hold off the commit until the end of tomorrow in case others have other comments. [~jianliang.wu], could you update the patch to address the Xiao's comment? > Make block increment deletion number configurable > - > > Key: HDFS-13831 > URL: https://issues.apache.org/jira/browse/HDFS-13831 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Yiqun Lin >Assignee: Ryan Wu >Priority: Major > Attachments: HDFS-13831.001.patch, HDFS-13831.002.patch, > HDFS-13831.003.patch > > > When NN deletes a large directory, it will hold the write lock long time. For > improving this, we remove the blocks in a batch way. So that other waiters > have a chance to get the lock. But right now, the batch number is a > hard-coded value. > {code} > static int BLOCK_DELETION_INCREMENT = 1000; > {code} > We can make this value configurable, so that we can control the frequency of > other waiters to get the lock chance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13831) Make block increment deletion number configurable
[ https://issues.apache.org/jira/browse/HDFS-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590631#comment-16590631 ] Xiao Chen commented on HDFS-13831: -- Thanks [~jianliang.wu] and [~linyiqun]. LGTM pending 1 thing: The convention was to postfix the config item with _KEY and _DEFAULT. So suggeest we {{s/DFS_NAMENODE_BLOCK_DELETION_INCREMENT/DFS_NAMENODE_BLOCK_DELETION_INCREMENT_KEY/g}} > Make block increment deletion number configurable > - > > Key: HDFS-13831 > URL: https://issues.apache.org/jira/browse/HDFS-13831 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Yiqun Lin >Assignee: Ryan Wu >Priority: Major > Attachments: HDFS-13831.001.patch, HDFS-13831.002.patch, > HDFS-13831.003.patch > > > When NN deletes a large directory, it will hold the write lock long time. For > improving this, we remove the blocks in a batch way. So that other waiters > have a chance to get the lock. But right now, the batch number is a > hard-coded value. > {code} > static int BLOCK_DELETION_INCREMENT = 1000; > {code} > We can make this value configurable, so that we can control the frequency of > other waiters to get the lock chance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13831) Make block increment deletion number configurable
[ https://issues.apache.org/jira/browse/HDFS-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590405#comment-16590405 ] Yiqun Lin commented on HDFS-13831: -- Thanks [~jianliang.wu] for updating the patch. Other than the checkstyle issue, others look good to me. Hi [~jojochuang] and [~xiaochen], any other comments for the latest patch? In my opinion, current change is a very safe change. > Make block increment deletion number configurable > - > > Key: HDFS-13831 > URL: https://issues.apache.org/jira/browse/HDFS-13831 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Yiqun Lin >Assignee: Ryan Wu >Priority: Major > Attachments: HDFS-13831.001.patch, HDFS-13831.002.patch, > HDFS-13831.003.patch > > > When NN deletes a large directory, it will hold the write lock long time. For > improving this, we remove the blocks in a batch way. So that other waiters > have a chance to get the lock. But right now, the batch number is a > hard-coded value. > {code} > static int BLOCK_DELETION_INCREMENT = 1000; > {code} > We can make this value configurable, so that we can control the frequency of > other waiters to get the lock chance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13831) Make block increment deletion number configurable
[ https://issues.apache.org/jira/browse/HDFS-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590173#comment-16590173 ] genericqa commented on HDFS-13831: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 40s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 3s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 5s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 22s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 55s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 47s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 54s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 55s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 630 unchanged - 2 fixed = 631 total (was 632) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 24s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 9s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 46s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 84m 17s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 31s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}142m 39s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.server.datanode.TestDirectoryScanner | | | hadoop.hdfs.TestLeaseRecovery2 | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 | | JIRA Issue | HDFS-13831 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12936802/HDFS-13831.003.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml | | uname | Linux 8fcde814d883 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 1ac0144 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_181 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/24850/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt | | unit |
[jira] [Commented] (HDFS-13831) Make block increment deletion number configurable
[ https://issues.apache.org/jira/browse/HDFS-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590024#comment-16590024 ] Ryan Wu commented on HDFS-13831: Thx [~linyiqun]. I have updated the patch and add the error-checking logic to ensure block-increment-deletion a positive integer. Moreover, we don't need to add the new test since the _TestLargeDirectoryDelete_ already __ covers the scenario_. S_o, I just set the block-increment-deletion via configuration in _TestLargeDirectoryDelete._ > Make block increment deletion number configurable > - > > Key: HDFS-13831 > URL: https://issues.apache.org/jira/browse/HDFS-13831 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Yiqun Lin >Assignee: Ryan Wu >Priority: Major > Attachments: HDFS-13831.001.patch, HDFS-13831.002.patch, > HDFS-13831.003.patch > > > When NN deletes a large directory, it will hold the write lock long time. For > improving this, we remove the blocks in a batch way. So that other waiters > have a chance to get the lock. But right now, the batch number is a > hard-coded value. > {code} > static int BLOCK_DELETION_INCREMENT = 1000; > {code} > We can make this value configurable, so that we can control the frequency of > other waiters to get the lock chance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13831) Make block increment deletion number configurable
[ https://issues.apache.org/jira/browse/HDFS-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16588294#comment-16588294 ] Yiqun Lin commented on HDFS-13831: -- The patch almost looks good to me, [~jianliang.wu]. Some minors comments: * Rename {{block.deletion.increment}} to {{dfs.namenode.block.deletion.increment}} * Rename to {{BLOCK_DELETION_INCREMENT}} to {{DFS_NAMENODE_BLOCK_DELETION_INCREMENT}}. * The description of the setting can update to following {noformat} The number of block deletion increment. This setting will control the block increment deletion rate to ensure that other waiters on the lock can get in. {noformat} > Make block increment deletion number configurable > - > > Key: HDFS-13831 > URL: https://issues.apache.org/jira/browse/HDFS-13831 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Yiqun Lin >Assignee: Ryan Wu >Priority: Major > Attachments: HDFS-13831.001.patch, HDFS-13831.002.patch > > > When NN deletes a large directory, it will hold the write lock long time. For > improving this, we remove the blocks in a batch way. So that other waiters > have a chance to get the lock. But right now, the batch number is a > hard-coded value. > {code} > static int BLOCK_DELETION_INCREMENT = 1000; > {code} > We can make this value configurable, so that we can control the frequency of > other waiters to get the lock chance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13831) Make block increment deletion number configurable
[ https://issues.apache.org/jira/browse/HDFS-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16587522#comment-16587522 ] genericqa commented on HDFS-13831: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 23s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 38s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 13s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 7s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 49s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 14m 44s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 23s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 59s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 630 unchanged - 2 fixed = 630 total (was 632) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 29s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 18s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red}105m 38s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 34s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}168m 30s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.TestLeaseRecovery2 | | | hadoop.hdfs.server.sps.TestExternalStoragePolicySatisfier | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 | | JIRA Issue | HDFS-13831 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12936430/HDFS-13831.002.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml | | uname | Linux fcaa37a3c405 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14 08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 54d0bf8 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_181 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/24824/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/24824/testReport/ | | Max.
[jira] [Commented] (HDFS-13831) Make block increment deletion number configurable
[ https://issues.apache.org/jira/browse/HDFS-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16587460#comment-16587460 ] genericqa commented on HDFS-13831: -- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 26s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 47s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 57s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 0s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 2s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 8s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 53s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 47s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 52s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 56s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch generated 4 new + 630 unchanged - 2 fixed = 634 total (was 632) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 58s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 16s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 45s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green}105m 57s{color} | {color:green} hadoop-hdfs in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 35s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}163m 4s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 | | JIRA Issue | HDFS-13831 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12936425/HDFS-13831.002.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml | | uname | Linux b9a51bbe70ba 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14 08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 54d0bf8 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_181 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/24823/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/24823/testReport/ | | Max. process+thread count | 3091 (vs. ulimit of 1) | | modules | C: hadoop-hdfs-project/hadoop-hdfs U:
[jira] [Commented] (HDFS-13831) Make block increment deletion number configurable
[ https://issues.apache.org/jira/browse/HDFS-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16587230#comment-16587230 ] genericqa commented on HDFS-13831: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 42s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m 27s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 3s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 6s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 12s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 35s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 56s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 38s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 37s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 37s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 59s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch generated 4 new + 630 unchanged - 2 fixed = 634 total (was 632) {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 41s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 3m 47s{color} | {color:red} patch has errors when building and testing our client artifacts. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 31s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 45s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 35s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 24s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 53m 14s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 | | JIRA Issue | HDFS-13831 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12936412/HDFS-13831.001.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml | | uname | Linux 86d4fb7f9b9c 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27 10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / d3fef7a | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_181 | | findbugs | v3.1.0-RC1 | | mvninstall | https://builds.apache.org/job/PreCommit-HDFS-Build/24821/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt | | compile | https://builds.apache.org/job/PreCommit-HDFS-Build/24821/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs.txt | | javac |
[jira] [Commented] (HDFS-13831) Make block increment deletion number configurable
[ https://issues.apache.org/jira/browse/HDFS-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16584583#comment-16584583 ] Yiqun Lin commented on HDFS-13831: -- Thanks for the help, Inigo and Xiao! [~jianliang.wu] will attach his patch next week very soon, :). > Make block increment deletion number configurable > - > > Key: HDFS-13831 > URL: https://issues.apache.org/jira/browse/HDFS-13831 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Yiqun Lin >Assignee: Ryan Wu >Priority: Major > > When NN deletes a large directory, it will hold the write lock long time. For > improving this, we remove the blocks in a batch way. So that other waiters > have a chance to get the lock. But right now, the batch number is a > hard-coded value. > {code} > static int BLOCK_DELETION_INCREMENT = 1000; > {code} > We can make this value configurable, so that we can control the frequency of > other waiters to get the lock chance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13831) Make block increment deletion number configurable
[ https://issues.apache.org/jira/browse/HDFS-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16584439#comment-16584439 ] Xiao Chen commented on HDFS-13831: -- +1 to the idea. It seems Inigo added Ryan already. [~linyiqun] I also added you in the committers role so you can do this in the future (by going to project settings -> users and roles > Make block increment deletion number configurable > - > > Key: HDFS-13831 > URL: https://issues.apache.org/jira/browse/HDFS-13831 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Yiqun Lin >Assignee: Ryan Wu >Priority: Major > > When NN deletes a large directory, it will hold the write lock long time. For > improving this, we remove the blocks in a batch way. So that other waiters > have a chance to get the lock. But right now, the batch number is a > hard-coded value. > {code} > static int BLOCK_DELETION_INCREMENT = 1000; > {code} > We can make this value configurable, so that we can control the frequency of > other waiters to get the lock chance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13831) Make block increment deletion number configurable
[ https://issues.apache.org/jira/browse/HDFS-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16583315#comment-16583315 ] Yiqun Lin commented on HDFS-13831: -- Anyone who can help add [~jianliang.wu] as a HDFS contributor? He is willing to work for this, :). > Make block increment deletion number configurable > - > > Key: HDFS-13831 > URL: https://issues.apache.org/jira/browse/HDFS-13831 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Yiqun Lin >Priority: Major > > When NN deletes a large directory, it will hold the write lock long time. For > improving this, we remove the blocks in a batch way. So that other waiters > have a chance to get the lock. But right now, the batch number is a > hard-coded value. > {code} > static int BLOCK_DELETION_INCREMENT = 1000; > {code} > We can make this value configurable, so that we can control the frequency of > other waiters to get the lock chance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13831) Make block increment deletion number configurable
[ https://issues.apache.org/jira/browse/HDFS-13831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16583307#comment-16583307 ] Wei-Chiu Chuang commented on HDFS-13831: +1 to make it configurable. Better, make it time-aware, say release the lock every 1 second. > Make block increment deletion number configurable > - > > Key: HDFS-13831 > URL: https://issues.apache.org/jira/browse/HDFS-13831 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Yiqun Lin >Priority: Major > > When NN deletes a large directory, it will hold the write lock long time. For > improving this, we remove the blocks in a batch way. So that other waiters > have a chance to get the lock. But right now, the batch number is a > hard-coded value. > {code} > static int BLOCK_DELETION_INCREMENT = 1000; > {code} > We can make this value configurable, so that we can control the frequency of > other waiters to get the lock chance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org