[jira] [Commented] (HDFS-14568) setStoragePolicy should check quota and update consume on storage type quota.
[ https://issues.apache.org/jira/browse/HDFS-14568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16981403#comment-16981403 ] Jinglun commented on HDFS-14568: Hi [~xyao] , just a minor reminder of this jira. Do you have time for v04 ? Appreciating all your comments ! Thanks a lot !!! > setStoragePolicy should check quota and update consume on storage type quota. > - > > Key: HDFS-14568 > URL: https://issues.apache.org/jira/browse/HDFS-14568 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Labels: imcompatible > Attachments: HDFS-14568-001.patch, HDFS-14568-unit-test.patch, > HDFS-14568.002.patch, HDFS-14568.003.patch, HDFS-14568.004.patch > > > At present when the storage policy of a directory or a file is changed, we > just simply change the recorded policy. But the change of storage policy also > change the consume, and that will cause violation of Quota, and cause the > consume recorded in DirectoryWithQuotaFeature out of date. > We should do the quota check and consume update when setStoragePolicy(): > # Compute the new consume and check quota on ancestors. If quota exceed > throwing the QuotaExceedException. Else go to 2. > # Update the consume to all ancestors with Quota. > > Related to HDFS-14633, see HDFS-14633 to get more discussions. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14568) setStoragePolicy should check quota and update consume on storage type quota.
[ https://issues.apache.org/jira/browse/HDFS-14568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16976496#comment-16976496 ] Jinglun commented on HDFS-14568: Hi [~xyao], would you like to help taking a review of v04 ? Thanks ! > setStoragePolicy should check quota and update consume on storage type quota. > - > > Key: HDFS-14568 > URL: https://issues.apache.org/jira/browse/HDFS-14568 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Labels: imcompatible > Attachments: HDFS-14568-001.patch, HDFS-14568-unit-test.patch, > HDFS-14568.002.patch, HDFS-14568.003.patch, HDFS-14568.004.patch > > > At present when the storage policy of a directory or a file is changed, we > just simply change the recorded policy. But the change of storage policy also > change the consume, and that will cause violation of Quota, and cause the > consume recorded in DirectoryWithQuotaFeature out of date. > We should do the quota check and consume update when setStoragePolicy(): > # Compute the new consume and check quota on ancestors. If quota exceed > throwing the QuotaExceedException. Else go to 2. > # Update the consume to all ancestors with Quota. > > Related to HDFS-14633, see HDFS-14633 to get more discussions. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14568) setStoragePolicy should check quota and update consume on storage type quota.
[ https://issues.apache.org/jira/browse/HDFS-14568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16945669#comment-16945669 ] Jinglun commented on HDFS-14568: Hi [~xyao], could you help to take a look at v04 ? Thanks ! > setStoragePolicy should check quota and update consume on storage type quota. > - > > Key: HDFS-14568 > URL: https://issues.apache.org/jira/browse/HDFS-14568 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Labels: imcompatible > Attachments: HDFS-14568-001.patch, HDFS-14568-unit-test.patch, > HDFS-14568.002.patch, HDFS-14568.003.patch, HDFS-14568.004.patch > > > At present when the storage policy of a directory or a file is changed, we > just simply change the recorded policy. But the change of storage policy also > change the consume, and that will cause violation of Quota, and cause the > consume recorded in DirectoryWithQuotaFeature out of date. > We should do the quota check and consume update when setStoragePolicy(): > # Compute the new consume and check quota on ancestors. If quota exceed > throwing the QuotaExceedException. Else go to 2. > # Update the consume to all ancestors with Quota. > > Related to HDFS-14633, see HDFS-14633 to get more discussions. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14568) setStoragePolicy should check quota and update consume on storage type quota.
[ https://issues.apache.org/jira/browse/HDFS-14568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16931056#comment-16931056 ] Jinglun commented on HDFS-14568: Hi [~xyao] [~surendrasingh], do you have time for this? Could you help to review v04? Thanks! > setStoragePolicy should check quota and update consume on storage type quota. > - > > Key: HDFS-14568 > URL: https://issues.apache.org/jira/browse/HDFS-14568 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Labels: imcompatible > Attachments: HDFS-14568-001.patch, HDFS-14568-unit-test.patch, > HDFS-14568.002.patch, HDFS-14568.003.patch, HDFS-14568.004.patch > > > The quota and consume of the file's ancestors are not handled when the > storage policy of the file is changed. For example: > 1. Set quota StorageType.SSD fileSpace-1 to the parent dir; > 2. Create a file size of fileSpace with storage policy \{DISK,DISK,DISK} > under it; > 3. Change the storage policy of the file to ALLSSD_STORAGE_POLICY_NAME and > expect a QuotaByStorageTypeExceededException. > Because the quota and consume is not handled, the expected exception is not > threw out. > > There are 3 reasons why we should handle the consume and the quota. > 1. Replication uses the new storage policy. Considering a file with BlockType > CONTIGUOUS. It's replication factor is 3 and it's storage policy is "HOT". > Now we change the policy to "ONE_SSD". If a DN goes down and the file needs > replication, the NN will choose storages in policy "ONE_SSD" and replicate > the block to a SSD storage. > 2. We acturally have a cluster storaging both HOT and COLD data. We have a > backgroud process searching all the files to find those that are not accessed > for a period of time. Then we set them to COLD and start a mover to move the > replicas. After moving, all the replicas are consistent with the storage > policy. > 3. The NameNode manages the global state of the cluster. If there is any > inconsistent situation, such as the replicas doesn't match the storage policy > of the file, we should take the NameNode as the standard and make the cluster > to match the NameNode. The block replication is a good example of the rule. > When we count the consume of a file(CONTIGUOUS), we multiply the replication > factor with the file's length, no matter the file is under replicated or > excessed. So does the storage type quota and consume. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14568) setStoragePolicy should check quota and update consume on storage type quota.
[ https://issues.apache.org/jira/browse/HDFS-14568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16926292#comment-16926292 ] Jinglun commented on HDFS-14568: Hi [~surendrasingh], sorry for my late response. Do you mean to set SSD storage quota to 10byte on a directory with 10GB DISK space consume? I think we shouldn't allow this because the setStoragePolicy() will cause a quota exceed. And I think any rpc causing quota exceed should end with a QuotaExceed exception. In patch-004 a RemoteException with QuotaByStorageTypeExceededException will be thrown. +1 the change would be incompatible because the method only throws IOException but now it will throw a QuotaExceedException. May be adding a switch to enable the quota check & consume update ? > setStoragePolicy should check quota and update consume on storage type quota. > - > > Key: HDFS-14568 > URL: https://issues.apache.org/jira/browse/HDFS-14568 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HDFS-14568-001.patch, HDFS-14568-unit-test.patch, > HDFS-14568.002.patch, HDFS-14568.003.patch, HDFS-14568.004.patch > > > The quota and consume of the file's ancestors are not handled when the > storage policy of the file is changed. For example: > 1. Set quota StorageType.SSD fileSpace-1 to the parent dir; > 2. Create a file size of fileSpace with storage policy \{DISK,DISK,DISK} > under it; > 3. Change the storage policy of the file to ALLSSD_STORAGE_POLICY_NAME and > expect a QuotaByStorageTypeExceededException. > Because the quota and consume is not handled, the expected exception is not > threw out. > > There are 3 reasons why we should handle the consume and the quota. > 1. Replication uses the new storage policy. Considering a file with BlockType > CONTIGUOUS. It's replication factor is 3 and it's storage policy is "HOT". > Now we change the policy to "ONE_SSD". If a DN goes down and the file needs > replication, the NN will choose storages in policy "ONE_SSD" and replicate > the block to a SSD storage. > 2. We acturally have a cluster storaging both HOT and COLD data. We have a > backgroud process searching all the files to find those that are not accessed > for a period of time. Then we set them to COLD and start a mover to move the > replicas. After moving, all the replicas are consistent with the storage > policy. > 3. The NameNode manages the global state of the cluster. If there is any > inconsistent situation, such as the replicas doesn't match the storage policy > of the file, we should take the NameNode as the standard and make the cluster > to match the NameNode. The block replication is a good example of the rule. > When we count the consume of a file(CONTIGUOUS), we multiply the replication > factor with the file's length, no matter the file is under replicated or > excessed. So does the storage type quota and consume. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14568) setStoragePolicy should check quota and update consume on storage type quota.
[ https://issues.apache.org/jira/browse/HDFS-14568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16923076#comment-16923076 ] Surendra Singh Lilhore commented on HDFS-14568: --- Thanks [~LiJinglun] for patch.. One doubt [~LiJinglun], What if my SSD storage quota is only 10 and already I have 10GB blocks on DISK storage, should I allow to change the quota ? {quote} * Firstly, This is a change in API behavior, Earlier setStoragePolicy() wasn't throwing exception, after this change it will throw. Can be considered as an incompatible change. Thus would require agreement from more people.{quote} Yes, this is incompatible change. > setStoragePolicy should check quota and update consume on storage type quota. > - > > Key: HDFS-14568 > URL: https://issues.apache.org/jira/browse/HDFS-14568 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HDFS-14568-001.patch, HDFS-14568-unit-test.patch, > HDFS-14568.002.patch, HDFS-14568.003.patch, HDFS-14568.004.patch > > > The quota and consume of the file's ancestors are not handled when the > storage policy of the file is changed. For example: > 1. Set quota StorageType.SSD fileSpace-1 to the parent dir; > 2. Create a file size of fileSpace with storage policy \{DISK,DISK,DISK} > under it; > 3. Change the storage policy of the file to ALLSSD_STORAGE_POLICY_NAME and > expect a QuotaByStorageTypeExceededException. > Because the quota and consume is not handled, the expected exception is not > threw out. > > There are 3 reasons why we should handle the consume and the quota. > 1. Replication uses the new storage policy. Considering a file with BlockType > CONTIGUOUS. It's replication factor is 3 and it's storage policy is "HOT". > Now we change the policy to "ONE_SSD". If a DN goes down and the file needs > replication, the NN will choose storages in policy "ONE_SSD" and replicate > the block to a SSD storage. > 2. We acturally have a cluster storaging both HOT and COLD data. We have a > backgroud process searching all the files to find those that are not accessed > for a period of time. Then we set them to COLD and start a mover to move the > replicas. After moving, all the replicas are consistent with the storage > policy. > 3. The NameNode manages the global state of the cluster. If there is any > inconsistent situation, such as the replicas doesn't match the storage policy > of the file, we should take the NameNode as the standard and make the cluster > to match the NameNode. The block replication is a good example of the rule. > When we count the consume of a file(CONTIGUOUS), we multiply the replication > factor with the file's length, no matter the file is under replicated or > excessed. So does the storage type quota and consume. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14568) setStoragePolicy should check quota and update consume on storage type quota.
[ https://issues.apache.org/jira/browse/HDFS-14568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16918431#comment-16918431 ] Jinglun commented on HDFS-14568: Hi [~xyao], do you have time to review this? Thanks ! > setStoragePolicy should check quota and update consume on storage type quota. > - > > Key: HDFS-14568 > URL: https://issues.apache.org/jira/browse/HDFS-14568 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HDFS-14568-001.patch, HDFS-14568-unit-test.patch, > HDFS-14568.002.patch, HDFS-14568.003.patch, HDFS-14568.004.patch > > > The quota and consume of the file's ancestors are not handled when the > storage policy of the file is changed. For example: > 1. Set quota StorageType.SSD fileSpace-1 to the parent dir; > 2. Create a file size of fileSpace with storage policy \{DISK,DISK,DISK} > under it; > 3. Change the storage policy of the file to ALLSSD_STORAGE_POLICY_NAME and > expect a QuotaByStorageTypeExceededException. > Because the quota and consume is not handled, the expected exception is not > threw out. > > There are 3 reasons why we should handle the consume and the quota. > 1. Replication uses the new storage policy. Considering a file with BlockType > CONTIGUOUS. It's replication factor is 3 and it's storage policy is "HOT". > Now we change the policy to "ONE_SSD". If a DN goes down and the file needs > replication, the NN will choose storages in policy "ONE_SSD" and replicate > the block to a SSD storage. > 2. We acturally have a cluster storaging both HOT and COLD data. We have a > backgroud process searching all the files to find those that are not accessed > for a period of time. Then we set them to COLD and start a mover to move the > replicas. After moving, all the replicas are consistent with the storage > policy. > 3. The NameNode manages the global state of the cluster. If there is any > inconsistent situation, such as the replicas doesn't match the storage policy > of the file, we should take the NameNode as the standard and make the cluster > to match the NameNode. The block replication is a good example of the rule. > When we count the consume of a file(CONTIGUOUS), we multiply the replication > factor with the file's length, no matter the file is under replicated or > excessed. So does the storage type quota and consume. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14568) setStoragePolicy should check quota and update consume on storage type quota.
[ https://issues.apache.org/jira/browse/HDFS-14568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16915592#comment-16915592 ] Hadoop QA commented on HDFS-14568: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 48s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m 7s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 45s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 9s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 34s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 22s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 58s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 55s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 55s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 36s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 12s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 27s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 17s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red}107m 12s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 35s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}168m 44s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.tools.TestDFSZKFailoverController | | | hadoop.hdfs.server.balancer.TestBalancer | | | hadoop.hdfs.TestDecommission | | | hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes | | | hadoop.hdfs.server.namenode.TestNameNodeMXBean | \\ \\ || Subsystem || Report/Notes || | Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e53b4 | | JIRA Issue | HDFS-14568 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12978546/HDFS-14568.004.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux ce7de88694fa 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24 10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / d2225c8 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_222 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/27673/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | | Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/27673/testReport/ | | Max. process+thread count | 3072 (vs. ulimit of 5500) | | modules | C: hado
[jira] [Commented] (HDFS-14568) setStoragePolicy should check quota and update consume on storage type quota.
[ https://issues.apache.org/jira/browse/HDFS-14568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16914938#comment-16914938 ] Jinglun commented on HDFS-14568: Upload patch-003, fix checkstyle. > setStoragePolicy should check quota and update consume on storage type quota. > - > > Key: HDFS-14568 > URL: https://issues.apache.org/jira/browse/HDFS-14568 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HDFS-14568-001.patch, HDFS-14568-unit-test.patch, > HDFS-14568.002.patch, HDFS-14568.003.patch > > > The quota and consume of the file's ancestors are not handled when the > storage policy of the file is changed. For example: > 1. Set quota StorageType.SSD fileSpace-1 to the parent dir; > 2. Create a file size of fileSpace with storage policy \{DISK,DISK,DISK} > under it; > 3. Change the storage policy of the file to ALLSSD_STORAGE_POLICY_NAME and > expect a QuotaByStorageTypeExceededException. > Because the quota and consume is not handled, the expected exception is not > threw out. > > There are 3 reasons why we should handle the consume and the quota. > 1. Replication uses the new storage policy. Considering a file with BlockType > CONTIGUOUS. It's replication factor is 3 and it's storage policy is "HOT". > Now we change the policy to "ONE_SSD". If a DN goes down and the file needs > replication, the NN will choose storages in policy "ONE_SSD" and replicate > the block to a SSD storage. > 2. We acturally have a cluster storaging both HOT and COLD data. We have a > backgroud process searching all the files to find those that are not accessed > for a period of time. Then we set them to COLD and start a mover to move the > replicas. After moving, all the replicas are consistent with the storage > policy. > 3. The NameNode manages the global state of the cluster. If there is any > inconsistent situation, such as the replicas doesn't match the storage policy > of the file, we should take the NameNode as the standard and make the cluster > to match the NameNode. The block replication is a good example of the rule. > When we count the consume of a file(CONTIGUOUS), we multiply the replication > factor with the file's length, no matter the file is under replicated or > excessed. So does the storage type quota and consume. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14568) setStoragePolicy should check quota and update consume on storage type quota.
[ https://issues.apache.org/jira/browse/HDFS-14568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16914043#comment-16914043 ] Hadoop QA commented on HDFS-14568: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 54s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m 45s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 56s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 42s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 4s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 46s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 20s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 57s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 13s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 4s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 4s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 43s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch generated 5 new + 39 unchanged - 0 fixed = 44 total (was 39) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 8s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 41s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 38s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red}115m 37s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 46s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}178m 16s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes | | | hadoop.hdfs.TestReconstructStripedFile | | | hadoop.hdfs.server.datanode.TestDataNodeUUID | | | hadoop.hdfs.qjournal.server.TestJournalNodeSync | \\ \\ || Subsystem || Report/Notes || | Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e53b4 | | JIRA Issue | HDFS-14568 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12978363/HDFS-14568.002.patch | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 176419087f09 4.15.0-48-generic #51-Ubuntu SMP Wed Apr 3 08:28:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / f6af7d0 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_212 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/27646/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt | | unit | https://builds.apache.org/job/PreCommit-HDFS-Build/27646/artifact/out/patch-unit-hadoop-hdfs-
[jira] [Commented] (HDFS-14568) setStoragePolicy should check quota and update consume on storage type quota.
[ https://issues.apache.org/jira/browse/HDFS-14568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913896#comment-16913896 ] Jinglun commented on HDFS-14568: Upload patch-002 fixing the setStoragePolicy. > setStoragePolicy should check quota and update consume on storage type quota. > - > > Key: HDFS-14568 > URL: https://issues.apache.org/jira/browse/HDFS-14568 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 3.1.0 >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HDFS-14568-001.patch, HDFS-14568-unit-test.patch, > HDFS-14568.002.patch > > > The quota and consume of the file's ancestors are not handled when the > storage policy of the file is changed. For example: > 1. Set quota StorageType.SSD fileSpace-1 to the parent dir; > 2. Create a file size of fileSpace with storage policy \{DISK,DISK,DISK} > under it; > 3. Change the storage policy of the file to ALLSSD_STORAGE_POLICY_NAME and > expect a QuotaByStorageTypeExceededException. > Because the quota and consume is not handled, the expected exception is not > threw out. > > There are 3 reasons why we should handle the consume and the quota. > 1. Replication uses the new storage policy. Considering a file with BlockType > CONTIGUOUS. It's replication factor is 3 and it's storage policy is "HOT". > Now we change the policy to "ONE_SSD". If a DN goes down and the file needs > replication, the NN will choose storages in policy "ONE_SSD" and replicate > the block to a SSD storage. > 2. We acturally have a cluster storaging both HOT and COLD data. We have a > backgroud process searching all the files to find those that are not accessed > for a period of time. Then we set them to COLD and start a mover to move the > replicas. After moving, all the replicas are consistent with the storage > policy. > 3. The NameNode manages the global state of the cluster. If there is any > inconsistent situation, such as the replicas doesn't match the storage policy > of the file, we should take the NameNode as the standard and make the cluster > to match the NameNode. The block replication is a good example of the rule. > When we count the consume of a file(CONTIGUOUS), we multiply the replication > factor with the file's length, no matter the file is under replicated or > excessed. So does the storage type quota and consume. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org