[jira] [Commented] (HDFS-15377) BlockScanner scans one part per round, expect full scans after several rounds
[ https://issues.apache.org/jira/browse/HDFS-15377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17118320#comment-17118320 ] Hadoop QA commented on HDFS-15377: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m 56s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m 29s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 19s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 58s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 27s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 19m 9s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 45s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 3m 23s{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 20s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 13s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 6s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 6s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 47s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 503 unchanged - 0 fixed = 504 total (was 503) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 17m 48s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 48s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m 10s{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red}116m 7s{color} | {color:red} hadoop-hdfs in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 37s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}198m 32s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:hadoop-hdfs-project/hadoop-hdfs | | | Inconsistent synchronization of org.apache.hadoop.hdfs.server.datanode.VolumeScanner.curPosition; locked 66% of time Unsynchronized access at VolumeScanner.java:66% of time Unsynchronized access at VolumeScanner.java:[line 457] | | Failed junit tests | hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy | | | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped | | | hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes | | | hadoop.hdfs.server.datanode.TestBPOfferService | | | hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup | | | hadoop.hdfs.TestReconstructStr
[jira] [Commented] (HDFS-14320) Support skipTrash for WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-14320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17118279#comment-17118279 ] Hadoop QA commented on HDFS-14320: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 51s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green} No case conflicting files found. {color} | | {color:blue}0{color} | {color:blue} markdownlint {color} | {color:blue} 0m 0s{color} | {color:blue} markdownlint was not available. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 2s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 12s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 34s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 0s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 10s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 18m 18s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 22s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 2m 50s{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 5s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 26s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 41s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 1m 10s{color} | {color:red} hadoop-hdfs-project in the patch failed. {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 10s{color} | {color:red} hadoop-hdfs-project in the patch failed. {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 52s{color} | {color:orange} hadoop-hdfs-project: The patch generated 1 new + 150 unchanged - 1 fixed = 151 total (was 151) {color} | | {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 42s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 4m 35s{color} | {color:red} patch has errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 10s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 56s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 12s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 49s{color} | {color:red} hadoop-hdfs in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 28s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 70m 8s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/PreCommit-HDFS-Build/29375/artifact/out/Dockerfile | | JIRA Issue | HDFS-14320 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12963035/HDFS-14320-
[jira] [Commented] (HDFS-13274) RBF: Extend RouterRpcClient to use multiple sockets
[ https://issues.apache.org/jira/browse/HDFS-13274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17118254#comment-17118254 ] Janus Chow commented on HDFS-13274: --- The mount table is mostly 3 layers deep, and we have 40 mount policy. The FederationRPC stats are as follows: {code:java} "name" : "Hadoop:service=Router,name=FederationRPC", "modelerType" : "org.apache.hadoop.hdfs.server.federation.metrics.FederationRPCMetrics", "ProxyOpFailureStandby" : 181, "ProxyOpFailureCommunicate" : 8314, "ProxyOpFailureClientOverloaded" : 0, "ProxyOpNotImplemented" : 0, "ProxyOpRetries" : 0, "RouterFailureStateStoreOps" : 0, "RouterFailureSafemodeOps" : 25, "RouterFailureReadOnlyOps" : 0, "RouterFailureLockedOps" : 0, "RpcServerCallQueue" : 0, "RpcServerNumOpenConnections" : 632, "RpcClientNumConnections" : 1023, "RpcClientNumActiveConnections" : 73, "RpcClientNumCreatingConnections" : 0, "RpcClientNumConnectionPools" : 161, {code} We set the pool-size to 100. > RBF: Extend RouterRpcClient to use multiple sockets > --- > > Key: HDFS-13274 > URL: https://issues.apache.org/jira/browse/HDFS-13274 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > > HADOOP-13144 introduces the ability to create multiple connections for the > same user and use different sockets. The RouterRpcClient should use this > approach to get a better throughput. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14320) Support skipTrash for WebHDFS
[ https://issues.apache.org/jira/browse/HDFS-14320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17118235#comment-17118235 ] Chao Sun commented on HDFS-14320: - Bumping up this as this seems to be an important feature. Curious what is the current status [~weichiu]. > Support skipTrash for WebHDFS > -- > > Key: HDFS-14320 > URL: https://issues.apache.org/jira/browse/HDFS-14320 > Project: Hadoop HDFS > Issue Type: New Feature > Components: namenode, webhdfs >Affects Versions: 3.2.0 >Reporter: Karthik Palanisamy >Assignee: Karthik Palanisamy >Priority: Major > Attachments: HDFS-14320-001.patch, HDFS-14320-002.patch, > HDFS-14320-003.patch, HDFS-14320-004.patch, HDFS-14320-005.patch, > HDFS-14320-006.patch, HDFS-14320-007.patch, HDFS-14320-008.patch > > > Files/Directories deleted via webhdfs rest call doesn't use the skiptrash > feature, it would be deleted permanently. This feature is very important us > because our user has deleted large directory accidentally. > By default, Skiptrash option is set to true, skiptrash=true. Any files, Using > CURL will be permanently deleted. > Example: > curl -iv -X DELETE > "http://:50070/webhdfs/v1/tmp/sampledata?op=DELETE&user.name=hdfs&recursive=true"; > > Use skiptrash=false, to move files to trash Instead. > Example: > curl -iv -X DELETE > "http://:50070/webhdfs/v1/tmp/sampledata?op=DELETE&user.name=hdfs&recursive=true&skiptrash=false"; > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15377) BlockScanner scans one part per round, expect full scans after several rounds
[ https://issues.apache.org/jira/browse/HDFS-15377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17118218#comment-17118218 ] Yang Yun commented on HDFS-15377: - Thanks [~elgoiri] for the review. I'm sorry for the mistake. updated to HDFS-15377.002.patch. I fixed one checkstyles issue, I think the other one private issue can be ignore for variable 'scanPartSize' is a member of class Conf. > BlockScanner scans one part per round, expect full scans after several rounds > - > > Key: HDFS-15377 > URL: https://issues.apache.org/jira/browse/HDFS-15377 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15377.002.patch > > > For reducing disk IO, one block is separated to multiple parts, BlockScanner > scans only one part per round. Expect that after several rounds, the full > block should be scanned > Add a new option "dfs.block.scanner.part.size". the maximum data size per > scan by the block scanner. this value should be the multiple of chunk size, > for example, 512, 1024, 4096 ... > Default value is -1, will disable partial scan. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15377) BlockScanner scans one part per round, expect full scans after several rounds
[ https://issues.apache.org/jira/browse/HDFS-15377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15377: Attachment: (was: HDFS-15377.001.path) > BlockScanner scans one part per round, expect full scans after several rounds > - > > Key: HDFS-15377 > URL: https://issues.apache.org/jira/browse/HDFS-15377 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15377.002.patch > > > For reducing disk IO, one block is separated to multiple parts, BlockScanner > scans only one part per round. Expect that after several rounds, the full > block should be scanned > Add a new option "dfs.block.scanner.part.size". the maximum data size per > scan by the block scanner. this value should be the multiple of chunk size, > for example, 512, 1024, 4096 ... > Default value is -1, will disable partial scan. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15377) BlockScanner scans one part per round, expect full scans after several rounds
[ https://issues.apache.org/jira/browse/HDFS-15377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15377: Status: Open (was: Patch Available) > BlockScanner scans one part per round, expect full scans after several rounds > - > > Key: HDFS-15377 > URL: https://issues.apache.org/jira/browse/HDFS-15377 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15377.002.patch > > > For reducing disk IO, one block is separated to multiple parts, BlockScanner > scans only one part per round. Expect that after several rounds, the full > block should be scanned > Add a new option "dfs.block.scanner.part.size". the maximum data size per > scan by the block scanner. this value should be the multiple of chunk size, > for example, 512, 1024, 4096 ... > Default value is -1, will disable partial scan. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15377) BlockScanner scans one part per round, expect full scans after several rounds
[ https://issues.apache.org/jira/browse/HDFS-15377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Yun updated HDFS-15377: Attachment: HDFS-15377.002.patch Status: Patch Available (was: Open) > BlockScanner scans one part per round, expect full scans after several rounds > - > > Key: HDFS-15377 > URL: https://issues.apache.org/jira/browse/HDFS-15377 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15377.002.patch > > > For reducing disk IO, one block is separated to multiple parts, BlockScanner > scans only one part per round. Expect that after several rounds, the full > block should be scanned > Add a new option "dfs.block.scanner.part.size". the maximum data size per > scan by the block scanner. this value should be the multiple of chunk size, > for example, 512, 1024, 4096 ... > Default value is -1, will disable partial scan. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15246) ArrayIndexOfboundsException in BlockManager CreateLocatedBlock
[ https://issues.apache.org/jira/browse/HDFS-15246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17118129#comment-17118129 ] Hadoop QA commented on HDFS-15246: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m 4s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m 37s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 19s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 52s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 25s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 16m 17s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 47s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 2m 56s{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 54s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 7s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 3s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 3s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 43s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 55s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red}108m 43s{color} | {color:red} hadoop-hdfs in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 41s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}179m 2s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics | | | hadoop.hdfs.TestSafeModeWithStripedFileWithRandomECPolicy | | | hadoop.hdfs.server.namenode.ha.TestHAAppend | | | hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes | | | hadoop.hdfs.TestReconstructStripedFile | | | hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy | | | hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup | | | hadoop.hdfs.client.impl.TestBlockReaderLocal | | | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped | | | hadoop.hdfs.TestReadStripedFileWithMissingBlocks | \\ \\ || Subsystem || Report/Notes || | Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://builds.apache.org/job/PreCommit-HDFS-Build/29373/artifact/out/Dockerfile | | JIRA Issue | HDFS-15246 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/13004166/HDFS-15246.001.patch | | Optional Tests | dupname asflicense compile javac java
[jira] [Commented] (HDFS-15362) FileWithSnapshotFeature#updateQuotaAndCollectBlocks should collect all distinct blocks
[ https://issues.apache.org/jira/browse/HDFS-15362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17117995#comment-17117995 ] Hudson commented on HDFS-15362: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18300 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/18300/]) HDFS-15362. FileWithSnapshotFeature#updateQuotaAndCollectBlocks should (inigoiri: rev 2148a8fe645333444c4e8110bb56acf0fb8e41b4) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestFileWithSnapshotFeature.java > FileWithSnapshotFeature#updateQuotaAndCollectBlocks should collect all > distinct blocks > -- > > Key: HDFS-15362 > URL: https://issues.apache.org/jira/browse/HDFS-15362 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Major > Fix For: 3.4.0 > > Attachments: HDFS-15362.001.patch, HDFS-15362.002.patch > > > FileWithSnapshotFeature#updateQuotaAndCollectBlocks uses list to collect > blocks > {code:java} > List allBlocks = new ArrayList(); > if (file.getBlocks() != null) { > allBlocks.addAll(Arrays.asList(file.getBlocks())); > }{code} > INodeFile#storagespaceConsumedContiguous collects all distinct blocks by set > {code:java} > // Collect all distinct blocks > Set allBlocks = new HashSet<>(Arrays.asList(getBlocks())); > DiffList diffs = sf.getDiffs().asList(); > for(FileDiff diff : diffs) { >BlockInfo[] diffBlocks = diff.getBlocks(); >if (diffBlocks != null) { > allBlocks.addAll(Arrays.asList(diffBlocks)); > } {code} > but on updating the reclaim context we subtract these both , so wrong quota > value can be updated > {code:java} > QuotaCounts current = file.storagespaceConsumed(bsp); > reclaimContext.quotaDelta().add(oldCounts.subtract(current)); {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15351) Blocks Scheduled Count was wrong on Truncate
[ https://issues.apache.org/jira/browse/HDFS-15351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17117993#comment-17117993 ] hemanthboyina commented on HDFS-15351: -- {quote} I've always hated this list to array interface... [~belugabehr] you are the expert on these things; any alternative? {quote} hi [~belugabehr] , you have any alternatives or suggestions for this ? > Blocks Scheduled Count was wrong on Truncate > - > > Key: HDFS-15351 > URL: https://issues.apache.org/jira/browse/HDFS-15351 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Major > Attachments: HDFS-15351.001.patch, HDFS-15351.002.patch, > HDFS-15351.003.patch > > > On truncate and append we remove the blocks from Reconstruction Queue > On removing the blocks from pending reconstruction , we need to decrement > Blocks Scheduled -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15246) ArrayIndexOfboundsException in BlockManager CreateLocatedBlock
[ https://issues.apache.org/jira/browse/HDFS-15246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hemanthboyina updated HDFS-15246: - Attachment: HDFS-15246.001.patch Status: Patch Available (was: Open) > ArrayIndexOfboundsException in BlockManager CreateLocatedBlock > -- > > Key: HDFS-15246 > URL: https://issues.apache.org/jira/browse/HDFS-15246 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Major > Attachments: HDFS-15246-testrepro.patch, HDFS-15246.001.patch > > > java.lang.ArrayIndexOutOfBoundsException: Index 1 out of bounds for length 1 > > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createLocatedBlock(BlockManager.java:1362) > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.createLocatedBlocks(BlockManager.java:1501) > at > org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getBlockLocations(FSDirStatAndListingOp.java:179) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:2047) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:770) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15375) Reconstruction Work should not happen for Corrupt Block
[ https://issues.apache.org/jira/browse/HDFS-15375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17117980#comment-17117980 ] hemanthboyina commented on HDFS-15375: -- ran test failures in local , seems not related org.apache.hadoop.hdfs.TestReconstructStripedFile.testErasureCodingWorkerXmitsWeight org.apache.hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy.testErasureCodingWorkerXmitsWeight these tests were failing even without this patch , following up on these tests , found they were failing continonusly [https://builds.apache.org/job/PreCommit-HDFS-Build/29368/] [https://builds.apache.org/job/PreCommit-HDFS-Build/29366/|https://builds.apache.org/job/PreCommit-HDFS-Build/29366/#showFailuresLink] [https://builds.apache.org/job/PreCommit-HDFS-Build/29358/] > Reconstruction Work should not happen for Corrupt Block > --- > > Key: HDFS-15375 > URL: https://issues.apache.org/jira/browse/HDFS-15375 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Major > Attachments: HDFS-15375-testrepro.patch, HDFS-15375.001.patch > > > In BlockManager#updateNeededReconstructions , while updating the > NeededReconstruction we are adding Pendingreconstruction blocks to live > replicas > {code:java} > int pendingNum = pendingReconstruction.getNumReplicas(block); > int curExpectedReplicas = getExpectedRedundancyNum(block); > if (!hasEnoughEffectiveReplicas(block, repl, pendingNum)) { > neededReconstruction.update(block, repl.liveReplicas() + > pendingNum,{code} > But if two replicas were in pending reconstruction (due to corruption) , and > if the third replica is corrupted the block should be in > QUEUE_WITH_CORRUPT_BLOCKS but because of above logic it was getting added in > QUEUE_LOW_REDUNDANCY , this makes the RedudancyMonitor to reconstruct a > corrupted block , which is wrong -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15377) BlockScanner scans one part per round, expect full scans after several rounds
[ https://issues.apache.org/jira/browse/HDFS-15377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17117974#comment-17117974 ] Íñigo Goiri commented on HDFS-15377: Thanks [~hadoop_yangyun] for the patch. Do you mind uploading with the right extension (path->patch). There's a also a few checkstyles and findbugs here. > BlockScanner scans one part per round, expect full scans after several rounds > - > > Key: HDFS-15377 > URL: https://issues.apache.org/jira/browse/HDFS-15377 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Yang Yun >Assignee: Yang Yun >Priority: Minor > Attachments: HDFS-15377.001.path > > > For reducing disk IO, one block is separated to multiple parts, BlockScanner > scans only one part per round. Expect that after several rounds, the full > block should be scanned > Add a new option "dfs.block.scanner.part.size". the maximum data size per > scan by the block scanner. this value should be the multiple of chunk size, > for example, 512, 1024, 4096 ... > Default value is -1, will disable partial scan. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15362) FileWithSnapshotFeature#updateQuotaAndCollectBlocks should collect all distinct blocks
[ https://issues.apache.org/jira/browse/HDFS-15362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17117971#comment-17117971 ] Íñigo Goiri commented on HDFS-15362: Thanks [~hemanthboyina] for the patch. Committed to trunk. > FileWithSnapshotFeature#updateQuotaAndCollectBlocks should collect all > distinct blocks > -- > > Key: HDFS-15362 > URL: https://issues.apache.org/jira/browse/HDFS-15362 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Major > Fix For: 3.4.0 > > Attachments: HDFS-15362.001.patch, HDFS-15362.002.patch > > > FileWithSnapshotFeature#updateQuotaAndCollectBlocks uses list to collect > blocks > {code:java} > List allBlocks = new ArrayList(); > if (file.getBlocks() != null) { > allBlocks.addAll(Arrays.asList(file.getBlocks())); > }{code} > INodeFile#storagespaceConsumedContiguous collects all distinct blocks by set > {code:java} > // Collect all distinct blocks > Set allBlocks = new HashSet<>(Arrays.asList(getBlocks())); > DiffList diffs = sf.getDiffs().asList(); > for(FileDiff diff : diffs) { >BlockInfo[] diffBlocks = diff.getBlocks(); >if (diffBlocks != null) { > allBlocks.addAll(Arrays.asList(diffBlocks)); > } {code} > but on updating the reclaim context we subtract these both , so wrong quota > value can be updated > {code:java} > QuotaCounts current = file.storagespaceConsumed(bsp); > reclaimContext.quotaDelta().add(oldCounts.subtract(current)); {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-15362) FileWithSnapshotFeature#updateQuotaAndCollectBlocks should collect all distinct blocks
[ https://issues.apache.org/jira/browse/HDFS-15362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-15362: --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed Status: Resolved (was: Patch Available) > FileWithSnapshotFeature#updateQuotaAndCollectBlocks should collect all > distinct blocks > -- > > Key: HDFS-15362 > URL: https://issues.apache.org/jira/browse/HDFS-15362 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Major > Fix For: 3.4.0 > > Attachments: HDFS-15362.001.patch, HDFS-15362.002.patch > > > FileWithSnapshotFeature#updateQuotaAndCollectBlocks uses list to collect > blocks > {code:java} > List allBlocks = new ArrayList(); > if (file.getBlocks() != null) { > allBlocks.addAll(Arrays.asList(file.getBlocks())); > }{code} > INodeFile#storagespaceConsumedContiguous collects all distinct blocks by set > {code:java} > // Collect all distinct blocks > Set allBlocks = new HashSet<>(Arrays.asList(getBlocks())); > DiffList diffs = sf.getDiffs().asList(); > for(FileDiff diff : diffs) { >BlockInfo[] diffBlocks = diff.getBlocks(); >if (diffBlocks != null) { > allBlocks.addAll(Arrays.asList(diffBlocks)); > } {code} > but on updating the reclaim context we subtract these both , so wrong quota > value can be updated > {code:java} > QuotaCounts current = file.storagespaceConsumed(bsp); > reclaimContext.quotaDelta().add(oldCounts.subtract(current)); {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13274) RBF: Extend RouterRpcClient to use multiple sockets
[ https://issues.apache.org/jira/browse/HDFS-13274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17117924#comment-17117924 ] Íñigo Goiri commented on HDFS-13274: The stats overall look pretty high, the main issues I see: * GetServerDefaults: this goes to all the NNs, so a slow NN will hog this for longer. * RenewLease: Similar to get server defaults. * GetListing: What does your mount table look like? You really need to do some profiling on the Router to see what the connections are doing. I can't remember right away but there were JMX metrics (FederationRPC) to see how many connections were there in parallel and overall (FederationRPCMetrics). > RBF: Extend RouterRpcClient to use multiple sockets > --- > > Key: HDFS-13274 > URL: https://issues.apache.org/jira/browse/HDFS-13274 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > > HADOOP-13144 introduces the ability to create multiple connections for the > same user and use different sockets. The RouterRpcClient should use this > approach to get a better throughput. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15346) RBF: Balance data across federation namespaces with DistCp and snapshot diff / Step 2: The DistCpFedBalance.
[ https://issues.apache.org/jira/browse/HDFS-15346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17117500#comment-17117500 ] Hadoop QA commented on HDFS-15346: -- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 40s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green} No case conflicting files found. {color} | | {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m 0s{color} | {color:blue} Shelldocs was not available. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 3 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 17s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m 47s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 22m 5s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m 16s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m 40s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 17m 38s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 22s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 6m 45s{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 32s{color} | {color:blue} branch/hadoop-project no findbugs output file (findbugsXml.xml) {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 32s{color} | {color:blue} branch/hadoop-assemblies no findbugs output file (findbugsXml.xml) {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 33s{color} | {color:blue} branch/hadoop-tools/hadoop-tools-dist no findbugs output file (findbugsXml.xml) {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 33s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m 15s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 18m 15s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 54s{color} | {color:orange} root: The patch generated 16 new + 0 unchanged - 0 fixed = 16 total (was 0) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m 0s{color} | {color:green} There were no new shellcheck issues. {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 6s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 15m 54s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 39s{color} | {color:green} the patch passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 31s{color} | {color:blue} hadoop-project has no data from findbugs {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 31s{color} | {color:blue} hadoop-assemblies has no data from findbugs {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}