[jira] [Updated] (HDFS-9608) Disk IO imbalance in HDFS with heterogeneous storages

[ https://issues.apache.org/jira/browse/HDFS-9608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zhou updated HDFS-9608: --- Attachment: HDFS-9608.01.patch This patch insure choosing volumes in RR mode for each storage type. Besides,

[jira] [Commented] (HDFS-8430) Erasure coding: update DFSClient.getFileChecksum() logic for stripe files

[ https://issues.apache.org/jira/browse/HDFS-8430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15080667#comment-15080667 ] Kai Zheng commented on HDFS-8430: - Thanks Nicholas for the correction. Yeah I misunderstood. It's smart to

[jira] [Commented] (HDFS-8767) RawLocalFileSystem.listStatus() returns null for UNIX pipefile

[ https://issues.apache.org/jira/browse/HDFS-8767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15080554#comment-15080554 ] Junping Du commented on HDFS-8767: -- Hi [~wheat9] and [~cnauroth], does this fix should land on branch-2.6

[jira] [Updated] (HDFS-8767) RawLocalFileSystem.listStatus() returns null for UNIX pipefile

[ https://issues.apache.org/jira/browse/HDFS-8767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-8767: - Target Version/s: 2.7.2, 2.6.4 > RawLocalFileSystem.listStatus() returns null for UNIX pipefile >

[jira] [Commented] (HDFS-6945) BlockManager should remove a block from excessReplicateMap and decrement ExcessBlocks metric when the block is removed

[ https://issues.apache.org/jira/browse/HDFS-6945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15080567#comment-15080567 ] Junping Du commented on HDFS-6945: -- I have cherry-pick the patch to branch-2.6. > BlockManager should

[jira] [Created] (HDFS-9608) Disk IO imbalance in HDFS with heterogeneous storages

Wei Zhou created HDFS-9608: -- Summary: Disk IO imbalance in HDFS with heterogeneous storages Key: HDFS-9608 URL: https://issues.apache.org/jira/browse/HDFS-9608 Project: Hadoop HDFS Issue Type: Bug

[jira] [Updated] (HDFS-9445) Datanode may deadlock while handling a bad volume

[ https://issues.apache.org/jira/browse/HDFS-9445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Walter Su updated HDFS-9445: Attachment: HDFS-9445-branch-2.6_02.patch yes. Uploaded HDFS-9445-branch-2.6_02.patch > Datanode may

[jira] [Updated] (HDFS-6945) BlockManager should remove a block from excessReplicateMap and decrement ExcessBlocks metric when the block is removed

[ https://issues.apache.org/jira/browse/HDFS-6945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-6945: - Fix Version/s: 2.6.4 > BlockManager should remove a block from excessReplicateMap and decrement >

[jira] [Updated] (HDFS-6945) BlockManager should remove a block from excessReplicateMap and decrement ExcessBlocks metric when the block is removed

[ https://issues.apache.org/jira/browse/HDFS-6945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-6945: - Target Version/s: 2.8.0, 2.6.4 (was: 2.8.0) > BlockManager should remove a block from excessReplicateMap

[jira] [Commented] (HDFS-9294) DFSClient deadlock when close file and failed to renew lease

[ https://issues.apache.org/jira/browse/HDFS-9294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15080582#comment-15080582 ] Junping Du commented on HDFS-9294: -- Hi [~szetszwo] and [~brahmareddy], shall we cherry-pick this fix to

[jira] [Updated] (HDFS-9294) DFSClient deadlock when close file and failed to renew lease

[ https://issues.apache.org/jira/browse/HDFS-9294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-9294: - Target Version/s: 2.7.2, 2.6.4 (was: 2.7.2) > DFSClient deadlock when close file and failed to renew

[jira] [Commented] (HDFS-8430) Erasure coding: update DFSClient.getFileChecksum() logic for stripe files

[ https://issues.apache.org/jira/browse/HDFS-8430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15080604#comment-15080604 ] Kai Zheng commented on HDFS-8430: - Thanks [~szetszwo] for the ideas! It looks like it's acceptable the file

[jira] [Updated] (HDFS-8676) Delayed rolling upgrade finalization can cause heartbeat expiration and write failures

[ https://issues.apache.org/jira/browse/HDFS-8676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-8676: - Target Version/s: 2.7.2, 2.6.4 (was: 2.7.2) > Delayed rolling upgrade finalization can cause heartbeat

[jira] [Commented] (HDFS-8676) Delayed rolling upgrade finalization can cause heartbeat expiration and write failures

[ https://issues.apache.org/jira/browse/HDFS-8676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15080557#comment-15080557 ] Junping Du commented on HDFS-8676: -- Hi [~walter.k.su] and [~kihwal], I tried to cherry-pick this patch to

[jira] [Commented] (HDFS-7645) Rolling upgrade is restoring blocks from trash multiple times

[ https://issues.apache.org/jira/browse/HDFS-7645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15080559#comment-15080559 ] Junping Du commented on HDFS-7645: -- Hi [~kihwal], do you suggest we should backport these three fixes

[jira] [Updated] (HDFS-8891) HDFS concat should keep srcs order

[ https://issues.apache.org/jira/browse/HDFS-8891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-8891: - Target Version/s: 2.6.4 > HDFS concat should keep srcs order > -- > >

[jira] [Commented] (HDFS-8891) HDFS concat should keep srcs order

[ https://issues.apache.org/jira/browse/HDFS-8891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15080579#comment-15080579 ] Junping Du commented on HDFS-8891: -- Hi [~chris.douglas] and [~jingzhao], shall we cherry-pick this fix to

[jira] [Commented] (HDFS-9294) DFSClient deadlock when close file and failed to renew lease

[ https://issues.apache.org/jira/browse/HDFS-9294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15080594#comment-15080594 ] Tsz Wo Nicholas Sze commented on HDFS-9294: --- +1 we should as well cherry-pick this to branch-2.6.

[jira] [Commented] (HDFS-8430) Erasure coding: update DFSClient.getFileChecksum() logic for stripe files

[ https://issues.apache.org/jira/browse/HDFS-8430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15080608#comment-15080608 ] Kai Zheng commented on HDFS-8430: - So given above approach is used, the involved change would be much

[jira] [Commented] (HDFS-9178) Slow datanode I/O can cause a wrong node to be marked bad

[ https://issues.apache.org/jira/browse/HDFS-9178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15080547#comment-15080547 ] Junping Du commented on HDFS-9178: -- Hi [~kihwal], I saw you already attached the patch for 2.6 branch.

[jira] [Updated] (HDFS-9178) Slow datanode I/O can cause a wrong node to be marked bad

[ https://issues.apache.org/jira/browse/HDFS-9178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-9178: - Target Version/s: 2.7.2, 2.6.4 (was: 2.7.2) > Slow datanode I/O can cause a wrong node to be marked bad >

[jira] [Commented] (HDFS-9607) Advance Hadoop Architecture (AHA) - HDFS

[ https://issues.apache.org/jira/browse/HDFS-9607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15080575#comment-15080575 ] Dinesh S. Atreya commented on HDFS-9607: Linking to the parent/umbrella JIRA. > Advance Hadoop

[jira] [Commented] (HDFS-4660) Block corruption can happen during pipeline recovery

[ https://issues.apache.org/jira/browse/HDFS-4660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15080542#comment-15080542 ] Junping Du commented on HDFS-4660: -- Hi [~kihwal], shall we backport this patch to 2.6.x branch? > Block

[jira] [Updated] (HDFS-4660) Block corruption can happen during pipeline recovery

[ https://issues.apache.org/jira/browse/HDFS-4660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-4660: - Target Version/s: 2.7.1, 2.6.4 (was: 2.7.1) > Block corruption can happen during pipeline recovery >

[jira] [Updated] (HDFS-8722) Optimize datanode writes for small writes and flushes

[ https://issues.apache.org/jira/browse/HDFS-8722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-8722: - Target Version/s: 2.7.2, 2.6.4 (was: 2.7.2) > Optimize datanode writes for small writes and flushes >

[jira] [Updated] (HDFS-8995) Flaw in registration bookeeping can make DN die on reconnect

[ https://issues.apache.org/jira/browse/HDFS-8995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-8995: - Target Version/s: 2.7.2, 2.6.4 (was: 2.7.2) > Flaw in registration bookeeping can make DN die on

[jira] [Commented] (HDFS-8995) Flaw in registration bookeeping can make DN die on reconnect

[ https://issues.apache.org/jira/browse/HDFS-8995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15080553#comment-15080553 ] Junping Du commented on HDFS-8995: -- Hi [~hitliuyi] and [~kihwal], as Sangjin's comments earlier, shall we

[jira] [Commented] (HDFS-8430) Erasure coding: update DFSClient.getFileChecksum() logic for stripe files

[ https://issues.apache.org/jira/browse/HDFS-8430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15080639#comment-15080639 ] Tsz Wo Nicholas Sze commented on HDFS-8430: --- > ... It looks like it's acceptable the file

[jira] [Updated] (HDFS-9106) Transfer failure during pipeline recovery causes permanent write failures

[ https://issues.apache.org/jira/browse/HDFS-9106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-9106: - Target Version/s: 2.8.0, 2.6.4 (was: 2.8.0) > Transfer failure during pipeline recovery causes permanent

[jira] [Commented] (HDFS-8722) Optimize datanode writes for small writes and flushes

[ https://issues.apache.org/jira/browse/HDFS-8722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15080543#comment-15080543 ] Junping Du commented on HDFS-8722: -- Hi [~kihwal], shall we backport the fix to 2.6 branch also? >

[jira] [Commented] (HDFS-9106) Transfer failure during pipeline recovery causes permanent write failures

[ https://issues.apache.org/jira/browse/HDFS-9106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15080549#comment-15080549 ] Junping Du commented on HDFS-9106: -- Hi [~hitliuyi], [~jingzhao] and [~kihwal], do we think this bug should

[jira] [Commented] (HDFS-9220) Reading small file (< 512 bytes) that is open for append fails due to incorrect checksum

[ https://issues.apache.org/jira/browse/HDFS-9220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15080570#comment-15080570 ] Junping Du commented on HDFS-9220: -- Hi [~jingzhao] and [~kihwal], shall we cherry-pick the fix to

[jira] [Commented] (HDFS-9607) Advance Hadoop Architecture (AHA) - HDFS

[ https://issues.apache.org/jira/browse/HDFS-9607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15080571#comment-15080571 ] Dinesh S. Atreya commented on HDFS-9607: Please choose which of the following API signatures are

[jira] [Updated] (HDFS-9220) Reading small file (< 512 bytes) that is open for append fails due to incorrect checksum

[ https://issues.apache.org/jira/browse/HDFS-9220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-9220: - Target Version/s: 2.7.2, 2.6.4 (was: 2.7.2) > Reading small file (< 512 bytes) that is open for append

[jira] [Updated] (HDFS-9445) Datanode may deadlock while handling a bad volume

[ https://issues.apache.org/jira/browse/HDFS-9445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HDFS-9445: - Target Version/s: 2.7.2, 2.6.4 (was: 2.7.2) > Datanode may deadlock while handling a bad volume >

[jira] [Commented] (HDFS-9445) Datanode may deadlock while handling a bad volume

[ https://issues.apache.org/jira/browse/HDFS-9445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15080586#comment-15080586 ] Junping Du commented on HDFS-9445: -- Hi [~walter.k.su], [~vinayrpet] and [~kihwal], does the same issue

[jira] [Updated] (HDFS-9445) Datanode may deadlock while handling a bad volume

[ https://issues.apache.org/jira/browse/HDFS-9445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Walter Su updated HDFS-9445: Status: Patch Available (was: Reopened) > Datanode may deadlock while handling a bad volume >

[jira] [Reopened] (HDFS-9445) Datanode may deadlock while handling a bad volume

[ https://issues.apache.org/jira/browse/HDFS-9445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Walter Su reopened HDFS-9445: - > Datanode may deadlock while handling a bad volume > - > >

[jira] [Commented] (HDFS-9445) Datanode may deadlock while handling a bad volume

[ https://issues.apache.org/jira/browse/HDFS-9445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15080679#comment-15080679 ] Hadoop QA commented on HDFS-9445: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem ||

[jira] [Commented] (HDFS-9607) Advance Hadoop Architecture (AHA) - HDFS

[ https://issues.apache.org/jira/browse/HDFS-9607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15080678#comment-15080678 ] Dinesh S. Atreya commented on HDFS-9607: A few checks that should be implemented by the method

[jira] [Commented] (HDFS-9445) Datanode may deadlock while handling a bad volume

[ https://issues.apache.org/jira/browse/HDFS-9445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15080680#comment-15080680 ] Hadoop QA commented on HDFS-9445: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem ||

[jira] [Commented] (HDFS-8430) Erasure coding: update DFSClient.getFileChecksum() logic for stripe files

[ https://issues.apache.org/jira/browse/HDFS-8430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15080672#comment-15080672 ] Walter Su commented on HDFS-8430: - bq. 1. Use CRC64 (or some other linear code) for block checksum instead

[jira] [Commented] (HDFS-9376) TestSeveralNameNodes fails occasionally

[ https://issues.apache.org/jira/browse/HDFS-9376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15080780#comment-15080780 ] Masatake Iwasaki commented on HDFS-9376: Thanks, [~cnauroth]. > TestSeveralNameNodes fails

[jira] [Updated] (HDFS-9601) NNThroughputBenchmark.BlockReportStats should handle NotReplicatedYetException on adding block

[ https://issues.apache.org/jira/browse/HDFS-9601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki updated HDFS-9601: --- Attachment: HDFS-9601.002.patch I attached 002. * simplified retrying * added comment * got

[jira] [Commented] (HDFS-9601) NNThroughputBenchmark.BlockReportStats should handle NotReplicatedYetException on adding block

[ https://issues.apache.org/jira/browse/HDFS-9601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15080778#comment-15080778 ] Masatake Iwasaki commented on HDFS-9601: Thanks for the comment, [~liuml07]. As you say, we can

[jira] [Commented] (HDFS-9607) Advance Hadoop Architecture (AHA) - HDFS

[ https://issues.apache.org/jira/browse/HDFS-9607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15080683#comment-15080683 ] Dinesh S. Atreya commented on HDFS-9607: Personal preference regarding API naming. * We can prefer

[jira] [Commented] (HDFS-8913) Documentation clarity regarding Secondary node, Checkpoint node & Backup node

[ https://issues.apache.org/jira/browse/HDFS-8913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15080782#comment-15080782 ] Jeff Zhang commented on HDFS-8913: -- +1, I think we do need to highlight the differences between these

[jira] [Created] (HDFS-9606) Unknown error encountered while tailing edits. Shutting down standby NN.

tangshangwen created HDFS-9606: -- Summary: Unknown error encountered while tailing edits. Shutting down standby NN. Key: HDFS-9606 URL: https://issues.apache.org/jira/browse/HDFS-9606 Project: Hadoop

[jira] [Commented] (HDFS-9607) Advance Hadoop Architecture (AHA) - HDFS

[ https://issues.apache.org/jira/browse/HDFS-9607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15080493#comment-15080493 ] Dinesh S. Atreya commented on HDFS-9607: This write feature proposal is a very restricted version

[jira] [Commented] (HDFS-9607) Advance Hadoop Architecture (AHA) - HDFS

[ https://issues.apache.org/jira/browse/HDFS-9607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15080498#comment-15080498 ] Dinesh S. Atreya commented on HDFS-9607: Can somebody delete the above comment and this one, since

[jira] [Commented] (HDFS-9607) Advance Hadoop Architecture (AHA) - HDFS

[ https://issues.apache.org/jira/browse/HDFS-9607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15080500#comment-15080500 ] Dinesh S. Atreya commented on HDFS-9607: This write feature proposal is a very restricted version

[jira] [Commented] (HDFS-8430) Erasure coding: update DFSClient.getFileChecksum() logic for stripe files

[ https://issues.apache.org/jira/browse/HDFS-8430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15080487#comment-15080487 ] Tsz Wo Nicholas Sze commented on HDFS-8430: --- [~drankye], I think it is a good start for the first

[jira] [Created] (HDFS-9607) Advance Hadoop Architecture (AHA) - HDFS

Dinesh S. Atreya created HDFS-9607: -- Summary: Advance Hadoop Architecture (AHA) - HDFS Key: HDFS-9607 URL: https://issues.apache.org/jira/browse/HDFS-9607 Project: Hadoop HDFS Issue Type: