[jira] [Commented] (HDFS-12643) HDFS maintenance state behaviour is confusing and not well documented

2021-09-28 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-12643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17421453#comment-17421453 ] Kihwal Lee commented on HDFS-12643: --- Probably the missing information is that the cluster nodes need to

[jira] [Updated] (HDFS-16198) Short circuit read leaks Slot objects when InvalidToken exception is thrown

2021-09-15 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-16198: -- Fix Version/s: 2.10.2 > Short circuit read leaks Slot objects when InvalidToken exception is thrown >

[jira] [Commented] (HDFS-16198) Short circuit read leaks Slot objects when InvalidToken exception is thrown

2021-09-15 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17415718#comment-17415718 ] Kihwal Lee commented on HDFS-16198: --- Cherry-picked to branch-2.10. > Short circuit read leaks Slot

[jira] [Commented] (HDFS-16164) Configuration to allow group with read-all privilege

2021-08-11 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397658#comment-17397658 ] Kihwal Lee commented on HDFS-16164: --- Maybe you can accomplish what you want with ACL. > Configuration

[jira] [Commented] (HDFS-16134) Optimize RPC throughput

2021-07-21 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17384959#comment-17384959 ] Kihwal Lee commented on HDFS-16134: --- [~daryn] > Optimize RPC throughput > --- > >

[jira] [Updated] (HDFS-16127) Improper pipeline close recovery causes a permanent write failure or data loss.

2021-07-16 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-16127: -- Fix Version/s: 3.3.2 3.2.3 3.4.0 Hadoop Flags: Reviewed

[jira] [Commented] (HDFS-16127) Improper pipeline close recovery causes a permanent write failure or data loss.

2021-07-14 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17380634#comment-17380634 ] Kihwal Lee commented on HDFS-16127: --- There is no unit test in the patch. It isn't easy to instrument

[jira] [Updated] (HDFS-16127) Improper pipeline close recovery causes a permanent write failure or data loss.

2021-07-13 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-16127: -- Attachment: HDFS-16127.patch > Improper pipeline close recovery causes a permanent write failure or

[jira] [Updated] (HDFS-16127) Improper pipeline close recovery causes a permanent write failure or data loss.

2021-07-13 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-16127: -- Status: Patch Available (was: Open) > Improper pipeline close recovery causes a permanent write

[jira] [Assigned] (HDFS-16127) Improper pipeline close recovery causes a permanent write failure or data loss.

2021-07-13 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee reassigned HDFS-16127: - Assignee: Kihwal Lee > Improper pipeline close recovery causes a permanent write failure or

[jira] [Commented] (HDFS-16127) Improper pipeline close recovery causes a permanent write failure or data loss.

2021-07-13 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17379964#comment-17379964 ] Kihwal Lee commented on HDFS-16127: --- The proposed solution is to check the size of {{ackQueue}} when

[jira] [Commented] (HDFS-16127) Improper pipeline close recovery causes a permanent write failure or data loss.

2021-07-13 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17379961#comment-17379961 ] Kihwal Lee commented on HDFS-16127: --- *How the bug manifests* In DataStreamer thread's main loop, it

[jira] [Updated] (HDFS-16127) Improper pipeline close recovery causes a permanent write failure or data loss.

2021-07-13 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-16127: -- Description: When a block is being closed, the data streamer in the client waits for the final ACK to

[jira] [Created] (HDFS-16127) Improper pipeline close recovery causes a permanent write failure or data loss.

2021-07-13 Thread Kihwal Lee (Jira)
Kihwal Lee created HDFS-16127: - Summary: Improper pipeline close recovery causes a permanent write failure or data loss. Key: HDFS-16127 URL: https://issues.apache.org/jira/browse/HDFS-16127 Project:

[jira] [Resolved] (HDFS-16123) NameNode standby checkpoint is abnormal, because the data disk is full, even if the disk space is restored

2021-07-12 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-16123. --- Resolution: Invalid > NameNode standby checkpoint is abnormal, because the data disk is full, even

[jira] [Commented] (HDFS-16124) NameNode standby checkpoint is abnormal, error is No image directories available!

2021-07-12 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17379191#comment-17379191 ] Kihwal Lee commented on HDFS-16124: --- Namenodes do not automatically restore failed storage directories

[jira] [Commented] (HDFS-16117) Add file count info in audit log to record the file count for delete and getListing RPC request to assist user trouble shooting when RPC cost is increasing

2021-07-09 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17378127#comment-17378127 ] Kihwal Lee commented on HDFS-16117: --- Traditionally audit log format changes have been marked

[jira] [Updated] (HDFS-16117) Add file count info in audit log to record the file count for delete and getListing RPC request to assist user trouble shooting when RPC cost is increasing

2021-07-09 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-16117: -- Hadoop Flags: Incompatible change > Add file count info in audit log to record the file count for

[jira] [Comment Edited] (HDFS-13671) Namenode deletes large dir slowly caused by FoldedTreeSet#removeAndGet

2021-06-18 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-13671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1736#comment-1736 ] Kihwal Lee edited comment on HDFS-13671 at 6/18/21, 3:30 PM: - EOL does not

[jira] [Commented] (HDFS-13671) Namenode deletes large dir slowly caused by FoldedTreeSet#removeAndGet

2021-06-18 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-13671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1736#comment-1736 ] Kihwal Lee commented on HDFS-13671: --- EOL does not mean no more commits. you can still commit stuff.

[jira] [Updated] (HDFS-15618) Improve datanode shutdown latency

2021-06-16 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-15618: -- Resolution: Fixed Status: Resolved (was: Patch Available) I've committed this to branch-2.10.

[jira] [Updated] (HDFS-15618) Improve datanode shutdown latency

2021-06-16 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-15618: -- Fix Version/s: 2.10.2 > Improve datanode shutdown latency > - > >

[jira] [Commented] (HDFS-15618) Improve datanode shutdown latency

2021-06-16 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17364369#comment-17364369 ] Kihwal Lee commented on HDFS-15618: --- +1 The 2.10 patch looks good. Thanks, Ahmed for working on the

[jira] [Commented] (HDFS-15963) Unreleased volume references cause an infinite loop

2021-06-16 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17364299#comment-17364299 ] Kihwal Lee commented on HDFS-15963: --- I've looked at heap dumps and confirm the analysis by

[jira] [Commented] (HDFS-15963) Unreleased volume references cause an infinite loop

2021-06-15 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17364000#comment-17364000 ] Kihwal Lee commented on HDFS-15963: --- We hit this in 2.10 recently. I've cherry-picked it to branch-2.10

[jira] [Updated] (HDFS-15963) Unreleased volume references cause an infinite loop

2021-06-15 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-15963: -- Fix Version/s: 2.10.2 > Unreleased volume references cause an infinite loop >

[jira] [Commented] (HDFS-13671) Namenode deletes large dir slowly caused by FoldedTreeSet#removeAndGet

2021-06-10 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-13671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17361135#comment-17361135 ] Kihwal Lee commented on HDFS-13671: --- Thanks for sharing the test result, [~huanghaibin]. I am still

[jira] [Commented] (HDFS-16042) DatanodeAdminMonitor scan should be delay based

2021-06-03 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17356692#comment-17356692 ] Kihwal Lee commented on HDFS-16042: --- It is not related to any incident. Since the default interval is

[jira] [Resolved] (HDFS-16034) Disk failures exceeding the DFIP threshold do not shutdown datanode

2021-05-21 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-16034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-16034. --- Resolution: Incomplete > Disk failures exceeding the DFIP threshold do not shutdown datanode >

[jira] [Created] (HDFS-16034) Disk failures exceeding the DFIP threshold do not shutdown datanode

2021-05-21 Thread Kihwal Lee (Jira)
Kihwal Lee created HDFS-16034: - Summary: Disk failures exceeding the DFIP threshold do not shutdown datanode Key: HDFS-16034 URL: https://issues.apache.org/jira/browse/HDFS-16034 Project: Hadoop HDFS

[jira] [Commented] (HDFS-8224) Schedule a block for scanning if its metadata file is corrupt

2021-04-06 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-8224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17315771#comment-17315771 ] Kihwal Lee commented on HDFS-8224: -- If you are sure the data block is not corrupt, you can use {{hdfs

[jira] [Comment Edited] (HDFS-15901) Solve the problem of DN repeated block reports occupying too many RPCs during Safemode

2021-03-19 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17304933#comment-17304933 ] Kihwal Lee edited comment on HDFS-15901 at 3/19/21, 2:18 PM: - [~jianghuazhu],

[jira] [Commented] (HDFS-15901) Solve the problem of DN repeated block reports occupying too many RPCs during Safemode

2021-03-19 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17304933#comment-17304933 ] Kihwal Lee commented on HDFS-15901: --- [~jianghuazhu], when it happens again, take a look at the

[jira] [Commented] (HDFS-15901) Solve the problem of DN repeated block reports occupying too many RPCs during Safemode

2021-03-19 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17304928#comment-17304928 ] Kihwal Lee commented on HDFS-15901: --- bq. look forward to your new block report flow control system, Do

[jira] [Commented] (HDFS-15868) Possible Resource Leak in EditLogFileOutputStream

2021-03-19 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17304911#comment-17304911 ] Kihwal Lee commented on HDFS-15868: --- It is a good change. It was probably neglected during the design

[jira] [Comment Edited] (HDFS-15901) Solve the problem of DN repeated block reports occupying too many RPCs during Safemode

2021-03-18 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17304503#comment-17304503 ] Kihwal Lee edited comment on HDFS-15901 at 3/18/21, 10:32 PM: -- The block

[jira] [Commented] (HDFS-15901) Solve the problem of DN repeated block reports occupying too many RPCs during Safemode

2021-03-18 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17304503#comment-17304503 ] Kihwal Lee commented on HDFS-15901: --- The block report lease feature is supposed to improve this, but it

[jira] [Commented] (HDFS-15837) Incorrect bytes causing block corruption after update pipeline and recovery failure

2021-02-17 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17286111#comment-17286111 ] Kihwal Lee commented on HDFS-15837: --- The successful pipeline update with 3072 bytes was at 03:42:40.

[jira] [Commented] (HDFS-15837) Incorrect bytes causing block corruption after update pipeline and recovery failure

2021-02-16 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17285448#comment-17285448 ] Kihwal Lee commented on HDFS-15837: --- What is the version of Hadoop you are using? We've seen similar

[jira] [Resolved] (HDFS-15825) Using a cryptographically weak Pseudo Random Number Generator (PRNG)

2021-02-08 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-15825. --- Resolution: Invalid > Using a cryptographically weak Pseudo Random Number Generator (PRNG) >

[jira] [Commented] (HDFS-15825) Using a cryptographically weak Pseudo Random Number Generator (PRNG)

2021-02-08 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17281142#comment-17281142 ] Kihwal Lee commented on HDFS-15825: --- [~Vicky Zhang], if you want to scan the code base, please do not

[jira] [Resolved] (HDFS-15824) Update to enable TLS >=1.2 as default secure protocols

2021-02-08 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-15824. --- Resolution: Invalid > Update to enable TLS >=1.2 as default secure protocols >

[jira] [Commented] (HDFS-15824) Update to enable TLS >=1.2 as default secure protocols

2021-02-08 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17281138#comment-17281138 ] Kihwal Lee commented on HDFS-15824: --- [~Vicky Zhang]  This is from an old branch that has been EOL'ed

[jira] [Commented] (HDFS-15822) Client retry mechanism may invalid when use hedgedRead

2021-02-05 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17279746#comment-17279746 ] Kihwal Lee commented on HDFS-15822: --- Hedged read has been known to be buggy. When there is an exception

[jira] [Commented] (HDFS-15813) DataStreamer: keep sending heartbeat packets while streaming

2021-02-04 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17279230#comment-17279230 ] Kihwal Lee commented on HDFS-15813: --- +1. Unit test failures seem unrelated.  If you can't find existing

[jira] [Comment Edited] (HDFS-15813) DataStreamer: keep sending heartbeat packets while streaming

2021-02-03 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17278224#comment-17278224 ] Kihwal Lee edited comment on HDFS-15813 at 2/3/21, 5:19 PM: As for the safety

[jira] [Updated] (HDFS-12763) DataStreamer should heartbeat during flush

2021-02-03 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-12763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-12763: -- Resolution: Duplicate Status: Resolved (was: Patch Available) > DataStreamer should heartbeat

[jira] [Commented] (HDFS-15813) DataStreamer: keep sending heartbeat packets while streaming

2021-02-03 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17278224#comment-17278224 ] Kihwal Lee commented on HDFS-15813: --- As for the safety of the change, I feel confident since this has

[jira] [Updated] (HDFS-15799) Make DisallowedDatanodeException terse

2021-02-03 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-15799: -- Fix Version/s: 3.2.3 3.4.0 3.3.1 Hadoop Flags: Reviewed

[jira] [Commented] (HDFS-15799) Make DisallowedDatanodeException terse

2021-02-03 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17278040#comment-17278040 ] Kihwal Lee commented on HDFS-15799: --- +1 lgtm. The failed tests don't seem related to the patch. > Make

[jira] [Commented] (HDFS-15789) Lease renewal does not require namesystem lock

2021-01-27 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17273129#comment-17273129 ] Kihwal Lee commented on HDFS-15789: --- This is a safe change. The FSN lock is only protecting NN

[jira] [Commented] (HDFS-10498) Intermittent test failure org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength.testSnapshotfileLength

2021-01-27 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-10498?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17273123#comment-17273123 ] Kihwal Lee commented on HDFS-10498: --- +1 > Intermittent test failure >

[jira] [Comment Edited] (HDFS-15794) IBR and FBR use different queues to load data.

2021-01-27 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272999#comment-17272999 ] Kihwal Lee edited comment on HDFS-15794 at 1/27/21, 4:58 PM: - {quote}The

[jira] [Commented] (HDFS-15794) IBR and FBR use different queues to load data.

2021-01-27 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272999#comment-17272999 ] Kihwal Lee commented on HDFS-15794: --- {quote}The problem here is that when the NameNode is blocked in

[jira] [Commented] (HDFS-15755) Add option to set retry times of checking failure volume and Import powermock for mock static method

2021-01-05 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17259042#comment-17259042 ] Kihwal Lee commented on HDFS-15755: --- I tried to add powermock a long time ago in HADOOP-7537 and

[jira] [Commented] (HDFS-15725) Lease Recovery never completes for a committed block which the DNs never finalize

2020-12-11 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247982#comment-17247982 ] Kihwal Lee commented on HDFS-15725: --- +1 > Lease Recovery never completes for a committed block which

[jira] [Commented] (HDFS-15727) RpcQueueTimeAvgTime of the NameNode increases after it becomes StandBy

2020-12-11 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247977#comment-17247977 ] Kihwal Lee commented on HDFS-15727: --- Standby namenode replays edits with the name system write lock

[jira] [Assigned] (HDFS-15726) Client should only complete a file if the last block is finalized

2020-12-10 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee reassigned HDFS-15726: - Assignee: Ahmed Hussein > Client should only complete a file if the last block is finalized >

[jira] [Created] (HDFS-15726) Client should only complete a file if the last block is finalized

2020-12-10 Thread Kihwal Lee (Jira)
Kihwal Lee created HDFS-15726: - Summary: Client should only complete a file if the last block is finalized Key: HDFS-15726 URL: https://issues.apache.org/jira/browse/HDFS-15726 Project: Hadoop HDFS

[jira] [Commented] (HDFS-15725) Lease Recovery never completes for a committed block which the DNs never finalize

2020-12-10 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247478#comment-17247478 ] Kihwal Lee commented on HDFS-15725: --- We have seen certain version of clients closing without completely

[jira] [Comment Edited] (HDFS-15725) Lease Recovery never completes for a committed block which the DNs never finalize

2020-12-10 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247455#comment-17247455 ] Kihwal Lee edited comment on HDFS-15725 at 12/10/20, 7:43 PM: -- We have seen

[jira] [Comment Edited] (HDFS-15725) Lease Recovery never completes for a committed block which the DNs never finalize

2020-12-10 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247455#comment-17247455 ] Kihwal Lee edited comment on HDFS-15725 at 12/10/20, 7:42 PM: -- We have seen

[jira] [Commented] (HDFS-15725) Lease Recovery never completes for a committed block which the DNs never finalize

2020-12-10 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17247455#comment-17247455 ] Kihwal Lee commented on HDFS-15725: --- We have seen this more with later versions of clients. The

[jira] [Updated] (HDFS-15725) Lease Recovery never completes for a committed block which the DNs never finalize

2020-12-10 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-15725: -- Attachment: lease_recovery_2_10.patch > Lease Recovery never completes for a committed block which the

[jira] [Updated] (HDFS-15622) Deleted blocks linger in the replications queue

2020-10-22 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-15622: -- Fix Version/s: 3.2.3 3.1.5 3.3.1 Hadoop Flags: Reviewed

[jira] [Updated] (HDFS-15622) Deleted blocks linger in the replications queue

2020-10-22 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-15622: -- Fix Version/s: 3.4.0 > Deleted blocks linger in the replications queue >

[jira] [Commented] (HDFS-15622) Deleted blocks linger in the replications queue

2020-10-22 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17219360#comment-17219360 ] Kihwal Lee commented on HDFS-15622: --- +1 on version 2 of the patch. > Deleted blocks linger in the

[jira] [Updated] (HDFS-15618) Improve datanode shutdown latency

2020-10-22 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-15618: -- Hadoop Flags: Reviewed Resolution: Fixed Status: Resolved (was: Patch Available) I've

[jira] [Updated] (HDFS-15618) Improve datanode shutdown latency

2020-10-22 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-15618: -- Fix Version/s: 3.2.3 3.1.5 > Improve datanode shutdown latency >

[jira] [Updated] (HDFS-15618) Improve datanode shutdown latency

2020-10-22 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-15618: -- Fix Version/s: 3.3.1 > Improve datanode shutdown latency > - > >

[jira] [Commented] (HDFS-15618) Improve datanode shutdown latency

2020-10-21 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17218127#comment-17218127 ] Kihwal Lee commented on HDFS-15618: --- I've committed this to trunk, but the patch does not work in

[jira] [Updated] (HDFS-15618) Improve datanode shutdown latency

2020-10-21 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-15618: -- Fix Version/s: 3.4.0 > Improve datanode shutdown latency > - > >

[jira] [Commented] (HDFS-15618) Improve datanode shutdown latency

2020-10-20 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17218120#comment-17218120 ] Kihwal Lee commented on HDFS-15618: --- +1 the version 4 of the patch looks good. > Improve datanode

[jira] [Commented] (HDFS-15622) Deleted blocks linger in the replications queue

2020-10-20 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17218087#comment-17218087 ] Kihwal Lee commented on HDFS-15622: --- The patch looks good. the only minor nit is the name of one of the

[jira] [Comment Edited] (HDFS-14941) Potential editlog race condition can cause corrupted file

2020-10-19 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-14941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17140608#comment-17140608 ] Kihwal Lee edited comment on HDFS-14941 at 10/19/20, 9:17 PM: -- Filed

[jira] [Commented] (HDFS-15628) HttpFS server throws NPE if a file is a symlink

2020-10-16 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17215497#comment-17215497 ] Kihwal Lee commented on HDFS-15628: --- I've committed this to trunk, branch-3.3, branch-3.2 and

[jira] [Updated] (HDFS-15628) HttpFS server throws NPE if a file is a symlink

2020-10-16 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-15628: -- Fix Version/s: 3.1.5 > HttpFS server throws NPE if a file is a symlink >

[jira] [Updated] (HDFS-15627) Audit log deletes before collecting blocks

2020-10-16 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-15627: -- Fix Version/s: 3.2.3 3.1.5 3.4.0 3.3.1

[jira] [Updated] (HDFS-15627) Audit log deletes before collecting blocks

2020-10-16 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-15627: -- Summary: Audit log deletes before collecting blocks (was: Audit log deletes after edit is written) >

[jira] [Commented] (HDFS-15627) Audit log deletes after edit is written

2020-10-16 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17215476#comment-17215476 ] Kihwal Lee commented on HDFS-15627: --- +1 lgtm > Audit log deletes after edit is written >

[jira] [Commented] (HDFS-15618) Improve datanode shutdown latency

2020-10-16 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17215454#comment-17215454 ] Kihwal Lee commented on HDFS-15618: --- It can be as fancy as adding a builder method for setting it (for

[jira] [Commented] (HDFS-15618) Improve datanode shutdown latency

2020-10-16 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17215452#comment-17215452 ] Kihwal Lee commented on HDFS-15618: --- In production, there is no harm in exiting after waiting for 5

[jira] [Updated] (HDFS-15628) HttpFS server throws NPE if a file is a symlink

2020-10-14 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-15628: -- Fix Version/s: 3.2.3 > HttpFS server throws NPE if a file is a symlink >

[jira] [Updated] (HDFS-15628) HttpFS server throws NPE if a file is a symlink

2020-10-14 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-15628: -- Fix Version/s: 3.4.0 3.3.1 Hadoop Flags: Reviewed Resolution: Fixed

[jira] [Updated] (HDFS-15628) HttpFS server throws NPE if a file is a symlink

2020-10-14 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-15628: -- Summary: HttpFS server throws NPE if a file is a symlink (was: https throws NPE if a file is a

[jira] [Commented] (HDFS-15628) https throws NPE if a file is a symlink

2020-10-14 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17214306#comment-17214306 ] Kihwal Lee commented on HDFS-15628: --- +1 LGTM. We've been running the same server-side change for over 2

[jira] [Commented] (HDFS-15618) Improve datanode shutdown latency

2020-10-14 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17214271#comment-17214271 ] Kihwal Lee commented on HDFS-15618: --- The patch looks okay. Although it is better than the previous

[jira] [Commented] (HDFS-15581) Access Controlled HTTPFS Proxy

2020-09-22 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200188#comment-17200188 ] Kihwal Lee commented on HDFS-15581: --- I've committed this to trunk, branch-3.3 and branch-3.2. Thanks

[jira] [Updated] (HDFS-15581) Access Controlled HTTPFS Proxy

2020-09-22 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-15581: -- Fix Version/s: 3.4.0 3.3.1 3.2.2 Hadoop Flags: Reviewed

[jira] [Commented] (HDFS-15581) Access Controlled HTTPFS Proxy

2020-09-22 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17200135#comment-17200135 ] Kihwal Lee commented on HDFS-15581: --- +1 The patch looks good. The documentation in

[jira] [Commented] (HDFS-15581) Access Controlled HTTPFS Proxy

2020-09-16 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197319#comment-17197319 ] Kihwal Lee commented on HDFS-15581: --- [~sodonnell], [~richard-ross] has already posted a patch.  Are you

[jira] [Assigned] (HDFS-15581) Access Controlled HTTPFS Proxy

2020-09-16 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee reassigned HDFS-15581: - Assignee: Richard > Access Controlled HTTPFS Proxy > -- > >

[jira] [Moved] (HDFS-15581) Access Controlled HTTPFS Proxy

2020-09-16 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee moved HADOOP-17264 to HDFS-15581: Component/s: (was: httpfs) httpfs

[jira] [Commented] (HDFS-15553) Improve NameNode RPC throughput with ReadWriteRpcCallQueue

2020-09-01 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17188580#comment-17188580 ] Kihwal Lee commented on HDFS-15553: --- It is fine to reorder user requests in general. As [~suxingfate]

[jira] [Commented] (HDFS-15474) HttpFS: WebHdfsFileSystem cannot renew an expired delegation token from HttpFS response

2020-07-17 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17159955#comment-17159955 ] Kihwal Lee commented on HDFS-15474: --- I believe the token ops are not implemented by HttpFS. They are

[jira] [Commented] (HDFS-15421) IBR leak causes standby NN to be stuck in safe mode

2020-06-24 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17143880#comment-17143880 ] Kihwal Lee commented on HDFS-15421: --- Patch 004 looks good to me. +1. > IBR leak causes standby NN to

[jira] [Commented] (HDFS-15421) IBR leak causes standby NN to be stuck in safe mode

2020-06-23 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17143331#comment-17143331 ] Kihwal Lee commented on HDFS-15421: --- Thanks, [~aajisaka] for the patch. I will also have a look soon.

[jira] [Updated] (HDFS-15422) Reported IBR is partially replaced with stored info when queuing.

2020-06-19 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-15422: -- Description: When queueing an IBR (incremental block report) on a standby namenode, some of the

[jira] [Commented] (HDFS-15422) Reported IBR is partially replaced with stored info when queuing.

2020-06-19 Thread Kihwal Lee (Jira)
[ https://issues.apache.org/jira/browse/HDFS-15422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17140618#comment-17140618 ] Kihwal Lee commented on HDFS-15422: --- The fix is simple. {code} @@ -2578,10 +2578,7 @@ private

[jira] [Created] (HDFS-15422) Reported IBR is partially replaced with stored info when queuing.

2020-06-19 Thread Kihwal Lee (Jira)
Kihwal Lee created HDFS-15422: - Summary: Reported IBR is partially replaced with stored info when queuing. Key: HDFS-15422 URL: https://issues.apache.org/jira/browse/HDFS-15422 Project: Hadoop HDFS

  1   2   3   4   5   6   7   8   9   10   >