[jira] [Created] (HDFS-6599) 2.4 addBlock is 10 to 20 times slower compared to 0.23

2014-06-24 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-6599: Summary: 2.4 addBlock is 10 to 20 times slower compared to 0.23 Key: HDFS-6599 URL: https://issues.apache.org/jira/browse/HDFS-6599 Project: Hadoop HDFS Issue Type:

[jira] [Updated] (HDFS-6599) 2.4 addBlock is 10 to 20 times slower compared to 0.23

2014-06-24 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6599: - Affects Version/s: 2.3.0 2.4 addBlock is 10 to 20 times slower compared to 0.23

[jira] [Commented] (HDFS-6599) 2.4 addBlock is 10 to 20 times slower compared to 0.23

2014-06-24 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14042481#comment-14042481 ] Kihwal Lee commented on HDFS-6599: -- This is caused by a change in

[jira] [Updated] (HDFS-6599) 2.4 addBlock is 10 to 20 times slower compared to 0.23

2014-06-24 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6599: - Priority: Blocker (was: Critical) 2.4 addBlock is 10 to 20 times slower compared to 0.23

[jira] [Created] (HDFS-6601) Issues in finalizing rolling upgrade when there is a layout version change

2014-06-24 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-6601: Summary: Issues in finalizing rolling upgrade when there is a layout version change Key: HDFS-6601 URL: https://issues.apache.org/jira/browse/HDFS-6601 Project: Hadoop HDFS

[jira] [Updated] (HDFS-6601) Issues in finalizing rolling upgrade when there is a layout version change

2014-06-24 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6601: - Status: Patch Available (was: Open) Issues in finalizing rolling upgrade when there is a layout version

[jira] [Updated] (HDFS-6601) Issues in finalizing rolling upgrade when there is a layout version change

2014-06-24 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6601: - Attachment: HDFS-6601.patch Issues in finalizing rolling upgrade when there is a layout version change

[jira] [Updated] (HDFS-6601) Issues in finalizing rolling upgrade when there is a layout version change

2014-06-24 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6601: - Target Version/s: 2.5.0 Affects Version/s: 2.5.0 Issues in finalizing rolling upgrade when there is

[jira] [Commented] (HDFS-6527) Edit log corruption due to defered INode removal

2014-06-25 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14043586#comment-14043586 ] Kihwal Lee commented on HDFS-6527: -- [~jingzhao] You are right. Since it reresolves inside

[jira] [Resolved] (HDFS-6527) Edit log corruption due to defered INode removal

2014-06-25 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-6527. -- Resolution: Fixed Edit log corruption due to defered INode removal

[jira] [Updated] (HDFS-6527) Edit log corruption due to defered INode removal

2014-06-25 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6527: - Fix Version/s: (was: 2.4.1) 2.5.0 3.0.0 Edit log corruption

[jira] [Commented] (HDFS-6527) Edit log corruption due to defered INode removal

2014-06-25 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14043596#comment-14043596 ] Kihwal Lee commented on HDFS-6527: -- Reverted it from branch-2.4.1 and also updated the

[jira] [Updated] (HDFS-6601) Issues in finalizing rolling upgrade when there is a layout version change

2014-06-25 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6601: - Priority: Blocker (was: Major) Issues in finalizing rolling upgrade when there is a layout version

[jira] [Created] (HDFS-6602) PendingDeletionBlocks on SBN keeps increasing

2014-06-25 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-6602: Summary: PendingDeletionBlocks on SBN keeps increasing Key: HDFS-6602 URL: https://issues.apache.org/jira/browse/HDFS-6602 Project: Hadoop HDFS Issue Type: Bug

[jira] [Commented] (HDFS-6602) PendingDeletionBlocks on SBN keeps increasing

2014-06-25 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14043736#comment-14043736 ] Kihwal Lee commented on HDFS-6602: -- Since {{ReplicationMonitor}} is not running on SBN,

[jira] [Comment Edited] (HDFS-6602) PendingDeletionBlocks on SBN keeps increasing

2014-06-25 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14043736#comment-14043736 ] Kihwal Lee edited comment on HDFS-6602 at 6/25/14 5:05 PM: --- Since

[jira] [Commented] (HDFS-6602) PendingDeletionBlocks on SBN keeps increasing

2014-06-25 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14043993#comment-14043993 ] Kihwal Lee commented on HDFS-6602: -- Block report processing is actually okay. All types of

[jira] [Resolved] (HDFS-6602) PendingDeletionBlocks on SBN keeps increasing

2014-06-25 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee resolved HDFS-6602. -- Resolution: Duplicate Assignee: Kihwal Lee It's already fixed by HDFS-6424!

[jira] [Commented] (HDFS-6569) OOB message can't be sent to the client when DataNode shuts down for upgrade

2014-06-26 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14044680#comment-14044680 ] Kihwal Lee commented on HDFS-6569: --

[jira] [Assigned] (HDFS-6569) OOB message can't be sent to the client when DataNode shuts down for upgrade

2014-06-26 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee reassigned HDFS-6569: Assignee: Kihwal Lee OOB message can't be sent to the client when DataNode shuts down for upgrade

[jira] [Commented] (HDFS-6601) Issues in finalizing rolling upgrade when there is a layout version change

2014-06-27 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14045990#comment-14045990 ] Kihwal Lee commented on HDFS-6601: -- [~jingzhao] Would you review the patch? We've tested

[jira] [Updated] (HDFS-6601) Issues in finalizing rolling upgrade when there is a layout version change

2014-06-28 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6601: - Resolution: Fixed Fix Version/s: 2.5.0 3.0.0 Hadoop Flags: Reviewed

[jira] [Commented] (HDFS-6558) Missing '\n' in the description of dfsadmin -rollingUpgrade

2014-06-30 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14047929#comment-14047929 ] Kihwal Lee commented on HDFS-6558: -- +1 lgtm Missing '\n' in the description of dfsadmin

[jira] [Commented] (HDFS-6558) Missing '\n' in the description of dfsadmin -rollingUpgrade

2014-06-30 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14047932#comment-14047932 ] Kihwal Lee commented on HDFS-6558: -- I've committed this to trunk and branch-2. Thanks for

[jira] [Updated] (HDFS-6558) Missing '\n' in the description of dfsadmin -rollingUpgrade

2014-06-30 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6558: - Resolution: Fixed Fix Version/s: 2.5.0 3.0.0 Hadoop Flags: Reviewed

[jira] [Updated] (HDFS-6415) Missing null check in FSImageSerialization#writePermissionStatus()

2014-06-30 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6415: - Resolution: Won't Fix Status: Resolved (was: Patch Available) Missing null check in

[jira] [Commented] (HDFS-6415) Missing null check in FSImageSerialization#writePermissionStatus()

2014-06-30 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14047944#comment-14047944 ] Kihwal Lee commented on HDFS-6415: -- I don't think we want to do anything special here. If

[jira] [Created] (HDFS-6618) Edit log corruption may still happen even after HDFS-6527

2014-07-01 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-6618: Summary: Edit log corruption may still happen even after HDFS-6527 Key: HDFS-6618 URL: https://issues.apache.org/jira/browse/HDFS-6618 Project: Hadoop HDFS Issue

[jira] [Updated] (HDFS-6618) Edit log corruption may still happen even after HDFS-6527

2014-07-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6618: - Attachment: HDFS-6618.patch The patch makes {{FileWithSnapshotFeature.isCurrentFileDeleted}} and

[jira] [Updated] (HDFS-6618) Edit log corruption may still happen even after HDFS-6527

2014-07-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6618: - Status: Patch Available (was: Open) Edit log corruption may still happen even after HDFS-6527

[jira] [Comment Edited] (HDFS-6618) Edit log corruption may still happen even after HDFS-6527

2014-07-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14049195#comment-14049195 ] Kihwal Lee edited comment on HDFS-6618 at 7/1/14 7:06 PM: -- bq. can

[jira] [Comment Edited] (HDFS-6618) Edit log corruption may still happen even after HDFS-6527

2014-07-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14049195#comment-14049195 ] Kihwal Lee edited comment on HDFS-6618 at 7/1/14 7:06 PM: -- bq. can

[jira] [Commented] (HDFS-6618) Edit log corruption may still happen even after HDFS-6527

2014-07-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14049195#comment-14049195 ] Kihwal Lee commented on HDFS-6618: -- bq. can we move the following code into the writelock

[jira] [Commented] (HDFS-6618) Edit log corruption may still happen even after HDFS-6527

2014-07-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14049309#comment-14049309 ] Kihwal Lee commented on HDFS-6618: -- I guess we can move it inside the first lock, since it

[jira] [Updated] (HDFS-6618) Edit log corruption may still happen even after HDFS-6527

2014-07-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6618: - Status: Open (was: Patch Available) Edit log corruption may still happen even after HDFS-6527

[jira] [Updated] (HDFS-6618) Edit log corruption may still happen even after HDFS-6527

2014-07-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6618: - Attachment: HDFS-6681.AbstractList.patch The new patch implements an idea for minimal change. This is

[jira] [Updated] (HDFS-6618) Edit log corruption may still happen even after HDFS-6527

2014-07-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6618: - Attachment: HDFS-6618.AbstractList.patch Edit log corruption may still happen even after HDFS-6527

[jira] [Updated] (HDFS-6618) Edit log corruption may still happen even after HDFS-6527

2014-07-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6618: - Attachment: (was: HDFS-6681.AbstractList.patch) Edit log corruption may still happen even after

[jira] [Updated] (HDFS-6618) Edit log corruption may still happen even after HDFS-6527

2014-07-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6618: - Attachment: HDFS-6618.inodeRemover.patch Here is a new patch that implements {{INodeRemover}}. It

[jira] [Updated] (HDFS-6618) Edit log corruption may still happen even after HDFS-6527

2014-07-01 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6618: - Status: Patch Available (was: Open) Edit log corruption may still happen even after HDFS-6527

[jira] [Updated] (HDFS-6618) Edit log corruption may still happen even after HDFS-6527

2014-07-02 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6618: - Attachment: HDFS-6618.inodeRemover.v2.patch Fixed the javadoc warning in v2. Edit log corruption may

[jira] [Updated] (HDFS-6618) Remove deleted INodes from INodeMap right away

2014-07-02 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6618: - Summary: Remove deleted INodes from INodeMap right away (was: Edit log corruption may still happen even

[jira] [Commented] (HDFS-6618) Remove deleted INodes from INodeMap right away

2014-07-02 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14050610#comment-14050610 ] Kihwal Lee commented on HDFS-6618: -- Unit test failure is unrelated to the patch. Changed

[jira] [Created] (HDFS-6622) Rename and AddBlock may race and produce invalid edits

2014-07-02 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-6622: Summary: Rename and AddBlock may race and produce invalid edits Key: HDFS-6622 URL: https://issues.apache.org/jira/browse/HDFS-6622 Project: Hadoop HDFS Issue Type:

[jira] [Updated] (HDFS-6622) Rename and AddBlock may race and produce invalid edits

2014-07-02 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6622: - Description: While investigating HDFS-6618, we have discovered that rename happening in the middle of

[jira] [Updated] (HDFS-6622) Rename and AddBlock may race and produce invalid edits

2014-07-02 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6622: - Status: Patch Available (was: Open) Rename and AddBlock may race and produce invalid edits

[jira] [Updated] (HDFS-6622) Rename and AddBlock may race and produce invalid edits

2014-07-02 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6622: - Attachment: HDFS-6622.patch Fix + test case. The test fails without the fix. Rename and AddBlock may

[jira] [Commented] (HDFS-6622) Rename and AddBlock may race and produce invalid edits

2014-07-02 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14050793#comment-14050793 ] Kihwal Lee commented on HDFS-6622: -- Yes, I meant HDFS-6618. Corrected the description.

[jira] [Assigned] (HDFS-6618) Remove deleted INodes from INodeMap right away

2014-07-03 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee reassigned HDFS-6618: Assignee: Kihwal Lee Remove deleted INodes from INodeMap right away

[jira] [Commented] (HDFS-6618) Remove deleted INodes from INodeMap right away

2014-07-03 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14051887#comment-14051887 ] Kihwal Lee commented on HDFS-6618: -- [~cmccabe], thanks for the review. bq. What happens if

[jira] [Comment Edited] (HDFS-6618) Remove deleted INodes from INodeMap right away

2014-07-03 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14051887#comment-14051887 ] Kihwal Lee edited comment on HDFS-6618 at 7/3/14 8:39 PM: --

[jira] [Updated] (HDFS-6622) Rename and AddBlock may race and produce invalid edits

2014-07-09 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6622: - Attachment: HDFS-6622.v2.patch Rename and AddBlock may race and produce invalid edits

[jira] [Assigned] (HDFS-6622) Rename and AddBlock may race and produce invalid edits

2014-07-09 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee reassigned HDFS-6622: Assignee: Kihwal Lee Rename and AddBlock may race and produce invalid edits

[jira] [Commented] (HDFS-6618) Remove deleted INodes from INodeMap right away

2014-07-09 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14056451#comment-14056451 ] Kihwal Lee commented on HDFS-6618: -- Ok, I will take the simple approach. Remove deleted

[jira] [Commented] (HDFS-6622) Rename and AddBlock may race and produce invalid edits

2014-07-09 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14056449#comment-14056449 ] Kihwal Lee commented on HDFS-6622: -- The reason I added the strict check was to prevent

[jira] [Updated] (HDFS-6618) Remove deleted INodes from INodeMap right away

2014-07-09 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6618: - Attachment: HDFS-6618.simpler.patch Remove deleted INodes from INodeMap right away

[jira] [Commented] (HDFS-6618) Remove deleted INodes from INodeMap right away

2014-07-09 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14056477#comment-14056477 ] Kihwal Lee commented on HDFS-6618: -- With the new patch {{removedINodes}} is passed to

[jira] [Created] (HDFS-6651) Deletion failure can leak inodes permanently.

2014-07-09 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-6651: Summary: Deletion failure can leak inodes permanently. Key: HDFS-6651 URL: https://issues.apache.org/jira/browse/HDFS-6651 Project: Hadoop HDFS Issue Type: Bug

[jira] [Commented] (HDFS-6618) Remove deleted INodes from INodeMap right away

2014-07-09 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14056499#comment-14056499 ] Kihwal Lee commented on HDFS-6618: -- Filed HDFS-6651 for the leak problem. Remove deleted

[jira] [Commented] (HDFS-6618) Remove deleted INodes from INodeMap right away

2014-07-09 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14056512#comment-14056512 ] Kihwal Lee commented on HDFS-6618: -- The build failed because of this. {panel} [exec]

[jira] [Commented] (HDFS-6647) Edit log corruption when pipeline recovery occurs for deleted file present in snapshot

2014-07-09 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14056773#comment-14056773 ] Kihwal Lee commented on HDFS-6647: -- It is already checking if the file is deleted. It's

[jira] [Updated] (HDFS-6647) Edit log corruption when pipeline recovery occurs for deleted file present in snapshot

2014-07-09 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6647: - Attachment: HDFS-6647.patch The patch adds {{isFileDeleted()}} method. This depends to HDFS-6618. I also

[jira] [Commented] (HDFS-6647) Edit log corruption when pipeline recovery occurs for deleted file present in snapshot

2014-07-09 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14056783#comment-14056783 ] Kihwal Lee commented on HDFS-6647: -- Marking HDFS-6618 as a dependency. I won't submit the

[jira] [Commented] (HDFS-6647) Edit log corruption when pipeline recovery occurs for deleted file present in snapshot

2014-07-09 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14056803#comment-14056803 ] Kihwal Lee commented on HDFS-6647: -- bq. does it not seem strange to anyone that we allow

[jira] [Commented] (HDFS-6647) Edit log corruption when pipeline recovery occurs for deleted file present in snapshot

2014-07-09 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14056815#comment-14056815 ] Kihwal Lee commented on HDFS-6647: -- bq. Oh, sorry, Kihwal - shouldn't this code also be

[jira] [Updated] (HDFS-6647) Edit log corruption when pipeline recovery occurs for deleted file present in snapshot

2014-07-09 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6647: - Attachment: HDFS-6647.v2.patch Edit log corruption when pipeline recovery occurs for deleted file

[jira] [Updated] (HDFS-6647) Edit log corruption when pipeline recovery occurs for deleted file present in snapshot

2014-07-10 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6647: - Attachment: HDFS-6647.v3.patch The new patch adds {{file.getParent() == null}} back to the deletion

[jira] [Updated] (HDFS-6660) Use int instead of object reference to DatanodeStorageInfo in BlockInfo triplets,

2014-07-10 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6660: - Target Version/s: 2.6.0 Fix Version/s: (was: 2.5.0) (was: 3.0.0)

[jira] [Updated] (HDFS-6659) Create a Block List

2014-07-10 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6659: - Target Version/s: 2.6.0 Fix Version/s: (was: 2.5.0) (was: 3.0.0)

[jira] [Updated] (HDFS-6658) Namenode memory optimization - Block replicas list

2014-07-10 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6658: - Target Version/s: 2.6.0 Namenode memory optimization - Block replicas list

[jira] [Updated] (HDFS-6658) Namenode memory optimization - Block replicas list

2014-07-10 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6658: - Fix Version/s: (was: 2.5.0) (was: 3.0.0) Namenode memory optimization - Block

[jira] [Updated] (HDFS-6661) Use BlockList instead of double linked list i.e BlockInfo triplets

2014-07-10 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6661: - Target Version/s: 2.6.0 Fix Version/s: (was: 2.5.0) (was: 3.0.0)

[jira] [Updated] (HDFS-6658) Namenode memory optimization - Block replicas list

2014-07-10 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6658: - Assignee: Amir Langer Namenode memory optimization - Block replicas list

[jira] [Updated] (HDFS-6660) Use int instead of object reference to DatanodeStorageInfo in BlockInfo triplets,

2014-07-10 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6660: - Assignee: Amir Langer Use int instead of object reference to DatanodeStorageInfo in BlockInfo

[jira] [Updated] (HDFS-6661) Use BlockList instead of double linked list i.e BlockInfo triplets

2014-07-10 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6661: - Assignee: Amir Langer Use BlockList instead of double linked list i.e BlockInfo triplets

[jira] [Updated] (HDFS-6659) Create a Block List

2014-07-10 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6659: - Assignee: Amir Langer Create a Block List --- Key: HDFS-6659

[jira] [Commented] (HDFS-6658) Namenode memory optimization - Block replicas list

2014-07-10 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14057575#comment-14057575 ] Kihwal Lee commented on HDFS-6658: -- [~langera], I've added you as contributor and assigned

[jira] [Commented] (HDFS-6658) Namenode memory optimization - Block replicas list

2014-07-10 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14057619#comment-14057619 ] Kihwal Lee commented on HDFS-6658: -- Is there any way to compact or free a set of sparse

[jira] [Created] (HDFS-6663) Admin command to track file and locations from block id

2014-07-10 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-6663: Summary: Admin command to track file and locations from block id Key: HDFS-6663 URL: https://issues.apache.org/jira/browse/HDFS-6663 Project: Hadoop HDFS Issue

[jira] [Updated] (HDFS-6647) Edit log corruption when pipeline recovery occurs for deleted file present in snapshot

2014-07-10 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6647: - Resolution: Fixed Fix Version/s: 2.5.0 3.0.0 Hadoop Flags: Reviewed

[jira] [Commented] (HDFS-6569) OOB message can't be sent to the client when DataNode shuts down for upgrade

2014-07-11 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14058836#comment-14058836 ] Kihwal Lee commented on HDFS-6569: -- Sorry for the delay. I looked at the patch and the

[jira] [Created] (HDFS-6696) Name node cannot start if the path of a file under construction contains .snapshot

2014-07-16 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-6696: Summary: Name node cannot start if the path of a file under construction contains .snapshot Key: HDFS-6696 URL: https://issues.apache.org/jira/browse/HDFS-6696 Project:

[jira] [Comment Edited] (HDFS-6696) Name node cannot start if the path of a file under construction contains .snapshot

2014-07-16 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14064180#comment-14064180 ] Kihwal Lee edited comment on HDFS-6696 at 7/16/14 9:56 PM: --- Here

[jira] [Commented] (HDFS-6696) Name node cannot start if the path of a file under construction contains .snapshot

2014-07-16 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14064180#comment-14064180 ] Kihwal Lee commented on HDFS-6696: -- {panel} java.io.FileNotFoundException: File does not

[jira] [Updated] (HDFS-6016) Update datanode replacement policy to make writes more robust

2014-08-13 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6016: - Resolution: Won't Fix Status: Resolved (was: Patch Available) Per Nicholas' comment, I won't fix

[jira] [Updated] (HDFS-6569) OOB message can't be sent to the client when DataNode shuts down for upgrade

2014-08-15 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6569: - Assignee: Brandon Li (was: Kihwal Lee) OOB message can't be sent to the client when DataNode shuts down

[jira] [Commented] (HDFS-6569) OOB message can't be sent to the client when DataNode shuts down for upgrade

2014-08-15 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14098582#comment-14098582 ] Kihwal Lee commented on HDFS-6569: -- The patch looks good in general. There are a couple of

[jira] [Commented] (HDFS-6825) Edit log corruption due to delayed block removal

2014-08-15 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14098583#comment-14098583 ] Kihwal Lee commented on HDFS-6825: -- I will take a look at it soon. Edit log corruption

[jira] [Commented] (HDFS-6825) Edit log corruption due to delayed block removal

2014-08-15 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14098657#comment-14098657 ] Kihwal Lee commented on HDFS-6825: -- Could we also check that this works with a recursive

[jira] [Commented] (HDFS-6825) Edit log corruption due to delayed block removal

2014-08-15 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14098943#comment-14098943 ] Kihwal Lee commented on HDFS-6825: -- [~yzhangal] I've verified that you are correct. The

[jira] [Commented] (HDFS-6569) OOB message can't be sent to the client when DataNode shuts down for upgrade

2014-08-18 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14101160#comment-14101160 ] Kihwal Lee commented on HDFS-6569: -- +1 OOB message can't be sent to the client when

[jira] [Commented] (HDFS-6293) Issues with OIV processing PB-based fsimages

2014-08-20 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14103861#comment-14103861 ] Kihwal Lee commented on HDFS-6293: -- [~fnie] Please see the release note. This is

[jira] [Commented] (HDFS-3689) Add support for variable length block

2014-08-20 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-3689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14103909#comment-14103909 ] Kihwal Lee commented on HDFS-3689: -- We also need to think about file checksum. I think we

[jira] [Created] (HDFS-6888) Remove audit logging of getFIleInfo()

2014-08-20 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-6888: Summary: Remove audit logging of getFIleInfo() Key: HDFS-6888 URL: https://issues.apache.org/jira/browse/HDFS-6888 Project: Hadoop HDFS Issue Type: Bug

[jira] [Created] (HDFS-6889) Provide an iterator-based listing API for FileSystem

2014-08-20 Thread Kihwal Lee (JIRA)
Kihwal Lee created HDFS-6889: Summary: Provide an iterator-based listing API for FileSystem Key: HDFS-6889 URL: https://issues.apache.org/jira/browse/HDFS-6889 Project: Hadoop HDFS Issue Type:

[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()

2014-08-21 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105604#comment-14105604 ] Kihwal Lee commented on HDFS-6888: -- See the patch in HDFS-3733. When this logging was

[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()

2014-08-21 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105698#comment-14105698 ] Kihwal Lee commented on HDFS-6888: -- What if we log it at debug level? Remove audit

[jira] [Commented] (HDFS-6888) Remove audit logging of getFIleInfo()

2014-08-21 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105722#comment-14105722 ] Kihwal Lee commented on HDFS-6888: -- We could have {{logAuditEvent()}} check {{cmd}}

[jira] [Commented] (HDFS-6914) Resolve huge memory consumption Issue with OIV processing PB-based fsimages

2014-08-22 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107095#comment-14107095 ] Kihwal Lee commented on HDFS-6914: -- We do this with the fsimage_oiv/oiv_legacy that was

[jira] [Commented] (HDFS-6965) NN continues to issue block locations for DNs with full disks

2014-08-28 Thread Kihwal Lee (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114065#comment-14114065 ] Kihwal Lee commented on HDFS-6965: -- I think there is a flaw in isGoodTarget(). When

  1   2   3   4   5   6   7   8   9   10   >