[jira] [Commented] (HDFS-6981) DN upgrade with layout version change should not use trash

2014-09-07 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124958#comment-14124958 ] James Thomas commented on HDFS-6981: +1, looks good to me. Getting a 404 when I try to

[jira] [Updated] (HDFS-6777) Supporting consistent edit log reads when in-progress edit log segments are included

2014-09-07 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6777: --- Resolution: Fixed Status: Resolved (was: Patch Available) Committed as part of HDFS-6634.

[jira] [Updated] (HDFS-6777) Supporting consistent edit log reads when in-progress edit log segments are included

2014-09-07 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6777: --- Fix Version/s: 2.6.0 Supporting consistent edit log reads when in-progress edit log segments are

[jira] [Updated] (HDFS-6777) Supporting consistent edit log reads when in-progress edit log segments are included

2014-09-07 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6777: --- Description: For inotify, we want to be able to read transactions from in-progress edit log segments

[jira] [Commented] (HDFS-6981) DN upgrade with layout version change should not use trash

2014-09-06 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14124731#comment-14124731 ] James Thomas commented on HDFS-6981: {code} + if

[jira] [Commented] (HDFS-6981) DN upgrade with layout version change should not use trash

2014-09-05 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14123471#comment-14123471 ] James Thomas commented on HDFS-6981: The marker file sounds like the best solution to

[jira] [Commented] (HDFS-6981) DN upgrade with layout version change should not use trash

2014-09-04 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14122262#comment-14122262 ] James Thomas commented on HDFS-6981: [~arpitagarwal], thanks for the patch. A couple

[jira] [Commented] (HDFS-6981) DN upgrade with layout version change should not use trash

2014-09-04 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14122378#comment-14122378 ] James Thomas commented on HDFS-6981: bq. Not an issue. If there was no layout version

[jira] [Commented] (HDFS-6634) inotify in HDFS

2014-09-02 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14119104#comment-14119104 ] James Thomas commented on HDFS-6634: Thanks guys inotify in HDFS ---

[jira] [Commented] (HDFS-6981) DN upgrade with layout version change should not use trash

2014-09-01 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117782#comment-14117782 ] James Thomas commented on HDFS-6981: [~arpitagarwal], thanks for filing this. Shouldn't

[jira] [Commented] (HDFS-6800) Support Datanode layout changes with rolling upgrade

2014-08-30 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14116257#comment-14116257 ] James Thomas commented on HDFS-6800: I tested that rolling upgrade to and rolling

[jira] [Commented] (HDFS-6800) Support Datanode layout changes with rolling upgrade

2014-08-30 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14116260#comment-14116260 ] James Thomas commented on HDFS-6800: Also, my comment only applies to putting this in

[jira] [Updated] (HDFS-6634) inotify in HDFS

2014-08-29 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6634: --- Attachment: HDFS-6634.8.patch [~andrew.wang], thanks for the review. Made all of the changes you

[jira] [Commented] (HDFS-6800) Support Datanode layout changes with rolling upgrade

2014-08-29 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14115878#comment-14115878 ] James Thomas commented on HDFS-6800: Thanks, Arpit. Here are the key issues I think we

[jira] [Updated] (HDFS-6634) inotify in HDFS

2014-08-29 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6634: --- Attachment: HDFS-6634.9.patch Updated. inotify in HDFS --- Key:

[jira] [Updated] (HDFS-6800) Determine how Datanode layout changes should interact with rolling upgrade

2014-08-28 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6800: --- Attachment: HDFS-6800.7.patch Uploaded patch that addresses case 2) above. [~arpitagarwal], thoughts?

[jira] [Updated] (HDFS-6800) Determine how Datanode layout changes should interact with rolling upgrade

2014-08-28 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6800: --- Attachment: HDFS-6800.8.patch Good call. Updated patch accordingly. Determine how Datanode layout

[jira] [Commented] (HDFS-6800) Determine how Datanode layout changes should interact with rolling upgrade

2014-08-28 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114356#comment-14114356 ] James Thomas commented on HDFS-6800: Great, thanks for your help on this, Arpit. I will

[jira] [Updated] (HDFS-6777) Supporting consistent edit log reads when in-progress edit log segments are included

2014-08-28 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6777: --- Attachment: 6777-design.2.pdf Updated the design doc to reflect changes mentioned at

[jira] [Commented] (HDFS-6800) Determine how Datanode layout changes should interact with rolling upgrade

2014-08-28 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114522#comment-14114522 ] James Thomas commented on HDFS-6800: Thanks, Arpit. Could you commit this? Colin is

[jira] [Commented] (HDFS-3528) Use native CRC32 in DFS write path

2014-08-28 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-3528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14114622#comment-14114622 ] James Thomas commented on HDFS-3528: Yep, think so -- for both ByteBuffers and byte

[jira] [Updated] (HDFS-6865) Byte array native checksumming on client side (HDFS changes)

2014-08-27 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6865: --- Attachment: HDFS-6865.6.patch [~tlipcon], thanks for the review. Made all of your suggested changes

[jira] [Updated] (HDFS-6865) Byte array native checksumming on client side (HDFS changes)

2014-08-27 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6865: --- Attachment: HDFS-6865.7.patch Turns out TestBlockToken was a related failure. The others seem

[jira] [Updated] (HDFS-6865) Byte array native checksumming on client side (HDFS changes)

2014-08-27 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6865: --- Attachment: HDFS-6865.8.patch Fixed patch. Byte array native checksumming on client side (HDFS

[jira] [Commented] (HDFS-6865) Byte array native checksumming on client side (HDFS changes)

2014-08-27 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14112595#comment-14112595 ] James Thomas commented on HDFS-6865: [~stack], thanks for taking a look. So you saw no

[jira] [Commented] (HDFS-6800) Determine how Datanode layout changes should interact with rolling upgrade

2014-08-27 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14112783#comment-14112783 ] James Thomas commented on HDFS-6800: [~arpitagarwal], thanks for the response. Good

[jira] [Commented] (HDFS-6800) Determine how Datanode layout changes should interact with rolling upgrade

2014-08-27 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113038#comment-14113038 ] James Thomas commented on HDFS-6800: [~arpitagarwal], realized what I wrote above is

[jira] [Commented] (HDFS-6800) Determine how Datanode layout changes should interact with rolling upgrade

2014-08-27 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113081#comment-14113081 ] James Thomas commented on HDFS-6800: Right, the scenario I'm talking about is the

[jira] [Updated] (HDFS-6800) Determine how Datanode layout changes should interact with rolling upgrade

2014-08-27 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6800: --- Attachment: HDFS-6800.6.patch Attached patch with implementation of strategy I proposed above.

[jira] [Commented] (HDFS-6800) Determine how Datanode layout changes should interact with rolling upgrade

2014-08-27 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14113276#comment-14113276 ] James Thomas commented on HDFS-6800: A couple of notes: 1) We do not need to modify

[jira] [Updated] (HDFS-6634) inotify in HDFS

2014-08-27 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6634: --- Attachment: HDFS-6634.7.patch Made the change Colin suggested and also removed the logic to choose

[jira] [Created] (HDFS-6939) Support path-based filtering of inotify events

2014-08-25 Thread James Thomas (JIRA)
James Thomas created HDFS-6939: -- Summary: Support path-based filtering of inotify events Key: HDFS-6939 URL: https://issues.apache.org/jira/browse/HDFS-6939 Project: Hadoop HDFS Issue Type:

[jira] [Updated] (HDFS-6634) inotify in HDFS

2014-08-23 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6634: --- Attachment: HDFS-6634.6.patch Responded to most of Colin's comments and made some changes to improve

[jira] [Updated] (HDFS-6800) Determine how Datanode layout changes should interact with rolling upgrade

2014-08-23 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6800: --- Attachment: HDFS-6800.5.patch Updated patch, using the strategy I described in my response to

[jira] [Commented] (HDFS-6800) Determine how Datanode layout changes should interact with rolling upgrade

2014-08-22 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107133#comment-14107133 ] James Thomas commented on HDFS-6800: [~cmccabe], thanks for checking this out. Why

[jira] [Updated] (HDFS-6865) Byte array native checksumming on client side (HDFS changes)

2014-08-22 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6865: --- Attachment: HDFS-6865.5.patch Fixed the TestBlockUnderConstruction timeout, which I missed last time.

[jira] [Commented] (HDFS-6800) Determine how Datanode layout changes should interact with rolling upgrade

2014-08-22 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107286#comment-14107286 ] James Thomas commented on HDFS-6800: [~arpitagarwal], thanks for the response. The

[jira] [Commented] (HDFS-6865) Byte array native checksumming on client side (HDFS changes)

2014-08-22 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107404#comment-14107404 ] James Thomas commented on HDFS-6865: I ran the tests that [~tlipcon] suggested and have

[jira] [Updated] (HDFS-6634) inotify in HDFS

2014-08-22 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6634: --- Attachment: inotify-design.4.pdf Integrated the content from inotify-intro into inotify-design as

[jira] [Commented] (HDFS-6800) Determine how Datanode layout changes should interact with rolling upgrade

2014-08-22 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14107594#comment-14107594 ] James Thomas commented on HDFS-6800: Right, that's what I figured. Will talk to

[jira] [Updated] (HDFS-6865) Byte array native checksumming on client side (HDFS changes)

2014-08-21 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6865: --- Attachment: HDFS-6865.4.patch Reattching same patch to retrigger Jenkins. Byte array native

[jira] [Updated] (HDFS-6865) Byte array native checksumming on client side (HDFS changes)

2014-08-20 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6865: --- Attachment: HDFS-6865.2.patch Attempted to resolve test failures. Byte array native checksumming on

[jira] [Updated] (HDFS-6865) Byte array native checksumming on client side (HDFS changes)

2014-08-20 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6865: --- Attachment: HDFS-6865.3.patch Fixed compilation error. Byte array native checksumming on client

[jira] [Commented] (HDFS-6800) Determine how Datanode layout changes should interact with rolling upgrade

2014-08-19 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14102980#comment-14102980 ] James Thomas commented on HDFS-6800: [~arpitagarwal], did you get a chance to look at

[jira] [Updated] (HDFS-6853) MiniDFSCluster.isClusterUp() should not check if null NameNodes are up

2014-08-18 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6853: --- Attachment: HDFS-6853.2.patch Reposted same patch to retrigger Jenkins.

[jira] [Updated] (HDFS-6800) Determine how Datanode layout changes should interact with rolling upgrade

2014-08-18 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6800: --- Attachment: HDFS-6800.4.patch Reposted same patch to retrigger Jenkins. Determine how Datanode

[jira] [Updated] (HDFS-6561) Byte array native checksumming on client side (Hadoop changes)

2014-08-18 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6561: --- Summary: Byte array native checksumming on client side (Hadoop changes) (was: Byte array native

[jira] [Created] (HDFS-6865) Byte array native checksumming on client side (HDFS changes)

2014-08-18 Thread James Thomas (JIRA)
James Thomas created HDFS-6865: -- Summary: Byte array native checksumming on client side (HDFS changes) Key: HDFS-6865 URL: https://issues.apache.org/jira/browse/HDFS-6865 Project: Hadoop HDFS

[jira] [Updated] (HDFS-6865) Byte array native checksumming on client side (HDFS changes)

2014-08-18 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6865: --- Component/s: (was: datanode) Byte array native checksumming on client side (HDFS changes)

[jira] [Updated] (HDFS-6865) Byte array native checksumming on client side (HDFS changes)

2014-08-18 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6865: --- Attachment: HDFS-6865.patch Won't submit till HDFS-6561 goes in. Byte array native checksumming on

[jira] [Updated] (HDFS-6634) inotify in HDFS

2014-08-18 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6634: --- Attachment: inotify-design.3.pdf Updated design doc to include current protobuf definitions and

[jira] [Updated] (HDFS-6865) Byte array native checksumming on client side (HDFS changes)

2014-08-18 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6865: --- Description: Refactor FSOutputSummer to buffer data and use the native checksum calculation

[jira] [Updated] (HDFS-6800) Determine how Datanode layout changes should interact with rolling upgrade

2014-08-15 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6800: --- Attachment: HDFS-6800.2.patch Updated the patch to delete the trash directory if the previous

[jira] [Updated] (HDFS-6800) Determine how Datanode layout changes should interact with rolling upgrade

2014-08-15 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6800: --- Attachment: HDFS-6800.3.patch Determine how Datanode layout changes should interact with rolling

[jira] [Created] (HDFS-6853) MiniDFSCluster.isClusterUp() should not check if null NameNodes are up

2014-08-14 Thread James Thomas (JIRA)
James Thomas created HDFS-6853: -- Summary: MiniDFSCluster.isClusterUp() should not check if null NameNodes are up Key: HDFS-6853 URL: https://issues.apache.org/jira/browse/HDFS-6853 Project: Hadoop HDFS

[jira] [Updated] (HDFS-6853) MiniDFSCluster.isClusterUp() should not check if null NameNodes are up

2014-08-14 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6853: --- Attachment: HDFS-6853.patch MiniDFSCluster.isClusterUp() should not check if null NameNodes are up

[jira] [Updated] (HDFS-6853) MiniDFSCluster.isClusterUp() should not check if null NameNodes are up

2014-08-14 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6853: --- Assignee: James Thomas Status: Patch Available (was: Open) MiniDFSCluster.isClusterUp()

[jira] [Updated] (HDFS-6634) inotify in HDFS

2014-08-14 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6634: --- Attachment: HDFS-6634.5.patch The TestCheckpoint failure looks like a flake even though it relates to

[jira] [Updated] (HDFS-6561) Byte array native checksumming on client side

2014-08-14 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6561: --- Attachment: HDFS-6561.4.patch Todd's patch with the second change suggested by Colin. [~cmccabe], as

[jira] [Updated] (HDFS-6561) Byte array native checksumming on client side

2014-08-14 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6561: --- Attachment: HDFS-6561.5.patch Thanks, Colin. Posted updated patch. Byte array native checksumming

[jira] [Commented] (HDFS-6853) MiniDFSCluster.isClusterUp() should not check if null NameNodes are up

2014-08-14 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14098034#comment-14098034 ] James Thomas commented on HDFS-6853: Looks like a few tests rely on this behavior ...

[jira] [Updated] (HDFS-6800) Determine how Datanode layout changes should interact with rolling upgrade

2014-08-14 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6800: --- Status: Patch Available (was: Open) Determine how Datanode layout changes should interact with

[jira] [Updated] (HDFS-6800) Determine how Datanode layout changes should interact with rolling upgrade

2014-08-14 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6800: --- Attachment: HDFS-6800.patch Attached a patch with our proposed solution. [~arpitagarwal] or

[jira] [Assigned] (HDFS-6800) Determine how Datanode layout changes should interact with rolling upgrade

2014-08-14 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas reassigned HDFS-6800: -- Assignee: James Thomas Determine how Datanode layout changes should interact with rolling

[jira] [Updated] (HDFS-6634) inotify in HDFS

2014-08-13 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6634: --- Attachment: HDFS-6634.4.patch Thanks for the comments, Andrew. Updated patch. inotify in HDFS

[jira] [Updated] (HDFS-6634) inotify in HDFS

2014-08-08 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6634: --- Attachment: HDFS-6634.3.patch Thanks for the comments, Andrew. Addressed them in patch #3. inotify

[jira] [Commented] (HDFS-6482) Use block ID-based block layout on datanodes

2014-08-06 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14088059#comment-14088059 ] James Thomas commented on HDFS-6482: [~arpitagarwal], yep, we're waiting on discussion

[jira] [Updated] (HDFS-6634) inotify in HDFS

2014-08-04 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6634: --- Attachment: HDFS-6634.2.patch inotify in HDFS --- Key: HDFS-6634

[jira] [Commented] (HDFS-6482) Use block ID-based block layout on datanodes

2014-08-01 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14083035#comment-14083035 ] James Thomas commented on HDFS-6482: [~cnauroth], apologies, thanks. Use block

[jira] [Updated] (HDFS-6634) inotify in HDFS

2014-08-01 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6634: --- Attachment: HDFS-6634.patch Work-in-progress patch including the changes submitted at HDFS-6777.

[jira] [Updated] (HDFS-6634) inotify in HDFS

2014-08-01 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6634: --- Attachment: inotify-design.2.pdf Updated design doc with new ProtoBuf definitions. inotify in HDFS

[jira] [Updated] (HDFS-6634) inotify in HDFS

2014-08-01 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6634: --- Target Version/s: 2.6.0 (was: 2.5.0) Status: Patch Available (was: Open) inotify in

[jira] [Commented] (HDFS-6634) inotify in HDFS

2014-08-01 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14083351#comment-14083351 ] James Thomas commented on HDFS-6634: All functionality described in the design doc

[jira] [Updated] (HDFS-6634) inotify in HDFS

2014-07-31 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6634: --- Attachment: inotify-design.pdf Posted design doc. inotify in HDFS ---

[jira] [Commented] (HDFS-6482) Use block ID-based block layout on datanodes

2014-07-31 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14081427#comment-14081427 ] James Thomas commented on HDFS-6482: [~sureshms] No, during rollback the directory

[jira] [Commented] (HDFS-6482) Use block ID-based block layout on datanodes

2014-07-31 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14081603#comment-14081603 ] James Thomas commented on HDFS-6482: [~sureshms] Is the documentation the Rollback

[jira] [Commented] (HDFS-6482) Use block ID-based block layout on datanodes

2014-07-31 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14081607#comment-14081607 ] James Thomas commented on HDFS-6482: Or [~kihwal]? Use block ID-based block layout on

[jira] [Updated] (HDFS-6777) Supporting consistent edit log reads when in-progress edit log segments are included

2014-07-31 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6777: --- Status: Patch Available (was: Open) Supporting consistent edit log reads when in-progress edit log

[jira] [Commented] (HDFS-6634) inotify in HDFS

2014-07-30 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14080017#comment-14080017 ] James Thomas commented on HDFS-6634: [~aw] What if the format of the audit log changes?

[jira] [Created] (HDFS-6777) Supporting consistent edit log reads when in-progress edit log segments are included

2014-07-29 Thread James Thomas (JIRA)
James Thomas created HDFS-6777: -- Summary: Supporting consistent edit log reads when in-progress edit log segments are included Key: HDFS-6777 URL: https://issues.apache.org/jira/browse/HDFS-6777

[jira] [Updated] (HDFS-6777) Supporting consistent edit log reads when in-progress edit log segments are included

2014-07-29 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6777: --- Description: For inotify, we want to be able to read transactions from in-progress edit log segments

[jira] [Updated] (HDFS-6777) Supporting consistent edit log reads when in-progress edit log segments are included

2014-07-29 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6777: --- Attachment: HDFS-6777.patch Supporting consistent edit log reads when in-progress edit log segments

[jira] [Updated] (HDFS-6777) Supporting consistent edit log reads when in-progress edit log segments are included

2014-07-29 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6777: --- Attachment: 6777-design.pdf Supporting consistent edit log reads when in-progress edit log segments

[jira] [Updated] (HDFS-6777) Supporting consistent edit log reads when in-progress edit log segments are included

2014-07-29 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6777: --- Description: For inotify, we want to be able to read transactions from in-progress edit log segments

[jira] [Commented] (HDFS-6482) Use block ID-based block layout on datanodes

2014-07-21 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14069541#comment-14069541 ] James Thomas commented on HDFS-6482: In response to [~cmccabe]'s comments: One thread

[jira] [Updated] (HDFS-6561) Byte array native checksumming on client side

2014-07-18 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6561: --- Attachment: HDFS-6561.3.patch Updated due to changes in HADOOP-10838. Byte array native

[jira] [Updated] (HDFS-6482) Use block ID-based block layout on datanodes

2014-07-18 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6482: --- Attachment: HDFS-6482.8.patch Fixed the new test and parallelized and nativized the hard link process

[jira] [Updated] (HDFS-6482) Use block ID-based block layout on datanodes

2014-07-18 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6482: --- Attachment: hadoop-24-datanode-dir.tgz HDFS-6482.9.patch Removed binary file from

[jira] [Updated] (HDFS-6482) Use block ID-based block layout on datanodes

2014-07-18 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6482: --- Attachment: (was: HDFS-6482.9.patch) Use block ID-based block layout on datanodes

[jira] [Updated] (HDFS-6482) Use block ID-based block layout on datanodes

2014-07-18 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6482: --- Attachment: HDFS-6482.9.patch It tried to apply the tgz .. reuploading patch. Use block ID-based

[jira] [Updated] (HDFS-6561) Byte array native checksumming on client side

2014-07-17 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6561: --- Attachment: HDFS-6561.2.patch Updated patch based on changes to HADOOP-10838. Byte array native

[jira] [Commented] (HDFS-6634) inotify in HDFS

2014-07-14 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14061255#comment-14061255 ] James Thomas commented on HDFS-6634: What I wrote above about ProtoBuf'ing isn't

[jira] [Commented] (HDFS-6634) inotify in HDFS

2014-07-10 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14058136#comment-14058136 ] James Thomas commented on HDFS-6634: Initially we will expose the entire stream to

[jira] [Updated] (HDFS-6560) Byte array native checksumming on DN side

2014-07-10 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6560: --- Status: Patch Available (was: Open) Byte array native checksumming on DN side

[jira] [Updated] (HDFS-6560) Byte array native checksumming on DN side

2014-07-10 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6560: --- Attachment: HDFS-6560.patch Addressed feedback. Colin, I kept the wrapper function because it will be

[jira] [Updated] (HDFS-6561) Byte array native checksumming on client side

2014-07-10 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6561: --- Attachment: HDFS-6561.patch Patch attached. Was finally able to get some perf stat output comparing

[jira] [Updated] (HDFS-6561) Byte array native checksumming on client side

2014-07-10 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6561: --- Status: Patch Available (was: Open) Byte array native checksumming on client side

[jira] [Commented] (HDFS-6561) Byte array native checksumming on client side

2014-07-10 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14058210#comment-14058210 ] James Thomas commented on HDFS-6561: Need to consolidate this patch with the one in

[jira] [Commented] (HDFS-6634) inotify in HDFS

2014-07-09 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14056474#comment-14056474 ] James Thomas commented on HDFS-6634: That would require this project to be a

[jira] [Updated] (HDFS-6634) inotify in HDFS

2014-07-09 Thread James Thomas (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas updated HDFS-6634: --- Attachment: inotify-intro.2.pdf Updated the design doc in response to Andrew's comments. I think we

  1   2   >