[jira] [Reopened] (HDFS-6391) Get the Key/IV from the NameNode for encrypted files in DFSClient

2014-06-20 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang reopened HDFS-6391: --- Get the Key/IV from the NameNode for encrypted files in DFSClient

[jira] [Commented] (HDFS-6391) Get the Key/IV from the NameNode for encrypted files in DFSClient

2014-06-20 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039459#comment-14039459 ] Andrew Wang commented on HDFS-6391: --- I'm starting on this JIRA, and was wondering about

[jira] [Commented] (HDFS-6391) Get the Key/IV from the NameNode for encrypted files in DFSClient

2014-06-20 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039524#comment-14039524 ] Andrew Wang commented on HDFS-6391: --- Ah, so the client would just initially only send the

[jira] [Commented] (HDFS-6391) Get the Key/IV from the NameNode for encrypted files in DFSClient

2014-06-20 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039579#comment-14039579 ] Andrew Wang commented on HDFS-6391: --- Gotcha, that works for me. I might rename CipherType

[jira] [Commented] (HDFS-5369) Support negative caching of user-group mapping

2014-06-23 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-5369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14041497#comment-14041497 ] Andrew Wang commented on HDFS-5369: --- Hey [~eddyxu], 30s sounds okay to me, maybe even a

[jira] [Commented] (HDFS-6561) Byte array native checksumming on client side

2014-06-23 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14041522#comment-14041522 ] Andrew Wang commented on HDFS-6561: --- +1 sounds good to me, pretty sure that flushes are

[jira] [Updated] (HDFS-6391) Get the Key/IV from the NameNode for encrypted files in DFSClient

2014-06-26 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6391: -- Attachment: hdfs-6391.002.patch Patch attached. This is based off of some work that [~clamb] did.

[jira] [Work started] (HDFS-6391) Get the Key/IV from the NameNode for encrypted files in DFSClient

2014-06-26 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-6391 started by Andrew Wang. Get the Key/IV from the NameNode for encrypted files in DFSClient

[jira] [Updated] (HDFS-6391) Get the Key/IV from the NameNode for encrypted files in DFSClient

2014-06-26 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6391: -- Attachment: hdfs-6391.003.patch New patch attached. I just renamed the key-id constant. PBHelper, since

[jira] [Updated] (HDFS-6391) Get the Key/IV from the NameNode for encrypted files in DFSClient

2014-06-27 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6391: -- Attachment: hdfs-6391.004.patch New patch fixes the whitespace issues and I moved around imports in

[jira] [Resolved] (HDFS-6391) Get the Key/IV from the NameNode for encrypted files in DFSClient

2014-06-27 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HDFS-6391. --- Resolution: Fixed Fix Version/s: fs-encryption (HADOOP-10150 and HDFS-6134) Committed to

[jira] [Created] (HDFS-6605) Client server negotiation of cipher suite

2014-06-27 Thread Andrew Wang (JIRA)
Andrew Wang created HDFS-6605: - Summary: Client server negotiation of cipher suite Key: HDFS-6605 URL: https://issues.apache.org/jira/browse/HDFS-6605 Project: Hadoop HDFS Issue Type: Sub-task

[jira] [Commented] (HDFS-6605) Client server negotiation of cipher suite

2014-06-27 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14046386#comment-14046386 ] Andrew Wang commented on HDFS-6605: --- There was some discussion on HDFS-6391 with

[jira] [Created] (HDFS-6613) Improve logging in caching classes

2014-06-30 Thread Andrew Wang (JIRA)
Andrew Wang created HDFS-6613: - Summary: Improve logging in caching classes Key: HDFS-6613 URL: https://issues.apache.org/jira/browse/HDFS-6613 Project: Hadoop HDFS Issue Type: Improvement

[jira] [Updated] (HDFS-6613) Improve logging in caching classes

2014-06-30 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6613: -- Attachment: HDFS-6613.001.patch Patch attached, almost purely logging changes. Improve logging in

[jira] [Work started] (HDFS-6613) Improve logging in caching classes

2014-06-30 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-6613 started by Andrew Wang. Improve logging in caching classes -- Key: HDFS-6613 URL:

[jira] [Commented] (HDFS-6612) MiniDFSNNTopology#simpleFederatedTopology(int) always hardcode nameservice ID

2014-06-30 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14048372#comment-14048372 ] Andrew Wang commented on HDFS-6612: --- +1 pending jenkins

[jira] [Commented] (HDFS-6610) TestShortCircuitLocalRead tests sometimes timeout on slow machines

2014-06-30 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14048382#comment-14048382 ] Andrew Wang commented on HDFS-6610: --- +1 pending Jenkins TestShortCircuitLocalRead tests

[jira] [Updated] (HDFS-6613) Improve logging in caching classes

2014-06-30 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6613: -- Status: Patch Available (was: In Progress) Improve logging in caching classes

[jira] [Created] (HDFS-6619) Clean up encryption-related tests

2014-07-01 Thread Andrew Wang (JIRA)
Andrew Wang created HDFS-6619: - Summary: Clean up encryption-related tests Key: HDFS-6619 URL: https://issues.apache.org/jira/browse/HDFS-6619 Project: Hadoop HDFS Issue Type: Sub-task

[jira] [Commented] (HDFS-6619) Clean up encryption-related tests

2014-07-01 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14049530#comment-14049530 ] Andrew Wang commented on HDFS-6619: --- Let's also try to isolate the tests to use the FS

[jira] [Updated] (HDFS-6605) Client server negotiation of cipher suite

2014-07-01 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6605: -- Attachment: hdfs-6605.001.patch Patch attached. Overall idea: - Client provides an ordered list of

[jira] [Commented] (HDFS-6605) Client server negotiation of cipher suite

2014-07-02 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14050320#comment-14050320 ] Andrew Wang commented on HDFS-6605: --- Hey Yi, thanks for the review! I'll handle it in the

[jira] [Commented] (HDFS-6612) MiniDFSNNTopology#simpleFederatedTopology(int) always hardcode nameservice ID

2014-07-02 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14050520#comment-14050520 ] Andrew Wang commented on HDFS-6612: --- LGTM will commit shortly.

[jira] [Updated] (HDFS-6612) MiniDFSNNTopology#simpleFederatedTopology(int) always hardcode nameservice ID

2014-07-02 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6612: -- Resolution: Fixed Fix Version/s: 2.5.0 Status: Resolved (was: Patch Available)

[jira] [Updated] (HDFS-6612) MiniDFSNNTopology#simpleFederatedTopology(int) always hardcode nameservice ID

2014-07-02 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6612: -- Description: MiniDFSNNTopology#simpleFederatedTopology(int numNameservices) always hardcode

[jira] [Commented] (HDFS-6610) TestShortCircuitLocalRead tests sometimes timeout on slow machines

2014-07-02 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14050563#comment-14050563 ] Andrew Wang commented on HDFS-6610: --- I manually bumped the buildbot, somehow it didn't

[jira] [Updated] (HDFS-6605) Client server negotiation of cipher suite

2014-07-02 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6605: -- Attachment: hdfs-6605.002.patch Thanks for the additional review Charles. New patch attached that

[jira] [Created] (HDFS-6624) TestBlockTokenWithDFS#testEnd2End fails sometimes

2014-07-02 Thread Andrew Wang (JIRA)
Andrew Wang created HDFS-6624: - Summary: TestBlockTokenWithDFS#testEnd2End fails sometimes Key: HDFS-6624 URL: https://issues.apache.org/jira/browse/HDFS-6624 Project: Hadoop HDFS Issue Type:

[jira] [Updated] (HDFS-6624) TestBlockTokenWithDFS#testEnd2End fails sometimes

2014-07-02 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6624: -- Attachment: PreCommit-HDFS-Build #7274 test - testEnd2End [Jenkins].html Attaching the html from the

[jira] [Commented] (HDFS-6610) TestShortCircuitLocalRead tests sometimes timeout on slow machines

2014-07-02 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14050836#comment-14050836 ] Andrew Wang commented on HDFS-6610: --- I couldn't repro either of these locally. The first

[jira] [Updated] (HDFS-6610) TestShortCircuitLocalRead tests sometimes timeout on slow machines

2014-07-02 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6610: -- Resolution: Fixed Fix Version/s: 2.5.0 Status: Resolved (was: Patch Available)

[jira] [Commented] (HDFS-6605) Client server negotiation of cipher suite

2014-07-02 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14050841#comment-14050841 ] Andrew Wang commented on HDFS-6605: --- Great, thanks Charles. I'll commit this shortly;

[jira] [Resolved] (HDFS-6605) Client server negotiation of cipher suite

2014-07-02 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HDFS-6605. --- Resolution: Fixed Fix Version/s: fs-encryption (HADOOP-10150 and HDFS-6134) Client server

[jira] [Updated] (HDFS-6613) Improve logging in caching classes

2014-07-02 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6613: -- Attachment: HDFS-6613.002.patch Good point Steve, I tried my hand at swapping in SLF4J in this new

[jira] [Updated] (HDFS-6613) Improve logging in caching classes

2014-07-03 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6613: -- Resolution: Fixed Fix Version/s: 2.6.0 Status: Resolved (was: Patch Available)

[jira] [Commented] (HDFS-6493) Propose to change dfs.namenode.startup.delay.block.deletion to second instead of millisecond

2014-07-03 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14051856#comment-14051856 ] Andrew Wang commented on HDFS-6493: --- +1 LGTM, will commit this shortly. Propose to

[jira] [Updated] (HDFS-6493) Change dfs.namenode.startup.delay.block.deletion to second instead of millisecond

2014-07-03 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6493: -- Summary: Change dfs.namenode.startup.delay.block.deletion to second instead of millisecond (was:

[jira] [Updated] (HDFS-6493) Change dfs.namenode.startup.delay.block.deletion to second instead of millisecond

2014-07-03 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6493: -- Resolution: Fixed Fix Version/s: 2.5.0 Target Version/s: 2.5.0 Status:

[jira] [Commented] (HDFS-6511) BlockManager#computeInvalidateWork() could do nothing

2014-07-03 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14051862#comment-14051862 ] Andrew Wang commented on HDFS-6511: --- +1 LGTM, will commit this shortly.

[jira] [Updated] (HDFS-6511) BlockManager#computeInvalidateWork() could do nothing

2014-07-03 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6511?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6511: -- Resolution: Fixed Fix Version/s: 2.6.0 Status: Resolved (was: Patch Available)

[jira] [Commented] (HDFS-6516) List of Encryption Zones should be based on inodes

2014-07-03 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14052039#comment-14052039 ] Andrew Wang commented on HDFS-6516: --- +1 List of Encryption Zones should be based on

[jira] [Commented] (HDFS-6633) Support reading new data in a being written file until the file is closed

2014-07-07 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14054351#comment-14054351 ] Andrew Wang commented on HDFS-6633: --- Hi Nicholas, few quick questions, - Are you

[jira] [Created] (HDFS-6635) Refactor encryption zone functionality into new EncryptionZoneManager class

2014-07-07 Thread Andrew Wang (JIRA)
Andrew Wang created HDFS-6635: - Summary: Refactor encryption zone functionality into new EncryptionZoneManager class Key: HDFS-6635 URL: https://issues.apache.org/jira/browse/HDFS-6635 Project: Hadoop

[jira] [Work started] (HDFS-6635) Refactor encryption zone functionality into new EncryptionZoneManager class

2014-07-07 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-6635 started by Andrew Wang. Refactor encryption zone functionality into new EncryptionZoneManager class

[jira] [Updated] (HDFS-6635) Refactor encryption zone functionality into new EncryptionZoneManager class

2014-07-07 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6635: -- Attachment: hdfs-6635.001.patch Patch attached. Pretty straightforward move. I also took the

[jira] [Updated] (HDFS-6635) Refactor encryption zone functionality into new EncryptionZoneManager class

2014-07-07 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6635: -- Attachment: hdfs-6635.002.patch Thanks for reviewing Charles, new patch attached. Will commit this

[jira] [Resolved] (HDFS-6635) Refactor encryption zone functionality into new EncryptionZoneManager class

2014-07-07 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HDFS-6635. --- Resolution: Fixed Fix Version/s: fs-encryption (HADOOP-10150 and HDFS-6134) Committed, thanks

[jira] [Commented] (HDFS-6634) inotify in HDFS

2014-07-08 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14055611#comment-14055611 ] Andrew Wang commented on HDFS-6634: --- Hi James, I took a quick look through the doc, had

[jira] [Created] (HDFS-6650) API to get the root of an encryption zone for a path

2014-07-09 Thread Andrew Wang (JIRA)
Andrew Wang created HDFS-6650: - Summary: API to get the root of an encryption zone for a path Key: HDFS-6650 URL: https://issues.apache.org/jira/browse/HDFS-6650 Project: Hadoop HDFS Issue Type:

[jira] [Created] (HDFS-6653) Delegate KeyProviderExtension#toString to underlying KeyProvider

2014-07-09 Thread Andrew Wang (JIRA)
Andrew Wang created HDFS-6653: - Summary: Delegate KeyProviderExtension#toString to underlying KeyProvider Key: HDFS-6653 URL: https://issues.apache.org/jira/browse/HDFS-6653 Project: Hadoop HDFS

[jira] [Updated] (HDFS-6653) Delegate KeyProviderExtension#toString to underlying KeyProvider

2014-07-09 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6653: -- Status: Patch Available (was: Open) Delegate KeyProviderExtension#toString to underlying KeyProvider

[jira] [Updated] (HDFS-6653) Delegate KeyProviderExtension#toString to underlying KeyProvider

2014-07-09 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6653: -- Attachment: HDFS-6653.001.patch Patch attached, tested by looking at the toString output Delegate

[jira] [Updated] (HDFS-6474) Namenode needs to get the actual keys and iv from the KeyProvider

2014-07-10 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6474: -- Attachment: hdfs-6474.001.patch Here's a mildly tested patch that implements the EDEK scheme with the

[jira] [Commented] (HDFS-6474) Namenode needs to get the actual keys and iv from the KeyProvider

2014-07-10 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14057247#comment-14057247 ] Andrew Wang commented on HDFS-6474: --- I'll give this my own re-review tomorrow, but I

[jira] [Commented] (HDFS-6474) Namenode needs to get the actual keys and iv from the KeyProvider

2014-07-10 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14058209#comment-14058209 ] Andrew Wang commented on HDFS-6474: --- Hey Charles, thanks for taking a look. I

[jira] [Created] (HDFS-6668) Add and expose KeyProvider-related metrics on NameNode

2014-07-11 Thread Andrew Wang (JIRA)
Andrew Wang created HDFS-6668: - Summary: Add and expose KeyProvider-related metrics on NameNode Key: HDFS-6668 URL: https://issues.apache.org/jira/browse/HDFS-6668 Project: Hadoop HDFS Issue

[jira] [Updated] (HDFS-6474) Namenode needs to get the actual keys and iv from the KeyProvider

2014-07-11 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6474: -- Attachment: hdfs-6474.002.patch Thanks for the reviews all. New patch attached incorporating Charles'

[jira] [Updated] (HDFS-6474) Namenode needs to get the actual keys and iv from the KeyProvider

2014-07-11 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6474: -- Attachment: hdfs-6474.002.patch new patch, fixed a small typo Namenode needs to get the actual keys

[jira] [Updated] (HDFS-6474) Namenode needs to get the actual keys and iv from the KeyProvider

2014-07-11 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6474: -- Attachment: (was: hdfs-6474.002.patch) Namenode needs to get the actual keys and iv from the

[jira] [Updated] (HDFS-6474) Namenode needs to get the actual keys and iv from the KeyProvider

2014-07-11 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6474: -- Attachment: hdfs-6474.003.patch New patch attached with doc fix, will commit shortly. Namenode needs

[jira] [Resolved] (HDFS-6474) Namenode needs to get the actual keys and iv from the KeyProvider

2014-07-11 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HDFS-6474. --- Resolution: Fixed Fix Version/s: fs-encryption (HADOOP-10150 and HDFS-6134) Namenode needs to

[jira] [Commented] (HDFS-6669) Fix 1 space misalignment in FileSystem class

2014-07-11 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14059417#comment-14059417 ] Andrew Wang commented on HDFS-6669: --- Hi Henry, This is the kind of thing we'd actually

[jira] [Updated] (HDFS-6669) Fix 1 space misalignment in FileSystem class

2014-07-11 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6669: -- Resolution: Won't Fix Status: Resolved (was: Patch Available) After discussion with Henry, we

[jira] [Commented] (HDFS-6678) MiniDFSCluster may still be partially running after initialization fails.

2014-07-14 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14061167#comment-14061167 ] Andrew Wang commented on HDFS-6678: --- +1 LGTM MiniDFSCluster may still be partially

[jira] [Updated] (HDFS-6619) Clean up encryption-related tests

2014-07-15 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6619: -- Attachment: hdfs-6619.001.patch Patch attached. High level sketch of changes: - TestHDFSEncryption

[jira] [Work started] (HDFS-6619) Clean up encryption-related tests

2014-07-15 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-6619 started by Andrew Wang. Clean up encryption-related tests - Key: HDFS-6619 URL:

[jira] [Updated] (HDFS-6509) distcp vs Data At Rest Encryption

2014-07-15 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6509: -- Affects Version/s: fs-encryption (HADOOP-10150 and HDFS-6134) distcp vs Data At Rest Encryption

[jira] [Commented] (HDFS-6134) Transparent data at rest encryption

2014-07-15 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14061756#comment-14061756 ] Andrew Wang commented on HDFS-6134: --- Charles posted a design doc for how distcp will work

[jira] [Resolved] (HDFS-6619) Clean up encryption-related tests

2014-07-15 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HDFS-6619. --- Resolution: Fixed Fix Version/s: fs-encryption (HADOOP-10150 and HDFS-6134) Thanks for the

[jira] [Commented] (HDFS-6688) Hadoop JMX stats are not refreshed

2014-07-15 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14062640#comment-14062640 ] Andrew Wang commented on HDFS-6688: --- Hi Biju, did you wait 10.5 minutes for the default

[jira] [Created] (HDFS-6690) Deduplicate xattr names in memory

2014-07-15 Thread Andrew Wang (JIRA)
Andrew Wang created HDFS-6690: - Summary: Deduplicate xattr names in memory Key: HDFS-6690 URL: https://issues.apache.org/jira/browse/HDFS-6690 Project: Hadoop HDFS Issue Type: Improvement

[jira] [Commented] (HDFS-6688) Hadoop JMX stats are not refreshed

2014-07-15 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14062771#comment-14062771 ] Andrew Wang commented on HDFS-6688: --- Hi Biju, The 10.5 minute dead node timeout has been

[jira] [Commented] (HDFS-6688) Hadoop JMX stats are not refreshed

2014-07-15 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14062772#comment-14062772 ] Andrew Wang commented on HDFS-6688: --- Also, please feel free to email the hdfs-dev@ list

[jira] [Commented] (HDFS-5464) Simplify block report diff calculation

2014-07-15 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-5464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14062798#comment-14062798 ] Andrew Wang commented on HDFS-5464: --- Considering we'll have 8 or 10TB disks in a few

[jira] [Updated] (HDFS-6690) Deduplicate xattr names in memory

2014-07-15 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6690: -- Attachment: hdfs-6690.001.patch Patch attached. I looked at some heap dumps to verify. With an image

[jira] [Updated] (HDFS-6690) Deduplicate xattr names in memory

2014-07-15 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6690: -- Status: Patch Available (was: Open) Deduplicate xattr names in memory

[jira] [Commented] (HDFS-6690) Deduplicate xattr names in memory

2014-07-15 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14062866#comment-14062866 ] Andrew Wang commented on HDFS-6690: --- [~cmccabe] or [~umamaheswararao], do you mind taking

[jira] [Resolved] (HDFS-6688) Hadoop JMX stats are not refreshed

2014-07-15 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HDFS-6688. --- Resolution: Not a Problem After discussion with Biju, we've decided that the stale nodes metric is

[jira] [Commented] (HDFS-6688) Hadoop JMX stats are not refreshed

2014-07-15 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14062875#comment-14062875 ] Andrew Wang commented on HDFS-6688: --- I admit we don't have the greatest (any?)

[jira] [Commented] (HDFS-6690) Deduplicate xattr names in memory

2014-07-15 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14062886#comment-14062886 ] Andrew Wang commented on HDFS-6690: --- It's a private variable, so I don't think it matters

[jira] [Created] (HDFS-6692) Add more HDFS encryption tests

2014-07-15 Thread Andrew Wang (JIRA)
Andrew Wang created HDFS-6692: - Summary: Add more HDFS encryption tests Key: HDFS-6692 URL: https://issues.apache.org/jira/browse/HDFS-6692 Project: Hadoop HDFS Issue Type: Sub-task Affects

[jira] [Commented] (HDFS-6405) Test Crypto streams in HDFS

2014-07-15 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14062986#comment-14062986 ] Andrew Wang commented on HDFS-6405: --- Hi Yi, overall looks good, few small comments: *

[jira] [Commented] (HDFS-6690) Deduplicate xattr names in memory

2014-07-16 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14063850#comment-14063850 ] Andrew Wang commented on HDFS-6690: --- Oops, we should wait for Jenkins first :)

[jira] [Updated] (HDFS-6690) Deduplicate xattr names in memory

2014-07-16 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6690: -- Attachment: hdfs-6690.002.patch Okay, switched it out to a Map. Will commit this shortly, thanks

[jira] [Commented] (HDFS-6422) getfattr in CLI doesn't throw exception or return non-0 return code when xattr doesn't exist

2014-07-16 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14063883#comment-14063883 ] Andrew Wang commented on HDFS-6422: --- Hey Charles, the TestWebHDFSXAttr test failures are

[jira] [Resolved] (HDFS-6405) Test Crypto streams in HDFS

2014-07-16 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HDFS-6405. --- Resolution: Fixed +1 thanks Yi, I committed this to the branch. Test Crypto streams in HDFS

[jira] [Commented] (HDFS-6546) Add non-superuser capability to get the encryption zone for a specific path

2014-07-16 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14063934#comment-14063934 ] Andrew Wang commented on HDFS-6546: --- We had a request from Hive for this feature. For a

[jira] [Commented] (HDFS-6509) distcp vs Data At Rest Encryption

2014-07-16 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14063947#comment-14063947 ] Andrew Wang commented on HDFS-6509: --- Hi Yi, thanks for looking over the doc. Some points

[jira] [Commented] (HDFS-6696) Name node cannot start if the path of a file under construction contains .snapshot

2014-07-16 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14063977#comment-14063977 ] Andrew Wang commented on HDFS-6696: --- Thanks for finding this Kihwal, lemme take a look.

[jira] [Assigned] (HDFS-6696) Name node cannot start if the path of a file under construction contains .snapshot

2014-07-16 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang reassigned HDFS-6696: - Assignee: Andrew Wang Name node cannot start if the path of a file under construction contains

[jira] [Commented] (HDFS-6690) Deduplicate xattr names in memory

2014-07-16 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14063983#comment-14063983 ] Andrew Wang commented on HDFS-6690: --- I think throwing down FSDirectory lock asserts in

[jira] [Updated] (HDFS-6422) getfattr in CLI doesn't throw exception or return non-0 return code when xattr doesn't exist

2014-07-16 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6422: -- Priority: Blocker (was: Major) I looked into this more with [~clamb] and there's a compatibility

[jira] [Updated] (HDFS-6690) Deduplicate xattr names in memory

2014-07-16 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6690: -- Attachment: hdfs-6690.003.patch I chatted with Colin a bit offline, and we agreed that adding some docs

[jira] [Commented] (HDFS-6696) Name node cannot start if the path of a file under construction contains .snapshot

2014-07-16 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14064206#comment-14064206 ] Andrew Wang commented on HDFS-6696: --- Poking around on JIRA, snapshots were added in

[jira] [Commented] (HDFS-6696) Name node cannot start if the path of a file under construction contains .snapshot

2014-07-16 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14064331#comment-14064331 ] Andrew Wang commented on HDFS-6696: --- Hey [~kihwal], what Hadoop version were you starting

[jira] [Commented] (HDFS-6626) Node is marked decommissioned if it becomes dead when it is being decommissioned

2014-07-16 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14064408#comment-14064408 ] Andrew Wang commented on HDFS-6626: --- Hi Ming, I think the main goal of decommissioning

[jira] [Updated] (HDFS-6690) Deduplicate xattr names in memory

2014-07-16 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6690: -- Resolution: Fixed Fix Version/s: 2.6.0 Status: Resolved (was: Patch Available)

[jira] [Commented] (HDFS-6268) Better sorting in NetworkTopology#pseudoSortByDistance when no local node is found

2014-07-17 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14065725#comment-14065725 ] Andrew Wang commented on HDFS-6268: --- Hi Ashwin, I think it's okay to make it fully

[jira] [Commented] (HDFS-6701) Make seed optional in NetworkTopology#sortByDistance

2014-07-18 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-6701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14066575#comment-14066575 ] Andrew Wang commented on HDFS-6701: --- Hi Ashwin, thanks for working on this. Overall looks

  1   2   3   4   5   6   7   8   9   10   >