[jira] [Commented] (HDFS-11689) New exception thrown by (private) DFSClient API isHDFSEncryptionEnabled broke hacky hive code

2017-04-21 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15979089#comment-15979089 ] Andrew Wang commented on HDFS-11689: Adding Deprecated will make Hive's build print a deprecation

[jira] [Commented] (HDFS-11402) HDFS Snapshots should capture point-in-time copies of OPEN files

2017-04-20 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977967#comment-15977967 ] Andrew Wang commented on HDFS-11402: Hi Manoj, took another look, mostly just code cleanup-type

[jira] [Created] (HDFS-11687) Add new public encryption APIs required by Hive

2017-04-20 Thread Andrew Wang (JIRA)
Andrew Wang created HDFS-11687: -- Summary: Add new public encryption APIs required by Hive Key: HDFS-11687 URL: https://issues.apache.org/jira/browse/HDFS-11687 Project: Hadoop HDFS Issue Type:

[jira] [Commented] (HDFS-11643) Balancer fencing fails when writing erasure coded lock file

2017-04-20 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977205#comment-15977205 ] Andrew Wang commented on HDFS-11643: Looks pretty close, wanted to clarify a few renames, +1 pending

[jira] [Updated] (HDFS-11660) TestFsDatasetCache#testPageRounder fails intermittently with AssertionError

2017-04-19 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11660: --- Resolution: Fixed Fix Version/s: 3.0.0-alpha3 2.9.0 Status:

[jira] [Updated] (HDFS-11671) TestReconstructStripedBlocks#test2RecoveryTasksForSameBlockGroup fails

2017-04-19 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11671: --- Resolution: Fixed Fix Version/s: 3.0.0-alpha3 Status: Resolved (was: Patch

[jira] [Updated] (HDFS-11660) TestFsDatasetCache#testPageRounder fails intermittently with AssertionError

2017-04-19 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11660: --- Attachment: HDFS-11660.003.patch Thanks for the review John, good points! This also reminded me that

[jira] [Commented] (HDFS-11643) Balancer fencing fails when writing erasure coded lock file

2017-04-19 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15975451#comment-15975451 ] Andrew Wang commented on HDFS-11643: Hi Sammi, took a deeper look at this, mostly just nitty and API

[jira] [Resolved] (HDFS-11683) getErasureCodingPolicyByName should throw a checked exception

2017-04-19 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HDFS-11683. Resolution: Not A Problem Looking more, it seems like this is actually pretty common in our API.

[jira] [Commented] (HDFS-11683) getErasureCodingPolicyByName should throw a checked exception

2017-04-19 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15975424#comment-15975424 ] Andrew Wang commented on HDFS-11683: I also see this same thing happening in setNewINodeStoragePolicy

[jira] [Commented] (HDFS-11683) getErasureCodingPolicyByName should throw a checked exception

2017-04-19 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15975414#comment-15975414 ] Andrew Wang commented on HDFS-11683: I think an InvalidRequestException might be more appropriate,

[jira] [Created] (HDFS-11683) getErasureCodingPolicyByName should throw a checked exception

2017-04-19 Thread Andrew Wang (JIRA)
Andrew Wang created HDFS-11683: -- Summary: getErasureCodingPolicyByName should throw a checked exception Key: HDFS-11683 URL: https://issues.apache.org/jira/browse/HDFS-11683 Project: Hadoop HDFS

[jira] [Commented] (HDFS-11682) TestBalancer#testBalancerWithStripedFile is flaky

2017-04-19 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15975282#comment-15975282 ] Andrew Wang commented on HDFS-11682: Eddy expressed interest in fixing this, assigned to him. >

[jira] [Assigned] (HDFS-11682) TestBalancer#testBalancerWithStripedFile is flaky

2017-04-19 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang reassigned HDFS-11682: -- Assignee: Lei (Eddy) Xu > TestBalancer#testBalancerWithStripedFile is flaky >

[jira] [Updated] (HDFS-11682) TestBalancer#testBalancerWithStripedFile is flaky

2017-04-19 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11682: --- Attachment: IndexOutOfBoundsException.log timeout.log Attached logs from precommit

[jira] [Created] (HDFS-11682) TestBalancer#testBalancerWithStripedFile is flaky

2017-04-19 Thread Andrew Wang (JIRA)
Andrew Wang created HDFS-11682: -- Summary: TestBalancer#testBalancerWithStripedFile is flaky Key: HDFS-11682 URL: https://issues.apache.org/jira/browse/HDFS-11682 Project: Hadoop HDFS Issue

[jira] [Updated] (HDFS-11671) TestReconstructStripedBlocks#test2RecoveryTasksForSameBlockGroup fails

2017-04-19 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11671: --- Attachment: HDFS-11671.002.patch Thanks John for reviewing! New patch, I tested compilation locally.

[jira] [Updated] (HDFS-11671) TestReconstructStripedBlocks#test2RecoveryTasksForSameBlockGroup fails

2017-04-18 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11671: --- Attachment: HDFS-11671.001.patch This was a fun one to trace down. I had a pretty reliable repro on

[jira] [Assigned] (HDFS-11671) TestReconstructStripedBlocks#test2RecoveryTasksForSameBlockGroup fails

2017-04-18 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang reassigned HDFS-11671: -- Assignee: Andrew Wang > TestReconstructStripedBlocks#test2RecoveryTasksForSameBlockGroup fails

[jira] [Updated] (HDFS-11671) TestReconstructStripedBlocks#test2RecoveryTasksForSameBlockGroup fails

2017-04-18 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11671: --- Status: Patch Available (was: Open) >

[jira] [Created] (HDFS-11672) Fix some misleading log messages related to EC block groups

2017-04-18 Thread Andrew Wang (JIRA)
Andrew Wang created HDFS-11672: -- Summary: Fix some misleading log messages related to EC block groups Key: HDFS-11672 URL: https://issues.apache.org/jira/browse/HDFS-11672 Project: Hadoop HDFS

[jira] [Updated] (HDFS-11660) TestFsDatasetCache#testPageRounder fails intermittently with AssertionError

2017-04-18 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11660: --- Attachment: HDFS-11660.002.patch New patch attached, uses a reader/writer lock which is simpler than

[jira] [Commented] (HDFS-11671) TestReconstructStripedBlocks#test2RecoveryTasksForSameBlockGroup fails

2017-04-18 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15973653#comment-15973653 ] Andrew Wang commented on HDFS-11671: Linking to an older JIRA with other test failures in this same

[jira] [Created] (HDFS-11671) TestReconstructStripedBlocks#test2RecoveryTasksForSameBlockGroup fails

2017-04-18 Thread Andrew Wang (JIRA)
Andrew Wang created HDFS-11671: -- Summary: TestReconstructStripedBlocks#test2RecoveryTasksForSameBlockGroup fails Key: HDFS-11671 URL: https://issues.apache.org/jira/browse/HDFS-11671 Project: Hadoop

[jira] [Commented] (HDFS-11665) HttpFSServerWebServer$deprecateEnv may leak secret

2017-04-18 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15973002#comment-15973002 ] Andrew Wang commented on HDFS-11665: +1 thanks John! > HttpFSServerWebServer$deprecateEnv may leak

[jira] [Assigned] (HDFS-7048) Incorrect Dispatcher#Source wait/notify leads to early termination

2017-04-18 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-7048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang reassigned HDFS-7048: - Assignee: (was: Chengbing Liu) > Incorrect Dispatcher#Source wait/notify leads to early

[jira] [Updated] (HDFS-11661) GetContentSummary uses excessive amounts of memory

2017-04-18 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11661: --- Affects Version/s: 3.0.0-alpha2 > GetContentSummary uses excessive amounts of memory >

[jira] [Updated] (HDFS-11661) GetContentSummary uses excessive amounts of memory

2017-04-18 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11661: --- Target Version/s: 3.0.0-alpha3, 2.8.1 (was: 2.8.1) > GetContentSummary uses excessive amounts of

[jira] [Updated] (HDFS-11652) Improve ECSchema and ErasureCodingPolicy toString, hashCode, equals

2017-04-17 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11652: --- Resolution: Fixed Fix Version/s: 3.0.0-alpha3 Status: Resolved (was: Patch

[jira] [Updated] (HDFS-11643) Balancer fencing fails when writing erasure coded lock file

2017-04-17 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11643: --- Priority: Blocker (was: Critical) Bumping to blocker for visibility. Looks like the patch needs a

[jira] [Updated] (HDFS-11644) DFSStripedOutputStream should not implement Syncable

2017-04-17 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11644: --- Priority: Blocker (was: Major) Bumping this to a blocker for visibility. Important for API

[jira] [Updated] (HDFS-11660) TestFsDataset#testPageRounder fails intermittently with AssertionError

2017-04-17 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11660: --- Attachment: HDFS-11660.001.patch Mockito mocking isn't threadsafe, so I think this happens when we

[jira] [Updated] (HDFS-11660) TestFsDatasetCache#testPageRounder fails intermittently with AssertionError

2017-04-17 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11660: --- Status: Patch Available (was: Open) > TestFsDatasetCache#testPageRounder fails intermittently with

[jira] [Updated] (HDFS-11660) TestFsDatasetCache#testPageRounder fails intermittently with AssertionError

2017-04-17 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11660: --- Summary: TestFsDatasetCache#testPageRounder fails intermittently with AssertionError (was:

[jira] [Created] (HDFS-11660) TestFsDataset#testPageRounder fails intermittently with AssertionError

2017-04-17 Thread Andrew Wang (JIRA)
Andrew Wang created HDFS-11660: -- Summary: TestFsDataset#testPageRounder fails intermittently with AssertionError Key: HDFS-11660 URL: https://issues.apache.org/jira/browse/HDFS-11660 Project: Hadoop

[jira] [Updated] (HDFS-11652) Improve ECSchema and ErasureCodingPolicy toString, hashCode, equals

2017-04-17 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11652: --- Attachment: HDFS-11652.004.patch Thanks for reviewing Wei-Chiu and Rakesh! New patch attached. >

[jira] [Commented] (HDFS-11402) HDFS Snapshots should capture point-in-time copies of OPEN files

2017-04-14 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969728#comment-15969728 ] Andrew Wang commented on HDFS-11402: Hi Manoj, thanks for working on this. I gave it a nit pass, still

[jira] [Assigned] (HDFS-11654) dfs.nameservice.id and dfs.ha.namenode.id are being ignored by HDFS clients

2017-04-13 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang reassigned HDFS-11654: -- Assignee: Andre Araujo > dfs.nameservice.id and dfs.ha.namenode.id are being ignored by HDFS

[jira] [Updated] (HDFS-11652) Improve ECSchema and ErasureCodingPolicy toString, hashCode, equals

2017-04-13 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11652: --- Attachment: HDFS-11652.003.patch Thanks for reviewing, good catch! I thought it just wanted odd

[jira] [Commented] (HDFS-11644) DFSStripedOutputStream should not implement Syncable

2017-04-13 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968245#comment-15968245 ] Andrew Wang commented on HDFS-11644: Hi Steve, Stack, thanks for replying! bq. I'd advocate what

[jira] [Updated] (HDFS-11652) Improve ECSchema and ErasureCodingPolicy toString, hashCode, equals

2017-04-12 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11652: --- Attachment: HDFS-11652.002.patch Few mistakes crept in, 002 patch. > Improve ECSchema and

[jira] [Updated] (HDFS-11652) Improve ECSchema and ErasureCodingPolicy toString, hashCode, equals

2017-04-12 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11652: --- Attachment: HDFS-11652.001.patch > Improve ECSchema and ErasureCodingPolicy toString, hashCode,

[jira] [Commented] (HDFS-11652) Improve ECSchema and ErasureCodingPolicy toString, hashCode, equals

2017-04-12 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15966986#comment-15966986 ] Andrew Wang commented on HDFS-11652: [~jojochuang] you noticed the ID issue over on another JIRA, mind

[jira] [Updated] (HDFS-11652) Improve ECSchema and ErasureCodingPolicy toString, hashCode, equals

2017-04-12 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11652: --- Attachment: (was: HDFS-11652.001.patch) > Improve ECSchema and ErasureCodingPolicy toString,

[jira] [Updated] (HDFS-11652) Improve ECSchema and ErasureCodingPolicy toString, hashCode, equals

2017-04-12 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11652: --- Status: Patch Available (was: Open) > Improve ECSchema and ErasureCodingPolicy toString, hashCode,

[jira] [Updated] (HDFS-11652) Improve ECSchema and ErasureCodingPolicy toString, hashCode, equals

2017-04-12 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11652: --- Attachment: HDFS-11652.001.patch Patch attached. * Switched over to using EqualsBuilder and

[jira] [Created] (HDFS-11652) Improve ECSchema and ErasureCodingPolicy toString, hashCode, equals

2017-04-12 Thread Andrew Wang (JIRA)
Andrew Wang created HDFS-11652: -- Summary: Improve ECSchema and ErasureCodingPolicy toString, hashCode, equals Key: HDFS-11652 URL: https://issues.apache.org/jira/browse/HDFS-11652 Project: Hadoop HDFS

[jira] [Commented] (HDFS-11644) DFSStripedOutputStream should not implement Syncable

2017-04-12 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15966925#comment-15966925 ] Andrew Wang commented on HDFS-11644: Good idea on isSyncable. I'm still peeved that checking

[jira] [Commented] (HDFS-11644) DFSStripedOutputStream should not implement Syncable

2017-04-12 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15966723#comment-15966723 ] Andrew Wang commented on HDFS-11644: Looking at this, it's not pretty. FileSystem returns an

[jira] [Commented] (HDFS-11643) Balancer fencing fails when writing erasure coded lock file

2017-04-12 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15966691#comment-15966691 ] Andrew Wang commented on HDFS-11643: Agree, at least for the balancer, I think for now we only need a

[jira] [Commented] (HDFS-9260) Improve the performance and GC friendliness of NameNode startup and full block reports

2017-04-12 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15966680#comment-15966680 ] Andrew Wang commented on HDFS-9260: --- One other question from our team, what's the typical block size on a

[jira] [Commented] (HDFS-11644) DFSStripedOutputStream should not implement Syncable

2017-04-12 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15966557#comment-15966557 ] Andrew Wang commented on HDFS-11644: [~manojg] want to pick this one up? > DFSStripedOutputStream

[jira] [Commented] (HDFS-11615) FSNamesystemLock metrics can be inaccurate due to millisecond precision

2017-04-12 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15966470#comment-15966470 ] Andrew Wang commented on HDFS-11615: Hi Erik, What's your proposal for new names? I'm guessing that

[jira] [Commented] (HDFS-11651) Add a public API for specifying an EC policy at create time

2017-04-12 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15966460#comment-15966460 ] Andrew Wang commented on HDFS-11651: One idea is that we add a new public interface to the builder for

[jira] [Commented] (HDFS-10996) Ability to specify per-file EC policy at create time

2017-04-12 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15966461#comment-15966461 ] Andrew Wang commented on HDFS-10996: FYI that I filed HDFS-11651 to expose this as a public API. I

[jira] [Created] (HDFS-11651) Add a public API for specifying an EC policy at create time

2017-04-12 Thread Andrew Wang (JIRA)
Andrew Wang created HDFS-11651: -- Summary: Add a public API for specifying an EC policy at create time Key: HDFS-11651 URL: https://issues.apache.org/jira/browse/HDFS-11651 Project: Hadoop HDFS

[jira] [Updated] (HDFS-10996) Ability to specify per-file EC policy at create time

2017-04-12 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-10996: --- Resolution: Fixed Fix Version/s: 3.0.0-alpha3 Status: Resolved (was: Patch

[jira] [Commented] (HDFS-11565) Use compact identifiers for built-in ECPolicies in HdfsFileStatus

2017-04-12 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15966451#comment-15966451 ] Andrew Wang commented on HDFS-11565: Committed to trunk, thanks again Wei-chiu for reviewing. > Use

[jira] [Updated] (HDFS-11565) Use compact identifiers for built-in ECPolicies in HdfsFileStatus

2017-04-12 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11565: --- Resolution: Fixed Fix Version/s: 3.0.0-alpha3 Release Note: Some of the existing fields in

[jira] [Commented] (HDFS-11565) Use compact identifiers for built-in ECPolicies in HdfsFileStatus

2017-04-12 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15966445#comment-15966445 ] Andrew Wang commented on HDFS-11565: I'll fix the unused import on commit, thanks for reviewing

[jira] [Commented] (HDFS-9260) Improve the performance and GC friendliness of NameNode startup and full block reports

2017-04-12 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15966423#comment-15966423 ] Andrew Wang commented on HDFS-9260: --- Hi Daryn, I talked to our people who help run this large customer

[jira] [Commented] (HDFS-10531) Add EC policy and storage policy related usage summarization function to dfs du command

2017-04-11 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15965223#comment-15965223 ] Andrew Wang commented on HDFS-10531: Hi Sammi, thanks for the comment, bq. From user's point of view,

[jira] [Resolved] (HDFS-11552) Erasure Coding: Support Parity Blocks placement onto same nodes hosting Data Blocks when DataNodes are insufficient

2017-04-11 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HDFS-11552. Resolution: Not A Problem > Erasure Coding: Support Parity Blocks placement onto same nodes hosting

[jira] [Commented] (HDFS-11552) Erasure Coding: Support Parity Blocks placement onto same nodes hosting Data Blocks when DataNodes are insufficient

2017-04-11 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15965214#comment-15965214 ] Andrew Wang commented on HDFS-11552: I think with the XOR-2-1 policy this also becomes less important,

[jira] [Commented] (HDFS-9342) Erasure coding: client should update and commit block based on acknowledged size

2017-04-11 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15965164#comment-15965164 ] Andrew Wang commented on HDFS-9342: --- LGTM, thanks Sammi for refining, only some minors: * Please add a

[jira] [Created] (HDFS-11644) DFSStripedOutputStream should not implement Syncable

2017-04-11 Thread Andrew Wang (JIRA)
Andrew Wang created HDFS-11644: -- Summary: DFSStripedOutputStream should not implement Syncable Key: HDFS-11644 URL: https://issues.apache.org/jira/browse/HDFS-11644 Project: Hadoop HDFS Issue

[jira] [Commented] (HDFS-10996) Ability to specify per-file EC policy at create time

2017-04-11 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15965109#comment-15965109 ] Andrew Wang commented on HDFS-10996: Hi Sammi, thanks for working on this, sorry for the slow review:

[jira] [Commented] (HDFS-11615) FSNamesystemLock metrics can be inaccurate due to millisecond precision

2017-04-11 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15965062#comment-15965062 ] Andrew Wang commented on HDFS-11615: These metrics are normally consumed by a metrics system and

[jira] [Updated] (HDFS-11565) Use compact identifiers for built-in ECPolicies in HdfsFileStatus

2017-04-11 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11565: --- Attachment: HDFS-11565.003.patch Sure, good idea Wei-chiu, added some Precondition checks and new

[jira] [Commented] (HDFS-11503) Integrate Chocolate Cloud RS coder implementation

2017-04-11 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15965022#comment-15965022 ] Andrew Wang commented on HDFS-11503: Hi [~sw0rdf1sh], We already have assignees actively working on

[jira] [Commented] (HDFS-11643) Balancer fencing fails when writing erasure coded lock file

2017-04-11 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15964941#comment-15964941 ] Andrew Wang commented on HDFS-11643: Heads up [~Sammi], this relates to our discussion about a

[jira] [Created] (HDFS-11643) Balancer fencing fails when writing erasure coded lock file

2017-04-11 Thread Andrew Wang (JIRA)
Andrew Wang created HDFS-11643: -- Summary: Balancer fencing fails when writing erasure coded lock file Key: HDFS-11643 URL: https://issues.apache.org/jira/browse/HDFS-11643 Project: Hadoop HDFS

[jira] [Commented] (HDFS-11565) Use compact identifiers for built-in ECPolicies in HdfsFileStatus

2017-04-10 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15963588#comment-15963588 ] Andrew Wang commented on HDFS-11565: There seems to be a versioning issue where the build is picking

[jira] [Commented] (HDFS-11637) Fix javac warning caused by the deprecated key used in TestDFSClientRetries#testFailuresArePerOperation

2017-04-10 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15963460#comment-15963460 ] Andrew Wang commented on HDFS-11637: +1 thanks [~linyiqun] > Fix javac warning caused by the

[jira] [Commented] (HDFS-11615) FSNamesystemLock metrics can be inaccurate due to millisecond precision

2017-04-10 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15963454#comment-15963454 ] Andrew Wang commented on HDFS-11615: If we have nanosecond precision, is there a reason to use

[jira] [Commented] (HDFS-11565) Use compact identifiers for built-in ECPolicies in HdfsFileStatus

2017-04-10 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15963323#comment-15963323 ] Andrew Wang commented on HDFS-11565: Could a reviewer try running TestPBHelper for me as well? It

[jira] [Updated] (HDFS-11565) Use compact identifiers for built-in ECPolicies in HdfsFileStatus

2017-04-07 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11565: --- Attachment: HDFS-11565.002.patch Patch attached. I also reverted the change to the PB tag numbers,

[jira] [Updated] (HDFS-11623) Move system erasure coding policies into hadoop-hdfs-client

2017-04-07 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11623: --- Resolution: Fixed Fix Version/s: 3.0.0-alpha3 Status: Resolved (was: Patch

[jira] [Commented] (HDFS-11623) Move system erasure coding policies into hadoop-hdfs-client

2017-04-07 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961568#comment-15961568 ] Andrew Wang commented on HDFS-11623: Filed HADOOP-14293 for the TestFsDatasetImpl failure, others are

[jira] [Updated] (HDFS-11633) FSImage failover disables all erasure coding policies

2017-04-07 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11633: --- Resolution: Fixed Fix Version/s: 3.0.0-alpha3 Status: Resolved (was: Patch

[jira] [Commented] (HDFS-11633) FSImage failover disables all erasure coding policies

2017-04-07 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961533#comment-15961533 ] Andrew Wang commented on HDFS-11633: +1 LGTM checking this in > FSImage failover disables all erasure

[jira] [Updated] (HDFS-11623) Move system erasure coding policies into hadoop-hdfs-client

2017-04-07 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11623: --- Attachment: HDFS-11623.006.patch Patch attached to fix TestOIV failure, have to use XOR for this. >

[jira] [Commented] (HDFS-11623) Move system erasure coding policies into hadoop-hdfs-client

2017-04-07 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961289#comment-15961289 ] Andrew Wang commented on HDFS-11623: Thanks for reviewing Wei-chiu! New patch to simplify the array as

[jira] [Updated] (HDFS-11623) Move system erasure coding policies into hadoop-hdfs-client

2017-04-07 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11623: --- Attachment: HDFS-11623.005.patch > Move system erasure coding policies into hadoop-hdfs-client >

[jira] [Commented] (HDFS-11633) FSImage failover disables all erasure coding policies

2017-04-07 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15961250#comment-15961250 ] Andrew Wang commented on HDFS-11633: Good catch! Yea, the clear is only supposed to clear things

[jira] [Commented] (HDFS-11623) Move system erasure coding policies into hadoop-hdfs-client

2017-04-06 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960113#comment-15960113 ] Andrew Wang commented on HDFS-11623: Aha, now I understand better your concern. I did an IDE refactor

[jira] [Commented] (HDFS-11565) Use compact identifiers for built-in ECPolicies in HdfsFileStatus

2017-04-06 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15960087#comment-15960087 ] Andrew Wang commented on HDFS-11565: Thanks for taking a look Wei-chiu! Like I said in my previous

[jira] [Updated] (HDFS-11623) Move system erasure coding policies into hadoop-hdfs-client

2017-04-06 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11623: --- Attachment: HDFS-11623.004.patch Forgot a closing parens... > Move system erasure coding policies

[jira] [Updated] (HDFS-11623) Move system erasure coding policies into hadoop-hdfs-client

2017-04-06 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11623: --- Attachment: HDFS-11623.003.patch New patch attached renaming to {{SystemErasureCodingPolicies}}. The

[jira] [Updated] (HDFS-11538) Move ClientProtocol HA proxies into hadoop-hdfs-client

2017-04-05 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11538: --- Fix Version/s: (was: 2.9.0) 2.8.1 Cool. I did the requisite JIRA stuff to make

[jira] [Updated] (HDFS-9276) Failed to Update HDFS Delegation Token for long running application in HA mode

2017-04-05 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-9276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-9276: -- Fix Version/s: (was: 2.9.0) 2.8.1 Backported this to 2.8.1, looks pretty critical

[jira] [Commented] (HDFS-11395) RequestHedgingProxyProvider#RequestHedgingInvocationHandler hides the Exception thrown from NameNode

2017-04-05 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15958034#comment-15958034 ] Andrew Wang commented on HDFS-11395: I backported this to branch-2.8 to make another backport easier,

[jira] [Updated] (HDFS-10683) Make class Token$PrivateToken private

2017-04-05 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-10683: --- Target Version/s: 3.0.0-alpha2, 2.9.0 (was: 2.9.0, 3.0.0-alpha2) Fix Version/s: (was:

[jira] [Resolved] (HDFS-11629) Revert HDFS-11431 hadoop-hdfs-client JAR does not include ConfiguredFailoverProxyProvider.

2017-04-05 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved HDFS-11629. Resolution: Fixed Fix Version/s: 2.8.1 > Revert HDFS-11431 hadoop-hdfs-client JAR does not

[jira] [Updated] (HDFS-11395) RequestHedgingProxyProvider#RequestHedgingInvocationHandler hides the Exception thrown from NameNode

2017-04-05 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11395: --- Fix Version/s: (was: 2.9.0) 2.8.1 3.0.0-alpha3 >

[jira] [Created] (HDFS-11629) Revert HDFS-11431 hadoop-hdfs-client JAR does not include ConfiguredFailoverProxyProvider.

2017-04-05 Thread Andrew Wang (JIRA)
Andrew Wang created HDFS-11629: -- Summary: Revert HDFS-11431 hadoop-hdfs-client JAR does not include ConfiguredFailoverProxyProvider. Key: HDFS-11629 URL: https://issues.apache.org/jira/browse/HDFS-11629

[jira] [Commented] (HDFS-11626) Deprecate oiv_legacy tool

2017-04-05 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957998#comment-15957998 ] Andrew Wang commented on HDFS-11626: We should definitely deprecate if it's not already. The name

[jira] [Updated] (HDFS-11596) hadoop-hdfs-client jar is in the wrong directory in release tarball

2017-04-05 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-11596: --- Resolution: Fixed Fix Version/s: 3.0.0-alpha3 Release Note: The scope of hadoop-hdfs's

[jira] [Commented] (HDFS-11596) hadoop-hdfs-client jar is in the wrong directory in release tarball

2017-04-05 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-11596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957974#comment-15957974 ] Andrew Wang commented on HDFS-11596: bq. Don't ask me, I don't understand maven I knew we should have

[jira] [Comment Edited] (HDFS-10848) Move hadoop-hdfs-native-client module into hadoop-hdfs-client

2017-04-05 Thread Andrew Wang (JIRA)
[ https://issues.apache.org/jira/browse/HDFS-10848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15957866#comment-15957866 ] Andrew Wang edited comment on HDFS-10848 at 4/5/17 10:20 PM: -

<    3   4   5   6   7   8   9   10   11   12   >