[jira] [Updated] (HADOOP-13876) S3Guard: better support for multi-bucket access

2017-02-03 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13876: -- Status: Patch Available (was: Open) > S3Guard: better support for multi-bucket access >

[jira] [Updated] (HADOOP-13876) S3Guard: better support for multi-bucket access

2017-02-03 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13876: -- Status: Open (was: Patch Available) > S3Guard: better support for multi-bucket access >

[jira] [Commented] (HADOOP-13904) DynamoDBMetadataStore to handle DDB throttling failures through retry policy

2017-01-31 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847856#comment-15847856 ] Aaron Fabbri commented on HADOOP-13904: --- Thank you for the helpful review [~ste...@apache.org]

[jira] [Commented] (HADOOP-13904) DynamoDBMetadataStore to handle DDB throttling failures through retry policy

2017-01-31 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15848052#comment-15848052 ] Aaron Fabbri commented on HADOOP-13904: --- The time it takes to run these varies depending on scale

[jira] [Commented] (HADOOP-13876) S3Guard: better support for multi-bucket access

2017-01-31 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847825#comment-15847825 ] Aaron Fabbri commented on HADOOP-13876: --- v4 patch tested in US West 2. No new failures, so I

[jira] [Commented] (HADOOP-13904) DynamoDBMetadataStore to handle DDB throttling failures through retry policy

2017-02-01 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15848103#comment-15848103 ] Aaron Fabbri commented on HADOOP-13904: --- Following up on determining whether or not the DynamoDB

[jira] [Updated] (HADOOP-14013) S3Guard: fix multi-bucket integration tests

2017-02-06 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14013: -- Resolution: Fixed Fix Version/s: HADOOP-13345 Status: Resolved (was: Patch

[jira] [Commented] (HADOOP-14028) S3A block output streams don't delete temporary files in multipart uploads

2017-02-06 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854968#comment-15854968 ] Aaron Fabbri commented on HADOOP-14028: --- Quick review of v4 patch {noformat} @@ -392,8 +391,15 @@

[jira] [Commented] (HADOOP-14041) CLI command to prune old metadata

2017-02-06 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854863#comment-15854863 ] Aaron Fabbri commented on HADOOP-14041: --- A couple of thoughts on this patch: 1. I think prune()

[jira] [Commented] (HADOOP-13345) S3Guard: Improved Consistency for S3A

2017-02-08 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15858758#comment-15858758 ] Aaron Fabbri commented on HADOOP-13345: --- S3Guard is specifically designed to allow multiple

[jira] [Commented] (HADOOP-14041) CLI command to prune old metadata

2017-02-08 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15858874#comment-15858874 ] Aaron Fabbri commented on HADOOP-14041: --- Thank you for the patch. Overall it looks pretty good.

[jira] [Commented] (HADOOP-13876) S3Guard: better support for multi-bucket access

2017-02-01 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15848840#comment-15848840 ] Aaron Fabbri commented on HADOOP-13876: --- [~ste...@apache.org] I think I addressed everything here.

[jira] [Updated] (HADOOP-13876) S3Guard: better support for multi-bucket access

2017-01-25 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13876: -- Attachment: HADOOP-13876-HADOOP-13345.001.patch Attaching v001 patch which fixes

[jira] [Commented] (HADOOP-13985) s3guard: add a version marker to every table

2017-01-26 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840258#comment-15840258 ] Aaron Fabbri commented on HADOOP-13985: --- [~ste...@apache.org] would you like me to commit this to

[jira] [Updated] (HADOOP-13876) S3Guard: better support for multi-bucket access

2017-01-26 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13876: -- Status: Patch Available (was: Open) > S3Guard: better support for multi-bucket access >

[jira] [Commented] (HADOOP-14020) Optimize dirListingUnion

2017-01-27 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15843695#comment-15843695 ] Aaron Fabbri commented on HADOOP-14020: --- Ah.. Thank you for the heads up. I have some stuff to

[jira] [Created] (HADOOP-14036) S3Guard: intermittent duplicate item keys failure

2017-01-27 Thread Aaron Fabbri (JIRA)
Aaron Fabbri created HADOOP-14036: - Summary: S3Guard: intermittent duplicate item keys failure Key: HADOOP-14036 URL: https://issues.apache.org/jira/browse/HADOOP-14036 Project: Hadoop Common

[jira] [Updated] (HADOOP-13876) S3Guard: better support for multi-bucket access

2017-01-27 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13876: -- Status: Open (was: Patch Available) > S3Guard: better support for multi-bucket access >

[jira] [Commented] (HADOOP-13876) S3Guard: better support for multi-bucket access

2017-01-27 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15843751#comment-15843751 ] Aaron Fabbri commented on HADOOP-13876: --- Same test results as before (those three failures.. note

[jira] [Updated] (HADOOP-13876) S3Guard: better support for multi-bucket access

2017-01-27 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13876: -- Attachment: HADOOP-13876-HADOOP-13345.002.patch Attaching v2 patch. Just rebases on latest

[jira] [Updated] (HADOOP-13876) S3Guard: better support for multi-bucket access

2017-01-27 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13876: -- Status: Patch Available (was: Open) > S3Guard: better support for multi-bucket access >

[jira] [Commented] (HADOOP-13936) S3Guard: DynamoDB can go out of sync with S3AFileSystem::delete operation

2017-01-25 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15839356#comment-15839356 ] Aaron Fabbri commented on HADOOP-13936: --- This is part of a general class of problems. In general,

[jira] [Commented] (HADOOP-14020) Optimize dirListingUnion

2017-01-25 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15839362#comment-15839362 ] Aaron Fabbri commented on HADOOP-14020: --- Thanks for doing this [~mackrorysd]. Looks really good.

[jira] [Commented] (HADOOP-13876) S3Guard: better support for multi-bucket access including read-only

2017-01-20 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15832303#comment-15832303 ] Aaron Fabbri commented on HADOOP-13876: --- Ok, I will clone this and create a new one that covers

[jira] [Commented] (HADOOP-13876) S3Guard: better support for multi-bucket access

2017-01-20 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15832317#comment-15832317 ] Aaron Fabbri commented on HADOOP-13876: --- Done.. Feel free to tweak descriptions, etc. > S3Guard:

[jira] [Created] (HADOOP-14013) S3Guard: fix multi-bucket integration tests

2017-01-20 Thread Aaron Fabbri (JIRA)
Aaron Fabbri created HADOOP-14013: - Summary: S3Guard: fix multi-bucket integration tests Key: HADOOP-14013 URL: https://issues.apache.org/jira/browse/HADOOP-14013 Project: Hadoop Common

[jira] [Updated] (HADOOP-14013) S3Guard: fix multi-bucket integration tests

2017-01-20 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14013: -- Description: Clone of HADOOP-13876 to address multi-bucket test failures separately. One of

[jira] [Updated] (HADOOP-13876) S3Guard: better support for multi-bucket access including read-only

2017-01-20 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13876: -- Description: HADOOP-13449 adds support for DynamoDBMetadataStore. The code currently

[jira] [Updated] (HADOOP-13876) S3Guard: better support for multi-bucket access

2017-01-20 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13876: -- Summary: S3Guard: better support for multi-bucket access (was: S3Guard: better support for

[jira] [Commented] (HADOOP-13876) S3Guard: better support for multi-bucket access

2017-01-26 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840264#comment-15840264 ] Aaron Fabbri commented on HADOOP-13876: --- [~ste...@apache.org], I propose we get your patch for

[jira] [Updated] (HADOOP-14020) Optimize dirListingUnion

2017-01-26 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14020: -- Status: Open (was: Patch Available) > Optimize dirListingUnion > >

[jira] [Updated] (HADOOP-14020) Optimize dirListingUnion

2017-01-26 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14020: -- Status: Patch Available (was: Open) > Optimize dirListingUnion > >

[jira] [Commented] (HADOOP-14020) Optimize dirListingUnion

2017-01-26 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15840550#comment-15840550 ] Aaron Fabbri commented on HADOOP-14020: --- Thanks! Besides the minor checkstyle issue... {code}

[jira] [Commented] (HADOOP-13985) s3guard: add a version marker to every table

2017-01-25 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15838473#comment-15838473 ] Aaron Fabbri commented on HADOOP-13985: --- Thanks for doing this. I agree this is a must-have.

[jira] [Commented] (HADOOP-14013) S3Guard: fix multi-bucket integration tests

2017-01-24 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15837045#comment-15837045 ] Aaron Fabbri commented on HADOOP-14013: --- This all sounds good to me. BTW, feel free to

[jira] [Commented] (HADOOP-13075) Add support for SSE-KMS and SSE-C in s3a filesystem

2017-01-25 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15838640#comment-15838640 ] Aaron Fabbri commented on HADOOP-13075: --- Thanks for doing this [~moist]. One quick question,

[jira] [Commented] (HADOOP-14087) S3A typo in pom.xml test exclusions

2017-02-15 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15869010#comment-15869010 ] Aaron Fabbri commented on HADOOP-14087: --- {quote} not clean Jenkins... is the patch based on

[jira] [Updated] (HADOOP-14087) S3A typo in pom.xml test exclusions

2017-02-15 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14087: -- Attachment: HADOOP-14087.002.patch v2 patch based off latest trunk. > S3A typo in pom.xml

[jira] [Commented] (HADOOP-13998) initial s3guard preview

2017-02-22 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15877812#comment-15877812 ] Aaron Fabbri commented on HADOOP-13998: --- [~ste...@apache.org] probably merge in latest trunk, do a

[jira] [Commented] (HADOOP-14041) CLI command to prune old metadata

2017-02-20 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15874906#comment-15874906 ] Aaron Fabbri commented on HADOOP-14041: --- Thanks for the folow-up patch [~mackrorysd]. This is

[jira] [Updated] (HADOOP-14041) CLI command to prune old metadata

2017-02-20 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14041: -- Attachment: HADOOP-14041-HADOOP-13345.008.patch > CLI command to prune old metadata >

[jira] [Commented] (HADOOP-13453) S3Guard: Instrument new functionality with Hadoop metrics.

2017-02-20 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15874958#comment-15874958 ] Aaron Fabbri commented on HADOOP-13453: --- I will comment as I see new places in the code that could

[jira] [Commented] (HADOOP-13904) DynamoDBMetadataStore to handle DDB throttling failures through retry policy

2017-02-20 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15874982#comment-15874982 ] Aaron Fabbri commented on HADOOP-13904: --- {quote} there's some TODO in the code. {quote} I'm

[jira] [Updated] (HADOOP-14096) s3guard: regression in dirListingUnion

2017-02-20 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14096: -- Attachment: HADOOP-14096-HADOOP-13345.001.patch Attaching a patch. Includes new unit test

[jira] [Updated] (HADOOP-13904) DynamoDBMetadataStore to handle DDB throttling failures through retry policy

2017-02-20 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13904: -- Attachment: HADOOP-13904-HADOOP-13345.004.patch v4 patch removes TODOs, adds @value javadocs

[jira] [Assigned] (HADOOP-14096) s3guard: regression in dirListingUnion

2017-02-20 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri reassigned HADOOP-14096: - Assignee: Aaron Fabbri > s3guard: regression in dirListingUnion >

[jira] [Commented] (HADOOP-14096) s3guard: regression in dirListingUnion

2017-02-20 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15875074#comment-15875074 ] Aaron Fabbri commented on HADOOP-14096: --- Starting work on the unit test and fix for this. >

[jira] [Updated] (HADOOP-14096) s3guard: regression in dirListingUnion

2017-02-20 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14096: -- Status: Patch Available (was: Open) > s3guard: regression in dirListingUnion >

[jira] [Updated] (HADOOP-14041) CLI command to prune old metadata

2017-02-20 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14041: -- Status: Open (was: Patch Available) > CLI command to prune old metadata >

[jira] [Updated] (HADOOP-14041) CLI command to prune old metadata

2017-02-20 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14041: -- Status: Patch Available (was: Open) > CLI command to prune old metadata >

[jira] [Updated] (HADOOP-14041) CLI command to prune old metadata

2017-02-20 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14041: -- Attachment: HADOOP-14041-HADOOP-13345.009.patch v9 patch.. - Rebase on latest feature branch

[jira] [Updated] (HADOOP-14041) CLI command to prune old metadata

2017-02-20 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14041: -- Resolution: Fixed Status: Resolved (was: Patch Available) Committed to HADOOP-13345

[jira] [Updated] (HADOOP-14096) s3guard: regression in dirListingUnion

2017-02-20 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14096: -- Resolution: Fixed Fix Version/s: HADOOP-13345 Status: Resolved (was: Patch

[jira] [Updated] (HADOOP-13904) DynamoDBMetadataStore to handle DDB throttling failures through retry policy

2017-02-20 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13904: -- Resolution: Fixed Fix Version/s: HADOOP-13345 Status: Resolved (was: Patch

[jira] [Commented] (HADOOP-13914) s3guard: improve S3AFileStatus#isEmptyDirectory handling

2017-02-17 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15872633#comment-15872633 ] Aaron Fabbri commented on HADOOP-13914: --- Ok.. I don't want to step on your toes [~liuml07]. I was

[jira] [Commented] (HADOOP-13345) S3Guard: Improved Consistency for S3A

2017-02-17 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15872384#comment-15872384 ] Aaron Fabbri commented on HADOOP-13345: --- Once we figure out the SSE test failure I'm +1 to do a

[jira] [Commented] (HADOOP-13904) DynamoDBMetadataStore to handle DDB throttling failures through retry policy

2017-02-17 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15872401#comment-15872401 ] Aaron Fabbri commented on HADOOP-13904: --- Jenkins above is clean except HADOOP-14030 (TestKDiag

[jira] [Commented] (HADOOP-14090) Allow users to specify region for DynamoDB table instead of endpoint

2017-02-16 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870984#comment-15870984 ] Aaron Fabbri commented on HADOOP-14090: --- Some quick comments before doing a more in-depth review:

[jira] [Commented] (HADOOP-14068) Add integration test version of TestMetadataStore for DynamoDB

2017-02-16 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870948#comment-15870948 ] Aaron Fabbri commented on HADOOP-14068: --- Thanks for doing this. It is clear there is value in

[jira] [Commented] (HADOOP-14041) CLI command to prune old metadata

2017-02-16 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870761#comment-15870761 ] Aaron Fabbri commented on HADOOP-14041: --- [~mackrorysd] I'm also fine with doing a followup

[jira] [Commented] (HADOOP-14046) Metastore destruction test creates table without version marker

2017-02-16 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870873#comment-15870873 ] Aaron Fabbri commented on HADOOP-14046: --- +1, and committed to feature branch. Thanks for your

[jira] [Updated] (HADOOP-14046) Metastore destruction test creates table without version marker

2017-02-16 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14046: -- Resolution: Fixed Release Note: +1, and committed to feature branch. Thanks for your

[jira] [Updated] (HADOOP-14046) Metastore destruction test creates table without version marker

2017-02-16 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14046: -- Release Note: (was: +1, and committed to feature branch. Thanks for your work on this

[jira] [Commented] (HADOOP-14041) CLI command to prune old metadata

2017-02-16 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870930#comment-15870930 ] Aaron Fabbri commented on HADOOP-14041: --- This usage also doesn't work: {noformat} $ hadoop s3a

[jira] [Commented] (HADOOP-14090) Allow users to specify region for DynamoDB table instead of endpoint

2017-02-17 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15872689#comment-15872689 ] Aaron Fabbri commented on HADOOP-14090: --- +1.. I'll add that I originally suggested we add support

[jira] [Commented] (HADOOP-14094) Rethink S3GuardTool options

2017-02-17 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15872699#comment-15872699 ] Aaron Fabbri commented on HADOOP-14094: --- Yes, s3a as the subcommand is confusing. e.g. {{hadoop

[jira] [Commented] (HADOOP-14041) CLI command to prune old metadata

2017-02-17 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15872846#comment-15872846 ] Aaron Fabbri commented on HADOOP-14041: --- Thanks for the follow-up patch [~mackrorysd]. Looks

[jira] [Commented] (HADOOP-13345) S3Guard: Improved Consistency for S3A

2017-02-17 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15872867#comment-15872867 ] Aaron Fabbri commented on HADOOP-13345: --- Thanks [~liuml07]! {quote} $ mvn -Dit.test='ITestS3A*'

[jira] [Issue Comment Deleted] (HADOOP-14102) Relax error message assertion in S3A test ITestS3AEncryptionSSEC::testCreateFileAndReadWithDifferentEncryptionKey

2017-02-22 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14102: -- Comment: was deleted (was: Thanks for taking the time to do this [~liuml07]. Looks good to

[jira] [Commented] (HADOOP-14102) Relax error message assertion in S3A test ITestS3AEncryptionSSEC::testCreateFileAndReadWithDifferentEncryptionKey

2017-02-22 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879118#comment-15879118 ] Aaron Fabbri commented on HADOOP-14102: --- Thanks for taking the time to do this [~liuml07]. Looks

[jira] [Commented] (HADOOP-14102) Relax error message assertion in S3A test ITestS3AEncryptionSSEC::testCreateFileAndReadWithDifferentEncryptionKey

2017-02-22 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879119#comment-15879119 ] Aaron Fabbri commented on HADOOP-14102: --- Thanks for taking the time to do this [~liuml07]. Looks

[jira] [Commented] (HADOOP-13345) S3Guard: Improved Consistency for S3A

2017-02-22 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15879422#comment-15879422 ] Aaron Fabbri commented on HADOOP-13345: --- Works for me, thanks [~liuml07] > S3Guard: Improved

[jira] [Commented] (HADOOP-14020) Optimize dirListingUnion

2017-02-19 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15874037#comment-15874037 ] Aaron Fabbri commented on HADOOP-14020: --- Shoot, I just noticed a bug this patch introduced.. The

[jira] [Created] (HADOOP-14096) s3guard: regression in dirListingUnion

2017-02-19 Thread Aaron Fabbri (JIRA)
Aaron Fabbri created HADOOP-14096: - Summary: s3guard: regression in dirListingUnion Key: HADOOP-14096 URL: https://issues.apache.org/jira/browse/HADOOP-14096 Project: Hadoop Common Issue

[jira] [Commented] (HADOOP-14096) s3guard: regression in dirListingUnion

2017-02-19 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15874045#comment-15874045 ] Aaron Fabbri commented on HADOOP-14096: --- I was able to exercise the bug by nuking my DDB table

[jira] [Commented] (HADOOP-14096) s3guard: regression in dirListingUnion

2017-02-19 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15874049#comment-15874049 ] Aaron Fabbri commented on HADOOP-14096: --- [~mackrorysd] FYI.. I'm in the middle of some stuff so

[jira] [Commented] (HADOOP-14068) Add integration test version of TestMetadataStore for DynamoDB

2017-02-21 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876895#comment-15876895 ] Aaron Fabbri commented on HADOOP-14068: --- Looks good for the most part.. two things: - Where did

[jira] [Commented] (HADOOP-13345) S3Guard: Improved Consistency for S3A

2017-02-21 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15876515#comment-15876515 ] Aaron Fabbri commented on HADOOP-13345: --- Sorry for delayed response.. {quote}

[jira] [Updated] (HADOOP-13904) DynamoDBMetadataStore to handle DDB throttling failures through retry policy

2017-02-15 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13904: -- Attachment: screenshot-1.png > DynamoDBMetadataStore to handle DDB throttling failures

[jira] [Commented] (HADOOP-13904) DynamoDBMetadataStore to handle DDB throttling failures through retry policy

2017-02-15 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868944#comment-15868944 ] Aaron Fabbri commented on HADOOP-13904: --- I did not find the SDK code that implements retries, but

[jira] [Updated] (HADOOP-13904) DynamoDBMetadataStore to handle DDB throttling failures through retry policy

2017-02-15 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13904: -- Status: Open (was: Patch Available) > DynamoDBMetadataStore to handle DDB throttling

[jira] [Updated] (HADOOP-13904) DynamoDBMetadataStore to handle DDB throttling failures through retry policy

2017-02-15 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13904: -- Status: Patch Available (was: Open) Rebased patch looks the same.. I'll just resubmit to

[jira] [Updated] (HADOOP-13904) DynamoDBMetadataStore to handle DDB throttling failures through retry policy

2017-02-15 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13904: -- Attachment: HADOOP-13904-HADOOP-13345.003.patch Attaching v3 patch that pulls out the

[jira] [Created] (HADOOP-14087) S3A typo in pom.xml test exclusions

2017-02-15 Thread Aaron Fabbri (JIRA)
Aaron Fabbri created HADOOP-14087: - Summary: S3A typo in pom.xml test exclusions Key: HADOOP-14087 URL: https://issues.apache.org/jira/browse/HADOOP-14087 Project: Hadoop Common Issue Type:

[jira] [Updated] (HADOOP-14087) S3A typo in pom.xml test exclusions

2017-02-15 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14087: -- Affects Version/s: (was: HADOOP-13345) 2.8.0

[jira] [Updated] (HADOOP-14087) S3A typo in pom.xml test exclusions

2017-02-15 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14087: -- Description: Noticed a copy/paste typo in hadoop-tools/hadoop-aws/pom.xml:

[jira] [Updated] (HADOOP-14087) S3A typo in pom.xml test exclusions

2017-02-15 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14087: -- Target Version/s: 2.8.0, 3.0.0-beta1 (was: HADOOP-13345) > S3A typo in pom.xml test

[jira] [Updated] (HADOOP-14087) S3A typo in pom.xml test exclusions

2017-02-15 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14087: -- Issue Type: Bug (was: Sub-task) Parent: (was: HADOOP-13345) > S3A typo in

[jira] [Updated] (HADOOP-14087) S3A typo in pom.xml test exclusions

2017-02-15 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14087: -- Attachment: HADOOP-14087.001.patch > S3A typo in pom.xml test exclusions >

[jira] [Updated] (HADOOP-14087) S3A typo in pom.xml test exclusions

2017-02-15 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14087: -- Status: Patch Available (was: Open) > S3A typo in pom.xml test exclusions >

[jira] [Commented] (HADOOP-13904) DynamoDBMetadataStore to handle DDB throttling failures through retry policy

2017-02-15 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868982#comment-15868982 ] Aaron Fabbri commented on HADOOP-13904: --- Filed HADOOP-14087. [~ste...@apache.org] any remaining

[jira] [Commented] (HADOOP-14030) PreCommit TestKDiag failure

2017-02-15 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868988#comment-15868988 ] Aaron Fabbri commented on HADOOP-14030: --- Also seeing these here HADOOP-14041 > PreCommit

[jira] [Commented] (HADOOP-14041) CLI command to prune old metadata

2017-02-15 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868994#comment-15868994 ] Aaron Fabbri commented on HADOOP-14041: --- Your latest patch looks good to me, +1. I'm fine with

[jira] [Commented] (HADOOP-14041) CLI command to prune old metadata

2017-02-16 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870696#comment-15870696 ] Aaron Fabbri commented on HADOOP-14041: --- Just recording results from my test runs last night: mvn

[jira] [Commented] (HADOOP-14041) CLI command to prune old metadata

2017-02-16 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15870704#comment-15870704 ] Aaron Fabbri commented on HADOOP-14041: --- Also, shouldn't this work? {noformat} $ hadoop s3a

[jira] [Commented] (HADOOP-13736) Change PathMetadata to hold S3AFileStatus instead of FileStatus.

2017-02-15 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868356#comment-15868356 ] Aaron Fabbri commented on HADOOP-13736: --- Interesting [~mackrorysd]. I thought this patch was just

[jira] [Commented] (HADOOP-14041) CLI command to prune old metadata

2017-02-15 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868367#comment-15868367 ] Aaron Fabbri commented on HADOOP-14041: --- [~mackrorysd] Any idea what is up with the jenkins unit

[jira] [Commented] (HADOOP-13736) Change PathMetadata to hold S3AFileStatus instead of FileStatus.

2017-02-15 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868602#comment-15868602 ] Aaron Fabbri commented on HADOOP-13736: --- Ah.. If the DDB code is asserting that it is getting an

[jira] [Commented] (HADOOP-13736) Change PathMetadata to hold S3AFileStatus instead of FileStatus.

2017-02-15 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868636#comment-15868636 ] Aaron Fabbri commented on HADOOP-13736: --- Yep.. I see that function doesn't need a S3AFileStatus at

[jira] [Commented] (HADOOP-13904) DynamoDBMetadataStore to handle DDB throttling failures through retry policy

2017-02-15 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15868754#comment-15868754 ] Aaron Fabbri commented on HADOOP-13904: --- Ok.. I started on this, again realizing what a crazy API

<    1   2   3   4   5   6   7   8   9   10   >