[jira] [Updated] (HADOOP-14738) Deprecate S3N in hadoop 3.0/2,9, target removal in Hadoop 3.1

2017-08-14 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14738: -- Priority: Blocker (was: Major) > Deprecate S3N in hadoop 3.0/2,9, target removal in Hadoop

[jira] [Commented] (HADOOP-14749) review s3guard docs & code prior to merge

2017-08-14 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16126207#comment-16126207 ] Aaron Fabbri commented on HADOOP-14749: --- Awesome, thanks for doing this [~ste...@apache.org] >

[jira] [Commented] (HADOOP-14749) review s3guard docs & code prior to merge

2017-08-11 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124000#comment-16124000 ] Aaron Fabbri commented on HADOOP-14749: --- Quick review of the v5 patch: {noformat} +### Security +

[jira] [Updated] (HADOOP-14756) S3Guard: expose capability query in MetadataStore and add tests of authoritative mode

2017-08-11 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14756: -- Summary: S3Guard: expose capability query in MetadataStore and add tests of authoritative

[jira] [Commented] (HADOOP-14756) S3Guard: instrument MetadataStore for asserting & testing MetadataStoreTestBase

2017-08-11 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123991#comment-16123991 ] Aaron Fabbri commented on HADOOP-14756: --- Thanks for filing this. FYI I wrote #3 in the code

[jira] [Commented] (HADOOP-14749) review s3guard docs & code prior to merge

2017-08-11 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16123792#comment-16123792 ] Aaron Fabbri commented on HADOOP-14749: --- {quote} If we added a field for each entry as to when the

[jira] [Commented] (HADOOP-14749) review s3guard docs & code prior to merge

2017-08-09 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120562#comment-16120562 ] Aaron Fabbri commented on HADOOP-14749: --- +1 on v3 patch. > review s3guard docs & code prior to

[jira] [Comment Edited] (HADOOP-14467) S3Guard: Improve FNFE message when opening a stream

2017-08-09 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120168#comment-16120168 ] Aaron Fabbri edited comment on HADOOP-14467 at 8/9/17 5:13 PM: --- I didn't

[jira] [Commented] (HADOOP-14467) S3Guard: Improve FNFE message when opening a stream

2017-08-09 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120168#comment-16120168 ] Aaron Fabbri commented on HADOOP-14467: --- I didn't find a nice clean way to add a new exception

[jira] [Commented] (HADOOP-14735) ITestS3AEncryptionSSEC failing in parallel s3guard runs

2017-08-09 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120140#comment-16120140 ] Aaron Fabbri commented on HADOOP-14735: --- +1 looks good to me. Thanks for doing this

[jira] [Comment Edited] (HADOOP-14154) Set isAuthoritative flag when creating DirListingMetadata in DynamoDBMetaStore

2017-08-09 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120131#comment-16120131 ] Aaron Fabbri edited comment on HADOOP-14154 at 8/9/17 3:56 PM: --- Hi

[jira] [Commented] (HADOOP-14154) Set isAuthoritative flag when creating DirListingMetadata in DynamoDBMetaStore

2017-08-09 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16120131#comment-16120131 ] Aaron Fabbri commented on HADOOP-14154: --- Hi [~ste...@apache.org]. I have a prototype patch

[jira] [Commented] (HADOOP-14749) review s3guard docs & code prior to merge

2017-08-09 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16119955#comment-16119955 ] Aaron Fabbri commented on HADOOP-14749: --- Thanks for the patch [~ste...@apache.org]. This is good

[jira] [Commented] (HADOOP-14738) Deprecate S3N in hadoop 3.0/2,9, target removal in Hadoop 3.1

2017-08-08 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118841#comment-16118841 ] Aaron Fabbri commented on HADOOP-14738: --- I'd vote to remove s3n in 3.0.. and make this JIRA a

[jira] [Commented] (HADOOP-14733) ITestS3GuardConcurrentOps failing with -Ddynamodblocal -Ds3guard

2017-08-08 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118368#comment-16118368 ] Aaron Fabbri commented on HADOOP-14733: --- +1 LGTM. > ITestS3GuardConcurrentOps failing with

[jira] [Comment Edited] (HADOOP-13345) S3Guard: Improved Consistency for S3A

2017-08-01 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16109690#comment-16109690 ] Aaron Fabbri edited comment on HADOOP-13345 at 8/1/17 9:00 PM: --- I have a

[jira] [Commented] (HADOOP-13345) S3Guard: Improved Consistency for S3A

2017-08-01 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16109690#comment-16109690 ] Aaron Fabbri commented on HADOOP-13345: --- I have a clean trunk merge commit ready to push.

[jira] [Commented] (HADOOP-13345) S3Guard: Improved Consistency for S3A

2017-07-28 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16105773#comment-16105773 ] Aaron Fabbri commented on HADOOP-13345: --- FYI I started working on merging latest trunk into

[jira] [Updated] (HADOOP-14467) S3Guard: Improve FNFE message when opening a stream

2017-07-25 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14467: -- Attachment: HADOOP-14467-HADOOP-13345.001.patch Attaching v1 patch that just adds a simple

[jira] [Commented] (HADOOP-14467) S3Guard: Improve FNFE message when opening a stream

2017-07-14 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16088268#comment-16088268 ] Aaron Fabbri commented on HADOOP-14467: --- I've created a simple integration test that exercises

[jira] [Commented] (HADOOP-14659) UGI getShortUserName does not need to search the Subject

2017-07-13 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16086732#comment-16086732 ] Aaron Fabbri commented on HADOOP-14659: --- +1 (non-binding) looks correct to me. I assume you're

[jira] [Commented] (HADOOP-14398) Modify documents for the FileSystem Builder API

2017-07-13 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16086461#comment-16086461 ] Aaron Fabbri commented on HADOOP-14398: --- Hi [~eddyxu]. Thank you for working on this

[jira] [Commented] (HADOOP-14397) Pull up the builder pattern to FileSystem and add AbstractContractCreateTest for it

2017-07-12 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084953#comment-16084953 ] Aaron Fabbri commented on HADOOP-14397: --- Code looks good to me. Nice test factorization. I can't

[jira] [Commented] (HADOOP-13998) initial s3guard preview

2017-07-12 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084812#comment-16084812 ] Aaron Fabbri commented on HADOOP-13998: --- +1, [~ste...@apache.org] > initial s3guard preview >

[jira] [Comment Edited] (HADOOP-14396) Add builder interface to FileContext

2017-07-12 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084808#comment-16084808 ] Aaron Fabbri edited comment on HADOOP-14396 at 7/12/17 10:03 PM: - Thanks

[jira] [Commented] (HADOOP-14396) Add builder interface to FileContext

2017-07-12 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084808#comment-16084808 ] Aaron Fabbri commented on HADOOP-14396: --- Thanks for the work on this [~eddyxu]. +1 (non-binding)

[jira] [Updated] (HADOOP-14633) S3Guard: optimize create codepath

2017-07-11 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14633: -- Attachment: HADOOP-14633-HADOOP-13345.003.patch v3 remaining checkstyle issue. (I blame the

[jira] [Commented] (HADOOP-14425) Add more s3guard metrics

2017-07-11 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16083171#comment-16083171 ] Aaron Fabbri commented on HADOOP-14425: --- Note from HADOOP-14633, we should increment put() failure

[jira] [Commented] (HADOOP-14633) S3Guard: optimize create codepath

2017-07-11 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16083174#comment-16083174 ] Aaron Fabbri commented on HADOOP-14633: --- Just noticed your request to increment a put failure

[jira] [Updated] (HADOOP-14633) S3Guard: optimize create codepath

2017-07-11 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14633: -- Attachment: HADOOP-14633-HADOOP-13345.002.patch v2 patch. Addresses checkstyle warnings,

[jira] [Commented] (HADOOP-14633) S3Guard: optimize create codepath

2017-07-11 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16082513#comment-16082513 ] Aaron Fabbri commented on HADOOP-14633: --- Will post a v2 patch with checkstyle fix. I'm getting

[jira] [Comment Edited] (HADOOP-14633) S3Guard: optimize create codepath

2017-07-07 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16078883#comment-16078883 ] Aaron Fabbri edited comment on HADOOP-14633 at 7/8/17 1:22 AM: --- Attaching

[jira] [Updated] (HADOOP-14633) S3Guard: optimize create codepath

2017-07-07 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14633: -- Fix Version/s: HADOOP-13345 Status: Patch Available (was: Open) > S3Guard:

[jira] [Commented] (HADOOP-14457) create() does not notify metadataStore of parent directories or ensure they're not existing files

2017-07-07 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16078884#comment-16078884 ] Aaron Fabbri commented on HADOOP-14457: --- FYI did some perf tests.. patch and results posted to

[jira] [Updated] (HADOOP-14633) S3Guard: optimize create codepath

2017-07-07 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14633: -- Attachment: HADOOP-14633-HADOOP-13345.001.patch Attaching v1 patch that gives almost 2x

[jira] [Commented] (HADOOP-14457) create() does not notify metadataStore of parent directories or ensure they're not existing files

2017-07-07 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16078476#comment-16078476 ] Aaron Fabbri commented on HADOOP-14457: --- I see this was already committed, so I filed HADOOP-14633

[jira] [Created] (HADOOP-14633) S3Guard: optimize create codepath

2017-07-07 Thread Aaron Fabbri (JIRA)
Aaron Fabbri created HADOOP-14633: - Summary: S3Guard: optimize create codepath Key: HADOOP-14633 URL: https://issues.apache.org/jira/browse/HADOOP-14633 Project: Hadoop Common Issue Type:

[jira] [Commented] (HADOOP-14457) create() does not notify metadataStore of parent directories or ensure they're not existing files

2017-07-07 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16078465#comment-16078465 ] Aaron Fabbri commented on HADOOP-14457: --- Thanks for the patch [~mackrorysd]. This looks really

[jira] [Commented] (HADOOP-14468) S3Guard: make short-circuit getFileStatus() configurable

2017-07-06 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16077499#comment-16077499 ] Aaron Fabbri commented on HADOOP-14468: --- {quote} That said, looking at all the places we call

[jira] [Updated] (HADOOP-14548) S3Guard: issues running parallel tests w/ S3N

2017-07-06 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14548: -- Fix Version/s: HADOOP-13345 Status: Patch Available (was: Open) > S3Guard: issues

[jira] [Updated] (HADOOP-14548) S3Guard: issues running parallel tests w/ S3N

2017-07-06 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14548: -- Attachment: HADOOP-14548-HADOOP-13345.001.patch Attaching documentation-only patch (v1). It

[jira] [Commented] (HADOOP-14620) S3A authentication failure for regions other than us-east-1

2017-07-06 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16077403#comment-16077403 ] Aaron Fabbri commented on HADOOP-14620: --- Looks like you are on a fairly old version. Can you

[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure

2017-07-05 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16075717#comment-16075717 ] Aaron Fabbri commented on HADOOP-14553: --- Integration tests passed in US West via `mvn -T 1C clean

[jira] [Commented] (HADOOP-14553) Add (parallelized) integration tests to hadoop-azure

2017-07-05 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16075714#comment-16075714 ] Aaron Fabbri commented on HADOOP-14553: --- Thank you for doing this work [~ste...@apache.org]. I'm

[jira] [Commented] (HADOOP-14576) DynamoDB tables may leave ACTIVE state after initial connection

2017-07-05 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16075123#comment-16075123 ] Aaron Fabbri commented on HADOOP-14576: --- Thanks for filing the JIRA, [~mackrorysd]. {quote} My

[jira] [Comment Edited] (HADOOP-14499) Findbugs warning in LocalMetadataStore

2017-06-21 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16058469#comment-16058469 ] Aaron Fabbri edited comment on HADOOP-14499 at 6/22/17 12:16 AM: - Yeah

[jira] [Updated] (HADOOP-14527) ITestS3GuardListConsistency is too slow

2017-06-21 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14527: -- Attachment: HADOOP-14527-HADOOP-13345.004.patch Thanks for the review [~mackrorysd].

[jira] [Commented] (HADOOP-14499) Findbugs warning in LocalMetadataStore

2017-06-21 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16058469#comment-16058469 ] Aaron Fabbri commented on HADOOP-14499: --- Yeah sorry for delay responding to your earlier comment.

[jira] [Commented] (HADOOP-14499) Findbugs warning in LocalMetadataStore

2017-06-21 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16058374#comment-16058374 ] Aaron Fabbri commented on HADOOP-14499: --- Hey [~mackrorysd], thank you for the patch. Can you

[jira] [Updated] (HADOOP-14548) S3Guard: issues running parallel tests w/ S3N

2017-06-20 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14548: -- Priority: Minor (was: Major) > S3Guard: issues running parallel tests w/ S3N >

[jira] [Commented] (HADOOP-14548) S3Guard: issues running parallel tests w/ S3N

2017-06-20 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056763#comment-16056763 ] Aaron Fabbri commented on HADOOP-14548: --- For the record, here are the failures on trunk (no

[jira] [Commented] (HADOOP-14457) create() does not notify metadataStore of parent directories or ensure they're not existing files

2017-06-20 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056703#comment-16056703 ] Aaron Fabbri commented on HADOOP-14457: --- [~mackrorysd] and I just chatted about this and I think

[jira] [Commented] (HADOOP-14548) S3Guard: issues running parallel tests w/ S3N

2017-06-20 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056553#comment-16056553 ] Aaron Fabbri commented on HADOOP-14548: --- Yeah, I will add something in the testing.md docs to that

[jira] [Commented] (HADOOP-14548) S3Guard: issues running parallel tests w/ S3N

2017-06-20 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055276#comment-16055276 ] Aaron Fabbri commented on HADOOP-14548: --- #5 failure was:

[jira] [Commented] (HADOOP-14548) S3Guard: issues running parallel tests w/ S3N

2017-06-20 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055269#comment-16055269 ] Aaron Fabbri commented on HADOOP-14548: --- I need to re-test this stuff on trunk to isolate any

[jira] [Assigned] (HADOOP-14548) S3Guard: issues running parallel tests w/ S3N

2017-06-20 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri reassigned HADOOP-14548: - Assignee: Aaron Fabbri > S3Guard: issues running parallel tests w/ S3N >

[jira] [Commented] (HADOOP-14548) S3Guard: issues running parallel tests w/ S3N

2017-06-19 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055026#comment-16055026 ] Aaron Fabbri commented on HADOOP-14548: --- My failures from `mvn verify -Dparallel-tests -Ds3guard

[jira] [Created] (HADOOP-14548) S3Guard: issues running parallel tests w/ S3N

2017-06-19 Thread Aaron Fabbri (JIRA)
Aaron Fabbri created HADOOP-14548: - Summary: S3Guard: issues running parallel tests w/ S3N Key: HADOOP-14548 URL: https://issues.apache.org/jira/browse/HADOOP-14548 Project: Hadoop Common

[jira] [Commented] (HADOOP-13786) Add S3Guard committer for zero-rename commits to S3 endpoints

2017-06-16 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16052550#comment-16052550 ] Aaron Fabbri commented on HADOOP-13786: --- Rebased v30 patch onto latest feature branch (shout if

[jira] [Updated] (HADOOP-14527) ITestS3GuardListConsistency is too slow

2017-06-16 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14527: -- Attachment: HADOOP-14527-HADOOP-13345.003.patch v3 patch fixes (existing) checkstyle issue.

[jira] [Updated] (HADOOP-14527) ITestS3GuardListConsistency is too slow

2017-06-15 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14527: -- Attachment: HADOOP-14527-HADOOP-13345.002.patch v2 patch, rebased on latest branch.

[jira] [Updated] (HADOOP-14527) ITestS3GuardListConsistency is too slow

2017-06-15 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14527: -- Status: Patch Available (was: Open) > ITestS3GuardListConsistency is too slow >

[jira] [Updated] (HADOOP-14527) ITestS3GuardListConsistency is too slow

2017-06-15 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14527: -- Description: I'm really glad to see folks adopting the inconsistency injection stuff and

[jira] [Updated] (HADOOP-14527) ITestS3GuardListConsistency is too slow

2017-06-15 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14527: -- Attachment: HADOOP-14527-HADOOP-13345.001.patch v1 patch.. Includes stopwatch instrumentation

[jira] [Commented] (HADOOP-14527) ITestS3GuardListConsistency is too slow

2017-06-15 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16051195#comment-16051195 ] Aaron Fabbri commented on HADOOP-14527: --- After my changes, runtime goes from ~8 minutes to 2:30.

[jira] [Commented] (HADOOP-14527) ITestS3GuardListConsistency is too slow

2017-06-15 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16051053#comment-16051053 ] Aaron Fabbri commented on HADOOP-14527: --- Added a method to clear any delayed keys from the

[jira] [Updated] (HADOOP-14488) s3guard w/ failure injection: listStatus fails after renaming file into directory

2017-06-14 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14488: -- Attachment: HADOOP-14488-HADOOP-13345-005.patch Actually attaching v5 patch. > s3guard w/

[jira] [Updated] (HADOOP-14488) s3guard w/ failure injection: listStatus fails after renaming file into directory

2017-06-14 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14488: -- Summary: s3guard w/ failure injection: listStatus fails after renaming file into directory

[jira] [Created] (HADOOP-14527) ITestS3GuardListConsistency is too slow

2017-06-14 Thread Aaron Fabbri (JIRA)
Aaron Fabbri created HADOOP-14527: - Summary: ITestS3GuardListConsistency is too slow Key: HADOOP-14527 URL: https://issues.apache.org/jira/browse/HADOOP-14527 Project: Hadoop Common Issue

[jira] [Commented] (HADOOP-14488) s3guard listStatus fails after renaming file into directory

2017-06-14 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049960#comment-16049960 ] Aaron Fabbri commented on HADOOP-14488: --- Thanks for the excellent work here [~mackrorysd]. I

[jira] [Commented] (HADOOP-14457) create() does not notify metadataStore of parent directories or ensure they're not existing files

2017-06-14 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049856#comment-16049856 ] Aaron Fabbri commented on HADOOP-14457: --- Good summary [~mackrorysd]. One thing I'll add about my

[jira] [Commented] (HADOOP-14457) create() does not notify metadataStore of parent directories or ensure they're not existing files

2017-06-12 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16047186#comment-16047186 ] Aaron Fabbri commented on HADOOP-14457: --- Thanks for interesting discussion [~mackrorysd]. {quote}

[jira] [Commented] (HADOOP-14457) create() does not notify metadataStore of parent directories or ensure they're not existing files

2017-06-09 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16045212#comment-16045212 ] Aaron Fabbri commented on HADOOP-14457: --- Thank you for updating your patch [~mackrorysd]. The v9

[jira] [Resolved] (HADOOP-14509) InconsistentAmazonS3Client adds extra paths to listStatus() after delete.

2017-06-09 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri resolved HADOOP-14509. --- Resolution: Duplicate Resolving as duplicate, thank you [~mackrorysd]. >

[jira] [Commented] (HADOOP-14488) s3guard listStatus fails after renaming file into directory

2017-06-08 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043789#comment-16043789 ] Aaron Fabbri commented on HADOOP-14488: --- Thanks for doing this [~mackrorysd]. Yep, looks like we

[jira] [Commented] (HADOOP-14457) create() does not notify metadataStore of parent directories or ensure they're not existing files

2017-06-08 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043693#comment-16043693 ] Aaron Fabbri commented on HADOOP-14457: --- Thank you for being flexible and splitting this up

[jira] [Updated] (HADOOP-14468) S3Guard: make short-circuit getFileStatus() configurable

2017-06-08 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14468: -- Priority: Minor (was: Major) > S3Guard: make short-circuit getFileStatus() configurable >

[jira] [Comment Edited] (HADOOP-14504) ProvidedFileStatusIterator#next() may throw IndexOutOfBoundsException

2017-06-08 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043583#comment-16043583 ] Aaron Fabbri edited comment on HADOOP-14504 at 6/8/17 10:57 PM:

[jira] [Updated] (HADOOP-14504) ProvidedFileStatusIterator#next() may throw IndexOutOfBoundsException

2017-06-08 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14504: -- Resolution: Fixed Fix Version/s: HADOOP-13345 Status: Resolved (was: Patch

[jira] [Commented] (HADOOP-14283) Upgrade AWS SDK to 1.11.134

2017-06-08 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043570#comment-16043570 ] Aaron Fabbri commented on HADOOP-14283: --- Thanks [~liuml07]. I ran into the same problem when

[jira] [Updated] (HADOOP-14476) make InconsistentAmazonS3Client usable in downstream tests

2017-06-08 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14476: -- Resolution: Fixed Fix Version/s: HADOOP-13345 Status: Resolved (was: Patch

[jira] [Commented] (HADOOP-14476) make InconsistentAmazonS3Client usable in downstream tests

2017-06-08 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043081#comment-16043081 ] Aaron Fabbri commented on HADOOP-14476: --- [~liuml07] are you available to review latest patch? I'd

[jira] [Commented] (HADOOP-14509) InconsistentAmazonS3Client adds extra paths to listStatus() after delete.

2017-06-08 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043078#comment-16043078 ] Aaron Fabbri commented on HADOOP-14509: --- Thank you [~mackrorysd] for agreeing to work on this.

[jira] [Assigned] (HADOOP-14509) InconsistentAmazonS3Client adds extra paths to listStatus() after delete.

2017-06-08 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri reassigned HADOOP-14509: - Assignee: Sean Mackrory (was: Aaron Fabbri) > InconsistentAmazonS3Client adds extra

[jira] [Created] (HADOOP-14509) InconsistentAmazonS3Client adds extra paths to listStatus() after delete.

2017-06-08 Thread Aaron Fabbri (JIRA)
Aaron Fabbri created HADOOP-14509: - Summary: InconsistentAmazonS3Client adds extra paths to listStatus() after delete. Key: HADOOP-14509 URL: https://issues.apache.org/jira/browse/HADOOP-14509

[jira] [Commented] (HADOOP-14488) s3guard listStatus fails after renaming file into directory

2017-06-08 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16042997#comment-16042997 ] Aaron Fabbri commented on HADOOP-14488: --- One thing that comes to mind is the bug I noticed in the

[jira] [Commented] (HADOOP-14476) make InconsistentAmazonS3Client usable in downstream tests

2017-06-07 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16042201#comment-16042201 ] Aaron Fabbri commented on HADOOP-14476: --- Also note v4 patch removes two SSE-related constants I

[jira] [Updated] (HADOOP-14476) make InconsistentAmazonS3Client usable in downstream tests

2017-06-07 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14476: -- Attachment: HADOOP-14476-HADOOP-13345.004.patch Thank you for the review [~liuml07].

[jira] [Commented] (HADOOP-14506) Add create() contract test that verifies ancestor dir creation

2017-06-07 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16042180#comment-16042180 ] Aaron Fabbri commented on HADOOP-14506: --- +1 (non-binding) on test code. If you re-roll patch,

[jira] [Commented] (HADOOP-14499) Findbugs warning in LocalMetadataStore

2017-06-07 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16042079#comment-16042079 ] Aaron Fabbri commented on HADOOP-14499: --- I feel like we need to define prune() semantics a little

[jira] [Updated] (HADOOP-14504) ProvidedFileStatusIterator#next() may throw IndexOutOfBoundsException

2017-06-07 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14504: -- Attachment: HADOOP-14504-HADOOP-13345.002.patch Attaching v2 patch: v1 was missing the test

[jira] [Updated] (HADOOP-14506) Add create() contract test that verifies ancestor dir creation

2017-06-07 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14506: -- Affects Version/s: (was: HADOOP-13345) 3.0.0-alpha4 > Add create()

[jira] [Updated] (HADOOP-14506) Add create() contract test that verifies ancestor dir creation

2017-06-07 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14506: -- Description: The semantics of {{FileSystem#create()}} appear to include implicit creation of

[jira] [Updated] (HADOOP-14506) Add create() contract test that verifies ancestor dir creation

2017-06-07 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14506: -- Issue Type: Improvement (was: Sub-task) Parent: (was: HADOOP-13345) > Add

[jira] [Commented] (HADOOP-14457) create() does not notify metadataStore of parent directories or ensure they're not existing files

2017-06-07 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16041907#comment-16041907 ] Aaron Fabbri commented on HADOOP-14457: --- Ok.. I created the following JIRAs for the items I list

[jira] [Created] (HADOOP-14506) Add create() contract test that verifies ancestor dir creation

2017-06-07 Thread Aaron Fabbri (JIRA)
Aaron Fabbri created HADOOP-14506: - Summary: Add create() contract test that verifies ancestor dir creation Key: HADOOP-14506 URL: https://issues.apache.org/jira/browse/HADOOP-14506 Project: Hadoop

[jira] [Created] (HADOOP-14505) simplify mkdirs() after S3Guard delete tracking change

2017-06-07 Thread Aaron Fabbri (JIRA)
Aaron Fabbri created HADOOP-14505: - Summary: simplify mkdirs() after S3Guard delete tracking change Key: HADOOP-14505 URL: https://issues.apache.org/jira/browse/HADOOP-14505 Project: Hadoop Common

[jira] [Updated] (HADOOP-14504) ProvidedFileStatusIterator#next() may throw IndexOutOfBoundsException

2017-06-07 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14504: -- Status: Patch Available (was: Open) > ProvidedFileStatusIterator#next() may throw

[jira] [Updated] (HADOOP-14504) ProvidedFileStatusIterator#next() may throw IndexOutOfBoundsException

2017-06-07 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14504: -- Attachment: HADOOP-14504-HADOOP-13345.001.patch Attaching v1 patch. Also includes unit test

[jira] [Created] (HADOOP-14504) ProvidedFileStatusIterator#next() may throw IndexOutOfBoundsException

2017-06-07 Thread Aaron Fabbri (JIRA)
Aaron Fabbri created HADOOP-14504: - Summary: ProvidedFileStatusIterator#next() may throw IndexOutOfBoundsException Key: HADOOP-14504 URL: https://issues.apache.org/jira/browse/HADOOP-14504 Project:

<    1   2   3   4   5   6   7   8   9   10   >