[jira] [Commented] (HADOOP-16830) Add public IOStatistics API; S3A to support

2020-07-22 Thread Aaron Fabbri (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17163131#comment-17163131 ] Aaron Fabbri commented on HADOOP-16830: --- Been following along. I should be able to finish a

[jira] [Commented] (HADOOP-16798) job commit failure in S3A MR magic committer test

2020-07-07 Thread Aaron Fabbri (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17153199#comment-17153199 ] Aaron Fabbri commented on HADOOP-16798: --- I missed the party on this one, but just had a thought..

[jira] [Commented] (HADOOP-13230) S3A to optionally retain directory markers; look under a marker for files when needEmptyDir=true

2020-03-10 Thread Aaron Fabbri (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-13230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17056471#comment-17056471 ] Aaron Fabbri commented on HADOOP-13230: --- [~ste...@apache.org] I'll take a look if you want. You

[jira] [Commented] (HADOOP-16792) Let s3 clients configure request timeout

2020-01-08 Thread Aaron Fabbri (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17011401#comment-17011401 ] Aaron Fabbri commented on HADOOP-16792: --- Hi [~mustafaiman] . All the site docs live here:

[jira] [Commented] (HADOOP-16725) s3guard prune can delete directories -leaving orphan children.

2019-11-25 Thread Aaron Fabbri (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16981786#comment-16981786 ] Aaron Fabbri commented on HADOOP-16725: --- Sounds good. Thanks for running through and checking

[jira] [Commented] (HADOOP-16605) NPE in TestAdlSdkConfiguration failing in yetus

2019-10-02 Thread Aaron Fabbri (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16943302#comment-16943302 ] Aaron Fabbri commented on HADOOP-16605: --- PR looks good to me. +1 > NPE in

[jira] [Commented] (HADOOP-15691) Add PathCapabilities to FS and FC to complement StreamCapabilities

2019-09-20 Thread Aaron Fabbri (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934199#comment-16934199 ] Aaron Fabbri commented on HADOOP-15691: --- +1 on latest pull request pending clean precommit

[jira] [Commented] (HADOOP-16547) s3guard prune command doesn't get AWS auth chain from FS

2019-09-17 Thread Aaron Fabbri (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931680#comment-16931680 ] Aaron Fabbri commented on HADOOP-16547: --- Current patch +1 LTGM after you address [~gabor.bota]'s

[jira] [Commented] (HADOOP-16430) S3AFilesystem.delete to incrementally update s3guard with deletions

2019-09-04 Thread Aaron Fabbri (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922903#comment-16922903 ] Aaron Fabbri commented on HADOOP-16430: --- Latest PR commits reviewed (+1). Aside: This is nice not

[jira] [Commented] (HADOOP-16430) S3AFilesystem.delete to incrementally update s3guard with deletions

2019-08-29 Thread Aaron Fabbri (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918773#comment-16918773 ] Aaron Fabbri commented on HADOOP-16430: --- +1 LGTM after your clarifying comments.. Thanks for the

[jira] [Commented] (HADOOP-16430) S3AFilesystem.delete to incrementally update s3guard with deletions

2019-08-28 Thread Aaron Fabbri (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918167#comment-16918167 ] Aaron Fabbri commented on HADOOP-16430: --- Thanks for the work on this. Looks pretty good. Made a

[jira] [Commented] (HADOOP-16382) Clock skew can cause S3Guard to think object metadata is out of date

2019-06-19 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16868071#comment-16868071 ] Aaron Fabbri commented on HADOOP-16382: ---   {quote}It'd be nice if we actually got a timestamp off

[jira] [Commented] (HADOOP-15183) S3Guard store becomes inconsistent after partial failure of rename

2019-06-12 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16862604#comment-16862604 ] Aaron Fabbri commented on HADOOP-15183: --- Made it through PR 951 except two files I still need to

[jira] [Commented] (HADOOP-16117) Upgrade to latest AWS SDK

2019-06-05 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16857270#comment-16857270 ] Aaron Fabbri commented on HADOOP-16117: --- +1 LGTM for trunk. Agreed extended testing is to be

[jira] [Comment Edited] (HADOOP-13980) S3Guard CLI: Add fsck check command

2019-06-05 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16857029#comment-16857029 ] Aaron Fabbri edited comment on HADOOP-13980 at 6/5/19 8:55 PM: --- Thanks

[jira] [Commented] (HADOOP-13980) S3Guard CLI: Add fsck check command

2019-06-05 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16857029#comment-16857029 ] Aaron Fabbri commented on HADOOP-13980: --- Thanks for your draft of FSCK requirements

[jira] [Commented] (HADOOP-15183) S3Guard store becomes inconsistent after partial failure of rename

2019-05-29 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16851518#comment-16851518 ] Aaron Fabbri commented on HADOOP-15183: --- Looking good [~ste...@apache.org]. A lot of nice

[jira] [Commented] (HADOOP-16324) S3A Delegation Token code to spell "Marshalled" as Marshaled

2019-05-22 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16846140#comment-16846140 ] Aaron Fabbri commented on HADOOP-16324: --- Blimey this one is easy peasy. Either option is

[jira] [Commented] (HADOOP-16279) S3Guard: Implement time-based (TTL) expiry for entries (and tombstones)

2019-05-22 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16846126#comment-16846126 ] Aaron Fabbri commented on HADOOP-16279: --- {quote}GB>I'm having a hard time implementing prune for

[jira] [Commented] (HADOOP-16279) S3Guard: Implement time-based (TTL) expiry for entries (and tombstones)

2019-05-22 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16846116#comment-16846116 ] Aaron Fabbri commented on HADOOP-16279: --- Good discussion, thanks. {quote}AF> ... when data from

[jira] [Commented] (HADOOP-16279) S3Guard: Implement time-based (TTL) expiry for entries (and tombstones)

2019-05-15 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840849#comment-16840849 ] Aaron Fabbri commented on HADOOP-16279: --- {quote}With HADOOP-15999 we changed a bit how

[jira] [Commented] (HADOOP-15999) S3Guard: Better support for out-of-band operations

2019-05-15 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840844#comment-16840844 ] Aaron Fabbri commented on HADOOP-15999: --- What about in-band delete (create tombstone) and then

[jira] [Comment Edited] (HADOOP-16279) S3Guard: Implement time-based (TTL) expiry for entries (and tombstones)

2019-05-15 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840799#comment-16840799 ] Aaron Fabbri edited comment on HADOOP-16279 at 5/15/19 9:34 PM:

[jira] [Commented] (HADOOP-16279) S3Guard: Implement time-based (TTL) expiry for entries (and tombstones)

2019-05-15 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840799#comment-16840799 ] Aaron Fabbri commented on HADOOP-16279: --- {quote}with HADOOP-15999 we changed a bit how

[jira] [Resolved] (HADOOP-16251) ABFS: add FSMainOperationsBaseTest

2019-05-10 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri resolved HADOOP-16251. --- Resolution: Fixed Fix Version/s: 3.2.1 > ABFS: add FSMainOperationsBaseTest >

[jira] [Commented] (HADOOP-16251) ABFS: add FSMainOperationsBaseTest

2019-05-10 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837600#comment-16837600 ] Aaron Fabbri commented on HADOOP-16251: --- Committed to trunk. Thank you for the contribution 

[jira] [Commented] (HADOOP-16251) ABFS: add FSMainOperationsBaseTest

2019-05-09 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16836626#comment-16836626 ] Aaron Fabbri commented on HADOOP-16251: --- I had one remaining question on your Ignore annotation

[jira] [Comment Edited] (HADOOP-16251) ABFS: add FSMainOperationsBaseTest

2019-05-08 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16831936#comment-16831936 ] Aaron Fabbri edited comment on HADOOP-16251 at 5/9/19 1:04 AM: --- Thanks

[jira] [Commented] (HADOOP-16251) ABFS: add FSMainOperationsBaseTest

2019-05-08 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16835999#comment-16835999 ] Aaron Fabbri commented on HADOOP-16251: --- Sorry for the confusion [~DanielZhou]. I misread that

[jira] [Commented] (HADOOP-16279) S3Guard: Implement time-based (TTL) expiry for entries (and tombstones)

2019-05-08 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16835944#comment-16835944 ] Aaron Fabbri commented on HADOOP-16279: --- Thanks for the work on this stuff [~gabor.bota]. I

[jira] [Commented] (HADOOP-16279) S3Guard: Implement time-based (TTL) expiry for entries (and tombstones)

2019-05-08 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16835785#comment-16835785 ] Aaron Fabbri commented on HADOOP-16279: --- [~ste...@apache.org] I'd argue LocalMetadataStore is

[jira] [Commented] (HADOOP-16278) With S3A Filesystem, Long Running services End up Doing lot of GC and eventually die

2019-05-08 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16835783#comment-16835783 ] Aaron Fabbri commented on HADOOP-16278: --- Agreed, +1 this simple patch stopping the quantiles on

[jira] [Comment Edited] (HADOOP-16278) With S3A Filesystem, Long Running services End up Doing lot of GC and eventually die

2019-05-08 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16835783#comment-16835783 ] Aaron Fabbri edited comment on HADOOP-16278 at 5/8/19 5:47 PM: --- Agreed,

[jira] [Commented] (HADOOP-16269) ABFS: add listFileStatus with StartFrom

2019-05-08 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16835778#comment-16835778 ] Aaron Fabbri commented on HADOOP-16269: --- This was on my todo list today but [~ste...@apache.org]

[jira] [Commented] (HADOOP-16291) HDFS Permissions Guide appears incorrect about getFileStatus()/getFileInfo()

2019-05-03 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832644#comment-16832644 ] Aaron Fabbri commented on HADOOP-16291: --- Thanks [~daryn]. Thought it was strange the docs were

[jira] [Commented] (HADOOP-16251) ABFS: add FSMainOperationsBaseTest

2019-05-02 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16831967#comment-16831967 ] Aaron Fabbri commented on HADOOP-16251: --- FYI: Filed HADOOP-16291 to address the errors in the FS

[jira] [Created] (HADOOP-16291) HDFS Permissions Guide appears incorrect about getFileStatus()/getFileInfo()

2019-05-02 Thread Aaron Fabbri (JIRA)
Aaron Fabbri created HADOOP-16291: - Summary: HDFS Permissions Guide appears incorrect about getFileStatus()/getFileInfo() Key: HADOOP-16291 URL: https://issues.apache.org/jira/browse/HADOOP-16291

[jira] [Updated] (HADOOP-16291) HDFS Permissions Guide appears incorrect about getFileStatus()/getFileInfo()

2019-05-02 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-16291: -- Labels: newbie (was: ) > HDFS Permissions Guide appears incorrect about

[jira] [Comment Edited] (HADOOP-16251) ABFS: add FSMainOperationsBaseTest

2019-05-02 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16831936#comment-16831936 ] Aaron Fabbri edited comment on HADOOP-16251 at 5/2/19 8:31 PM: --- Thanks

[jira] [Commented] (HADOOP-16251) ABFS: add FSMainOperationsBaseTest

2019-05-02 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16831936#comment-16831936 ] Aaron Fabbri commented on HADOOP-16251: --- Thanks for the patch [~DanielZhou]. We really appreciate

[jira] [Commented] (HADOOP-16221) S3Guard: fail write that doesn't update metadata store

2019-04-30 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830848#comment-16830848 ] Aaron Fabbri commented on HADOOP-16221: --- Sorry I didn't see this until now. Thanks for the

[jira] [Commented] (HADOOP-16252) Use configurable dynamo table name prefix in S3Guard tests

2019-04-17 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820403#comment-16820403 ] Aaron Fabbri commented on HADOOP-16252: --- +1 LGTM. You're only affecting the table names used for

[jira] [Commented] (HADOOP-16118) S3Guard to support on-demand DDB tables

2019-04-10 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815057#comment-16815057 ] Aaron Fabbri commented on HADOOP-16118: --- +1 on the patch. Thank you for the contribution

[jira] [Commented] (HADOOP-16193) add extra S3A MPU test to see what happens if a file is created during the MPU

2019-04-04 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16810320#comment-16810320 ] Aaron Fabbri commented on HADOOP-16193: --- Patch looks good to me but I had a failure running the

[jira] [Commented] (HADOOP-16188) s3a rename failed during copy, "Unable to copy part" + 200 error code

2019-03-14 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16793058#comment-16793058 ] Aaron Fabbri commented on HADOOP-16188: --- uhhh.. I'm shaking my head here. S3 is sending a

[jira] [Commented] (HADOOP-16169) ABFS: Bug fix for getPathProperties

2019-03-07 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16787507#comment-16787507 ] Aaron Fabbri commented on HADOOP-16169: --- +1 Patch looks good to me [~DanielZhou]. Yetus is happy

[jira] [Commented] (HADOOP-16119) KMS on Hadoop RPC Engine

2019-02-28 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16780873#comment-16780873 ] Aaron Fabbri commented on HADOOP-16119: --- Thank you for writing this up [~jojochuang]. The doc

[jira] [Commented] (HADOOP-16085) S3Guard: use object version to protect against inconsistent read after replace/overwrite

2019-02-07 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763078#comment-16763078 ] Aaron Fabbri commented on HADOOP-16085: --- {quote}My other concern is that this requires enabling

[jira] [Commented] (HADOOP-16085) S3Guard: use object version to protect against inconsistent read after replace/overwrite

2019-01-31 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16757837#comment-16757837 ] Aaron Fabbri commented on HADOOP-16085: --- Hi guys. We've thought about this issue a little in the

[jira] [Commented] (HADOOP-15229) Add FileSystem builder-based openFile() API to match createFile() + S3 Select

2019-01-24 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16751744#comment-16751744 ] Aaron Fabbri commented on HADOOP-15229: --- I looked at the latest large patch today. Also played

[jira] [Commented] (HADOOP-15229) Add FileSystem builder-based openFile() API to match createFile() + S3 Select

2019-01-18 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16746835#comment-16746835 ] Aaron Fabbri commented on HADOOP-15229: --- Thanks for the discussion [~ste...@apache.org]. No

[jira] [Commented] (HADOOP-15229) Add FileSystem builder-based openFile() API to match createFile()

2018-12-20 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16726307#comment-16726307 ] Aaron Fabbri commented on HADOOP-15229: --- Finished the rest of the v14 diff. Looks good overall. I

[jira] [Commented] (HADOOP-15819) S3A integration test failures: FileSystem is closed! - without parallel test run

2018-12-20 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16726305#comment-16726305 ] Aaron Fabbri commented on HADOOP-15819: --- Just catching up on this as I hit the bug again. Sounds

[jira] [Commented] (HADOOP-15229) Add FileSystem builder-based openFile() API to match createFile()

2018-12-20 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16726235#comment-16726235 ] Aaron Fabbri commented on HADOOP-15229: --- Nice work here, as usual [~ste...@apache.org].

[jira] [Commented] (HADOOP-15229) Add FileSystem builder-based openFile() API to match createFile()

2018-12-20 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16726157#comment-16726157 ] Aaron Fabbri commented on HADOOP-15229: --- Applying and testing the v14 patch now while I read over

[jira] [Commented] (HADOOP-16015) Add bouncycastle jars to hadoop-aws as test dependencies

2018-12-19 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-16015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16725504#comment-16725504 ] Aaron Fabbri commented on HADOOP-16015: --- +1 Looks good to me. > Add bouncycastle jars to

[jira] [Commented] (HADOOP-14556) S3A to support Delegation Tokens

2018-12-19 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16725483#comment-16725483 ] Aaron Fabbri commented on HADOOP-14556: --- Hi [~ste...@apache.org]. Looks like you are needing

[jira] [Commented] (HADOOP-15999) [s3a] Better support for out-of-band operations

2018-12-12 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16719456#comment-16719456 ] Aaron Fabbri commented on HADOOP-15999: --- Was going to link HADOOP-15780 here, but [~gabor.bota]

[jira] [Commented] (HADOOP-15229) Add FileSystem builder-based openFile() API to match createFile()

2018-11-30 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16705291#comment-16705291 ] Aaron Fabbri commented on HADOOP-15229: --- Looks good overall [~ste...@apache.org]. Good

[jira] [Commented] (HADOOP-15229) Add FileSystem builder-based openFile() API to match createFile()

2018-11-27 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701357#comment-16701357 ] Aaron Fabbri commented on HADOOP-15229: --- Hey [~ste...@apache.org] been on vacation but will try

[jira] [Commented] (HADOOP-15621) S3Guard: Implement time-based (TTL) expiry for Authoritative Directory Listing

2018-10-03 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636549#comment-16636549 ] Aaron Fabbri commented on HADOOP-15621: --- Apologies, my mistake. Should be fixed now. > S3Guard:

[jira] [Updated] (HADOOP-15621) S3Guard: Implement time-based (TTL) expiry for Authoritative Directory Listing

2018-10-02 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-15621: -- Resolution: Fixed Fix Version/s: 3.3.0 Status: Resolved (was: Patch

[jira] [Commented] (HADOOP-15621) S3Guard: Implement time-based (TTL) expiry for Authoritative Directory Listing

2018-10-02 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636139#comment-16636139 ] Aaron Fabbri commented on HADOOP-15621: --- v2 patch looks great. I'm really happy with how small

[jira] [Created] (HADOOP-15780) S3Guard: document how to deal with non-S3Guard processes writing data to S3Guarded buckets

2018-09-20 Thread Aaron Fabbri (JIRA)
Aaron Fabbri created HADOOP-15780: - Summary: S3Guard: document how to deal with non-S3Guard processes writing data to S3Guarded buckets Key: HADOOP-15780 URL: https://issues.apache.org/jira/browse/HADOOP-15780

[jira] [Commented] (HADOOP-15489) S3Guard to self update on directory listings of S3

2018-09-20 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16622914#comment-16622914 ] Aaron Fabbri commented on HADOOP-15489: --- [~ste...@apache.org] that was the original behavior and

[jira] [Updated] (HADOOP-15489) S3Guard to self update on directory listings of S3

2018-09-20 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-15489: -- Priority: Minor (was: Major) > S3Guard to self update on directory listings of S3 >

[jira] [Updated] (HADOOP-15779) S3guard: add inconsistency detection metrics

2018-09-20 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-15779: -- Issue Type: Sub-task (was: Improvement) Parent: HADOOP-15619 > S3guard: add

[jira] [Created] (HADOOP-15779) S3guard: add inconsistency detection metrics

2018-09-20 Thread Aaron Fabbri (JIRA)
Aaron Fabbri created HADOOP-15779: - Summary: S3guard: add inconsistency detection metrics Key: HADOOP-15779 URL: https://issues.apache.org/jira/browse/HADOOP-15779 Project: Hadoop Common

[jira] [Updated] (HADOOP-15779) S3guard: add inconsistency detection metrics

2018-09-20 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-15779: -- Issue Type: Improvement (was: Bug) > S3guard: add inconsistency detection metrics >

[jira] [Commented] (HADOOP-15748) S3 listing inconsistency can raise NPE in globber

2018-09-19 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16621174#comment-16621174 ] Aaron Fabbri commented on HADOOP-15748: --- Easy +1 by inspection (haven't tested it): looks like an

[jira] [Commented] (HADOOP-15773) ABFS: Fix issues raised by Yetus

2018-09-19 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16620850#comment-16620850 ] Aaron Fabbri commented on HADOOP-15773: --- Patch looks good, +1. I did not apply the patch on top

[jira] [Commented] (HADOOP-15754) s3guard: testDynamoTableTagging should clear existing config

2018-09-13 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613804#comment-16613804 ] Aaron Fabbri commented on HADOOP-15754: --- wild guess: Gabor and I have a region set in our test

[jira] [Commented] (HADOOP-15426) Make S3guard client resilient to DDB throttle events and network failures

2018-09-12 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16613007#comment-16613007 ] Aaron Fabbri commented on HADOOP-15426: --- filed INFRA-17015 for the Hadoop-trunk-Commit failure.

[jira] [Commented] (HADOOP-15426) Make S3guard client resilient to DDB throttle events and network failures

2018-09-12 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612991#comment-16612991 ] Aaron Fabbri commented on HADOOP-15426: --- post-commit test failure seems to be an infra issue?

[jira] [Updated] (HADOOP-15426) Make S3guard client resilient to DDB throttle events and network failures

2018-09-12 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-15426: -- Resolution: Fixed Fix Version/s: 3.2.0 Status: Resolved (was: Patch

[jira] [Commented] (HADOOP-14734) add option to tag DDB table(s) created

2018-09-12 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612981#comment-16612981 ] Aaron Fabbri commented on HADOOP-14734: --- FYI [~gabor.bota] just filed HADOOP-15754 which is low

[jira] [Created] (HADOOP-15754) s3guard: testDynamoTableTagging should clear existing config

2018-09-12 Thread Aaron Fabbri (JIRA)
Aaron Fabbri created HADOOP-15754: - Summary: s3guard: testDynamoTableTagging should clear existing config Key: HADOOP-15754 URL: https://issues.apache.org/jira/browse/HADOOP-15754 Project: Hadoop

[jira] [Commented] (HADOOP-15426) Make S3guard client resilient to DDB throttle events and network failures

2018-09-12 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612872#comment-16612872 ] Aaron Fabbri commented on HADOOP-15426: --- Yep. Testing and reviewing now. > Make S3guard client

[jira] [Updated] (HADOOP-14734) add option to tag DDB table(s) created

2018-09-12 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14734: -- Resolution: Fixed Fix Version/s: 3.2.0 Status: Resolved (was: Patch

[jira] [Commented] (HADOOP-14734) add option to tag DDB table(s) created

2018-09-12 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612840#comment-16612840 ] Aaron Fabbri commented on HADOOP-14734: --- I tested this in us-west-2 by: * Running all

[jira] [Commented] (HADOOP-14734) add option to tag DDB table(s) created

2018-09-12 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612754#comment-16612754 ] Aaron Fabbri commented on HADOOP-14734: --- Thanks for picking this up [~gabor.bota]. Your patch

[jira] [Updated] (HADOOP-15709) Move S3Guard LocalMetadataStore constants to org.apache.hadoop.fs.s3a.Constants

2018-09-07 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-15709: -- Fix Version/s: 3.2.0 > Move S3Guard LocalMetadataStore constants to >

[jira] [Updated] (HADOOP-15709) Move S3Guard LocalMetadataStore constants to org.apache.hadoop.fs.s3a.Constants

2018-09-07 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-15709: -- Resolution: Fixed Status: Resolved (was: Patch Available) Committed to trunk.

[jira] [Commented] (HADOOP-15709) Move S3Guard LocalMetadataStore constants to org.apache.hadoop.fs.s3a.Constants

2018-09-07 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16607402#comment-16607402 ] Aaron Fabbri commented on HADOOP-15709: --- +1 LGTM. > Move S3Guard LocalMetadataStore constants to

[jira] [Updated] (HADOOP-15621) s3guard: implement time-based (TTL) expiry for DynamoDB Metadata Store

2018-09-06 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-15621: -- Description: Similar to HADOOP-13649, I think we should add a TTL (time to live) feature to

[jira] [Commented] (HADOOP-15621) s3guard: implement time-based (TTL) expiry for DynamoDB Metadata Store

2018-09-06 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16606497#comment-16606497 ] Aaron Fabbri commented on HADOOP-15621: --- Thanks for the v1 patch. You've done some nice work

[jira] [Commented] (HADOOP-15426) Make S3guard client resilient to DDB throttle events and network failures

2018-09-04 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603775#comment-16603775 ] Aaron Fabbri commented on HADOOP-15426: --- FYI I also ran through the scale tests without any

[jira] [Commented] (HADOOP-15426) Make S3guard client resilient to DDB throttle events and network failures

2018-09-04 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603774#comment-16603774 ] Aaron Fabbri commented on HADOOP-15426: --- Looking at v9 patch. Integration tests passed in

[jira] [Commented] (HADOOP-14630) Contract Tests to verify create, mkdirs and rename under a file is forbidden

2018-08-29 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596817#comment-16596817 ] Aaron Fabbri commented on HADOOP-14630: --- +1 on the patch as soon as you get a green result from

[jira] [Commented] (HADOOP-15107) Stabilize/tune S3A committers; review correctness & docs

2018-08-29 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596784#comment-16596784 ] Aaron Fabbri commented on HADOOP-15107: --- +1 on the v4 patch. Code all looks good. {noformat} +

[jira] [Commented] (HADOOP-15107) Stabilize/tune S3A committers; review correctness & docs

2018-08-29 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596683#comment-16596683 ] Aaron Fabbri commented on HADOOP-15107: --- I don't want to rob others of the joys of learning the

[jira] [Commented] (HADOOP-15621) s3guard: implement time-based (TTL) expiry for DynamoDB Metadata Store

2018-08-28 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595930#comment-16595930 ] Aaron Fabbri commented on HADOOP-15621: --- Hey, thank you for working on this. {quote} The

[jira] [Updated] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)

2018-08-17 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14154: -- Resolution: Fixed Fix Version/s: 3.2.0 Status: Resolved (was: Patch

[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)

2018-08-16 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16583310#comment-16583310 ] Aaron Fabbri commented on HADOOP-14154: --- v7 patch: minor checkstyle cleanups. > Persist

[jira] [Updated] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)

2018-08-16 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14154: -- Attachment: HADOOP-14154.007.patch > Persist isAuthoritative bit in DynamoDBMetaStore

[jira] [Updated] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)

2018-08-16 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14154: -- Attachment: HADOOP-14154.006.patch > Persist isAuthoritative bit in DynamoDBMetaStore

[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)

2018-08-16 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16583161#comment-16583161 ] Aaron Fabbri commented on HADOOP-14154: --- Nice work [~gabor.bota]. v5 patch looks good.  I'm

[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)

2018-08-14 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16580248#comment-16580248 ] Aaron Fabbri commented on HADOOP-14154: --- Ah, interesting.  The existing code appears to work ok

[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)

2018-08-13 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16579034#comment-16579034 ] Aaron Fabbri commented on HADOOP-14154: --- Interesting results.  The fact that you had to make that

[jira] [Commented] (HADOOP-14154) Persist isAuthoritative bit in DynamoDBMetaStore (authoritative mode support)

2018-08-08 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16574200#comment-16574200 ] Aaron Fabbri commented on HADOOP-14154: --- Attached a tiny patch and script (perf-eval-v1.diff and

  1   2   3   4   5   6   7   8   9   10   >