[jira] [Comment Edited] (HADOOP-14927) ITestS3GuardTool failures in testDestroyNoBucket()

2018-02-13 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363379#comment-16363379 ] Aaron Fabbri edited comment on HADOOP-14927 at 2/14/18 2:21 AM: v1

[jira] [Updated] (HADOOP-14927) ITestS3GuardTool failures in testDestroyNoBucket()

2018-02-13 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14927: -- Attachment: HADOOP-14927.001.patch > ITestS3GuardTool failures in testDestroyNoBucket() >

[jira] [Updated] (HADOOP-14927) ITestS3GuardTool failures in testDestroyNoBucket()

2018-02-13 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14927: -- Status: Patch Available (was: In Progress) v1 patch. Also fix a nearby javadoc error >

[jira] [Work started] (HADOOP-14927) ITestS3GuardTool failures in testDestroyNoBucket()

2018-02-13 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-14927 started by Aaron Fabbri. - > ITestS3GuardTool failures in testDestroyNoBucket() >

[jira] [Commented] (HADOOP-14927) ITestS3GuardTool failures in testDestroyNoBucket()

2018-02-13 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363190#comment-16363190 ] Aaron Fabbri commented on HADOOP-14927: --- I do see it occasionally, just haven't bothered to post a

[jira] [Commented] (HADOOP-15216) S3AInputStream to handle reconnect on read() failure better

2018-02-09 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16359139#comment-16359139 ] Aaron Fabbri commented on HADOOP-15216: --- {quote} We could think about extending the fault

[jira] [Commented] (HADOOP-15216) S3AInputStream to handle reconnect on read() failure better

2018-02-08 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16357725#comment-16357725 ] Aaron Fabbri commented on HADOOP-15216: --- I'm working on a patch that uses {{retry()}} for

[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-02-07 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356378#comment-16356378 ] Aaron Fabbri commented on HADOOP-13761: --- Thanks for the feedback on the scale test patch

[jira] [Commented] (HADOOP-15183) S3Guard store becomes inconsistent after partial failure of rename

2018-02-06 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16354870#comment-16354870 ] Aaron Fabbri commented on HADOOP-15183: --- Looking at rename path for HADOOP-13761.  Rename is

[jira] [Commented] (HADOOP-15040) Upgrade AWS SDK: NPE bug spams logs w/ Yarn Log Aggregation

2018-02-02 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351133#comment-16351133 ] Aaron Fabbri commented on HADOOP-15040: --- Ping folks for easy review. [~sean_impala_9b93]

[jira] [Commented] (HADOOP-15040) Upgrade AWS SDK: NPE bug spams logs w/ Yarn Log Aggregation

2018-02-02 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351122#comment-16351122 ] Aaron Fabbri commented on HADOOP-15040: --- Tests w/o S3Guard look good.  I think we should get this

[jira] [Updated] (HADOOP-15040) Upgrade AWS SDK: NPE bug spams logs w/ Yarn Log Aggregation

2018-02-02 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-15040: -- Status: Patch Available (was: In Progress) Attaching patch to bump AWS SDK to latest

[jira] [Updated] (HADOOP-15040) Upgrade AWS SDK: NPE bug spams logs w/ Yarn Log Aggregation

2018-02-02 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-15040: -- Attachment: HADOOP-15040.001.patch > Upgrade AWS SDK: NPE bug spams logs w/ Yarn Log

[jira] [Updated] (HADOOP-15040) Upgrade AWS SDK: NPE bug spams logs w/ Yarn Log Aggregation

2018-02-02 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-15040: -- Attachment: HADOOP-12502-07.patch > Upgrade AWS SDK: NPE bug spams logs w/ Yarn Log

[jira] [Updated] (HADOOP-15040) Upgrade AWS SDK: NPE bug spams logs w/ Yarn Log Aggregation

2018-02-02 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-15040: -- Attachment: (was: HADOOP-12502-07.patch) > Upgrade AWS SDK: NPE bug spams logs w/ Yarn

[jira] [Resolved] (HADOOP-15207) Hadoop performance Issues

2018-02-01 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri resolved HADOOP-15207. --- Resolution: Invalid Can someone please ban this user? Similar SEO spam has been pasted on

[jira] [Commented] (HADOOP-15207) Hadoop performance Issues

2018-02-01 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349859#comment-16349859 ] Aaron Fabbri commented on HADOOP-15207: --- Please refrain from posting SEO spam to the Apache JIRA:

[jira] [Comment Edited] (HADOOP-15191) Add Private/Unstable BulkDelete operations to supporting object stores for DistCP

2018-02-01 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349840#comment-16349840 ] Aaron Fabbri edited comment on HADOOP-15191 at 2/2/18 6:17 AM: --- I like the

[jira] [Commented] (HADOOP-15191) Add Private/Unstable BulkDelete operations to supporting object stores for DistCP

2018-02-01 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349840#comment-16349840 ] Aaron Fabbri commented on HADOOP-15191: --- I like the idea of an optional batch delete interface,

[jira] [Commented] (HADOOP-15040) Upgrade AWS SDK: NPE bug spams logs w/ Yarn Log Aggregation

2018-01-30 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346069#comment-16346069 ] Aaron Fabbri commented on HADOOP-15040: --- Doing some testing with latest SDK 1.11.271 today.

[jira] [Updated] (HADOOP-15040) Upgrade AWS SDK: NPE bug spams logs w/ Yarn Log Aggregation

2018-01-30 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-15040: -- Summary: Upgrade AWS SDK: NPE bug spams logs w/ Yarn Log Aggregation (was: AWS SDK NPE bug

[jira] [Work started] (HADOOP-15040) AWS SDK NPE bug spams logs w/ Yarn Log Aggregation

2018-01-30 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-15040 started by Aaron Fabbri. - > AWS SDK NPE bug spams logs w/ Yarn Log Aggregation >

[jira] [Comment Edited] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-01-26 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341807#comment-16341807 ] Aaron Fabbri edited comment on HADOOP-13761 at 1/26/18 11:52 PM: - v2

[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-01-26 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16341807#comment-16341807 ] Aaron Fabbri commented on HADOOP-13761: --- v2 patch is same as v1 (just a test which tries to

[jira] [Updated] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-01-26 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13761: -- Attachment: HADOOP-13761.002.patch > S3Guard: implement retries for DDB failures and

[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-01-25 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16340137#comment-16340137 ] Aaron Fabbri commented on HADOOP-13761: --- Attached v1 RFC patch that just adds a test that attempts

[jira] [Updated] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-01-25 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13761: -- Attachment: HADOOP-13761.001.patch > S3Guard: implement retries for DDB failures and

[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-01-24 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16338852#comment-16338852 ] Aaron Fabbri commented on HADOOP-13761: --- Wrote a test to check above hypothesis on batch size vs.

[jira] [Commented] (HADOOP-14918) remove the Local Dynamo DB test option

2018-01-24 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16338645#comment-16338645 ] Aaron Fabbri commented on HADOOP-14918: --- You should be able to just change it to an ITest and set

[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-01-24 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16338609#comment-16338609 ] Aaron Fabbri commented on HADOOP-13761: --- {quote} Managed to break tests when working with a bucket

[jira] [Commented] (HADOOP-14918) remove the Local Dynamo DB test option

2018-01-24 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16338413#comment-16338413 ] Aaron Fabbri commented on HADOOP-14918: --- Patch looks good except we want to keep

[jira] [Commented] (HADOOP-12502) SetReplication OutOfMemoryError

2018-01-23 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-12502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336994#comment-16336994 ] Aaron Fabbri commented on HADOOP-12502: --- Thanks for the updated v9 patch. This one has some good

[jira] [Commented] (HADOOP-12502) SetReplication OutOfMemoryError

2018-01-19 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-12502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16332968#comment-16332968 ] Aaron Fabbri commented on HADOOP-12502: --- Thank you for the new patch [~vinayrpet] and for working

[jira] [Commented] (HADOOP-13974) S3Guard CLI to support list/purge of pending multipart commits

2018-01-17 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16329685#comment-16329685 ] Aaron Fabbri commented on HADOOP-13974: --- Hey [~ste...@apache.org] just noticed some unrelated

[jira] [Assigned] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-01-17 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri reassigned HADOOP-13761: - Assignee: Aaron Fabbri (was: Steve Loughran) > S3Guard: implement retries for DDB

[jira] [Commented] (HADOOP-15176) Enhance IAM assumed role support in S3A client

2018-01-17 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16329310#comment-16329310 ] Aaron Fabbri commented on HADOOP-15176: --- {quote} Patch 001, still WiP {quote} Ok, I won't

[jira] [Updated] (HADOOP-15141) Support IAM Assumed roles in S3A

2018-01-17 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-15141: -- Resolution: Fixed Fix Version/s: 3.1.0 Status: Resolved (was: Patch

[jira] [Commented] (HADOOP-15141) Support IAM Assumed roles in S3A

2018-01-17 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328447#comment-16328447 ] Aaron Fabbri commented on HADOOP-15141: --- Test failures look unrelated (TestCopyFromLocal works

[jira] [Commented] (HADOOP-15141) Support IAM Assumed roles in S3A

2018-01-16 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328290#comment-16328290 ] Aaron Fabbri commented on HADOOP-15141: --- v6 patch: just fixing checkstyle and whitespace.  Tested

[jira] [Updated] (HADOOP-15141) Support IAM Assumed roles in S3A

2018-01-16 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-15141: -- Attachment: HADOOP-15141-006.patch > Support IAM Assumed roles in S3A >

[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-01-16 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328003#comment-16328003 ] Aaron Fabbri commented on HADOOP-13761: --- [~ste...@apache.org] sure. > S3Guard: implement retries

[jira] [Commented] (HADOOP-15079) ITestS3AFileOperationCost#testFakeDirectoryDeletion failing after OutputCommitter patch

2018-01-12 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324829#comment-16324829 ] Aaron Fabbri commented on HADOOP-15079: --- I'm +1 on v2 patch assuming clean Yetus (i can restart

[jira] [Commented] (HADOOP-15079) ITestS3AFileOperationCost#testFakeDirectoryDeletion failing after OutputCommitter patch

2018-01-10 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16321714#comment-16321714 ] Aaron Fabbri commented on HADOOP-15079: --- Integration tests all passed with S3Guard (local) and

[jira] [Assigned] (HADOOP-14927) ITestS3GuardTool failures in testDestroyNoBucket()

2018-01-10 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri reassigned HADOOP-14927: - Assignee: Aaron Fabbri > ITestS3GuardTool failures in testDestroyNoBucket() >

[jira] [Commented] (HADOOP-14927) ITestS3GuardTool failures in testDestroyNoBucket()

2018-01-10 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16321710#comment-16321710 ] Aaron Fabbri commented on HADOOP-14927: --- Just hit this again so I will put up a patch soon...

[jira] [Updated] (HADOOP-14927) ITestS3GuardTool failures in testDestroyNoBucket()

2018-01-10 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14927: -- Issue Type: Sub-task (was: Bug) Parent: HADOOP-14825 > ITestS3GuardTool failures in

[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-01-10 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16321643#comment-16321643 ] Aaron Fabbri commented on HADOOP-13761: --- Reminder comment here: We should also: - Update

[jira] [Commented] (HADOOP-15079) ITestS3AFileOperationCost#testFakeDirectoryDeletion failing after OutputCommitter patch

2018-01-10 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16321527#comment-16321527 ] Aaron Fabbri commented on HADOOP-15079: --- Fragile bits of code.. thanks for digging in to this.

[jira] [Commented] (HADOOP-15141) Support IAM Assumed roles in S3A

2018-01-09 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319502#comment-16319502 ] Aaron Fabbri commented on HADOOP-15141: --- Javac are deprecation warnings on AWS API. Checkstyle

[jira] [Commented] (HADOOP-15140) S3guard mistakes root URI without / as non-absolute path

2018-01-09 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319489#comment-16319489 ] Aaron Fabbri commented on HADOOP-15140: --- I believe [~abrahamfine] was looking for noob JIRAs to

[jira] [Commented] (HADOOP-15141) Support IAM Assumed roles in S3A

2018-01-08 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16317596#comment-16317596 ] Aaron Fabbri commented on HADOOP-15141: --- Thanks for the contribution. I just ran integration tests

[jira] [Commented] (HADOOP-15006) Encrypt S3A data client-side with Hadoop libraries & Hadoop KMS

2018-01-05 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314171#comment-16314171 ] Aaron Fabbri commented on HADOOP-15006: --- Thanks again for writing this up [~moist]--it is very

[jira] [Commented] (HADOOP-13282) S3 blob etags to be made visible in status/getFileChecksum() calls

2017-12-20 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299308#comment-16299308 ] Aaron Fabbri commented on HADOOP-13282: --- Nice. This looks good. I see that checksum equality

[jira] [Commented] (HADOOP-15113) NPE in S3A getFileStatus: null instrumentation on using closed instance

2017-12-20 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16299130#comment-16299130 ] Aaron Fabbri commented on HADOOP-15113: --- Looks good to me, +1. (Tested in us west 2) Volatile

[jira] [Commented] (HADOOP-14965) s3a input stream "normal" fadvise mode to be adaptive

2017-12-19 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297868#comment-16297868 ] Aaron Fabbri commented on HADOOP-14965: --- Nice small patch, thank you. This looks good to me. +1

[jira] [Updated] (HADOOP-13974) S3a CLI to support list/purge of pending multipart commits

2017-12-15 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13974: -- Attachment: HADOOP-13974.007.patch v7 patch: Add two more eventually() wrappers to other test

[jira] [Commented] (HADOOP-13974) S3a CLI to support list/purge of pending multipart commits

2017-12-15 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16293366#comment-16293366 ] Aaron Fabbri commented on HADOOP-13974: --- Ok, I think this is ready. I did more testing and this

[jira] [Commented] (HADOOP-12502) SetReplication OutOfMemoryError

2017-12-14 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-12502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16291918#comment-16291918 ] Aaron Fabbri commented on HADOOP-12502: --- Hi, thanks for the work on this [~vinayrpet] and

[jira] [Updated] (HADOOP-13974) S3a CLI to support list/purge of pending multipart commits

2017-12-14 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13974: -- Attachment: HADOOP-13974.006.patch [~ste...@apache.org] this is almost ready. I had to add

[jira] [Updated] (HADOOP-13974) S3a CLI to support list/purge of pending multipart commits

2017-12-12 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13974: -- Attachment: HADOOP-13974.005.patch Posting v5 patch with feedback from [~ste...@apache.org].

[jira] [Commented] (HADOOP-15006) Encrypt S3A data client-side with Hadoop libraries & Hadoop KMS

2017-12-11 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16286869#comment-16286869 ] Aaron Fabbri commented on HADOOP-15006: --- Thanks for the reminder.. Will try to take a look this

[jira] [Updated] (HADOOP-13974) S3a CLI to support list/purge of pending multipart commits

2017-12-07 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13974: -- Attachment: HADOOP-13974.004.patch Attaching v4 patch. - Reworks code after related changes

[jira] [Comment Edited] (HADOOP-13974) S3a CLI to support list/purge of pending multipart commits

2017-12-07 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16282675#comment-16282675 ] Aaron Fabbri edited comment on HADOOP-13974 at 12/7/17 11:06 PM: -

[jira] [Updated] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file

2017-12-05 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14475: -- Resolution: Fixed Fix Version/s: 3.1.0 Status: Resolved (was: Patch

[jira] [Commented] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file

2017-12-05 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16279060#comment-16279060 ] Aaron Fabbri commented on HADOOP-14475: --- Also apology for the typo, I misspelled your name in the

[jira] [Commented] (HADOOP-15071) s3a troubleshooting docs to add a couple more failure modes

2017-12-04 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277823#comment-16277823 ] Aaron Fabbri commented on HADOOP-15071: --- +1 LGTM. > s3a troubleshooting docs to add a couple more

[jira] [Commented] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file

2017-12-04 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16277782#comment-16277782 ] Aaron Fabbri commented on HADOOP-14475: --- Latest patch looks good to me. Tested w/ and w/o s3guard

[jira] [Commented] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file

2017-11-30 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16273606#comment-16273606 ] Aaron Fabbri commented on HADOOP-14475: --- v15 patch looks good with minor change: {noformat} +its

[jira] [Commented] (HADOOP-13974) S3a CLI to support list/purge of pending multipart commits

2017-11-29 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16271677#comment-16271677 ] Aaron Fabbri commented on HADOOP-13974: --- This patch probably needs significant rewrite now that

[jira] [Commented] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file

2017-11-29 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16271668#comment-16271668 ] Aaron Fabbri commented on HADOOP-14475: --- I added some instrumentation to detect use after close

[jira] [Commented] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file

2017-11-28 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270034#comment-16270034 ] Aaron Fabbri commented on HADOOP-14475: --- Testing v15 patch I'm seeing occasional NPEs in test

[jira] [Commented] (HADOOP-15003) Merge S3A committers into trunk: Yetus patch checker

2017-11-21 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261384#comment-16261384 ] Aaron Fabbri commented on HADOOP-15003: --- My testing in us-west-2 looks good. +1 latest patch.

[jira] [Commented] (HADOOP-15003) Merge S3A committers into trunk: Yetus patch checker

2017-11-21 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261328#comment-16261328 ] Aaron Fabbri commented on HADOOP-15003: --- Testing latest patch. Latest commits look good (trivial

[jira] [Commented] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file

2017-11-17 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16257757#comment-16257757 ] Aaron Fabbri commented on HADOOP-14475: --- Some initial comments on v11-- in addition to above. - I

[jira] [Commented] (HADOOP-15003) Merge S3A committers into trunk: Yetus patch checker

2017-11-17 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16257444#comment-16257444 ] Aaron Fabbri commented on HADOOP-15003: --- Good point about this being a potential failure mode. +1

[jira] [Updated] (HADOOP-15003) Merge S3A committers into trunk: Yetus patch checker

2017-11-17 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-15003: -- Status: Open (was: Patch Available) > Merge S3A committers into trunk: Yetus patch checker >

[jira] [Updated] (HADOOP-15003) Merge S3A committers into trunk: Yetus patch checker

2017-11-17 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-15003: -- Attachment: HADOOP-15033-testfix-1.diff (Attaching a small patch.) Busy day today but I

[jira] [Commented] (HADOOP-15003) Merge S3A committers into trunk: Yetus patch checker

2017-11-16 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16255965#comment-16255965 ] Aaron Fabbri commented on HADOOP-15003: --- The cause of the {{ITestS3AHugeMagicCommits}} failure I'm

[jira] [Commented] (HADOOP-15003) Merge S3A committers into trunk: Yetus patch checker

2017-11-15 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16254854#comment-16254854 ] Aaron Fabbri commented on HADOOP-15003: --- FYI looks like I may have reproduced the parallel test

[jira] [Commented] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file

2017-11-15 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16254647#comment-16254647 ] Aaron Fabbri commented on HADOOP-14475: --- There is a learning curve for this Metrics2 stuff. I

[jira] [Commented] (HADOOP-14475) Metrics of S3A don't print out when enable it in Hadoop metrics property file

2017-11-15 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16254283#comment-16254283 ] Aaron Fabbri commented on HADOOP-14475: --- I will take a look at v9 patch. Noticing some checkstyle

[jira] [Created] (HADOOP-15040) AWS SDK NPE bug spams logs w/ Yarn Log Aggregation

2017-11-14 Thread Aaron Fabbri (JIRA)
Aaron Fabbri created HADOOP-15040: - Summary: AWS SDK NPE bug spams logs w/ Yarn Log Aggregation Key: HADOOP-15040 URL: https://issues.apache.org/jira/browse/HADOOP-15040 Project: Hadoop Common

[jira] [Commented] (HADOOP-15038) Abstract MetadataStore in S3Guard into a common module.

2017-11-14 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252646#comment-16252646 ] Aaron Fabbri commented on HADOOP-15038: --- I am happy to move {{MetadataStore}} and related

[jira] [Commented] (HADOOP-14098) AliyunOSS: improve the performance of object metadata operation

2017-11-14 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252610#comment-16252610 ] Aaron Fabbri commented on HADOOP-14098: --- Thanks for sharing your POC work. For reasoning about

[jira] [Commented] (HADOOP-15003) Merge S3A committers into trunk: Yetus patch checker

2017-11-14 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252559#comment-16252559 ] Aaron Fabbri commented on HADOOP-15003: --- I completed 20 runs of scale and integration tests last

[jira] [Commented] (HADOOP-15003) Merge S3A committers into trunk: Yetus patch checker

2017-11-14 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16252549#comment-16252549 ] Aaron Fabbri commented on HADOOP-15003: --- Javadoc: First one looks like an infra issue. Second one

[jira] [Commented] (HADOOP-15003) Merge S3A committers into trunk: Yetus patch checker

2017-11-13 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16250698#comment-16250698 ] Aaron Fabbri commented on HADOOP-15003: --- I'm running the latest patch through my "repeat

[jira] [Commented] (HADOOP-15003) Merge S3A committers into trunk: Yetus patch checker

2017-11-08 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16245303#comment-16245303 ] Aaron Fabbri commented on HADOOP-15003: --- Ran through the parallel integration tests about 5 times

[jira] [Commented] (HADOOP-15003) Merge S3A committers into trunk: Yetus patch checker

2017-11-08 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244539#comment-16244539 ] Aaron Fabbri commented on HADOOP-15003: --- Interesting. Sorry I didn't have more useful comments

[jira] [Comment Edited] (HADOOP-15003) Merge S3A committers into trunk: Yetus patch checker

2017-11-07 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16243269#comment-16243269 ] Aaron Fabbri edited comment on HADOOP-15003 at 11/8/17 5:26 AM:

[jira] [Commented] (HADOOP-15003) Merge S3A committers into trunk: Yetus patch checker

2017-11-07 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16243269#comment-16243269 ] Aaron Fabbri commented on HADOOP-15003: --- {quote}That's not a minor failure though{quote} I was

[jira] [Commented] (HADOOP-15003) Merge S3A committers into trunk: Yetus patch checker

2017-11-07 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242825#comment-16242825 ] Aaron Fabbri commented on HADOOP-15003: --- Latest changes look OK. javac warnings are deprecation

[jira] [Commented] (HADOOP-15003) Merge S3A committers into trunk: Yetus patch checker

2017-11-07 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16242581#comment-16242581 ] Aaron Fabbri commented on HADOOP-15003: --- I agree trunk is the right place for this now that we've

[jira] [Commented] (HADOOP-15003) Merge S3A committers into trunk: Yetus patch checker

2017-11-06 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241246#comment-16241246 ] Aaron Fabbri commented on HADOOP-15003: --- Ok I reviewed all the newest changes on the [pull

[jira] [Commented] (HADOOP-15003) Merge S3A committers into trunk: Yetus patch checker

2017-11-06 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240664#comment-16240664 ] Aaron Fabbri commented on HADOOP-15003: --- Testing / reviewing the latest here. Your pull request

[jira] [Commented] (HADOOP-15003) Merge S3A committers into trunk: Yetus patch checker

2017-11-03 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16238047#comment-16238047 ] Aaron Fabbri commented on HADOOP-15003: --- Shout if you want help with getting these to be green. I

[jira] [Updated] (HADOOP-15003) Merge S3A committers into trunk: Yetus patch checker

2017-11-02 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-15003: -- Status: Patch Available (was: Open) > Merge S3A committers into trunk: Yetus patch checker >

[jira] [Commented] (HADOOP-14971) Merge S3A committers into trunk

2017-11-02 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16236906#comment-16236906 ] Aaron Fabbri commented on HADOOP-14971: --- Another random thing I noticed: {noformat} case

[jira] [Commented] (HADOOP-14576) s3guard DynamoDB resource not found: tables not ACTIVE state after initial connection

2017-11-02 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16236463#comment-16236463 ] Aaron Fabbri commented on HADOOP-14576: --- I've seen similar issues intermittently just running S3A

[jira] [Updated] (HADOOP-14576) s3guard DynamoDB resource not found: tables not ACTIVE state after initial connection

2017-11-02 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14576: -- Summary: s3guard DynamoDB resource not found: tables not ACTIVE state after initial

<    1   2   3   4   5   6   7   8   9   10   >