[jira] [Commented] (HADOOP-15141) Support IAM Assumed roles in S3A

2018-01-16 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16328290#comment-16328290 ] Aaron Fabbri commented on HADOOP-15141: --- v6 patch: just fixing checkstyle and whit

[jira] [Commented] (HADOOP-15141) Support IAM Assumed roles in S3A

2018-01-17 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16328447#comment-16328447 ] Aaron Fabbri commented on HADOOP-15141: --- Test failures look unrelated (TestCopyFro

[jira] [Updated] (HADOOP-15141) Support IAM Assumed roles in S3A

2018-01-17 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-15141: -- Resolution: Fixed Fix Version/s: 3.1.0 Status: Resolved (was: Patch Availabl

[jira] [Commented] (HADOOP-15176) Enhance IAM assumed role support in S3A client

2018-01-17 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16329310#comment-16329310 ] Aaron Fabbri commented on HADOOP-15176: --- {quote} Patch 001, still WiP {quote} O

[jira] [Assigned] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-01-17 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri reassigned HADOOP-13761: - Assignee: Aaron Fabbri (was: Steve Loughran) > S3Guard: implement retries for DDB fail

[jira] [Commented] (HADOOP-13974) S3Guard CLI to support list/purge of pending multipart commits

2018-01-17 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16329685#comment-16329685 ] Aaron Fabbri commented on HADOOP-13974: --- Hey [~ste...@apache.org] just noticed som

[jira] [Commented] (HADOOP-12502) SetReplication OutOfMemoryError

2018-01-19 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-12502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16332968#comment-16332968 ] Aaron Fabbri commented on HADOOP-12502: --- Thank you for the new patch [~vinayrpet]

[jira] [Commented] (HADOOP-12502) SetReplication OutOfMemoryError

2018-01-23 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-12502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16336994#comment-16336994 ] Aaron Fabbri commented on HADOOP-12502: --- Thanks for the updated v9 patch. This one

[jira] [Commented] (HADOOP-14918) remove the Local Dynamo DB test option

2018-01-24 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16338413#comment-16338413 ] Aaron Fabbri commented on HADOOP-14918: --- Patch looks good except we want to keep T

[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-01-24 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16338609#comment-16338609 ] Aaron Fabbri commented on HADOOP-13761: --- {quote} Managed to break tests when worki

[jira] [Commented] (HADOOP-14918) remove the Local Dynamo DB test option

2018-01-24 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16338645#comment-16338645 ] Aaron Fabbri commented on HADOOP-14918: --- You should be able to just change it to a

[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-01-24 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16338852#comment-16338852 ] Aaron Fabbri commented on HADOOP-13761: --- Wrote a test to check above hypothesis on

[jira] [Updated] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-01-25 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13761: -- Attachment: HADOOP-13761.001.patch > S3Guard: implement retries for DDB failures and throttlin

[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-01-25 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16340137#comment-16340137 ] Aaron Fabbri commented on HADOOP-13761: --- Attached v1 RFC patch that just adds a te

[jira] [Updated] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-01-26 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13761: -- Attachment: HADOOP-13761.002.patch > S3Guard: implement retries for DDB failures and throttlin

[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-01-26 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16341807#comment-16341807 ] Aaron Fabbri commented on HADOOP-13761: --- v2 patch is same as v1 (just a test which

[jira] [Comment Edited] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-01-26 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16341807#comment-16341807 ] Aaron Fabbri edited comment on HADOOP-13761 at 1/26/18 11:52 PM: -

[jira] [Work started] (HADOOP-15040) AWS SDK NPE bug spams logs w/ Yarn Log Aggregation

2018-01-30 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-15040 started by Aaron Fabbri. - > AWS SDK NPE bug spams logs w/ Yarn Log Aggregation > --

[jira] [Updated] (HADOOP-15040) Upgrade AWS SDK: NPE bug spams logs w/ Yarn Log Aggregation

2018-01-30 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-15040: -- Summary: Upgrade AWS SDK: NPE bug spams logs w/ Yarn Log Aggregation (was: AWS SDK NPE bug sp

[jira] [Commented] (HADOOP-15040) Upgrade AWS SDK: NPE bug spams logs w/ Yarn Log Aggregation

2018-01-30 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16346069#comment-16346069 ] Aaron Fabbri commented on HADOOP-15040: --- Doing some testing with latest SDK 1.11.2

[jira] [Commented] (HADOOP-15191) Add Private/Unstable BulkDelete operations to supporting object stores for DistCP

2018-02-01 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16349840#comment-16349840 ] Aaron Fabbri commented on HADOOP-15191: --- I like the idea of an optional batch dele

[jira] [Comment Edited] (HADOOP-15191) Add Private/Unstable BulkDelete operations to supporting object stores for DistCP

2018-02-01 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16349840#comment-16349840 ] Aaron Fabbri edited comment on HADOOP-15191 at 2/2/18 6:17 AM: ---

[jira] [Commented] (HADOOP-15207) Hadoop performance Issues

2018-02-01 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16349859#comment-16349859 ] Aaron Fabbri commented on HADOOP-15207: --- Please refrain from posting SEO spam to t

[jira] [Resolved] (HADOOP-15207) Hadoop performance Issues

2018-02-01 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri resolved HADOOP-15207. --- Resolution: Invalid Can someone please ban this user? Similar SEO spam has been pasted on ot

[jira] [Updated] (HADOOP-15040) Upgrade AWS SDK: NPE bug spams logs w/ Yarn Log Aggregation

2018-02-02 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-15040: -- Attachment: HADOOP-12502-07.patch > Upgrade AWS SDK: NPE bug spams logs w/ Yarn Log Aggregatio

[jira] [Updated] (HADOOP-15040) Upgrade AWS SDK: NPE bug spams logs w/ Yarn Log Aggregation

2018-02-02 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-15040: -- Attachment: (was: HADOOP-12502-07.patch) > Upgrade AWS SDK: NPE bug spams logs w/ Yarn Log

[jira] [Updated] (HADOOP-15040) Upgrade AWS SDK: NPE bug spams logs w/ Yarn Log Aggregation

2018-02-02 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-15040: -- Attachment: HADOOP-15040.001.patch > Upgrade AWS SDK: NPE bug spams logs w/ Yarn Log Aggregati

[jira] [Updated] (HADOOP-15040) Upgrade AWS SDK: NPE bug spams logs w/ Yarn Log Aggregation

2018-02-02 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-15040: -- Status: Patch Available (was: In Progress) Attaching patch to bump AWS SDK to latest version.

[jira] [Commented] (HADOOP-15040) Upgrade AWS SDK: NPE bug spams logs w/ Yarn Log Aggregation

2018-02-02 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351122#comment-16351122 ] Aaron Fabbri commented on HADOOP-15040: --- Tests w/o S3Guard look good.  I think we

[jira] [Commented] (HADOOP-15040) Upgrade AWS SDK: NPE bug spams logs w/ Yarn Log Aggregation

2018-02-02 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16351133#comment-16351133 ] Aaron Fabbri commented on HADOOP-15040: --- Ping folks for easy review. [~sean_impala

[jira] [Commented] (HADOOP-15183) S3Guard store becomes inconsistent after partial failure of rename

2018-02-06 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354870#comment-16354870 ] Aaron Fabbri commented on HADOOP-15183: --- Looking at rename path for HADOOP-13761. 

[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-02-07 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16356378#comment-16356378 ] Aaron Fabbri commented on HADOOP-13761: --- Thanks for the feedback on the scale test

[jira] [Commented] (HADOOP-15216) S3AInputStream to handle reconnect on read() failure better

2018-02-08 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16357725#comment-16357725 ] Aaron Fabbri commented on HADOOP-15216: --- I'm working on a patch that uses {{retry(

[jira] [Commented] (HADOOP-15216) S3AInputStream to handle reconnect on read() failure better

2018-02-09 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16359139#comment-16359139 ] Aaron Fabbri commented on HADOOP-15216: --- {quote} We could think about extending th

[jira] [Commented] (HADOOP-14927) ITestS3GuardTool failures in testDestroyNoBucket()

2018-02-13 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363190#comment-16363190 ] Aaron Fabbri commented on HADOOP-14927: --- I do see it occasionally, just haven't bo

[jira] [Work started] (HADOOP-14927) ITestS3GuardTool failures in testDestroyNoBucket()

2018-02-13 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-14927 started by Aaron Fabbri. - > ITestS3GuardTool failures in testDestroyNoBucket() > --

[jira] [Updated] (HADOOP-14927) ITestS3GuardTool failures in testDestroyNoBucket()

2018-02-13 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14927: -- Attachment: HADOOP-14927.001.patch > ITestS3GuardTool failures in testDestroyNoBucket() >

[jira] [Updated] (HADOOP-14927) ITestS3GuardTool failures in testDestroyNoBucket()

2018-02-13 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-14927: -- Status: Patch Available (was: In Progress) v1 patch. Also fix a nearby javadoc error > ITest

[jira] [Comment Edited] (HADOOP-14927) ITestS3GuardTool failures in testDestroyNoBucket()

2018-02-13 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363379#comment-16363379 ] Aaron Fabbri edited comment on HADOOP-14927 at 2/14/18 2:21 AM: --

[jira] [Updated] (HADOOP-15040) Upgrade AWS SDK to 1.11.271: NPE bug spams logs w/ Yarn Log Aggregation

2018-02-13 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-15040: -- Resolution: Fixed Status: Resolved (was: Patch Available) Committed to trunk. Thank y

[jira] [Updated] (HADOOP-15040) Upgrade AWS SDK to 1.11.271: NPE bug spams logs w/ Yarn Log Aggregation

2018-02-13 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-15040: -- Fix Version/s: 3.1.0 > Upgrade AWS SDK to 1.11.271: NPE bug spams logs w/ Yarn Log Aggregation

[jira] [Commented] (HADOOP-15040) Upgrade AWS SDK to 1.11.271: NPE bug spams logs w/ Yarn Log Aggregation

2018-02-13 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363462#comment-16363462 ] Aaron Fabbri commented on HADOOP-15040: --- Will also commit branch-3.0 in a moment,

[jira] [Updated] (HADOOP-15040) Upgrade AWS SDK to 1.11.271: NPE bug spams logs w/ Yarn Log Aggregation

2018-02-13 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-15040: -- Fix Version/s: 3.0.2 > Upgrade AWS SDK to 1.11.271: NPE bug spams logs w/ Yarn Log Aggregation

[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-02-13 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16363517#comment-16363517 ] Aaron Fabbri commented on HADOOP-13761: --- Some entertaining logs from the new scale

[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-02-14 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364725#comment-16364725 ] Aaron Fabbri commented on HADOOP-13761: --- Finishing up the fail-on-read after open(

[jira] [Commented] (HADOOP-15040) Upgrade AWS SDK to 1.11.271: NPE bug spams logs w/ Yarn Log Aggregation

2018-02-14 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364925#comment-16364925 ] Aaron Fabbri commented on HADOOP-15040: --- I did not realize there was also a 3.1 br

[jira] [Updated] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-02-14 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13761: -- Attachment: HADOOP-13761.003.patch > S3Guard: implement retries for DDB failures and throttlin

[jira] [Updated] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-02-14 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13761: -- Attachment: HADOOP-13761.003.patch > S3Guard: implement retries for DDB failures and throttlin

[jira] [Updated] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-02-14 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13761: -- Attachment: (was: HADOOP-13761.003.patch) > S3Guard: implement retries for DDB failures an

[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-02-14 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16365040#comment-16365040 ] Aaron Fabbri commented on HADOOP-13761: --- Just posted v3 patch. {noformat} Feedback

[jira] [Updated] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-02-14 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13761: -- Attachment: HADOOP-13761.004.patch > S3Guard: implement retries for DDB failures and throttlin

[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-02-14 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16365059#comment-16365059 ] Aaron Fabbri commented on HADOOP-13761: --- v4 patch: v3 was missing a couple of chun

[jira] [Comment Edited] (HADOOP-15040) Upgrade AWS SDK to 1.11.271: NPE bug spams logs w/ Yarn Log Aggregation

2018-02-14 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16364925#comment-16364925 ] Aaron Fabbri edited comment on HADOOP-15040 at 2/15/18 2:22 AM: --

[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-02-14 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16365068#comment-16365068 ] Aaron Fabbri commented on HADOOP-13761: --- tested in us-west-2, with and without s3g

[jira] [Commented] (HADOOP-15176) Enhance IAM assumed role support in S3A client

2018-02-14 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16365136#comment-16365136 ] Aaron Fabbri commented on HADOOP-15176: --- {noformat} public static final String

[jira] [Commented] (HADOOP-15076) Enhance s3a troubleshooting docs, add perf section

2018-02-14 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16365139#comment-16365139 ] Aaron Fabbri commented on HADOOP-15076: --- {noformat} It is very easy to run out of

[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-02-15 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16366211#comment-16366211 ] Aaron Fabbri commented on HADOOP-13761: --- Your changes all look straightforward, th

[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-02-15 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16366433#comment-16366433 ] Aaron Fabbri commented on HADOOP-13761: --- [~ste...@apache.org] on your question abo

[jira] [Commented] (HADOOP-14759) S3GuardTool prune to prune specific bucket entries

2018-02-15 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16366435#comment-16366435 ] Aaron Fabbri commented on HADOOP-14759: --- I thought we already did this for some re

[jira] [Commented] (HADOOP-15210) Handle FNFE from S3Guard.getMetadataStore() in S3A initialize()

2018-02-15 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16366439#comment-16366439 ] Aaron Fabbri commented on HADOOP-15210: --- Seems like some customers *would* want th

[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-02-15 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16366622#comment-16366622 ] Aaron Fabbri commented on HADOOP-13761: --- [~ste...@apache.org] was this part of you

[jira] [Updated] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-02-15 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13761: -- Attachment: HADOOP-13761-005-to-006-approx.diff.txt > S3Guard: implement retries for DDB failu

[jira] [Updated] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-02-15 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13761: -- Attachment: HADOOP-13761-006.diff > S3Guard: implement retries for DDB failures and throttling

[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-02-15 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16366647#comment-16366647 ] Aaron Fabbri commented on HADOOP-13761: --- About to post v6 patch. Changes from v5:

[jira] [Updated] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-02-15 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13761: -- Attachment: (was: HADOOP-13761-006.diff) > S3Guard: implement retries for DDB failures and

[jira] [Updated] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-02-15 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13761: -- Attachment: HADOOP-13761-006.patch > S3Guard: implement retries for DDB failures and throttlin

[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-02-16 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16367954#comment-16367954 ] Aaron Fabbri commented on HADOOP-13761: --- {quote}Mark up the seek/read calls with @

[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-02-21 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16372014#comment-16372014 ] Aaron Fabbri commented on HADOOP-13761: --- I will try to post an updated patch today

[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-02-21 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16372377#comment-16372377 ] Aaron Fabbri commented on HADOOP-13761: --- v7 patch incoming.  Since last revision:

[jira] [Updated] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-02-21 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13761: -- Attachment: HADOOP-13761-007.patch > S3Guard: implement retries for DDB failures and throttlin

[jira] [Updated] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-02-21 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13761: -- Attachment: (was: HADOOP-13761-007.patch) > S3Guard: implement retries for DDB failures an

[jira] [Updated] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-02-21 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13761: -- Status: Open (was: Patch Available) > S3Guard: implement retries for DDB failures and throttl

[jira] [Updated] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-02-21 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13761: -- Attachment: HADOOP-13761-007.patch > S3Guard: implement retries for DDB failures and throttlin

[jira] [Updated] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-02-21 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13761: -- Status: Patch Available (was: Open) > S3Guard: implement retries for DDB failures and throttl

[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-02-22 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16373419#comment-16373419 ] Aaron Fabbri commented on HADOOP-13761: --- Cleaning up checkstyle / findbugs now.. n

[jira] [Updated] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-02-22 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13761: -- Attachment: HADOOP-13761-008.patch > S3Guard: implement retries for DDB failures and throttlin

[jira] [Updated] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-02-22 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13761: -- Attachment: HADOOP-13761-009.patch > S3Guard: implement retries for DDB failures and throttlin

[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-02-23 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16374799#comment-16374799 ] Aaron Fabbri commented on HADOOP-13761: --- Thanks.. I added some @ annotations for c

[jira] [Commented] (HADOOP-15264) AWS "shaded" SDK 1.271 is pulling in netty 4.1.17

2018-02-27 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16379319#comment-16379319 ] Aaron Fabbri commented on HADOOP-15264: --- +1 LGTM. No testing but I pulled trunk of

[jira] [Commented] (HADOOP-15269) S3 returning 400 on the directory /test/ GET of getFileStatus

2018-02-27 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16379327#comment-16379327 ] Aaron Fabbri commented on HADOOP-15269: --- Do you, or have you ever had, object vers

[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-03-01 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16383066#comment-16383066 ] Aaron Fabbri commented on HADOOP-13761: --- Thanks [~leftnoteasy] I've been travellin

[jira] [Updated] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-03-01 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13761: -- Attachment: HADOOP-13761-012.patch > S3Guard: implement retries for DDB failures and throttlin

[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-03-01 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16383164#comment-16383164 ] Aaron Fabbri commented on HADOOP-13761: --- v12 patch attached which eliminates neste

[jira] [Updated] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-03-02 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-13761: -- Status: Patch Available (was: Open) > S3Guard: implement retries for DDB failures and throttl

[jira] [Commented] (HADOOP-13761) S3Guard: implement retries for DDB failures and throttling; translate exceptions

2018-03-02 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-13761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16384246#comment-16384246 ] Aaron Fabbri commented on HADOOP-13761: --- Findbugs seems to be smoking crack and/or

[jira] [Commented] (HADOOP-14927) ITestS3GuardTool failures in testDestroyNoBucket()

2018-03-05 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16387187#comment-16387187 ] Aaron Fabbri commented on HADOOP-14927: --- Ok.. For some reason I do not hit this FN

[jira] [Commented] (HADOOP-15206) BZip2 drops and duplicates records when input split size is small

2018-03-07 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16390429#comment-16390429 ] Aaron Fabbri commented on HADOOP-15206: --- Just reviewing this as part of a backport

[jira] [Commented] (HADOOP-15273) distcp can't handle remote stores with different checksum algorithms

2018-03-07 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16390602#comment-16390602 ] Aaron Fabbri commented on HADOOP-15273: --- LGTM. +1 Testing: I only ran the TestCo

[jira] [Commented] (HADOOP-12502) SetReplication OutOfMemoryError

2018-03-07 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-12502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16390614#comment-16390614 ] Aaron Fabbri commented on HADOOP-12502: --- Thanks for the updated patch.. Will try t

[jira] [Commented] (HADOOP-15206) BZip2 drops and duplicates records when input split size is small

2018-03-08 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391796#comment-16391796 ] Aaron Fabbri commented on HADOOP-15206: --- Ah.. yes, I didn't notice the read() call

[jira] [Comment Edited] (HADOOP-15209) DistCp to eliminate needless deletion of files under already-deleted directories

2018-03-08 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392038#comment-16392038 ] Aaron Fabbri edited comment on HADOOP-15209 at 3/8/18 10:42 PM: --

[jira] [Commented] (HADOOP-15209) DistCp to eliminate needless deletion of files under already-deleted directories

2018-03-08 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392038#comment-16392038 ] Aaron Fabbri commented on HADOOP-15209: --- I will try to review / test this today.

[jira] [Commented] (HADOOP-15209) DistCp to eliminate needless deletion of files under already-deleted directories

2018-03-08 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16392299#comment-16392299 ] Aaron Fabbri commented on HADOOP-15209: --- Noticed you just mentioned cancelling the

[jira] [Commented] (HADOOP-15278) log s3a at info

2018-03-14 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16399442#comment-16399442 ] Aaron Fabbri commented on HADOOP-15278: --- +1 I think this is a good change. I feel

[jira] [Updated] (HADOOP-15278) log s3a at info

2018-03-14 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron Fabbri updated HADOOP-15278: -- Resolution: Fixed Fix Version/s: 3.2.0 3.1.0 Status: Resol

[jira] [Commented] (HADOOP-15209) DistCp to eliminate needless deletion of files under already-deleted directories

2018-03-14 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16399897#comment-16399897 ] Aaron Fabbri commented on HADOOP-15209: --- Nice work here [~ste...@apache.org]. I d

[jira] [Commented] (HADOOP-15209) DistCp to eliminate needless deletion of files under already-deleted directories

2018-03-15 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16400841#comment-16400841 ] Aaron Fabbri commented on HADOOP-15209: --- My testing all looked good. +1 from me.

[jira] [Commented] (HADOOP-15314) Scheme assertion in S3Guard DynamoDBMetadataStore::checkPath is unnecessarily restrictive

2018-03-16 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-15314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16403083#comment-16403083 ] Aaron Fabbri commented on HADOOP-15314: --- The DynamoDB implementation (DynamoDBMeta

[jira] [Commented] (HADOOP-14759) S3GuardTool prune to prune specific bucket entries

2018-03-23 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16412235#comment-16412235 ] Aaron Fabbri commented on HADOOP-14759: --- Thanks for the patch [~gabor.bota]. Look

[jira] [Commented] (HADOOP-14759) S3GuardTool prune to prune specific bucket entries

2018-04-04 Thread Aaron Fabbri (JIRA)
[ https://issues.apache.org/jira/browse/HADOOP-14759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16426073#comment-16426073 ] Aaron Fabbri commented on HADOOP-14759: --- Thanks for the updated patch [~gabor.bota

<    1   2   3   4   5   6   7   8   9   10   >