[jira] [Updated] (HADOOP-16923) mkdir on s3a should not be sensitive to trailing '/'

2020-11-30 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-16923: Priority: Minor (was: Major) > mkdir on s3a should not be sensitive to trailing '/' > --

[jira] [Assigned] (HADOOP-16923) mkdir on s3a should not be sensitive to trailing '/'

2020-11-30 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reassigned HADOOP-16923: --- Assignee: Steve Loughran > mkdir on s3a should not be sensitive to trailing '/' >

[jira] [Resolved] (HADOOP-16923) mkdir on s3a should not be sensitive to trailing '/'

2020-11-30 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-16923. - Resolution: Duplicate > mkdir on s3a should not be sensitive to trailing '/' >

[jira] [Updated] (HADOOP-17402) Add GCS FS impl reference to core-default.xml

2020-11-30 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-17402: Summary: Add GCS FS impl reference to core-default.xml (was: Add GCS FS defaults to core

[jira] [Commented] (HADOOP-17402) Add GCS FS defaults to core-default.xml

2020-11-30 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17240941#comment-17240941 ] Steve Loughran commented on HADOOP-17402: - done because it's separate. If the

[jira] [Comment Edited] (HADOOP-17402) Add GCS FS defaults to core-default.xml

2020-11-30 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17240941#comment-17240941 ] Steve Loughran edited comment on HADOOP-17402 at 11/30/20, 6:04 PM: -

[jira] [Created] (HADOOP-17403) S3A ITestPartialRenamesDeletes failure: missing directory marker

2020-11-30 Thread Steve Loughran (Jira)
Steve Loughran created HADOOP-17403: --- Summary: S3A ITestPartialRenamesDeletes failure: missing directory marker Key: HADOOP-17403 URL: https://issues.apache.org/jira/browse/HADOOP-17403 Project: Had

[jira] [Commented] (HADOOP-17403) S3A ITestPartialRenamesDeletes failure: missing directory marker

2020-11-30 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17240956#comment-17240956 ] Steve Loughran commented on HADOOP-17403: - {code} [ERROR] testRenameDirFailsIn

[jira] [Commented] (HADOOP-17403) S3A ITestPartialRenamesDeletes failure: missing directory marker

2020-11-30 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17240958#comment-17240958 ] Steve Loughran commented on HADOOP-17403: - I think there's some test setup conf

[jira] [Updated] (HADOOP-17403) S3A ITestPartialRenamesDeletes failure: missing directory marker

2020-11-30 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-17403: Component/s: test > S3A ITestPartialRenamesDeletes failure: missing directory marker > --

[jira] [Commented] (HADOOP-17402) Add GCS FS impl reference to core-default.xml

2020-12-01 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17241893#comment-17241893 ] Steve Loughran commented on HADOOP-17402: - > GCS connector has hadoop.fs.Filesy

[jira] [Updated] (HADOOP-17134) S3AFileSystem.listLocatedStatus(file) does a LIST even with S3Guard

2020-12-02 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-17134: Summary: S3AFileSystem.listLocatedStatus(file) does a LIST even with S3Guard (was: S3AFi

[jira] [Updated] (HADOOP-17134) S3AFileSystem.listLocatedStatus(file) does a LIST even with S3Guard

2020-12-02 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-17134: Parent Issue: HADOOP-17400 (was: HADOOP-16829) > S3AFileSystem.listLocatedStatus(file) d

[jira] [Updated] (HADOOP-17136) ITestS3ADirectoryPerformance.testListOperations failing

2020-12-02 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-17136: Parent Issue: HADOOP-17400 (was: HADOOP-16829) > ITestS3ADirectoryPerformance.testListOp

[jira] [Updated] (HADOOP-17335) s3a listing operation will fail in async prefetch if fs closed

2020-12-02 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-17335: Parent Issue: HADOOP-17400 (was: HADOOP-16829) > s3a listing operation will fail in asyn

[jira] [Updated] (HADOOP-17400) Optimize S3A for maximum performance in directory listings

2020-12-02 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-17400: Description: Make listing in applications as fast as we can get it especially for query

[jira] [Updated] (HADOOP-17400) Optimize S3A for maximum performance in directory listings

2020-12-02 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-17400: Description: Make listing in applications as fast as we can get it especially for query

[jira] [Created] (HADOOP-17406) enable s3a magic committer by default

2020-12-02 Thread Steve Loughran (Jira)
Steve Loughran created HADOOP-17406: --- Summary: enable s3a magic committer by default Key: HADOOP-17406 URL: https://issues.apache.org/jira/browse/HADOOP-17406 Project: Hadoop Common Issue T

[jira] [Commented] (HADOOP-15460) S3A FS to add "s3a:no-existence-checks" to the builder file creation option set

2020-12-02 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-15460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17242798#comment-17242798 ] Steve Loughran commented on HADOOP-15460: - now s3 is consistent we don't need t

[jira] [Resolved] (HADOOP-16863) Report on S3A cached 404 recovery better

2020-12-03 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-16863. - Resolution: Not A Problem AWS S3 is now consistent. No more 404 caching. > Report on S

[jira] [Resolved] (HADOOP-16804) s3a mkdir path/ can add 404 to S3 load balancers

2020-12-03 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-16804. - Resolution: Not A Problem AWS S3 is now consistent. No more 404 caching. > s3a mkdir p

[jira] [Resolved] (HADOOP-16879) s3a mkdirs() to not check dest for a dir marker

2020-12-03 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-16879. - Resolution: Duplicate > s3a mkdirs() to not check dest for a dir marker > -

[jira] [Resolved] (HADOOP-17332) S3A marker tool mixes up -min and -max

2020-12-03 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-17332. - Fix Version/s: 3.3.1 Resolution: Fixed > S3A marker tool mixes up -min and -max

[jira] [Updated] (HADOOP-17343) Upgrade aws-java-sdk to 1.11.901

2020-12-03 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-17343: Fix Version/s: 3.2.2 > Upgrade aws-java-sdk to 1.11.901 > ---

[jira] [Created] (HADOOP-17409) Remove S3Guard - no longer needed

2020-12-03 Thread Steve Loughran (Jira)
Steve Loughran created HADOOP-17409: --- Summary: Remove S3Guard - no longer needed Key: HADOOP-17409 URL: https://issues.apache.org/jira/browse/HADOOP-17409 Project: Hadoop Common Issue Type:

[jira] [Commented] (HADOOP-16776) backport HADOOP-16775: distcp copies to s3 are randomly corrupted

2020-12-04 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17243979#comment-17243979 ] Steve Loughran commented on HADOOP-16776: - Note: consistent S3 renders this fix

[jira] [Commented] (HADOOP-16775) DistCp reuses the same temp file within the task attempt for different files.

2020-12-04 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17243980#comment-17243980 ] Steve Loughran commented on HADOOP-16775: - Note: consistent S3 renders this fix

[jira] [Commented] (HADOOP-17402) Add GCS FS impl reference to core-default.xml

2020-12-04 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17243991#comment-17243991 ] Steve Loughran commented on HADOOP-17402: - I think we've always hoped to do the

[jira] [Updated] (HADOOP-17337) NetworkBinding has a runtime class dependency on a third-party shaded class

2020-12-04 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-17337: Fix Version/s: 3.3.1 > NetworkBinding has a runtime class dependency on a third-party sha

[jira] [Commented] (HADOOP-17359) [Hadoop-Tools]S3A MultiObjectDeleteException after uploading a file

2020-12-04 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17243993#comment-17243993 ] Steve Loughran commented on HADOOP-17359: - [~renxunsaky] -have you had any luck

[jira] [Updated] (HADOOP-17337) NetworkBinding has a runtime class dependency on a third-party shaded class

2020-12-04 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-17337: Priority: Blocker (was: Major) > NetworkBinding has a runtime class dependency on a thir

[jira] [Commented] (HADOOP-17337) NetworkBinding has a runtime class dependency on a third-party shaded class

2020-12-04 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17243994#comment-17243994 ] Steve Loughran commented on HADOOP-17337: - Checking up on this. I'd like to mak

[jira] [Commented] (HADOOP-17412) When `fs.s3a.connection.ssl.enabled=true`, Error when visit S3A with AKSK

2020-12-07 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17245162#comment-17245162 ] Steve Loughran commented on HADOOP-17412: - * please, no screenshots, paste a st

[jira] [Comment Edited] (HADOOP-17412) When `fs.s3a.connection.ssl.enabled=true`, Error when visit S3A with AKSK

2020-12-07 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17245162#comment-17245162 ] Steve Loughran edited comment on HADOOP-17412 at 12/7/20, 12:21 PM: -

[jira] [Commented] (HADOOP-17412) When `fs.s3a.connection.ssl.enabled=true`, Error when visit S3A with AKSK

2020-12-07 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17245216#comment-17245216 ] Steve Loughran commented on HADOOP-17412: - see also HADOOP-17017 > When `fs.s3

[jira] [Commented] (HADOOP-17412) When `fs.s3a.connection.ssl.enabled=true`, Error when visit S3A with AKSK

2020-12-07 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17245220#comment-17245220 ] Steve Loughran commented on HADOOP-17412: - also: what do you mean by "AKSK"? >

[jira] [Created] (HADOOP-17414) Magic committer files don't have the count of bytes written collected by spark

2020-12-07 Thread Steve Loughran (Jira)
Steve Loughran created HADOOP-17414: --- Summary: Magic committer files don't have the count of bytes written collected by spark Key: HADOOP-17414 URL: https://issues.apache.org/jira/browse/HADOOP-17414

[jira] [Commented] (HADOOP-17414) Magic committer files don't have the count of bytes written collected by spark

2020-12-07 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17245237#comment-17245237 ] Steve Loughran commented on HADOOP-17414: - Proposed: When we create the 0-byte

[jira] [Commented] (HADOOP-17412) When `fs.s3a.connection.ssl.enabled=true`, Error when visit S3A with AKSK

2020-12-07 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17245525#comment-17245525 ] Steve Loughran commented on HADOOP-17412: - +are you using wildfly + openssl,

[jira] [Created] (HADOOP-17415) Use S3 content-range header to update length of an object during reads

2020-12-08 Thread Steve Loughran (Jira)
Steve Loughran created HADOOP-17415: --- Summary: Use S3 content-range header to update length of an object during reads Key: HADOOP-17415 URL: https://issues.apache.org/jira/browse/HADOOP-17415 Projec

[jira] [Work started] (HADOOP-17414) Magic committer files don't have the count of bytes written collected by spark

2020-12-10 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-17414 started by Steve Loughran. --- > Magic committer files don't have the count of bytes written collected

[jira] [Updated] (HADOOP-17414) Magic committer files don't have the count of bytes written collected by spark

2020-12-10 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-17414: Description: The spark statistics tracking doesn't correctly assess the size of the uploa

[jira] [Updated] (HADOOP-17414) Magic committer files don't have the count of bytes written collected by spark

2020-12-10 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-17414: Description: The spark statistics tracking doesn't correctly assess the size of the uploa

[jira] [Commented] (HADOOP-17430) There is no way to clear Text bytes now

2020-12-11 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17248020#comment-17248020 ] Steve Loughran commented on HADOOP-17430: - # worth pulling in [~belugabehr] on

[jira] [Commented] (HADOOP-15875) S3AInputStream.seek should throw EOFException if seeking past the end of file

2020-12-12 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-15875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17248357#comment-17248357 ] Steve Loughran commented on HADOOP-15875: - Note that with the openFile() change

[jira] [Resolved] (HADOOP-17412) When `fs.s3a.connection.ssl.enabled=true`, Error when visit S3A with AKSK

2020-12-14 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-17412. - Resolution: Duplicate This is a duplicate of HADOOP-17017; you can't have "." in your b

[jira] [Created] (HADOOP-17433) Skipping network I/O in S3A getFileStatus(/) breaks ITestAssumeRole

2020-12-14 Thread Steve Loughran (Jira)
Steve Loughran created HADOOP-17433: --- Summary: Skipping network I/O in S3A getFileStatus(/) breaks ITestAssumeRole Key: HADOOP-17433 URL: https://issues.apache.org/jira/browse/HADOOP-17433 Project:

[jira] [Updated] (HADOOP-17433) Skipping network I/O in S3A getFileStatus(/) breaks ITestAssumeRole

2020-12-14 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-17433: Parent: HADOOP-17400 Issue Type: Sub-task (was: Bug) > Skipping network I/O in S

[jira] [Commented] (HADOOP-17433) Skipping network I/O in S3A getFileStatus(/) breaks ITestAssumeRole

2020-12-14 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17249141#comment-17249141 ] Steve Loughran commented on HADOOP-17433: - {code} [ERROR] Tests run: 26, Failur

[jira] [Commented] (HADOOP-17433) Skipping network I/O in S3A getFileStatus(/) breaks ITestAssumeRole

2020-12-14 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17249143#comment-17249143 ] Steve Loughran commented on HADOOP-17433: - The cause is there's an intercept wh

[jira] [Created] (HADOOP-17434) Improve S3A upload statistics collection from ProgressEvent callbacks

2020-12-15 Thread Steve Loughran (Jira)
Steve Loughran created HADOOP-17434: --- Summary: Improve S3A upload statistics collection from ProgressEvent callbacks Key: HADOOP-17434 URL: https://issues.apache.org/jira/browse/HADOOP-17434 Project

[jira] [Updated] (HADOOP-17437) Update Hadoop Documentation with a new AWS Credential Provider used with EKS

2020-12-17 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-17437: Component/s: fs/s3 > Update Hadoop Documentation with a new AWS Credential Provider used

[jira] [Commented] (HADOOP-17437) Update Hadoop Documentation with a new AWS Credential Provider used with EKS

2020-12-17 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17251159#comment-17251159 ] Steve Loughran commented on HADOOP-17437: - Happy to take a github PR against ha

[jira] [Commented] (HADOOP-15691) Add PathCapabilities to FS and FC to complement StreamCapabilities

2020-12-17 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-15691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17251164#comment-17251164 ] Steve Loughran commented on HADOOP-15691: - Sorry, missed that I'd pulled this b

[jira] [Commented] (HADOOP-17439) No shade guava in trunk

2020-12-18 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17251913#comment-17251913 ] Steve Loughran commented on HADOOP-17439: - # we haven't removed transient guava

[jira] [Resolved] (HADOOP-17338) Intermittent S3AInputStream failures: Premature end of Content-Length delimited message body etc

2020-12-18 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-17338. - Fix Version/s: 3.3.1 Resolution: Fixed > Intermittent S3AInputStream failures: P

[jira] [Updated] (HADOOP-17338) Intermittent S3AInputStream failures: Premature end of Content-Length delimited message body etc

2020-12-18 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-17338: Parent: HADOOP-16829 Issue Type: Sub-task (was: Bug) > Intermittent S3AInputStre

[jira] [Commented] (HADOOP-17338) Intermittent S3AInputStream failures: Premature end of Content-Length delimited message body etc

2020-12-19 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17252170#comment-17252170 ] Steve Loughran commented on HADOOP-17338: - ok, merged it into trunk after doing

[jira] [Moved] (HADOOP-17441) Update Jetty hadoop dependency

2020-12-21 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran moved HDFS-15742 to HADOOP-17441: Component/s: (was: build) build

[jira] [Commented] (HADOOP-17441) Update Jetty hadoop dependency

2020-12-21 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17252834#comment-17252834 ] Steve Loughran commented on HADOOP-17441: - You got a github PR for this? We won

[jira] [Commented] (HADOOP-16492) Support HuaweiCloud Object Storage as a Hadoop Backend File System

2020-12-21 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17252843#comment-17252843 ] Steve Loughran commented on HADOOP-16492: - bq. There are some optimizations li

[jira] [Commented] (HADOOP-17439) No shade guava in trunk

2020-12-21 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17252848#comment-17252848 ] Steve Loughran commented on HADOOP-17439: - Whatever is done with Guava, somethi

[jira] [Commented] (HADOOP-17440) Downgrade guava version in trunk

2020-12-21 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17252851#comment-17252851 ] Steve Loughran commented on HADOOP-17440: - -1. we upgraded Guava for a CVE: htt

[jira] [Commented] (HADOOP-17439) No shade guava in trunk

2020-12-21 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17252853#comment-17252853 ] Steve Loughran commented on HADOOP-17439: - Actually, one issue we need to under

[jira] [Updated] (HADOOP-16109) Parquet reading S3AFileSystem causes EOF

2020-12-23 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-16109: Fix Version/s: 3.1.4 > Parquet reading S3AFileSystem causes EOF > ---

[jira] [Commented] (HADOOP-16109) Parquet reading S3AFileSystem causes EOF

2020-12-23 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17254164#comment-17254164 ] Steve Loughran commented on HADOOP-16109: - It's in branch-3.1; hadoop-3.1.4 at

[jira] [Updated] (HADOOP-16830) Add Public IOStatistics API

2020-12-29 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-16830: Summary: Add Public IOStatistics API (was: Add public IOStatistics API) > Add Public IOS

[jira] [Created] (HADOOP-17450) hadoop-common to add IOStatistics API

2020-12-30 Thread Steve Loughran (Jira)
Steve Loughran created HADOOP-17450: --- Summary: hadoop-common to add IOStatistics API Key: HADOOP-17450 URL: https://issues.apache.org/jira/browse/HADOOP-17450 Project: Hadoop Common Issue T

[jira] [Work started] (HADOOP-17450) hadoop-common to add IOStatistics API

2020-12-30 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-17450 started by Steve Loughran. --- > hadoop-common to add IOStatistics API > --

[jira] [Resolved] (HADOOP-17450) hadoop-common to add IOStatistics API

2020-12-31 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-17450. - Fix Version/s: 3.4.0 Resolution: Fixed merged to trunk will backport once the re

[jira] [Work started] (HADOOP-17271) S3A statistics to support IOStatistics

2020-12-31 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HADOOP-17271 started by Steve Loughran. --- > S3A statistics to support IOStatistics > -

[jira] [Resolved] (HADOOP-17271) S3A statistics to support IOStatistics

2020-12-31 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-17271. - Fix Version/s: 3.4.0 Resolution: Fixed Merged in to trunk -will go into branch-3

[jira] [Updated] (HADOOP-17271) S3A statistics to support IOStatistics

2020-12-31 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-17271: Description: S3A to rework statistics with * API + Implementation split of the interface

[jira] [Created] (HADOOP-17451) intermittent failure of S3A huge file upload tests: count of bytes uploaded == 0

2020-12-31 Thread Steve Loughran (Jira)
Steve Loughran created HADOOP-17451: --- Summary: intermittent failure of S3A huge file upload tests: count of bytes uploaded == 0 Key: HADOOP-17451 URL: https://issues.apache.org/jira/browse/HADOOP-17451

[jira] [Commented] (HADOOP-17451) intermittent failure of S3A huge file upload tests: count of bytes uploaded == 0

2020-12-31 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17257105#comment-17257105 ] Steve Loughran commented on HADOOP-17451: - came with HADOOP-17271 and the S3A s

[jira] [Assigned] (HADOOP-17451) intermittent failure of S3A huge file upload tests: count of bytes uploaded == 0

2020-12-31 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reassigned HADOOP-17451: --- Assignee: Steve Loughran > intermittent failure of S3A huge file upload tests: cou

[jira] [Commented] (HADOOP-16830) Add Public IOStatistics API

2020-12-31 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17257106#comment-17257106 ] Steve Loughran commented on HADOOP-16830: - OK: core hadoop-common and S3A chang

[jira] [Comment Edited] (HADOOP-17433) Skipping network I/O in S3A getFileStatus(/) breaks ITestAssumeRole

2021-01-03 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17249143#comment-17249143 ] Steve Loughran edited comment on HADOOP-17433 at 1/3/21, 1:05 PM: ---

[jira] [Commented] (HADOOP-16415) Speed up S3A test runs

2021-01-04 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17258415#comment-17258415 ] Steve Loughran commented on HADOOP-16415: - h3. Huge tests we have too many of

[jira] [Commented] (HADOOP-16415) Speed up S3A test runs

2021-01-04 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17258420#comment-17258420 ] Steve Loughran commented on HADOOP-16415: - v1 list API calls are 100s. Do we ne

[jira] [Updated] (HADOOP-17455) [s3a] Intermittent failure of ITestS3ADeleteCost.testDeleteSingleFileInDir

2021-01-05 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-17455: Parent: HADOOP-16829 Issue Type: Sub-task (was: Improvement) > [s3a] Intermitten

[jira] [Commented] (HADOOP-17455) [s3a] Intermittent failure of ITestS3ADeleteCost.testDeleteSingleFileInDir

2021-01-05 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17258982#comment-17258982 ] Steve Loughran commented on HADOOP-17455: - object_delete_objects expected:<2> b

[jira] [Commented] (HADOOP-17403) S3A ITestPartialRenamesDeletes failure: missing directory marker

2021-01-05 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17258986#comment-17258986 ] Steve Loughran commented on HADOOP-17403: - seen during some backporting and mad

[jira] [Resolved] (HADOOP-17403) S3A ITestPartialRenamesDeletes failure: missing directory marker

2021-01-05 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-17403. - Fix Version/s: 3.3.1 Assignee: Steve Loughran Resolution: Workaround >

[jira] [Updated] (HADOOP-17403) S3A ITestPartialRenamesDeletes.testRenameDirFailsInDelete failure: missing directory marker

2021-01-05 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-17403: Summary: S3A ITestPartialRenamesDeletes.testRenameDirFailsInDelete failure: missing direc

[jira] [Created] (HADOOP-17456) S3A ITestPartialRenamesDeletes.testPartialDirDelete[bulk-delete=true] failure

2021-01-05 Thread Steve Loughran (Jira)
Steve Loughran created HADOOP-17456: --- Summary: S3A ITestPartialRenamesDeletes.testPartialDirDelete[bulk-delete=true] failure Key: HADOOP-17456 URL: https://issues.apache.org/jira/browse/HADOOP-17456

[jira] [Updated] (HADOOP-17454) [s3a] Disable bucket existence check - set fs.s3a.bucket.probe to 0

2021-01-05 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-17454: Release Note: S3A bucket existence check is disabled (fs.s3a.bucket.probe is 0), so there

[jira] [Updated] (HADOOP-17456) S3A ITestPartialRenamesDeletes.testPartialDirDelete[bulk-delete=true] failure

2021-01-05 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17456?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-17456: Parent Issue: HADOOP-16830 (was: HADOOP-16829) > S3A ITestPartialRenamesDeletes.testPart

[jira] [Commented] (HADOOP-17451) intermittent failure of S3A huge file upload tests: count of bytes uploaded == 0

2021-01-05 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17259055#comment-17259055 ] Steve Loughran commented on HADOOP-17451: - put byte count == 0 {code} [ERROR]

[jira] [Commented] (HADOOP-17451) intermittent failure of S3A huge file upload tests: count of bytes uploaded == 0

2021-01-05 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17259121#comment-17259121 ] Steve Loughran commented on HADOOP-17451: - I see the problem. The StorageStatis

[jira] [Commented] (HADOOP-17455) [s3a] Intermittent failure of ITestS3ADeleteCost.testDeleteSingleFileInDir

2021-01-05 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17259198#comment-17259198 ] Steve Loughran commented on HADOOP-17455: - Happening to me in a test run (not I

[jira] [Updated] (HADOOP-17455) [s3a] Intermittent failure of ITestS3ADeleteCost.testDeleteSingleFileInDir

2021-01-05 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-17455: Parent Issue: HADOOP-16830 (was: HADOOP-16829) > [s3a] Intermittent failure of ITestS3AD

[jira] [Updated] (HADOOP-17430) Restore ability to set Text to empty byte array

2021-01-05 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-17430: Summary: Restore ability to set Text to empty byte array (was: There is no way to clear

[jira] [Assigned] (HADOOP-17430) Restore ability to set Text to empty byte array

2021-01-05 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reassigned HADOOP-17430: --- Assignee: gaozhan ding > Restore ability to set Text to empty byte array > --

[jira] [Resolved] (HADOOP-17430) Restore ability to set Text to empty byte array

2021-01-05 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-17430. - Fix Version/s: 3.3.1 Resolution: Fixed > Restore ability to set Text to empty b

[jira] [Updated] (HADOOP-17457) Seeing test ITestS3AInconsistency.testGetFileStatus failure.

2021-01-06 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-17457: Description: {code} [*ERROR*] *Tests* *run: 3*, *Failures: 1*, Errors: 0, Skipped: 0, Tim

[jira] [Commented] (HADOOP-17457) Seeing test ITestS3AInconsistency.testGetFileStatus failure.

2021-01-06 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17259598#comment-17259598 ] Steve Loughran commented on HADOOP-17457: - I'm not seeing this. It could be bri

[jira] [Commented] (HADOOP-16711) S3A bucket existence checks to support v2 API and "no checks at all"

2021-01-06 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-16711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17259599#comment-17259599 ] Steve Loughran commented on HADOOP-16711: - HADOOP-17454 is a follow on -we chan

[jira] [Commented] (HADOOP-17371) Bump Jetty to the latest version 9.4.35

2021-01-06 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17259630#comment-17259630 ] Steve Loughran commented on HADOOP-17371: - this seems to have triggered warning

[jira] [Commented] (HADOOP-17451) intermittent failure of S3A huge file upload tests: count of bytes uploaded == 0

2021-01-06 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/HADOOP-17451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17259716#comment-17259716 ] Steve Loughran commented on HADOOP-17451: - got some failures in a test run; loo

<    1   2   3   4   5   6   7   8   9   10   >