[
https://issues.apache.org/jira/browse/HADOOP-14837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17816887#comment-17816887
]
ASF GitHub Bot commented on HADOOP-14837:
-----------------------------------------
bpahuja commented on code in PR #6407:
URL: https://github.com/apache/hadoop/pull/6407#discussion_r1487363104
##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:
##########
@@ -577,6 +583,11 @@ public void initialize(URI name, Configuration
originalConf)
s3aInternals = createS3AInternals();
+ s3ObjectStorageClassFilter =
Optional.of(conf.getTrimmed(READ_RESTORED_GLACIER_OBJECTS,
Review Comment:
By Default, we read all objects coming as part of listObjects API call and
when an engine tries to access the archived file it fails with
software.amazon.awssdk.services.s3.model.InvalidObjectStateException: The
operation is not valid for the object's storage class (Service: S3, Status
Code: 403, Request ID: X05JDR633AAK4TBQ, Extended Request ID:
uOxWdN4giUAuB9a4YWvnyrXPYCi2U35P5BrHhFO3aLSLLe4GtWhXGXCEJ/Ld5EyGr5b6VezTzeI=):InvalidObjectState
at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:243)
##########
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java:
##########
@@ -577,6 +583,11 @@ public void initialize(URI name, Configuration
originalConf)
s3aInternals = createS3AInternals();
+ s3ObjectStorageClassFilter =
Optional.of(conf.getTrimmed(READ_RESTORED_GLACIER_OBJECTS,
Review Comment:
By Default, we read all objects coming as part of listObjects API call and
when an engine tries to access the archived file it fails with
```
software.amazon.awssdk.services.s3.model.InvalidObjectStateException: The
operation is not valid for the object's storage class (Service: S3, Status
Code: 403, Request ID: X05JDR633AAK4TBQ, Extended Request ID:
uOxWdN4giUAuB9a4YWvnyrXPYCi2U35P5BrHhFO3aLSLLe4GtWhXGXCEJ/Ld5EyGr5b6VezTzeI=):InvalidObjectState
at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:243)
```
> Handle S3A "glacier" data
> -------------------------
>
> Key: HADOOP-14837
> URL: https://issues.apache.org/jira/browse/HADOOP-14837
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.0.0-beta1
> Reporter: Steve Loughran
> Assignee: Bhavay Pahuja
> Priority: Minor
> Labels: pull-request-available
>
> SPARK-21797 covers how if you have AWS S3 set to copy some files to glacier,
> they appear in the listing but GETs fail, and so does everything else
> We should think about how best to handle this.
> # report better
> # if listings can identify files which are glaciated then maybe we could have
> an option to filter them out
> # test & see what happens
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
