[
https://issues.apache.org/jira/browse/HDDS-9762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17790131#comment-17790131
]
Mladjan Gadzic commented on HDDS-9762:
--------------------------------------
There seems to be two underlaying issues. The first being AWSS3IOException and
the second FileAlreadyExistsException. Both coming from Hadoop code base.
The first one could be bypassed by setting property
fs.s3a.directory.marker.retention to keep.
The second one is tricky. It seems that Hadoop recognizes file as dir (which is
being recognized by Ozone as file). Overwrite needs to be enabled. During the
call to
[innerGetFileStatus|https://github.com/apache/hadoop/blob/8cd8e435fb43a251467ca74fadcb14f21a3e8163/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L1682],
[probes|https://github.com/apache/hadoop/blob/8cd8e435fb43a251467ca74fadcb14f21a3e8163/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L3580]
need to be set to
[ALL|https://github.com/apache/hadoop/blob/49f8ae965e9631a9b524484d4a1fdc3c97b5739f/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/impl/StatusProbeEnum.java#L40]
in order for innerGetFileStatus method to return
[S3AFileStatus|https://github.com/apache/hadoop/blob/19b99c1eccb913f21bdae5e80846f938d87e6b1a/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileStatus.java#L34]
where isDir field (of a parent class) is set to false. After that
[status.isDirectory()|https://github.com/apache/hadoop/blob/8cd8e435fb43a251467ca74fadcb14f21a3e8163/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L1688]
would be false and error would go away resulting in successful execution of
freon dfsg via s3g/s3a.
> [FSO] Hadoop dfs s3a protocol does not work with FSO buckets
> ------------------------------------------------------------
>
> Key: HDDS-9762
> URL: https://issues.apache.org/jira/browse/HDDS-9762
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 1.3.0
> Reporter: Mladjan Gadzic
> Priority: Major
>
> Trying to exercise freon dfsg over s3a results in exception.
> Command:
>
> {code:java}
> OZONE_CLASSPATH=/opt/hadoop/share/ozone/lib/aws-java-sdk-bundle-1.11.1026.jar:/opt/hadoop/share/ozone/lib/hadoop-aws-3.3.2.jar:$(ozone
> classpath ozone-common) ozone freon
> \-Dfs.s3a.endpoint=http://host.docker.internal:9878
> \-Dfs.s3a.etag.checksum.enabled=false \-Dfs.s3a.path.style.access=true
> \-Dfs.s3a.change.detection.source=versionid
> \-Dfs.s3a.change.detection.mode=client
> \-Dfs.s3a.change.detection.version.required=false \dfsg -s102400 -n10000 -t10
> --path=s3a://fso/ --prefix="s3-1GB" {code}
>
> Exception (command first run)
> {code:java}
> 2023-11-22 18:34:19,180 [s3a-transfer-fso-unbounded-pool4-t1] DEBUG
> impl.BulkDeleteRetryHandler: Retrying on error during bulk delete
> :org.apache.hadoop.fs.s3a.AWSS3IOException: delete:
> com.amazonaws.services.s3.model.MultiObjectDeleteException: One or more
> objects could not be deleted (Service: null; Status Code: 200; Error Code:
> null; Request ID: 0bcdb9b8-40f8-402f-b8d1-b5bdb8159823; S3 Extended Request
> ID: DwT29rWRhtYS; Proxy: null), S3 Extended Request ID: DwT29rWRhtYS:null:
> InternalError: s3-1GB/: Directory is not empty. Key:s3-1GB
> : One or more objects could not be deleted (Service: null; Status Code: 200;
> Error Code: null; Request ID: 0bcdb9b8-40f8-402f-b8d1-b5bdb8159823; S3
> Extended Request ID: DwT29rWRhtYS; Proxy: null)
> at
> org.apache.hadoop.fs.s3a.impl.MultiObjectDeleteSupport.translateDeleteException(MultiObjectDeleteSupport.java:117)
> at
> org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:312)
> at
> org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:426)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.deleteObjects(S3AFileSystem.java:2775)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.removeKeysS3(S3AFileSystem.java:3022)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.removeKeys(S3AFileSystem.java:3121)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.removeKeys(S3AFileSystem.java:3078)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.deleteUnnecessaryFakeDirectories(S3AFileSystem.java:4498)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$finishedWrite$31(S3AFileSystem.java:4403)
> at
> org.apache.hadoop.fs.s3a.impl.CallableSupplier.get(CallableSupplier.java:87)
> at
> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)
> Caused by: com.amazonaws.services.s3.model.MultiObjectDeleteException: One or
> more objects could not be deleted (Service: null; Status Code: 200; Error
> Code: null; Request ID: 0bcdb9b8-40f8-402f-b8d1-b5bdb8159823; S3 Extended
> Request ID: DwT29rWRhtYS; Proxy: null), S3 Extended Request ID: DwT29rWRhtYS
> at
> com.amazonaws.services.s3.AmazonS3Client.deleteObjects(AmazonS3Client.java:2345)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$deleteObjects$16(S3AFileSystem.java:2785)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528)
> at
> org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:414)
> ... 11 more{code}
> In consecutive run (command second run), there is a different exception
> {code:java}
> 2023-11-22 18:39:36,543 [pool-2-thread-9] ERROR freon.BaseFreonGenerator:
> Error on executing task 7
> :org.apache.hadoop.fs.FileAlreadyExistsException: s3a://fso/s3-1GB/7 is a
> directory
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.innerCreateFile(S3AFileSystem.java:1690)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$create$6(S3AFileSystem.java:1646)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:449)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2337)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2356)
> at org.apache.hadoop.fs.s3a.S3AFileSystem.create(S3AFileSystem.java:1645)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1233)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1210)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1091)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1078)
> at
> org.apache.hadoop.ozone.freon.HadoopFsGenerator.lambda$createFile$0(HadoopFsGenerator.java:112)
> at com.codahale.metrics.Timer.time(Timer.java:101)
> at
> org.apache.hadoop.ozone.freon.HadoopFsGenerator.createFile(HadoopFsGenerator.java:111)
> at
> org.apache.hadoop.ozone.freon.BaseFreonGenerator.tryNextTask(BaseFreonGenerator.java:220)
> at
> org.apache.hadoop.ozone.freon.BaseFreonGenerator.taskLoop(BaseFreonGenerator.java:200)
> at
> org.apache.hadoop.ozone.freon.BaseFreonGenerator.lambda$startTaskRunners$0(BaseFreonGenerator.java:174)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829) {code}
> Ozone SHA f34d347af1f7b9c1eb82cf27fbe8231c85493628.
> Libs from Hadoop 3.3.2 version.
> It is reproducible using unsecure Ozone Docker cluster with 3DNs.
> Steps to reproduce the issue:
> # bring up unsecure Ozone Docker cluster
> # exec into OM container
> # add env variables
> AWS_ACCESS_KEY_ID=random
> AWS_SECRET_KEY=random
> OZONE_ROOT_LOGGER=debug,console
> # create bucket named "fso" with FSO layout
> # run mentioned command (first time)
> # check output
> # run mentioned command (second time)
> # check output
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
