Mladjan Gadzic created HDDS-9762:
------------------------------------
Summary: [FSO] Hadoop dfs s3a protocol does not work with FSO
buckets
Key: HDDS-9762
URL: https://issues.apache.org/jira/browse/HDDS-9762
Project: Apache Ozone
Issue Type: Bug
Affects Versions: 1.3.0
Reporter: Mladjan Gadzic
Trying to exercise freon dfsg over s3a results in exception.
Command:
{code:java}
OZONE_CLASSPATH=/opt/hadoop/share/ozone/lib/aws-java-sdk-bundle-1.11.1026.jar:/opt/hadoop/share/ozone/lib/hadoop-aws-3.3.2.jar:$(ozone
classpath ozone-common) ozone freon
\-Dfs.s3a.endpoint=http://host.docker.internal:9878
\-Dfs.s3a.etag.checksum.enabled=false \-Dfs.s3a.path.style.access=true
\-Dfs.s3a.change.detection.source=versionid
\-Dfs.s3a.change.detection.mode=client
\-Dfs.s3a.change.detection.version.required=false \dfsg -s102400 -n10000 -t10
--path=s3a://fso/ --prefix="s3-1GB" {code}
Exception (command first run)
{code:java}
2023-11-22 18:34:19,180 [s3a-transfer-fso-unbounded-pool4-t1] DEBUG
impl.BulkDeleteRetryHandler: Retrying on error during bulk delete
:org.apache.hadoop.fs.s3a.AWSS3IOException: delete:
com.amazonaws.services.s3.model.MultiObjectDeleteException: One or more objects
could not be deleted (Service: null; Status Code: 200; Error Code: null;
Request ID: 0bcdb9b8-40f8-402f-b8d1-b5bdb8159823; S3 Extended Request ID:
DwT29rWRhtYS; Proxy: null), S3 Extended Request ID: DwT29rWRhtYS:null:
InternalError: s3-1GB/: Directory is not empty. Key:s3-1GB
: One or more objects could not be deleted (Service: null; Status Code: 200;
Error Code: null; Request ID: 0bcdb9b8-40f8-402f-b8d1-b5bdb8159823; S3 Extended
Request ID: DwT29rWRhtYS; Proxy: null)
at
org.apache.hadoop.fs.s3a.impl.MultiObjectDeleteSupport.translateDeleteException(MultiObjectDeleteSupport.java:117)
at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:312)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:426)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.deleteObjects(S3AFileSystem.java:2775)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.removeKeysS3(S3AFileSystem.java:3022)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.removeKeys(S3AFileSystem.java:3121)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.removeKeys(S3AFileSystem.java:3078)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.deleteUnnecessaryFakeDirectories(S3AFileSystem.java:4498)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$finishedWrite$31(S3AFileSystem.java:4403)
at
org.apache.hadoop.fs.s3a.impl.CallableSupplier.get(CallableSupplier.java:87)
at
java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.amazonaws.services.s3.model.MultiObjectDeleteException: One or
more objects could not be deleted (Service: null; Status Code: 200; Error Code:
null; Request ID: 0bcdb9b8-40f8-402f-b8d1-b5bdb8159823; S3 Extended Request ID:
DwT29rWRhtYS; Proxy: null), S3 Extended Request ID: DwT29rWRhtYS
at
com.amazonaws.services.s3.AmazonS3Client.deleteObjects(AmazonS3Client.java:2345)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$deleteObjects$16(S3AFileSystem.java:2785)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:414)
... 11 more{code}
In consecutive run (command second run), there is a different exception
{code:java}
2023-11-22 18:39:36,543 [pool-2-thread-9] ERROR freon.BaseFreonGenerator: Error
on executing task 7
:org.apache.hadoop.fs.FileAlreadyExistsException: s3a://fso/s3-1GB/7 is a
directory
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerCreateFile(S3AFileSystem.java:1690)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$create$6(S3AFileSystem.java:1646)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:449)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2337)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2356)
at org.apache.hadoop.fs.s3a.S3AFileSystem.create(S3AFileSystem.java:1645)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1233)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1210)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1091)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1078)
at
org.apache.hadoop.ozone.freon.HadoopFsGenerator.lambda$createFile$0(HadoopFsGenerator.java:112)
at com.codahale.metrics.Timer.time(Timer.java:101)
at
org.apache.hadoop.ozone.freon.HadoopFsGenerator.createFile(HadoopFsGenerator.java:111)
at
org.apache.hadoop.ozone.freon.BaseFreonGenerator.tryNextTask(BaseFreonGenerator.java:220)
at
org.apache.hadoop.ozone.freon.BaseFreonGenerator.taskLoop(BaseFreonGenerator.java:200)
at
org.apache.hadoop.ozone.freon.BaseFreonGenerator.lambda$startTaskRunners$0(BaseFreonGenerator.java:174)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829) {code}
Ozone SHA f34d347af1f7b9c1eb82cf27fbe8231c85493628.
Libs from Hadoop 3.3.2 version.
It is reproducible using unsecure Ozone Docker cluster with 3DNs.
Steps to reproduce the issue:
# bring up unsecure Ozone Docker cluster
# exec into OM container
# add env variables
AWS_ACCESS_KEY_ID=random
AWS_SECRET_KEY=random
OZONE_ROOT_LOGGER=debug,console
# create bucket named "fso" with FSO layout
# run mentioned command (first time)
# check output
# run mentioned command (second time)
# check output
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
