[
https://issues.apache.org/jira/browse/HDDS-9762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17793211#comment-17793211
]
Ashish Kumar commented on HDDS-9762:
------------------------------------
I run freon command including HDDS-9825 fix and it runs normally even in
second execution time.
First time execution:
{code:java}
OZONE_CLASSPATH=/Users/ashishk/Downloads/Hadoop-3.3.2/hadoop-3.3.2/share/hadoop/tools/lib/*.jar:$(ozone
classpath ozone-common) ozone freon \-Dfs.s3a.endpoint=http://localhostl:9878
\-Dfs.s3a.etag.checksum.enabled=false \-Dfs.s3a.path.style.access=true
\-Dfs.s3a.change.detection.source=versionid
\-Dfs.s3a.change.detection.mode=client
\-Dfs.s3a.change.detection.version.required=false
\-Dfs.s3a.directory.marker.retention=keep \dfsg -s102400 -n10000 -t10
--path=s3a://fso/ --prefix="s3-1GB"
2023-12-05 15:32:01,470 [main] INFO impl.MetricsConfig: Loaded properties from
hadoop-metrics2.properties
2023-12-05 15:32:01,518 [main] INFO impl.MetricsSystemImpl: Scheduled Metric
snapshot period at 10 second(s).
2023-12-05 15:32:01,518 [main] INFO impl.MetricsSystemImpl: ozone-freon metrics
system started
2023-12-05 15:32:01,589 [main] INFO freon.BaseFreonGenerator: Executing test
with prefix s3-1GB and number-of-tests 10000
0.00% |█
| 0/10000 Time: 0:00:00Disabling FS cache:
fs.s3a.impl.disable.cache
2023-12-05 15:32:01,757 [main] INFO impl.MetricsConfig: Loaded properties from
hadoop-metrics2.properties
2023-12-05 15:32:01,757 [main] INFO impl.MetricsSystemImpl: Scheduled Metric
snapshot period at 10 second(s).
2023-12-05 15:32:01,757 [main] INFO impl.MetricsSystemImpl: s3a-file-system
metrics system started
2023-12-05 15:32:02,176 [main] INFO impl.DirectoryPolicyImpl: Directory markers
will be kept
2023-12-05 15:32:02,315 [main] INFO impl.MetricsSystemImpl: Stopping
s3a-file-system metrics system...
2023-12-05 15:32:02,315 [main] INFO impl.MetricsSystemImpl: s3a-file-system
metrics system stopped.
2023-12-05 15:32:02,315 [main] INFO impl.MetricsSystemImpl: s3a-file-system
metrics system shutdown complete.
2023-12-05 15:32:02,329 [pool-2-thread-7] INFO impl.MetricsConfig: Loaded
properties from hadoop-metrics2.properties
2023-12-05 15:32:02,330 [pool-2-thread-7] INFO impl.MetricsSystemImpl:
Scheduled Metric snapshot period at 10 second(s).
2023-12-05 15:32:02,330 [pool-2-thread-7] INFO impl.MetricsSystemImpl:
s3a-file-system metrics system started
2023-12-05 15:32:02,337 [pool-2-thread-7] INFO impl.DirectoryPolicyImpl:
Directory markers will be kept
2023-12-05 15:32:02,338 [pool-2-thread-2] INFO impl.DirectoryPolicyImpl:
Directory markers will be kept
2023-12-05 15:32:02,338 [pool-2-thread-5] INFO impl.DirectoryPolicyImpl:
Directory markers will be kept
2023-12-05 15:32:02,338 [pool-2-thread-3] INFO impl.DirectoryPolicyImpl:
Directory markers will be kept
2023-12-05 15:32:02,339 [pool-2-thread-1] INFO impl.DirectoryPolicyImpl:
Directory markers will be kept
2023-12-05 15:32:02,340 [pool-2-thread-8] INFO impl.DirectoryPolicyImpl:
Directory markers will be kept
2023-12-05 15:32:02,341 [pool-2-thread-6] INFO impl.DirectoryPolicyImpl:
Directory markers will be kept
2023-12-05 15:32:02,344 [pool-2-thread-4] INFO impl.DirectoryPolicyImpl:
Directory markers will be kept
2023-12-05 15:32:02,344 [pool-2-thread-10] INFO impl.DirectoryPolicyImpl:
Directory markers will be kept
2023-12-05 15:32:02,344 [pool-2-thread-9] INFO impl.DirectoryPolicyImpl:
Directory markers will be kept
99.83%
|████████████████████████████████████████████████████████████████████████████████████████████████████
| 9983/10000 Time: 0:04:102023-12-05 15:36:12,985 [pool-2-thread-9] INFO
impl.MetricsSystemImpl: Stopping s3a-file-system metrics system...
2023-12-05 15:36:12,985 [pool-2-thread-9] INFO impl.MetricsSystemImpl:
s3a-file-system metrics system stopped.
2023-12-05 15:36:12,985 [pool-2-thread-9] INFO impl.MetricsSystemImpl:
s3a-file-system metrics system shutdown complete.
100.00%
|█████████████████████████████████████████████████████████████████████████████████████████████████████|
10000/10000 Time: 0:04:11
5/12/23 3:36:13 PM =============================================================
-- Timers ----------------------------------------------------------------------
file-create
count = 10000
mean rate = 39.82 calls/second
1-minute rate = 40.27 calls/second
5-minute rate = 39.48 calls/second
15-minute rate = 39.08 calls/second
min = 32.11 milliseconds
max = 487.80 milliseconds
mean = 247.85 milliseconds
stddev = 118.18 milliseconds
median = 248.47 milliseconds
75% <= 341.58 milliseconds
95% <= 432.07 milliseconds
98% <= 449.96 milliseconds
99% <= 461.53 milliseconds
99.9% <= 483.49 milliseconds
Total execution time (sec): 252
Failures: 0
Successful executions: 10000 {code}
Second time execution:
{code:java}
OZONE_CLASSPATH=/Users/ashishk/Downloads/Hadoop-3.3.2/hadoop-3.3.2/share/hadoop/tools/lib/*.jar:$(ozone
classpath ozone-common) ozone freon \-Dfs.s3a.endpoint=http://0.0.0.0:9878
\-Dfs.s3a.etag.checksum.enabled=false \-Dfs.s3a.path.style.access=true
\-Dfs.s3a.change.detection.source=versionid
\-Dfs.s3a.change.detection.mode=client
\-Dfs.s3a.change.detection.version.required=false
\-Dfs.s3a.directory.marker.retention=keep \dfsg -s102400 -n10000 -t10
--path=s3a://fso/ --prefix="s3-1GB"
2023-12-05 15:42:40,939 [main] INFO impl.MetricsConfig: Loaded properties from
hadoop-metrics2.properties
2023-12-05 15:42:40,993 [main] INFO impl.MetricsSystemImpl: Scheduled Metric
snapshot period at 10 second(s).
2023-12-05 15:42:40,993 [main] INFO impl.MetricsSystemImpl: ozone-freon metrics
system started
2023-12-05 15:42:41,063 [main] INFO freon.BaseFreonGenerator: Executing test
with prefix s3-1GB and number-of-tests 10000
0.00% |█
| 0/10000 Time: 0:00:00Disabling FS cache:
fs.s3a.impl.disable.cache
2023-12-05 15:42:41,236 [main] INFO impl.MetricsConfig: Loaded properties from
hadoop-metrics2.properties
2023-12-05 15:42:41,237 [main] INFO impl.MetricsSystemImpl: Scheduled Metric
snapshot period at 10 second(s).
2023-12-05 15:42:41,237 [main] INFO impl.MetricsSystemImpl: s3a-file-system
metrics system started
2023-12-05 15:42:41,659 [main] INFO impl.DirectoryPolicyImpl: Directory markers
will be kept
2023-12-05 15:42:41,838 [main] INFO impl.MetricsSystemImpl: Stopping
s3a-file-system metrics system...
2023-12-05 15:42:41,838 [main] INFO impl.MetricsSystemImpl: s3a-file-system
metrics system stopped.
2023-12-05 15:42:41,838 [main] INFO impl.MetricsSystemImpl: s3a-file-system
metrics system shutdown complete.
2023-12-05 15:42:41,852 [pool-2-thread-10] INFO impl.MetricsConfig: Loaded
properties from hadoop-metrics2.properties
2023-12-05 15:42:41,853 [pool-2-thread-10] INFO impl.MetricsSystemImpl:
Scheduled Metric snapshot period at 10 second(s).
2023-12-05 15:42:41,853 [pool-2-thread-10] INFO impl.MetricsSystemImpl:
s3a-file-system metrics system started
2023-12-05 15:42:41,859 [pool-2-thread-2] INFO impl.DirectoryPolicyImpl:
Directory markers will be kept
2023-12-05 15:42:41,859 [pool-2-thread-10] INFO impl.DirectoryPolicyImpl:
Directory markers will be kept
2023-12-05 15:42:41,860 [pool-2-thread-4] INFO impl.DirectoryPolicyImpl:
Directory markers will be kept
2023-12-05 15:42:41,861 [pool-2-thread-8] INFO impl.DirectoryPolicyImpl:
Directory markers will be kept
2023-12-05 15:42:41,861 [pool-2-thread-1] INFO impl.DirectoryPolicyImpl:
Directory markers will be kept
2023-12-05 15:42:41,863 [pool-2-thread-5] INFO impl.DirectoryPolicyImpl:
Directory markers will be kept
2023-12-05 15:42:41,865 [pool-2-thread-9] INFO impl.DirectoryPolicyImpl:
Directory markers will be kept
2023-12-05 15:42:41,867 [pool-2-thread-6] INFO impl.DirectoryPolicyImpl:
Directory markers will be kept
2023-12-05 15:42:41,867 [pool-2-thread-3] INFO impl.DirectoryPolicyImpl:
Directory markers will be kept
2023-12-05 15:42:41,868 [pool-2-thread-7] INFO impl.DirectoryPolicyImpl:
Directory markers will be kept
99.65%
|████████████████████████████████████████████████████████████████████████████████████████████████████
| 9965/10000 Time: 0:04:082023-12-05 15:46:50,934 [pool-2-thread-8] INFO
impl.MetricsSystemImpl: Stopping s3a-file-system metrics system...
2023-12-05 15:46:50,935 [pool-2-thread-8] INFO impl.MetricsSystemImpl:
s3a-file-system metrics system stopped.
2023-12-05 15:46:50,935 [pool-2-thread-8] INFO impl.MetricsSystemImpl:
s3a-file-system metrics system shutdown complete.
100.00%
|█████████████████████████████████████████████████████████████████████████████████████████████████████|
10000/10000 Time: 0:04:09
5/12/23 3:46:51 PM =============================================================
-- Timers ----------------------------------------------------------------------
file-create
count = 10000
mean rate = 40.02 calls/second
1-minute rate = 40.42 calls/second
5-minute rate = 38.29 calls/second
15-minute rate = 36.85 calls/second
min = 35.06 milliseconds
max = 486.16 milliseconds
mean = 247.89 milliseconds
stddev = 119.66 milliseconds
median = 248.93 milliseconds
75% <= 353.23 milliseconds
95% <= 434.81 milliseconds
98% <= 452.78 milliseconds
99% <= 466.03 milliseconds
99.9% <= 480.77 milliseconds
Total execution time (sec): 251
Failures: 0
Successful executions: 10000{code}
> [FSO] Hadoop dfs s3a protocol does not work with FSO buckets
> ------------------------------------------------------------
>
> Key: HDDS-9762
> URL: https://issues.apache.org/jira/browse/HDDS-9762
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 1.4.0
> Reporter: Mladjan Gadzic
> Priority: Blocker
> Attachments: 2023-12-02.png
>
>
> Trying to exercise freon dfsg over s3a results in exception.
> Command:
>
> {code:java}
> OZONE_CLASSPATH=/opt/hadoop/share/ozone/lib/aws-java-sdk-bundle-1.11.1026.jar:/opt/hadoop/share/ozone/lib/hadoop-aws-3.3.2.jar:$(ozone
> classpath ozone-common) ozone freon
> \-Dfs.s3a.endpoint=http://host.docker.internal:9878
> \-Dfs.s3a.etag.checksum.enabled=false \-Dfs.s3a.path.style.access=true
> \-Dfs.s3a.change.detection.source=versionid
> \-Dfs.s3a.change.detection.mode=client
> \-Dfs.s3a.change.detection.version.required=false \dfsg -s102400 -n10000 -t10
> --path=s3a://fso/ --prefix="s3-1GB" {code}
>
> Exception (command first run)
> {code:java}
> 2023-11-22 18:34:19,180 [s3a-transfer-fso-unbounded-pool4-t1] DEBUG
> impl.BulkDeleteRetryHandler: Retrying on error during bulk delete
> :org.apache.hadoop.fs.s3a.AWSS3IOException: delete:
> com.amazonaws.services.s3.model.MultiObjectDeleteException: One or more
> objects could not be deleted (Service: null; Status Code: 200; Error Code:
> null; Request ID: 0bcdb9b8-40f8-402f-b8d1-b5bdb8159823; S3 Extended Request
> ID: DwT29rWRhtYS; Proxy: null), S3 Extended Request ID: DwT29rWRhtYS:null:
> InternalError: s3-1GB/: Directory is not empty. Key:s3-1GB
> : One or more objects could not be deleted (Service: null; Status Code: 200;
> Error Code: null; Request ID: 0bcdb9b8-40f8-402f-b8d1-b5bdb8159823; S3
> Extended Request ID: DwT29rWRhtYS; Proxy: null)
> at
> org.apache.hadoop.fs.s3a.impl.MultiObjectDeleteSupport.translateDeleteException(MultiObjectDeleteSupport.java:117)
> at
> org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:312)
> at
> org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:426)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.deleteObjects(S3AFileSystem.java:2775)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.removeKeysS3(S3AFileSystem.java:3022)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.removeKeys(S3AFileSystem.java:3121)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.removeKeys(S3AFileSystem.java:3078)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.deleteUnnecessaryFakeDirectories(S3AFileSystem.java:4498)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$finishedWrite$31(S3AFileSystem.java:4403)
> at
> org.apache.hadoop.fs.s3a.impl.CallableSupplier.get(CallableSupplier.java:87)
> at
> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)
> Caused by: com.amazonaws.services.s3.model.MultiObjectDeleteException: One or
> more objects could not be deleted (Service: null; Status Code: 200; Error
> Code: null; Request ID: 0bcdb9b8-40f8-402f-b8d1-b5bdb8159823; S3 Extended
> Request ID: DwT29rWRhtYS; Proxy: null), S3 Extended Request ID: DwT29rWRhtYS
> at
> com.amazonaws.services.s3.AmazonS3Client.deleteObjects(AmazonS3Client.java:2345)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$deleteObjects$16(S3AFileSystem.java:2785)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528)
> at
> org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:414)
> ... 11 more{code}
> In consecutive run (command second run), there is a different exception
> {code:java}
> 2023-11-22 18:39:36,543 [pool-2-thread-9] ERROR freon.BaseFreonGenerator:
> Error on executing task 7
> :org.apache.hadoop.fs.FileAlreadyExistsException: s3a://fso/s3-1GB/7 is a
> directory
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.innerCreateFile(S3AFileSystem.java:1690)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$create$6(S3AFileSystem.java:1646)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:449)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2337)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2356)
> at org.apache.hadoop.fs.s3a.S3AFileSystem.create(S3AFileSystem.java:1645)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1233)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1210)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1091)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1078)
> at
> org.apache.hadoop.ozone.freon.HadoopFsGenerator.lambda$createFile$0(HadoopFsGenerator.java:112)
> at com.codahale.metrics.Timer.time(Timer.java:101)
> at
> org.apache.hadoop.ozone.freon.HadoopFsGenerator.createFile(HadoopFsGenerator.java:111)
> at
> org.apache.hadoop.ozone.freon.BaseFreonGenerator.tryNextTask(BaseFreonGenerator.java:220)
> at
> org.apache.hadoop.ozone.freon.BaseFreonGenerator.taskLoop(BaseFreonGenerator.java:200)
> at
> org.apache.hadoop.ozone.freon.BaseFreonGenerator.lambda$startTaskRunners$0(BaseFreonGenerator.java:174)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829) {code}
> Ozone SHA f34d347af1f7b9c1eb82cf27fbe8231c85493628.
> Libs from Hadoop 3.3.2 version.
> It is reproducible using unsecure Ozone Docker cluster with 3DNs.
> Steps to reproduce the issue:
> # bring up unsecure Ozone Docker cluster
> # exec into OM container
> # add env variables
> AWS_ACCESS_KEY_ID=random
> AWS_SECRET_KEY=random
> OZONE_ROOT_LOGGER=debug,console
> # create bucket named "fso" with FSO layout
> # run mentioned command (first time)
> # check output
> # run mentioned command (second time)
> # check output
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
