[
https://issues.apache.org/jira/browse/HDDS-9762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17793009#comment-17793009
]
Mladjan Gadzic commented on HDDS-9762:
--------------------------------------
This is the full log of a single request that fails and keeps retrying
{code:java}
2023-12-04 19:06:45,982 [pool-2-thread-9] DEBUG amazonaws.request: Retrying
Request: GET http://host.docker.internal:9878 /fso1/ Parameters:
({"list-type":["2"],"delimiter":["/"],"max-keys":["2"],"prefix":["s3-1GB/7/"],"fetch-owner":["false"]}Headers:
(amz-sdk-invocation-id: c3d2d4ab-bdb0-fd76-3b48-4a3d09e856b1, Content-Type:
application/octet-stream, Referer:
https://audit.example.org/hadoop/1/op_create/a0f121f5-dd97-4c01-8452-9f6fe4cb68d8-00000031/?op=op_create&p1=s3-1GB/7&pr=hadoop&ps=5cf4f097-2ad4-45be-b4af-7fd566423db7&id=a0f121f5-dd97-4c01-8452-9f6fe4cb68d8-00000031&t0=39&fs=a0f121f5-dd97-4c01-8452-9f6fe4cb68d8&t1=39&ts=1701716594843,
User-Agent: Hadoop 3.3.6, aws-sdk-java/1.12.261 Linux/6.4.16-linuxkit
OpenJDK_64-Bit_Server_VM/11.0.17+8-LTS java/11.0.17 kotlin/1.6.21
vendor/Red_Hat,_Inc. cfg/retry-mode/legacy, )
2023-12-04 19:06:46,638 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0
out of 1000)
2023-12-04 19:06:46,820 [pool-2-thread-5] DEBUG auth.AWS4Signer: AWS4 Canonical
Request: '"GET
/fso1/
delimiter=%2F&fetch-owner=false&list-type=2&max-keys=2&prefix=s3-1GB%2F4%2F
amz-sdk-invocation-id:41d7a059-a3d6-6d1b-0604-d98764c463b8
amz-sdk-request:attempt=13;max=21
amz-sdk-retry:12/15281/340
content-type:application/octet-stream
host:host.docker.internal:9878
referer:https://audit.example.org/hadoop/1/op_create/791712e0-5363-4ab7-8124-643c849a3415-00000020/?op=op_create&p1=s3-1GB/4&pr=hadoop&ps=5cf4f097-2ad4-45be-b4af-7fd566423db7&id=791712e0-5363-4ab7-8124-643c849a3415-00000020&t0=35&fs=791712e0-5363-4ab7-8124-643c849a3415&t1=35&ts=1701716594794
user-agent:Hadoop 3.3.6, aws-sdk-java/1.12.261 Linux/6.4.16-linuxkit
OpenJDK_64-Bit_Server_VM/11.0.17+8-LTS java/11.0.17 kotlin/1.6.21
vendor/Red_Hat,_Inc. cfg/retry-mode/legacy
x-amz-content-sha256:e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
x-amz-date:20231204T190646Zamz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;referer;user-agent;x-amz-content-sha256;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
2023-12-04 19:06:46,820 [pool-2-thread-5] DEBUG auth.AWS4Signer: AWS4 String to
Sign: '"AWS4-HMAC-SHA256
20231204T190646Z
20231204/us-east-1/s3/aws4_request
192bbb52ed2b116411a3e7bfc4e3abab1eb659d4d02ead32eda9c0374727b2b4"
2023-12-04 19:06:46,821 [pool-2-thread-5] DEBUG protocol.RequestAddCookies:
CookieSpec selected: default
2023-12-04 19:06:46,821 [pool-2-thread-5] DEBUG protocol.RequestAuthCache: Auth
cache not set in the context
2023-12-04 19:06:46,822 [pool-2-thread-5] DEBUG
conn.PoolingHttpClientConnectionManager: Connection request: [route:
{}->http://host.docker.internal:9878][total available: 0; route allocated: 0 of
96; total allocated: 0 of 96]
2023-12-04 19:06:46,822 [pool-2-thread-5] DEBUG
conn.PoolingHttpClientConnectionManager: Connection leased: [id: 365][route:
{}->http://host.docker.internal:9878][total available: 0; route allocated: 1 of
96; total allocated: 1 of 96]
2023-12-04 19:06:46,822 [pool-2-thread-5] DEBUG execchain.MainClientExec:
Opening connection {}->http://host.docker.internal:9878
2023-12-04 19:06:46,822 [pool-2-thread-5] DEBUG
conn.DefaultHttpClientConnectionOperator: Connecting to
host.docker.internal/192.168.65.254:9878
2023-12-04 19:06:46,825 [pool-2-thread-5] DEBUG
conn.DefaultHttpClientConnectionOperator: Connection established
172.18.0.6:51500<->192.168.65.254:9878
2023-12-04 19:06:46,825 [pool-2-thread-5] DEBUG
conn.DefaultManagedHttpClientConnection: http-outgoing-365: set socket timeout
to 200000
2023-12-04 19:06:46,825 [pool-2-thread-5] DEBUG execchain.MainClientExec:
Executing request GET
/fso1/?list-type=2&delimiter=%2F&max-keys=2&prefix=s3-1GB%2F4%2F&fetch-owner=false
HTTP/1.1
2023-12-04 19:06:46,825 [pool-2-thread-5] DEBUG execchain.MainClientExec: Proxy
auth state: UNCHALLENGED
2023-12-04 19:06:46,825 [pool-2-thread-5] DEBUG http.headers: http-outgoing-365
>> GET
/fso1/?list-type=2&delimiter=%2F&max-keys=2&prefix=s3-1GB%2F4%2F&fetch-owner=false
HTTP/1.1
2023-12-04 19:06:46,826 [pool-2-thread-5] DEBUG http.headers: http-outgoing-365
>> Host: host.docker.internal:9878
2023-12-04 19:06:46,826 [pool-2-thread-5] DEBUG http.headers: http-outgoing-365
>> amz-sdk-invocation-id: 41d7a059-a3d6-6d1b-0604-d98764c463b8
2023-12-04 19:06:46,826 [pool-2-thread-5] DEBUG http.headers: http-outgoing-365
>> amz-sdk-request: attempt=13;max=21
2023-12-04 19:06:46,826 [pool-2-thread-5] DEBUG http.headers: http-outgoing-365
>> amz-sdk-retry: 12/15281/340
2023-12-04 19:06:46,826 [pool-2-thread-5] DEBUG http.headers: http-outgoing-365
>> Authorization: AWS4-HMAC-SHA256
Credential=random/20231204/us-east-1/s3/aws4_request,
SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;referer;user-agent;x-amz-content-sha256;x-amz-date,
Signature=7943440ddcd19bc97fa486e9ab3b870982350223aed46227d9cbad4b69767d48
2023-12-04 19:06:46,826 [pool-2-thread-5] DEBUG http.headers: http-outgoing-365
>> Content-Type: application/octet-stream
2023-12-04 19:06:46,827 [pool-2-thread-5] DEBUG http.headers: http-outgoing-365
>> Referer:
https://audit.example.org/hadoop/1/op_create/791712e0-5363-4ab7-8124-643c849a3415-00000020/?op=op_create&p1=s3-1GB/4&pr=hadoop&ps=5cf4f097-2ad4-45be-b4af-7fd566423db7&id=791712e0-5363-4ab7-8124-643c849a3415-00000020&t0=35&fs=791712e0-5363-4ab7-8124-643c849a3415&t1=35&ts=1701716594794
2023-12-04 19:06:46,827 [pool-2-thread-5] DEBUG http.headers: http-outgoing-365
>> User-Agent: Hadoop 3.3.6, aws-sdk-java/1.12.261 Linux/6.4.16-linuxkit
OpenJDK_64-Bit_Server_VM/11.0.17+8-LTS java/11.0.17 kotlin/1.6.21
vendor/Red_Hat,_Inc. cfg/retry-mode/legacy
2023-12-04 19:06:46,827 [pool-2-thread-5] DEBUG http.headers: http-outgoing-365
>> x-amz-content-sha256:
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2023-12-04 19:06:46,827 [pool-2-thread-5] DEBUG http.headers: http-outgoing-365
>> X-Amz-Date: 20231204T190646Z
2023-12-04 19:06:46,827 [pool-2-thread-5] DEBUG http.headers: http-outgoing-365
>> Content-Length: 0
2023-12-04 19:06:46,827 [pool-2-thread-5] DEBUG http.headers: http-outgoing-365
>> Connection: Keep-Alive
2023-12-04 19:06:46,827 [pool-2-thread-5] DEBUG http.wire: http-outgoing-365 >>
"GET
/fso1/?list-type=2&delimiter=%2F&max-keys=2&prefix=s3-1GB%2F4%2F&fetch-owner=false
HTTP/1.1[\r][\n]"
2023-12-04 19:06:46,827 [pool-2-thread-5] DEBUG http.wire: http-outgoing-365 >>
"Host: host.docker.internal:9878[\r][\n]"
2023-12-04 19:06:46,828 [pool-2-thread-5] DEBUG http.wire: http-outgoing-365 >>
"amz-sdk-invocation-id: 41d7a059-a3d6-6d1b-0604-d98764c463b8[\r][\n]"
2023-12-04 19:06:46,828 [pool-2-thread-5] DEBUG http.wire: http-outgoing-365 >>
"amz-sdk-request: attempt=13;max=21[\r][\n]"
2023-12-04 19:06:46,828 [pool-2-thread-5] DEBUG http.wire: http-outgoing-365 >>
"amz-sdk-retry: 12/15281/340[\r][\n]"
2023-12-04 19:06:46,828 [pool-2-thread-5] DEBUG http.wire: http-outgoing-365 >>
"Authorization: AWS4-HMAC-SHA256
Credential=random/20231204/us-east-1/s3/aws4_request,
SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;referer;user-agent;x-amz-content-sha256;x-amz-date,
Signature=7943440ddcd19bc97fa486e9ab3b870982350223aed46227d9cbad4b69767d48[\r][\n]"
2023-12-04 19:06:46,828 [pool-2-thread-5] DEBUG http.wire: http-outgoing-365 >>
"Content-Type: application/octet-stream[\r][\n]"
2023-12-04 19:06:46,828 [pool-2-thread-5] DEBUG http.wire: http-outgoing-365 >>
"Referer:
https://audit.example.org/hadoop/1/op_create/791712e0-5363-4ab7-8124-643c849a3415-00000020/?op=op_create&p1=s3-1GB/4&pr=hadoop&ps=5cf4f097-2ad4-45be-b4af-7fd566423db7&id=791712e0-5363-4ab7-8124-643c849a3415-00000020&t0=35&fs=791712e0-5363-4ab7-8124-643c849a3415&t1=35&ts=1701716594794[\r][\n]";
2023-12-04 19:06:46,828 [pool-2-thread-5] DEBUG http.wire: http-outgoing-365 >>
"User-Agent: Hadoop 3.3.6, aws-sdk-java/1.12.261 Linux/6.4.16-linuxkit
OpenJDK_64-Bit_Server_VM/11.0.17+8-LTS java/11.0.17 kotlin/1.6.21
vendor/Red_Hat,_Inc. cfg/retry-mode/legacy[\r][\n]"
2023-12-04 19:06:46,828 [pool-2-thread-5] DEBUG http.wire: http-outgoing-365 >>
"x-amz-content-sha256:
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855[\r][\n]"
2023-12-04 19:06:46,829 [pool-2-thread-5] DEBUG http.wire: http-outgoing-365 >>
"X-Amz-Date: 20231204T190646Z[\r][\n]"
2023-12-04 19:06:46,829 [pool-2-thread-5] DEBUG http.wire: http-outgoing-365 >>
"Content-Length: 0[\r][\n]"
2023-12-04 19:06:46,829 [pool-2-thread-5] DEBUG http.wire: http-outgoing-365 >>
"Connection: Keep-Alive[\r][\n]"
2023-12-04 19:06:46,829 [pool-2-thread-5] DEBUG http.wire: http-outgoing-365 >>
"[\r][\n]"
2023-12-04 19:06:46,836 [pool-2-thread-5] DEBUG http.wire: http-outgoing-365 <<
"HTTP/1.1 500 Server Error[\r][\n]"
2023-12-04 19:06:46,836 [pool-2-thread-5] DEBUG http.wire: http-outgoing-365 <<
"Pragma: no-cache[\r][\n]"
2023-12-04 19:06:46,836 [pool-2-thread-5] DEBUG http.wire: http-outgoing-365 <<
"X-Content-Type-Options: nosniff[\r][\n]"
2023-12-04 19:06:46,836 [pool-2-thread-5] DEBUG http.wire: http-outgoing-365 <<
"X-XSS-Protection: 1; mode=block[\r][\n]"
2023-12-04 19:06:46,837 [pool-2-thread-5] DEBUG http.wire: http-outgoing-365 <<
"X-FRAME-OPTIONS: SAMEORIGIN[\r][\n]"
2023-12-04 19:06:46,837 [pool-2-thread-5] DEBUG http.wire: http-outgoing-365 <<
"Cache-Control: must-revalidate,no-cache,no-store[\r][\n]"
2023-12-04 19:06:46,837 [pool-2-thread-5] DEBUG http.wire: http-outgoing-365 <<
"Content-Type: text/html;charset=iso-8859-1[\r][\n]"
2023-12-04 19:06:46,837 [pool-2-thread-5] DEBUG http.wire: http-outgoing-365 <<
"Content-Length: 384[\r][\n]"
2023-12-04 19:06:46,837 [pool-2-thread-5] DEBUG http.wire: http-outgoing-365 <<
"Connection: close[\r][\n]"
2023-12-04 19:06:46,837 [pool-2-thread-5] DEBUG http.wire: http-outgoing-365 <<
"[\r][\n]"
2023-12-04 19:06:46,837 [pool-2-thread-5] DEBUG http.wire: http-outgoing-365 <<
"<html>[\n]"
2023-12-04 19:06:46,838 [pool-2-thread-5] DEBUG http.wire: http-outgoing-365 <<
"<head>[\n]"
2023-12-04 19:06:46,838 [pool-2-thread-5] DEBUG http.wire: http-outgoing-365 <<
"<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>[\n]"
2023-12-04 19:06:46,838 [pool-2-thread-5] DEBUG http.wire: http-outgoing-365 <<
"<title>Error 500 Request failed.</title>[\n]"
2023-12-04 19:06:46,838 [pool-2-thread-5] DEBUG http.wire: http-outgoing-365 <<
"</head>[\n]"
2023-12-04 19:06:46,838 [pool-2-thread-5] DEBUG http.wire: http-outgoing-365 <<
"<body><h2>HTTP ERROR 500 Request failed.</h2>[\n]"
2023-12-04 19:06:46,838 [pool-2-thread-5] DEBUG http.wire: http-outgoing-365 <<
"<table>[\n]"
2023-12-04 19:06:46,838 [pool-2-thread-5] DEBUG http.wire: http-outgoing-365 <<
"<tr><th>URI:</th><td>/fso1/</td></tr>[\n]"
2023-12-04 19:06:46,838 [pool-2-thread-5] DEBUG http.wire: http-outgoing-365 <<
"<tr><th>STATUS:</th><td>500</td></tr>[\n]"
2023-12-04 19:06:46,838 [pool-2-thread-5] DEBUG http.wire: http-outgoing-365 <<
"<tr><th>MESSAGE:</th><td>Request failed.</td></tr>[\n]"
2023-12-04 19:06:46,839 [pool-2-thread-5] DEBUG http.wire: http-outgoing-365 <<
"<tr><th>SERVLET:</th><td>jaxrs</td></tr>[\n]"
2023-12-04 19:06:46,839 [pool-2-thread-5] DEBUG http.wire: http-outgoing-365 <<
"</table>[\n]"
2023-12-04 19:06:46,839 [pool-2-thread-5] DEBUG http.wire: http-outgoing-365 <<
"[\n]"
2023-12-04 19:06:46,839 [pool-2-thread-5] DEBUG http.wire: http-outgoing-365 <<
"</body>[\n]"
2023-12-04 19:06:46,839 [pool-2-thread-5] DEBUG http.wire: http-outgoing-365 <<
"</html>[\n]"
2023-12-04 19:06:46,839 [pool-2-thread-5] DEBUG http.headers: http-outgoing-365
<< HTTP/1.1 500 Server Error
2023-12-04 19:06:46,839 [pool-2-thread-5] DEBUG http.headers: http-outgoing-365
<< Pragma: no-cache
2023-12-04 19:06:46,839 [pool-2-thread-5] DEBUG http.headers: http-outgoing-365
<< X-Content-Type-Options: nosniff
2023-12-04 19:06:46,840 [pool-2-thread-5] DEBUG http.headers: http-outgoing-365
<< X-XSS-Protection: 1; mode=block
2023-12-04 19:06:46,840 [pool-2-thread-5] DEBUG http.headers: http-outgoing-365
<< X-FRAME-OPTIONS: SAMEORIGIN
2023-12-04 19:06:46,840 [pool-2-thread-5] DEBUG http.headers: http-outgoing-365
<< Cache-Control: must-revalidate,no-cache,no-store
2023-12-04 19:06:46,840 [pool-2-thread-5] DEBUG http.headers: http-outgoing-365
<< Content-Type: text/html;charset=iso-8859-1
2023-12-04 19:06:46,840 [pool-2-thread-5] DEBUG http.headers: http-outgoing-365
<< Content-Length: 384
2023-12-04 19:06:46,840 [pool-2-thread-5] DEBUG http.headers: http-outgoing-365
<< Connection: close
2023-12-04 19:06:46,840 [pool-2-thread-5] DEBUG
conn.DefaultManagedHttpClientConnection: http-outgoing-365: Close connection
2023-12-04 19:06:46,841 [pool-2-thread-5] DEBUG execchain.MainClientExec:
Connection discarded
2023-12-04 19:06:46,841 [pool-2-thread-5] DEBUG
conn.PoolingHttpClientConnectionManager: Connection released: [id: 365][route:
{}->http://host.docker.internal:9878][total available: 0; route allocated: 0 of
96; total allocated: 0 of 96]
2023-12-04 19:06:46,841 [pool-2-thread-5] DEBUG amazonaws.request: Received
error response: com.amazonaws.services.s3.model.AmazonS3Exception: Server Error
(Service: Amazon S3; Status Code: 500; Error Code: 500 Server Error; Request
ID: null; S3 Extended Request ID: null; Proxy: null), S3 Extended Request ID:
null {code}
> [FSO] Hadoop dfs s3a protocol does not work with FSO buckets
> ------------------------------------------------------------
>
> Key: HDDS-9762
> URL: https://issues.apache.org/jira/browse/HDDS-9762
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 1.4.0
> Reporter: Mladjan Gadzic
> Priority: Blocker
> Attachments: 2023-12-02.png
>
>
> Trying to exercise freon dfsg over s3a results in exception.
> Command:
>
> {code:java}
> OZONE_CLASSPATH=/opt/hadoop/share/ozone/lib/aws-java-sdk-bundle-1.11.1026.jar:/opt/hadoop/share/ozone/lib/hadoop-aws-3.3.2.jar:$(ozone
> classpath ozone-common) ozone freon
> \-Dfs.s3a.endpoint=http://host.docker.internal:9878
> \-Dfs.s3a.etag.checksum.enabled=false \-Dfs.s3a.path.style.access=true
> \-Dfs.s3a.change.detection.source=versionid
> \-Dfs.s3a.change.detection.mode=client
> \-Dfs.s3a.change.detection.version.required=false \dfsg -s102400 -n10000 -t10
> --path=s3a://fso/ --prefix="s3-1GB" {code}
>
> Exception (command first run)
> {code:java}
> 2023-11-22 18:34:19,180 [s3a-transfer-fso-unbounded-pool4-t1] DEBUG
> impl.BulkDeleteRetryHandler: Retrying on error during bulk delete
> :org.apache.hadoop.fs.s3a.AWSS3IOException: delete:
> com.amazonaws.services.s3.model.MultiObjectDeleteException: One or more
> objects could not be deleted (Service: null; Status Code: 200; Error Code:
> null; Request ID: 0bcdb9b8-40f8-402f-b8d1-b5bdb8159823; S3 Extended Request
> ID: DwT29rWRhtYS; Proxy: null), S3 Extended Request ID: DwT29rWRhtYS:null:
> InternalError: s3-1GB/: Directory is not empty. Key:s3-1GB
> : One or more objects could not be deleted (Service: null; Status Code: 200;
> Error Code: null; Request ID: 0bcdb9b8-40f8-402f-b8d1-b5bdb8159823; S3
> Extended Request ID: DwT29rWRhtYS; Proxy: null)
> at
> org.apache.hadoop.fs.s3a.impl.MultiObjectDeleteSupport.translateDeleteException(MultiObjectDeleteSupport.java:117)
> at
> org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:312)
> at
> org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:426)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.deleteObjects(S3AFileSystem.java:2775)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.removeKeysS3(S3AFileSystem.java:3022)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.removeKeys(S3AFileSystem.java:3121)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.removeKeys(S3AFileSystem.java:3078)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.deleteUnnecessaryFakeDirectories(S3AFileSystem.java:4498)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$finishedWrite$31(S3AFileSystem.java:4403)
> at
> org.apache.hadoop.fs.s3a.impl.CallableSupplier.get(CallableSupplier.java:87)
> at
> java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)
> Caused by: com.amazonaws.services.s3.model.MultiObjectDeleteException: One or
> more objects could not be deleted (Service: null; Status Code: 200; Error
> Code: null; Request ID: 0bcdb9b8-40f8-402f-b8d1-b5bdb8159823; S3 Extended
> Request ID: DwT29rWRhtYS; Proxy: null), S3 Extended Request ID: DwT29rWRhtYS
> at
> com.amazonaws.services.s3.AmazonS3Client.deleteObjects(AmazonS3Client.java:2345)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$deleteObjects$16(S3AFileSystem.java:2785)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528)
> at
> org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:414)
> ... 11 more{code}
> In consecutive run (command second run), there is a different exception
> {code:java}
> 2023-11-22 18:39:36,543 [pool-2-thread-9] ERROR freon.BaseFreonGenerator:
> Error on executing task 7
> :org.apache.hadoop.fs.FileAlreadyExistsException: s3a://fso/s3-1GB/7 is a
> directory
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.innerCreateFile(S3AFileSystem.java:1690)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$create$6(S3AFileSystem.java:1646)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528)
> at
> org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:449)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2337)
> at
> org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2356)
> at org.apache.hadoop.fs.s3a.S3AFileSystem.create(S3AFileSystem.java:1645)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1233)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1210)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1091)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1078)
> at
> org.apache.hadoop.ozone.freon.HadoopFsGenerator.lambda$createFile$0(HadoopFsGenerator.java:112)
> at com.codahale.metrics.Timer.time(Timer.java:101)
> at
> org.apache.hadoop.ozone.freon.HadoopFsGenerator.createFile(HadoopFsGenerator.java:111)
> at
> org.apache.hadoop.ozone.freon.BaseFreonGenerator.tryNextTask(BaseFreonGenerator.java:220)
> at
> org.apache.hadoop.ozone.freon.BaseFreonGenerator.taskLoop(BaseFreonGenerator.java:200)
> at
> org.apache.hadoop.ozone.freon.BaseFreonGenerator.lambda$startTaskRunners$0(BaseFreonGenerator.java:174)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829) {code}
> Ozone SHA f34d347af1f7b9c1eb82cf27fbe8231c85493628.
> Libs from Hadoop 3.3.2 version.
> It is reproducible using unsecure Ozone Docker cluster with 3DNs.
> Steps to reproduce the issue:
> # bring up unsecure Ozone Docker cluster
> # exec into OM container
> # add env variables
> AWS_ACCESS_KEY_ID=random
> AWS_SECRET_KEY=random
> OZONE_ROOT_LOGGER=debug,console
> # create bucket named "fso" with FSO layout
> # run mentioned command (first time)
> # check output
> # run mentioned command (second time)
> # check output
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
