[
https://issues.apache.org/jira/browse/HDDS-8925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Doroszlai updated HDDS-8925:
-----------------------------------
Description:
* Worker thread sets {{completed=true}} on attempting {{n+1}}th task.
* Progress bar is {{terminate}}d if there are failed attempts, {{shutdown}} if
not.
The problem happens if there are tasks in progress when progressbar
shutdown/termination is initiated AND one of these in-progress tasks fail.
{code:title=repro}
$ docker-compose exec -T s3g ozone freon s3kg -n 1 -t 10 -b no-such-bucket
...
2023-06-25 13:30:46,493 [main] INFO freon.BaseFreonGenerator: Executing test
with prefix p0l2uvfgg7 and number-of-tests 1
2023-06-25 13:30:46,507 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0
out of 1)
2023-06-25 13:30:46,792 [pool-2-thread-1] ERROR freon.BaseFreonGenerator: Error
on executing task 0
com.amazonaws.SdkClientException: Unable to load AWS credentials from
environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY
(or AWS_SECRET_ACCESS_KEY))
at
com.amazonaws.auth.EnvironmentVariableCredentialsProvider.getCredentials(EnvironmentVariableCredentialsProvider.java:50)
...
at
org.apache.hadoop.ozone.freon.S3KeyGenerator.createKey(S3KeyGenerator.java:111)
at
org.apache.hadoop.ozone.freon.BaseFreonGenerator.tryNextTask(BaseFreonGenerator.java:220)
at
org.apache.hadoop.ozone.freon.BaseFreonGenerator.taskLoop(BaseFreonGenerator.java:200)
at
org.apache.hadoop.ozone.freon.BaseFreonGenerator.lambda$startTaskRunners$0(BaseFreonGenerator.java:174)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2023-06-25 13:30:47,509 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0
out of 1)
2023-06-25 13:30:48,510 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0
out of 1)
2023-06-25 13:30:49,511 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0
out of 1)
2023-06-25 13:30:50,513 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0
out of 1)
2023-06-25 13:30:51,514 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0
out of 1)
2023-06-25 13:30:52,515 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0
out of 1)
2023-06-25 13:30:53,517 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0
out of 1)
2023-06-25 13:30:54,518 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0
out of 1)
2023-06-25 13:30:55,519 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0
out of 1)
2023-06-25 13:30:56,520 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0
out of 1)
...
{code}
> BaseFreonGenerator may not complete with only failed attempt
> ------------------------------------------------------------
>
> Key: HDDS-8925
> URL: https://issues.apache.org/jira/browse/HDDS-8925
> Project: Apache Ozone
> Issue Type: Bug
> Components: freon
> Reporter: Attila Doroszlai
> Assignee: Attila Doroszlai
> Priority: Major
>
> * Worker thread sets {{completed=true}} on attempting {{n+1}}th task.
> * Progress bar is {{terminate}}d if there are failed attempts, {{shutdown}}
> if not.
> The problem happens if there are tasks in progress when progressbar
> shutdown/termination is initiated AND one of these in-progress tasks fail.
> {code:title=repro}
> $ docker-compose exec -T s3g ozone freon s3kg -n 1 -t 10 -b no-such-bucket
> ...
> 2023-06-25 13:30:46,493 [main] INFO freon.BaseFreonGenerator: Executing test
> with prefix p0l2uvfgg7 and number-of-tests 1
> 2023-06-25 13:30:46,507 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 %
> (0 out of 1)
> 2023-06-25 13:30:46,792 [pool-2-thread-1] ERROR freon.BaseFreonGenerator:
> Error on executing task 0
> com.amazonaws.SdkClientException: Unable to load AWS credentials from
> environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and
> AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
> at
> com.amazonaws.auth.EnvironmentVariableCredentialsProvider.getCredentials(EnvironmentVariableCredentialsProvider.java:50)
> ...
> at
> org.apache.hadoop.ozone.freon.S3KeyGenerator.createKey(S3KeyGenerator.java:111)
> at
> org.apache.hadoop.ozone.freon.BaseFreonGenerator.tryNextTask(BaseFreonGenerator.java:220)
> at
> org.apache.hadoop.ozone.freon.BaseFreonGenerator.taskLoop(BaseFreonGenerator.java:200)
> at
> org.apache.hadoop.ozone.freon.BaseFreonGenerator.lambda$startTaskRunners$0(BaseFreonGenerator.java:174)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)
> 2023-06-25 13:30:47,509 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 %
> (0 out of 1)
> 2023-06-25 13:30:48,510 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 %
> (0 out of 1)
> 2023-06-25 13:30:49,511 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 %
> (0 out of 1)
> 2023-06-25 13:30:50,513 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 %
> (0 out of 1)
> 2023-06-25 13:30:51,514 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 %
> (0 out of 1)
> 2023-06-25 13:30:52,515 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 %
> (0 out of 1)
> 2023-06-25 13:30:53,517 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 %
> (0 out of 1)
> 2023-06-25 13:30:54,518 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 %
> (0 out of 1)
> 2023-06-25 13:30:55,519 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 %
> (0 out of 1)
> 2023-06-25 13:30:56,520 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 %
> (0 out of 1)
> ...
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
