[
https://issues.apache.org/jira/browse/HDDS-8925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Doroszlai updated HDDS-8925:
-----------------------------------
Description:
* Worker thread sets {{completed=true}} on attempting n+1. task.
* Progress bar is terminated if there are failed attempts, shutdown if not.
The problem happens if there are tasks in progress when progressbar shutdown is
initiated AND one of these in-progress tasks is the first to fail.
{code:title=repro}
$ docker-compose exec -T s3g ozone freon s3kg -n 1 -t 10 -b no-such-bucket
...
2023-06-25 13:30:46,493 [main] INFO freon.BaseFreonGenerator: Executing test
with prefix p0l2uvfgg7 and number-of-tests 1
2023-06-25 13:30:46,507 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0
out of 1)
2023-06-25 13:30:46,792 [pool-2-thread-1] ERROR freon.BaseFreonGenerator: Error
on executing task 0
com.amazonaws.SdkClientException: Unable to load AWS credentials from
environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY
(or AWS_SECRET_ACCESS_KEY))
at
com.amazonaws.auth.EnvironmentVariableCredentialsProvider.getCredentials(EnvironmentVariableCredentialsProvider.java:50)
...
at
org.apache.hadoop.ozone.freon.S3KeyGenerator.createKey(S3KeyGenerator.java:111)
at
org.apache.hadoop.ozone.freon.BaseFreonGenerator.tryNextTask(BaseFreonGenerator.java:220)
at
org.apache.hadoop.ozone.freon.BaseFreonGenerator.taskLoop(BaseFreonGenerator.java:200)
at
org.apache.hadoop.ozone.freon.BaseFreonGenerator.lambda$startTaskRunners$0(BaseFreonGenerator.java:174)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2023-06-25 13:30:47,509 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0
out of 1)
2023-06-25 13:30:48,510 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0
out of 1)
2023-06-25 13:30:49,511 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0
out of 1)
2023-06-25 13:30:50,513 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0
out of 1)
2023-06-25 13:30:51,514 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0
out of 1)
2023-06-25 13:30:52,515 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0
out of 1)
2023-06-25 13:30:53,517 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0
out of 1)
2023-06-25 13:30:54,518 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0
out of 1)
2023-06-25 13:30:55,519 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0
out of 1)
2023-06-25 13:30:56,520 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0
out of 1)
...
{code}
was:
* Worker thread sets {{completed=true}} on attempting n+1. task.
* Progress bar is terminated if there are failed attempts, shutdown if not.
The problem happens if there are tasks in progress when progressbar
shutdown/termination is initiated AND one of these in-progress tasks fail.
{code:title=repro}
$ docker-compose exec -T s3g ozone freon s3kg -n 1 -t 10 -b no-such-bucket
...
2023-06-25 13:30:46,493 [main] INFO freon.BaseFreonGenerator: Executing test
with prefix p0l2uvfgg7 and number-of-tests 1
2023-06-25 13:30:46,507 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0
out of 1)
2023-06-25 13:30:46,792 [pool-2-thread-1] ERROR freon.BaseFreonGenerator: Error
on executing task 0
com.amazonaws.SdkClientException: Unable to load AWS credentials from
environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY
(or AWS_SECRET_ACCESS_KEY))
at
com.amazonaws.auth.EnvironmentVariableCredentialsProvider.getCredentials(EnvironmentVariableCredentialsProvider.java:50)
...
at
org.apache.hadoop.ozone.freon.S3KeyGenerator.createKey(S3KeyGenerator.java:111)
at
org.apache.hadoop.ozone.freon.BaseFreonGenerator.tryNextTask(BaseFreonGenerator.java:220)
at
org.apache.hadoop.ozone.freon.BaseFreonGenerator.taskLoop(BaseFreonGenerator.java:200)
at
org.apache.hadoop.ozone.freon.BaseFreonGenerator.lambda$startTaskRunners$0(BaseFreonGenerator.java:174)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2023-06-25 13:30:47,509 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0
out of 1)
2023-06-25 13:30:48,510 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0
out of 1)
2023-06-25 13:30:49,511 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0
out of 1)
2023-06-25 13:30:50,513 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0
out of 1)
2023-06-25 13:30:51,514 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0
out of 1)
2023-06-25 13:30:52,515 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0
out of 1)
2023-06-25 13:30:53,517 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0
out of 1)
2023-06-25 13:30:54,518 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0
out of 1)
2023-06-25 13:30:55,519 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0
out of 1)
2023-06-25 13:30:56,520 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 % (0
out of 1)
...
{code}
> BaseFreonGenerator may not complete with only failed attempt
> ------------------------------------------------------------
>
> Key: HDDS-8925
> URL: https://issues.apache.org/jira/browse/HDDS-8925
> Project: Apache Ozone
> Issue Type: Bug
> Components: freon
> Reporter: Attila Doroszlai
> Assignee: Attila Doroszlai
> Priority: Major
>
> * Worker thread sets {{completed=true}} on attempting n+1. task.
> * Progress bar is terminated if there are failed attempts, shutdown if not.
> The problem happens if there are tasks in progress when progressbar shutdown
> is initiated AND one of these in-progress tasks is the first to fail.
> {code:title=repro}
> $ docker-compose exec -T s3g ozone freon s3kg -n 1 -t 10 -b no-such-bucket
> ...
> 2023-06-25 13:30:46,493 [main] INFO freon.BaseFreonGenerator: Executing test
> with prefix p0l2uvfgg7 and number-of-tests 1
> 2023-06-25 13:30:46,507 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 %
> (0 out of 1)
> 2023-06-25 13:30:46,792 [pool-2-thread-1] ERROR freon.BaseFreonGenerator:
> Error on executing task 0
> com.amazonaws.SdkClientException: Unable to load AWS credentials from
> environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and
> AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
> at
> com.amazonaws.auth.EnvironmentVariableCredentialsProvider.getCredentials(EnvironmentVariableCredentialsProvider.java:50)
> ...
> at
> org.apache.hadoop.ozone.freon.S3KeyGenerator.createKey(S3KeyGenerator.java:111)
> at
> org.apache.hadoop.ozone.freon.BaseFreonGenerator.tryNextTask(BaseFreonGenerator.java:220)
> at
> org.apache.hadoop.ozone.freon.BaseFreonGenerator.taskLoop(BaseFreonGenerator.java:200)
> at
> org.apache.hadoop.ozone.freon.BaseFreonGenerator.lambda$startTaskRunners$0(BaseFreonGenerator.java:174)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)
> 2023-06-25 13:30:47,509 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 %
> (0 out of 1)
> 2023-06-25 13:30:48,510 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 %
> (0 out of 1)
> 2023-06-25 13:30:49,511 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 %
> (0 out of 1)
> 2023-06-25 13:30:50,513 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 %
> (0 out of 1)
> 2023-06-25 13:30:51,514 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 %
> (0 out of 1)
> 2023-06-25 13:30:52,515 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 %
> (0 out of 1)
> 2023-06-25 13:30:53,517 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 %
> (0 out of 1)
> 2023-06-25 13:30:54,518 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 %
> (0 out of 1)
> 2023-06-25 13:30:55,519 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 %
> (0 out of 1)
> 2023-06-25 13:30:56,520 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 %
> (0 out of 1)
> ...
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
