adoroszlai opened a new pull request, #4975:
URL: https://github.com/apache/ozone/pull/4975
## What changes were proposed in this pull request?
Fix edge case in `BaseFreonGenerator` which happens if there are tasks in
progress when `ProgressBar` shutdown/termination is initiated AND one of these
in-progress tasks fail.
Repro:
```
$ docker-compose exec -T s3g ozone freon s3kg -n 1 -t 10 -b no-such-bucket
...
2023-06-25 13:30:46,493 [main] INFO freon.BaseFreonGenerator: Executing test
with prefix p0l2uvfgg7 and number-of-tests 1
2023-06-25 13:30:46,507 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 %
(0 out of 1)
2023-06-25 13:30:46,792 [pool-2-thread-1] ERROR freon.BaseFreonGenerator:
Error on executing task 0
com.amazonaws.SdkClientException: Unable to load AWS credentials from
environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY
(or AWS_SECRET_ACCESS_KEY))
at
com.amazonaws.auth.EnvironmentVariableCredentialsProvider.getCredentials(EnvironmentVariableCredentialsProvider.java:50)
...
at
org.apache.hadoop.ozone.freon.S3KeyGenerator.createKey(S3KeyGenerator.java:111)
at
org.apache.hadoop.ozone.freon.BaseFreonGenerator.tryNextTask(BaseFreonGenerator.java:220)
at
org.apache.hadoop.ozone.freon.BaseFreonGenerator.taskLoop(BaseFreonGenerator.java:200)
at
org.apache.hadoop.ozone.freon.BaseFreonGenerator.lambda$startTaskRunners$0(BaseFreonGenerator.java:174)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2023-06-25 13:30:47,509 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 %
(0 out of 1)
2023-06-25 13:30:48,510 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 %
(0 out of 1)
2023-06-25 13:30:49,511 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 %
(0 out of 1)
2023-06-25 13:30:50,513 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 %
(0 out of 1)
2023-06-25 13:30:51,514 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 %
(0 out of 1)
2023-06-25 13:30:52,515 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 %
(0 out of 1)
2023-06-25 13:30:53,517 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 %
(0 out of 1)
2023-06-25 13:30:54,518 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 %
(0 out of 1)
2023-06-25 13:30:55,519 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 %
(0 out of 1)
2023-06-25 13:30:56,520 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 %
(0 out of 1)
...
```
https://issues.apache.org/jira/browse/HDDS-8925
## How was this patch tested?
Added integration test.
Also tested manually:
```
$ docker-compose exec -T s3g ozone freon s3kg -n 1 -t 10 -b no-such-bucket
...
2023-06-25 16:08:04,029 [main] INFO freon.BaseFreonGenerator: Executing test
with prefix x423upnr6h and number-of-tests 1
2023-06-25 16:08:04,044 [Thread-3] INFO freon.ProgressBar: Progress: 0.00 %
(0 out of 1)
2023-06-25 16:08:04,338 [pool-2-thread-1] ERROR freon.BaseFreonGenerator:
Error on executing task 0
com.amazonaws.SdkClientException: Unable to load AWS credentials from
environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY
(or AWS_SECRET_ACCESS_KEY))
at
com.amazonaws.auth.EnvironmentVariableCredentialsProvider.getCredentials(EnvironmentVariableCredentialsProvider.java:50)
...
at
org.apache.hadoop.ozone.freon.S3KeyGenerator.lambda$createKey$0(S3KeyGenerator.java:145)
at com.codahale.metrics.Timer.time(Timer.java:101)
at
org.apache.hadoop.ozone.freon.S3KeyGenerator.createKey(S3KeyGenerator.java:111)
at
org.apache.hadoop.ozone.freon.BaseFreonGenerator.tryNextTask(BaseFreonGenerator.java:220)
at
org.apache.hadoop.ozone.freon.BaseFreonGenerator.taskLoop(BaseFreonGenerator.java:200)
at
org.apache.hadoop.ozone.freon.BaseFreonGenerator.lambda$startTaskRunners$0(BaseFreonGenerator.java:174)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2023-06-25 16:08:05,045 [Thread-3] INFO freon.ProgressBar: Progress: 100.00
% (1 out of 1)
One ore more freon test is failed.
2023-06-25 16:08:05,062 [shutdown-hook-0] INFO metrics: type=TIMER,
name=key-create, count=1, min=13.293068, max=13.293068, mean=13.293068,
stddev=0.0, median=13.293068, p75=13.293068, p95=13.293068, p98=13.293068,
p99=13.293068, p999=13.293068, mean_rate=1.3469914857611078, m1=0.0, m5=0.0,
m15=0.0, rate_unit=events/second, duration_unit=milliseconds
2023-06-25 16:08:05,063 [shutdown-hook-0] INFO freon.BaseFreonGenerator:
Total execution time (sec): 1
2023-06-25 16:08:05,063 [shutdown-hook-0] INFO freon.BaseFreonGenerator:
Failures: 1
2023-06-25 16:08:05,063 [shutdown-hook-0] INFO freon.BaseFreonGenerator:
Successful executions: 0
2023-06-25 16:08:05,063 [shutdown-hook-0] INFO freon.BaseFreonGenerator:
Expected 1 --number-of-tests objects!, successfully executed 0
```
CI:
https://github.com/adoroszlai/hadoop-ozone/actions/runs/5370663046
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
