adoroszlai opened a new pull request, #11829:
URL: https://github.com/apache/ozone/pull/11829
## What changes were proposed in this pull request?
Store Hadoop and Ranger tarballs in `populate-cache` for use in acceptance
tests, to avoid repeated downloads from `apache.org`.` Both workflows use
`apache/infrastructure-actions/stash` instead of `actions/cache` for these new
artifacts to avoid the 10GB repository-wide cache limit. (Follow-up HDDS-17211
will replace other usage of `actions/cache`.) Tarballs are stored in
`~/.cache/ozone`.
Add retry for download using `curl` on two levels:
- `curl --retry 3`
- re-run `curl` if needed
Update `populate-cache` trigger branches to reflect current maintenance
releases. (This needs to be cherry-picked to those branches to take effect.)
`compose_testlib.bats` is dropped, because bats test are executed on
sources, where unsubstituted `${hadoop.version}`-like placeholders (normally
replaced at build time) cause errors. The functions tested by
`compose_testlib.bats` will soon be removed as well
(https://github.com/apache/ozone/pull/11722#issuecomment-6411176942).
Hard-coded reference to `acceptance` script in `check.yml` will be removed
in HDDS-17509 (replace reusable workflow with custom action). Hard-coded
reference to specific splits can be removed at that time by duplicating checks,
probably not worth the effort.
https://issues.apache.org/jira/browse/HDDS-17517
## How was this patch tested?
1. Pushed to `master` in my fork: `populate-cache` downloaded and stored the
tarballs.
```
Downloading /home/runner/.cache/ozone/hadoop-3.4.3-src.tar.gz from
https://www.apache.org/dyn/closer.lua?action=download&filename=hadoop/common/hadoop-3.4.3/hadoop-3.4.3-src.tar.gz,
attempt 1
...
Downloading /home/runner/.cache/ozone/apache-ranger-2.9.0.tar.gz from
https://www.apache.org/dyn/closer.lua?action=download&filename=ranger/2.9.0/apache-ranger-2.9.0.tar.gz,
attempt 1
...
Downloading /home/runner/.cache/ozone/ranger-2.9.0-ozone-plugin.tar.gz from
https://www.apache.org/dyn/closer.lua?action=download&filename=ranger/2.9.0/plugins/ozone/ranger-2.9.0-ozone-plugin.tar.gz,
attempt 1
```
2. CI run:
```
Executing test ozone/test-s3a.sh
...
/home/runner/.cache/ozone/hadoop-3.4.3-src.tar.gz already downloaded
```
https://github.com/adoroszlai/ozone/actions/runs/40739228220/job/123019441263#step:17:25
```
Executing test ozonesecure-ha/test-ranger.sh
...
/home/runner/.cache/ozone/apache-ranger-2.9.0.tar.gz already downloaded
/home/runner/.cache/ozone/ranger-2.9.0-ozone-plugin.tar.gz already downloaded
```
https://github.com/adoroszlai/ozone/actions/runs/40739228220/job/123028903587#step:17:25
3. Ran `test-s3a.sh` and `test-ranger.sh` locally to verify download without
pre-existing tarballs.
Full CI:
https://github.com/adoroszlai/ozone/actions/runs/40739228220
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
