adoroszlai opened a new pull request, #4750: URL: https://github.com/apache/ozone/pull/4750
## What changes were proposed in this pull request? So far each compose environment had a single test script and cluster config. If the same basic environment needs to be tested with custom configuration (e.g. different default bucket type, or using the Vault-based S3 secret store), then the environment is duplicated with minor changes. The PR updates the acceptance test library (`testlib.sh`) to allow running more than one test script per environment. Each of these can provide their own configuration and tweaks for the cluster, in a similar way to how the profiling/monitoring extension is implemented for the `ozone` environment. This helps avoid duplication, and lets us merge `ozone-legacy-bucket` into `ozone`, and `ozonesecure-vault` into `ozonesecure`. https://issues.apache.org/jira/browse/HDDS-8640 ## How was this patch tested? CI: https://github.com/adoroszlai/hadoop-ozone/actions/runs/5031215247 Verified that both test scripts were executed in each of the merged environments: ``` Executing test ozone/test-legacy-bucket.sh ... Executing test ozone/test.sh ``` https://github.com/adoroszlai/hadoop-ozone/actions/runs/5031215247/jobs/9024361030#step:5:47 https://github.com/adoroszlai/hadoop-ozone/actions/runs/5031215247/jobs/9024361030#step:5:162 ``` Executing test ozonesecure/test-vault.sh ... Executing test ozonesecure/test.sh ``` https://github.com/adoroszlai/hadoop-ozone/actions/runs/5031215247/jobs/9024360968#step:5:47 https://github.com/adoroszlai/hadoop-ozone/actions/runs/5031215247/jobs/9024360968#step:5:352 Downloaded test artifacts ([unsecure](https://github.com/adoroszlai/hadoop-ozone/suites/13028510650/artifacts/705954369), [secure](https://github.com/adoroszlai/hadoop-ozone/suites/13028510650/artifacts/705954368)), and verified that cluster logs for each run are separated, but there is a single summary (`log.html`, etc.): ``` $ unzip -t acceptance-unsecure.zip Archive: acceptance-unsecure.zip testing: log.html OK testing: output.log OK testing: ozone-legacy-bucket.xml OK testing: ozone.xml OK testing: ozone/ OK testing: report.html OK testing: summary.html OK testing: ozone/dn-audit-6b41d5ad2bd5.log OK testing: ozone/dn-audit-8587624f6389.log OK testing: ozone/dn-audit-953bb5afb4eb.log OK testing: ozone/dn-audit-ff8406a8f84a.log OK testing: ozone/docker-ozone-ozone-lib-scm.log OK testing: ozone/legacy-bucket/ OK testing: ozone/om-audit-97c71ea089d5.log OK testing: ozone/s3g-audit-7b5d545bdf0c.log OK testing: ozone/scm-audit-5e70e8ffb16d.log OK testing: ozone/legacy-bucket/docker-ozone-ozone-recon-nssummary-scm.log OK testing: ozone/legacy-bucket/om-audit-3978fb22b397.log OK testing: ozone/legacy-bucket/scm-audit-0694e8b95722.log OK ``` ``` $ unzip -t acceptance-secure.zip Archive: acceptance-secure.zip testing: log.html OK testing: output.log OK testing: ozonesecure-vault.xml OK testing: ozonesecure.xml OK testing: ozonesecure/ OK testing: report.html OK testing: summary.html OK testing: ozonesecure/dn-audit-2c8a131019ec.log OK testing: ozonesecure/dn-audit-63a37720394c.log OK testing: ozonesecure/dn-audit-a32fed614881.log OK testing: ozonesecure/docker-ozonesecure-ozonesecure-kinit-scm.log OK testing: ozonesecure/kms-audit.log OK testing: ozonesecure/om-audit-om.log OK testing: ozonesecure/s3g-audit-s3g.log OK testing: ozonesecure/scm-audit-scm.log OK testing: ozonesecure/vault/ OK testing: ozonesecure/vault/docker-ozonesecure-ozonesecure-s3-scm.log OK testing: ozonesecure/vault/om-audit-om.log OK testing: ozonesecure/vault/s3g-audit-s3g.log OK testing: ozonesecure/vault/scm-audit-scm.log OK ``` Verified that `log.html` contains the results for both test runs in each environment. Verified that the clusters for the merged tests had the expected configuration: ``` $ unzip -j acceptance-unsecure.zip ozone/legacy-bucket/docker-ozone-ozone-recon-nssummary-scm.log $ grep -o 'ozone.default.bucket.layout=[^,]*' docker-ozone-ozone-recon-nssummary-scm.log | sort -u ozone.default.bucket.layout=LEGACY $ unzip -j acceptance-secure.zip ozonesecure/vault/docker-ozonesecure-ozonesecure-s3-scm.log $ grep -o 'ozone.secret.s3.store.provider=[^,]*' docker-ozonesecure-ozonesecure-s3-scm.log | sort -u ozone.secret.s3.store.provider=org.apache.hadoop.ozone.s3.remote.vault.VaultS3SecretStorageProvider ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
