Ryan Blough created HDDS-16397:
----------------------------------
Summary: Investigate wall-clock improvement for Ozone Github CI
workflows
Key: HDDS-16397
URL: https://issues.apache.org/jira/browse/HDDS-16397
Project: Apache Ozone
Issue Type: Improvement
Components: CI
Affects Versions: 2.2.0, 2.3.0
Reporter: Ryan Blough
I have been doing some experimenting to try and speed up the ci-cd pipeline.
The core motivation is that we do not seem to use the resources of the default
github runners very heavily; in my checks it looks like ~6GB out of 15.6GB peak
memory consumption, and only during short bursts does CPU utilization get to
the high 90th-percentile.
The ci pipeline appears to have several options for greater or lesser caching,
parallelism, and work scoping, but they are not currently being exploited. The
plan is to experiment with these options and see if we can get lower wall-clock
time for some of these workflows. The target is stuff like:
- Implement mvn -T 1C for reactor parallelism. Chief target is the build job.
- Activate the parallel-tests profile and try up to 4 test threads.
- Further partition the om, ozone, hdds integration profiles.
- check.yml accepts 'split' that does not seem to go anywhere - follow through
to a work division property
- populate_cache.yml has a 'workflow_call' that no workflow invokes
- JUnit has parallel execution options, but we don't have thread-safety
verified for many tests; undertake thread safety verification to allow parallel
execution from JUnit
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
