[airflow] branch main updated: Force higher parallelism for waiting for images in CI (#28209)

[potiuk]

This is an automated email from the ASF dual-hosted git repository.

potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git


The following commit(s) were added to refs/heads/main by this push:
     new 1ed01b5875 Force higher parallelism for waiting for images in CI 
(#28209)
1ed01b5875 is described below

commit 1ed01b58752650985d67127acfb19705ca0c967f
Author: Jarek Potiuk <ja...@potiuk.com>
AuthorDate: Thu Dec 8 09:40:46 2022 +0100

    Force higher parallelism for waiting for images in CI (#28209)
    
    Default parallelism for public runners is 2 because they have 2
    CPUS. However image pulling is mostly about I/O (CPU is only really
    used when images are extracted and when tests are run - a bit).
    
    With parallelism = 2 the 4 images are serializing (first 2 images
    are pulled and tested and then 2 remaining ones)
    
    By setting parallelism to 6 we are allowing all 4 images to run in
    parallel and we are safe for 3.11 when it is out to also run in
    parallel). That should save ~2 minutes for image pulling.
---
 .github/workflows/ci.yml                              | 4 ++++
 dev/breeze/src/airflow_breeze/utils/common_options.py | 2 +-
 2 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 87013de0c9..6be92cab64 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -511,6 +511,8 @@ jobs:
     env:
       RUNS_ON: "${{needs.build-info.outputs.runs-on}}"
       BACKEND: sqlite
+      # Force more parallelism for pull even on public images
+      PARALLELISM: 6
     steps:
       - name: Cleanup repo
         run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm 
-rf /workspace/*"
@@ -1049,6 +1051,8 @@ jobs:
       RUNS_ON: "${{needs.build-info.outputs.runs-on}}"
       BACKEND: sqlite
       PYTHON_MAJOR_MINOR_VERSION: 
"${{needs.build-info.outputs.default-python-version}}"
+      # Force more parallelism for pull even on public images
+      PARALLELISM: 6
     steps:
       - name: Cleanup repo
         run: docker run -v "${GITHUB_WORKSPACE}:/workspace" -u 0:0 bash -c "rm 
-rf /workspace/*"
diff --git a/dev/breeze/src/airflow_breeze/utils/common_options.py 
b/dev/breeze/src/airflow_breeze/utils/common_options.py
index 82bbde6f7e..9caf98b007 100644
--- a/dev/breeze/src/airflow_breeze/utils/common_options.py
+++ b/dev/breeze/src/airflow_breeze/utils/common_options.py
@@ -439,7 +439,7 @@ option_run_in_parallel = click.option(
 option_parallelism = click.option(
     "--parallelism",
     help="Maximum number of processes to use while running the operation in 
parallel.",
-    type=click.IntRange(1, mp.cpu_count() * 2 if not 
generating_command_images() else 8),
+    type=click.IntRange(1, mp.cpu_count() * 3 if not 
generating_command_images() else 8),
     default=mp.cpu_count() if not generating_command_images() else 4,
     envvar="PARALLELISM",
     show_default=True,