This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 94875e836e Make better use of parallelism for K8S tests (#31594)
94875e836e is described below
commit 94875e836e64342500a36fce1c25c5446a319d7f
Author: Jarek Potiuk <[email protected]>
AuthorDate: Mon May 29 13:44:17 2023 +0200
Make better use of parallelism for K8S tests (#31594)
The parallelism in K8S tests is not utilised well for big machines with
8 CPUS. Those machines should be able to run easily 3 running
k8s tests in parallel but the "// 4" rule made them limited to run
just 2 by default. We are changing a bit the algorithm to make sure
that the parallelism characteristics leads to better utilisation and
shorter waiting time for the tests to complete.
---
dev/breeze/src/airflow_breeze/commands/kubernetes_commands.py | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/dev/breeze/src/airflow_breeze/commands/kubernetes_commands.py
b/dev/breeze/src/airflow_breeze/commands/kubernetes_commands.py
index d96cb3e20d..17ada45871 100644
--- a/dev/breeze/src/airflow_breeze/commands/kubernetes_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/kubernetes_commands.py
@@ -186,8 +186,8 @@ option_upgrade = click.option(
option_parallelism_cluster = click.option(
"--parallelism",
help="Maximum number of processes to use while running the operation in
parallel for cluster operations.",
- type=click.IntRange(1, max(1, mp.cpu_count() // 4) if not
generating_command_images() else 4),
- default=max(1, mp.cpu_count() // 4) if not generating_command_images()
else 2,
+ type=click.IntRange(1, max(1, (mp.cpu_count() + 1) // 3) if not
generating_command_images() else 4),
+ default=max(1, (mp.cpu_count() + 1) // 3) if not
generating_command_images() else 2,
envvar="PARALLELISM",
show_default=True,
)
