potiuk commented on PR #35828:
URL: https://github.com/apache/airflow/pull/35828#issuecomment-1824908372
Few comments:
Let's see. Tried that before but it was introducing a lot of random failures
(likely 4 k8s simply use too much memory).
Likely we could think about switching a bit memory utilisation on our
self-hosted runners (we allocate a lot of memory for tmpfs to do things more
stable/faster - so whole docker installation runs using docker backed by tmpfs
- not by "regular" filesystems on our self-hosted runners. Currently 85% of
memory (of 64 GB) is used as tmpfs and only remaining 15% is used for regular
memory of the system. Maybe some optimisations of images and size could allow
us to decrease the % memory used for filesystem and then running 4 k8s will be
fine ?
https://github.com/apache/airflow-ci-infra/blob/main/github-runner-ami/packer/files/mounts_setup.sh
One more comment - doing it this way is good for experimenting, but not
good for merging. Hardcoding parallelism should not happen - because when it
runs on self-hosted runners it's fine to run 3 but for public runners even 2 is
too much - so we have special parallelism for k8s tests that has a bit weird
formula to fine-tune behaviour of automated parallelism calculation
```
option_parallelism_cluster = click.option(
"--parallelism",
help="Maximum number of processes to use while running the operation in
parallel for cluster operations.",
type=click.IntRange(1, max(1, (mp.cpu_count() + 1) // 3) if not
generating_command_images() else 4),
default=max(1, (mp.cpu_count() + 1) // 3) if not
generating_command_images() else 2,
envvar="PARALLELISM",
show_default=True,
)
```
So eventually if we want to change parallelism - we will have to change
default calculation here, or have a conditional PARALLELISM setting in CI
depending on type of worker (I'd very much prefer the default calculation as
conditionals in GH aations CI tend to look strange and be prone to errors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
