potiuk commented on issue #23084: URL: https://github.com/apache/airflow/issues/23084#issuecomment-1117372001
Not too much. It's more about converting bash parallelism to Python one in this case. It's mostly to rewrite what's being currently run in https://github.com/apache/airflow/blob/main/.github/workflows/ci.yml#L1477 - eventually it's just running a sequence of commands and making sure they are "ok" :) it does have a good deal of parallelism involved - we are running tests in parallel in self-hosted runners so there is chance to learn a bit about that. I have alreeady impleemented some paralellism and it is WAY simpler in Python than it was in Bash (the bash way is almost incomprehensible even for me :D and Python parallelism how I implemented it is smpler - but it still handles adjusting parallelism level to the number of CPUs available etc.). You can see how i implemented "pull" in parallel. Start here: https://github.com/apache/airflow/blob/673f8940786b33663a5eb47047af2d7c2a14df99/dev/breeze/src/airflow_breeze/commands/ci_image_tools.py#L248 I decided that everything that we run evenually in CI will have a corresponding `breeze` command - this makes everyone's live easier, so in this case we need to have a `breeze run-helm-upgrade-tests` command with some options that will allow it to run in paralllel. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
