The GitHub Actions job "Tests" on airflow.git has failed. Run started by GitHub user potiuk (triggered by potiuk).
Head commit for run: 4495de73bc8ede5717e631942477384655a4ce0d / Jarek Potiuk <[email protected]> Modify breeze commands and flags for standalone shell commands (#35862) Breeze `shell` command has been foreseen eventually to also allow single commands in CI and pre-commit, and we are preparing it to be able to do so with all the bells and whistles. There are a few improvements needed to be able to do it smoothly. There are a few things that were preventing using `breeze shell` as general-purpose commmand executor: * we used defailt `docker-compose` project for running compose runs (derived from `docker-compose` parent folder of the `basy.yml` compose file. If we want to run standalone commands, we need to run them in separate projects in order to allow to run them while `breeze` shell is already running * when runing command in pre-commit hook, there is no terminal available because this command is run deep under-the-hood in git internals and it is run as command detached from the terminal it runs in. For that automated terminal allocation strategy used by docker compose was not working * breeze prints a lot of diagnostics of what's going on when you run the entrypoint by default - including diagnostics information printed by docker compose and automated docker builds. This is polluting the output of automated scripts (especially in pre-commits) * breeze command checks by defalt if image needs to be upgraded and prints information about it and allows the user to trigger the upgrade by answering "yes" for 10 seconds. This add extra delay and unnecessary information when running the scripts in pre-commits and generally in unattended mode. * Breeze CI entrypoint uses additional environment initialization when entering the container - it waits until DBs are available and performs few other initializaiton checks (for example loading scripts from `init.sh`. All that is not needed and should be skipped when running standalone scripts. * Sometimes it's not obious what internal commands are run by breeze shell - and it's difficult to debug/diagnose it. We run CI commands with `--verbose` flags so those command should not be printed even if `--verbose` flag is used though as they are polluting the output. * By default some flags are persistently stored after used (python version, backend, backend version) and they are "cached" - they are automatically used next time you start breeze - to keep the environment you use for the last time. Those shoudl not be persisted when running standalone command and want to override Python version or backend only in the command you want to execute. The PR implements the following changes to improve those aspects: * Allow specifying `--project-name` when running `breeze`, `breze shell` `breeze start-airflow` and `breeze down` commands. Default project is `breeze` now and user can use any project but predefined ones are also `pre-commit` (to be used in pre-commits) and legacy `docker-compose` if one would like to use `breeze down` to shut down some remaining containers from the legacy project. * Add `--restart` (aliased by `--remove-orphans`) flag that might be used when starting the command, which will also remove orphan containers (for example postgres container running while we are switching to sqlite). * Add `--tty` option ("auto", "disabled", "enabled") in `breeze shell` command. This flag allows to determine strategy of terminal allocation by the underlying docker-compose flags. * Add `--quiet` flag that silences the output of entrypoint diagnostics including the output of docker-compose and docker build when they are running as part of the command. NOTE: We also need to upgrade min docker-compose and docker version as we are using features available only in recent versions of these that allow to suppress some output. * Add `--skip-image-upgrade-check` in `breeze shell` to avoid checking if upgrade is needed. However if the image is not present, the latest version will be upgraded anyway - this allows to run pre-commits even if breeze has not been used before or `docker system prune -a` was used. We had `--skip-image-check` before, but we rename it to explain what check we are talking about. * Add `--skip-environment-initialization` in `breeze shell` to skip all initialization happening for interactive breeze entering. * Add `--verbose-commands` flag in `breeze shell`. For a long time we had `VERBOSE_COMMANDS` env variable support doing it (and printing commands as they are executed) but adding it explicitly as flag makes it easier discoverable and usable. * Add `SKIP_SAVING_CHOICES` environment variable. When this variable is set, the choices you specify when runing breeze command are not stored in cache and only used for the particular command you execute. Few other cleanups and minor fixes were also implemented as part of this change: * the click decorators and corresponding parameters in breeze/shell/start-airflow commands have been sorted to easier manage them (adding/updating the option became a real pain for those as they have a lot of them) * version_suffix_for_pypi default for CI image has been set up to dev0 to make automated upgrade after pressing `Y` use the same cache as `docker ci-image build` * added PYTHONWARNINGS to be forwardeable from host environment to container - which allows to control which warnings to display. * only print "Github Actions" group end when we are running inside GitHub Actions. * switch from distutls Versions to packaging Version - used to check pre-commit version (better Python 3.12 compatibility) * clarified that executor flag for "shell" command can accept more values than the one in "start-airflow" * when config options are not synchronized with "rich-click" help groups, proper location is displayed in error messsages (it asked to fix the issue in non-existing files). Report URL: https://github.com/apache/airflow/actions/runs/6997613569 With regards, GitHub Actions via GitBox --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
