potiuk commented on pull request #20407: URL: https://github.com/apache/airflow/pull/20407#issuecomment-997451067
> @potiuk. From what I see our CI workflows, not only involve Python dependencies but bash script(which we are trying to change), tests, database, document validation. I understand that to build all dependencies only in Python virtualenv enough, for the CI of other components we need docker to replicate the real environment in small containers that are also reusable. For the smaller scripts we will not need docker, virtualenv will be enough. The thing is that our small python scripts will only use "small" set of dependencies which are ok to have their own virtualenv. Those scripts will come from `dev` folder and we make sure that the scripts are taken from`main` and not from PR. So we will not use Docker for those. For example you can see here: https://github.com/apache/airflow/blob/4ac35d723b73d02875d56bf000aafd2235ef0f4a/.github/workflows/ci.yml#L240 This is the action that is runs pytest tests of our "Breeze2". What it does: * it sets working directory to ".dev/breeze" * it sets-up python in version 3.7 with dependency cache that understands pip install * It installs dependencies specified in "./dev/breeze"' setup.* files * and runs tests afterwards This will be very similar pattern for most of our jobs that do not need full "550" dependencies of airflow installed. In our case the CI image is really needed to run airflow tests, to run mypy and flake and few other things (for example building providers - which also needs to be done inside the container for security). Most of the other scripts of ours (free space for example) are perfectly fine to use the virtualenv support provided by GithubActions like below: ``` run-new-breeze-tests: timeout-minutes: 10 name: Breeze2 tests runs-on: ${{ fromJson(needs.build-info.outputs.runsOn) }} needs: [build-info] defaults: run: shell: bash working-directory: ./dev/breeze steps: - uses: actions/checkout@v2 with: persist-credentials: false - uses: actions/setup-python@v2 with: python-version: '3.7' cache: 'pip' - run: pip install -e . - run: python3 -m pytest -n auto --color=yes ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
