o-nikolas commented on code in PR #29469: URL: https://github.com/apache/airflow/pull/29469#discussion_r1103374145
########## dev/README_RELEASE_PROVIDER_PACKAGES.md: ########## @@ -561,14 +561,41 @@ Or update it if you already checked it out: svn update . ``` -Optionally you can use `check_files.py` script to verify that all expected files are -present in SVN. This script may help also with verifying installation of the packages. +Optionally you can use the [`check_files.py`](https://github.com/apache/airflow/blob/main/dev/check_files.py) +script to verify that all expected files are present in SVN. This script will produce a `Dockerfile.pmc` which +may help with verifying installation of the packages. ```shell script # Copy the list of packages (pypi urls) into `packages.txt` then run: python check_files.py providers -p {PATH_TO_SVN} ``` +After the above script completes you can build `Dockerfile.pmc` to trigger an installation of each provider +package: + +```shell script +docker build - < Dockerfile.pmc +``` + +**Note**: This may fail to install some providers. For example, if they require some system level dependencies +that aren't present in the image. If you wish to investigate you may update the Dockerfile to install any +missing dependencies and then try to build again. + +For example, currently `apache-airflow-providers-apache-hive` requires the `libsasl2` system dependency. To Review Comment: I actually noticed something weird. The install of the RC providers works when using the ci image as the dockerfile base image. But when I run `airflow info` I don't see the RC versions being used: #### The build: ``` $ docker build -f Dockerfile.pmc --tag pmc_ci_image . [+] Building 153.3s (21/21) FINISHED => [internal] load .dockerignore 0.0s => => transferring context: 2B 0.0s => [internal] load build definition from Dockerfile.pmc 0.0s => => transferring dockerfile: 1.35kB 0.0s => [internal] load metadata for ghcr.io/apache/airflow/main/ci/python3.10:latest 0.0s => [ 1/17] FROM ghcr.io/apache/airflow/main/ci/python3.10 0.1s => [ 2/17] RUN pip install 'apache-airflow-providers-amazon==7.2.0rc1' 9.4s => [ 3/17] RUN pip install 'apache-airflow-providers-apache-beam==4.2.0rc1' 8.5s => [ 4/17] RUN pip install 'apache-airflow-providers-apache-hive==5.1.2rc1' 8.8s => [ 5/17] RUN pip install 'apache-airflow-providers-arangodb==2.1.1rc1' 8.6s => [ 6/17] RUN pip install 'apache-airflow-providers-cncf-kubernetes==5.2.0rc1' 11.4s => [ 7/17] RUN pip install 'apache-airflow-providers-dbt-cloud==3.0.0rc1' 8.7s => [ 8/17] RUN pip install 'apache-airflow-providers-elasticsearch==4.4.0rc1' 9.1s => [ 9/17] RUN pip install 'apache-airflow-providers-ftp==3.3.1rc1' 7.3s => [10/17] RUN pip install 'apache-airflow-providers-google==8.9.0rc1' 16.2s => [11/17] RUN pip install 'apache-airflow-providers-microsoft-azure==5.2.0rc1' 9.9s => [12/17] RUN pip install 'apache-airflow-providers-mysql==4.0.1rc1' 9.4s => [13/17] RUN pip install 'apache-airflow-providers-presto==4.2.2rc1' 9.1s => [14/17] RUN pip install 'apache-airflow-providers-sftp==4.2.2rc1' 9.4s => [15/17] RUN pip install 'apache-airflow-providers-snowflake==4.0.3rc1' 9.0s => [16/17] RUN pip install 'apache-airflow-providers-tableau==4.1.0rc1' 8.8s => [17/17] RUN pip install 'apache-airflow-providers-trino==4.3.2rc1' 8.8s => exporting to image 0.7s => => exporting layers 0.6s => => writing image sha256:d549269c2ffa35f57fb54b8eef371bcbd5d60e98b5ca39b5eaed171c81674bd1 0.0s => => naming to docker.io/library/pmc_ci_image ``` #### Checking the version (I see the incorrect version) ``` $ docker run --rm --entrypoint airflow pmc_ci_image info | grep amazon apache-airflow-providers-amazon | 7.1.0 ``` #### Checking the versions with the image based on `apache:airflow/latest` (I see the correct RC version) ``` $ docker run --rm pmc airflow info | grep amazon apache-airflow-providers-amazon | 7.2.0rc1 ``` Any idea what's going on @potiuk? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
