potiuk commented on PR #37345: URL: https://github.com/apache/airflow/pull/37345#issuecomment-1937897742
Ok. That should be it. I have now changed the hatch_build hook, so that the complete list of all the 3rd-party dependencies that we can install for airflow is expoxed as 'devel-ci` extra. This should do the trick: 1) `devel-ci` extra has currently 321 dependencies to install (not including transitive) 2) when we prepare cache during image building we will be able to install them all via `apache-airflow[devel-ci] @ https://github.com/apache/airflow/archive/main.tar.gz"` and all of thme will get pre-installed 3) we do not need to copy pyproject.toml for it, so changed pyproject.toml will not invalidate those cached deps - they will stay stable installed in the cached image until we change `AIRFLOW_CI_BUILD_EPOCH` or until new Python base image 4) then, every time pyproject.toml changes, only differential installation will happen - for the dependencies that were updated or added since last cache rebuilding Tested it nicely now finally found a way to reproduce URL installation 1) `git archive HEAD --format tar.gz > /tmp/archive.tar.gz` 2) `pip install "/tmp/archive.tar.gz[devel-ci]"` Result: ``` Successfully installed JPype1-1.5.0 PyEnchant-3.2.2 adal-1.2.7 adlfs-2024.2.0 aiobotocore-2.11.2 aioitertools-0.11.0 alabaster-0.7.13 alibabacloud-adb20211201-1.2.8 alibabacloud-credentials-0.3.2 alibabacloud-endpoint-util-0.0.3 alibabacloud-gateway-spi-0.0.1 alibabacloud-openapi-util-0.2.2 alibabacloud-tea-0.3.5 alibabacloud-tea-openapi-0.3.8 alibabacloud-tea-util-0.3.11 alibabacloud-tea-xml-0.0.2 aliyun-python-sdk-core-2.14.0 aliyun-python-sdk-kms-2.16.2 amqp-5.2.0 analytics-python-1.2.9 anyascii-0.3.2 apache-airflow-2.9.0.dev0 apache-beam-2.53.0 apprise-1.7.2 asana-3.2.2 asn1crypto-1.5.1 astroid-2.15.8 asyncssh-2.14.2 atlasclient-1.0.0 atlassian-python-api-3.41.10 authlib-1.3.0 aws-sam-translator-1.84.0 aws-xray-sdk-2.12.1 azure-batch-14.1.0 azure-common-1.1.28 azure-core-1.30.0 azure-cosmos-4.5.1 azure-datalake-store-0.0.53 azure-identity-1.15.0 azure-keyvault-secrets-4.7.0 azure-kusto-data-4.3.1 azure-mgmt-containerinstance-10.1.0 azure-mgmt-containerregistry-10.3.0 azure-mgm t-core-1.4.0 azure-mgmt-cosmosdb-9.4.0 azure-mgmt-datafactory-5.0.0 azure-mgmt-datalake-nspkg-3.0.1 azure-mgmt-datalake-store-0.5.0 azure-mgmt-nspkg-3.0.2 azure-mgmt-resource-23.0.1 azure-mgmt-storage-21.1.0 azure-nspkg-3.0.2 azure-servicebus-7.11.4 azure-storage-blob-12.19.0 azure-storage-file-datalake-12.14.0 azure-storage-file-share-12.15.0 azure-synapse-artifacts-0.18.0 azure-synapse-spark-0.7.0 bcrypt-4.1.2 billiard-4.2.0 bitarray-2.9.2 boto3-1.34.34 botocore-1.34.34 cassandra-driver-3.29.0 cattrs-23.2.3 celery-5.3.6 cfn-lint-0.85.1 cgroupspy-0.2.2 chardet-5.2.0 checksumdir-1.2.0 ciso8601-2.3.1 click-didyoumean-0.3.0 click-plugins-1.1.1 click-repl-0.3.0 cloudant-2.15.0 cloudpickle-2.2.1 cohere-4.46 comm-0.2.1 confluent-kafka-2.3.0 crcmod-1.7 cryptography-41.0.7 curlify-2.2.1 databricks-sql-connector-2.9.3 datadog-0.48.0 db-dtypes-1.2.0 debugpy-1.8.1 deltalake-0.15.3 dill-0.3.1.1 distro-1.9.0 docker-7.0.0 docopt-0.6.2 docutils-0.16 ecdsa-0.18.0 elastic-transport-8.12.0 elasticse arch-8.12.0 entrypoints-0.4 enum34-1.1.10 eralchemy2-1.3.8 et-xmlfile-1.1.0 eventlet-0.35.1 facebook-business-19.0.0 fastavro-1.9.3 fasteners-0.19 fastjsonschema-2.19.1 flask-bcrypt-1.0.1 flower-2.0.1 fsspec-2023.12.2 future-0.18.3 gcloud-aio-auth-4.2.3 gcloud-aio-bigquery-7.0.0 gcloud-aio-storage-9.1.0 gcsfs-2023.12.2.post1 geomet-0.2.1.post1 gevent-23.9.1 google-ads-23.0.0 google-analytics-admin-0.22.5 google-api-core-2.17.0 google-api-python-client-2.117.0 google-auth-httplib2-0.2.0 google-auth-oauthlib-1.2.0 google-cloud-aiplatform-1.41.0 google-cloud-appengine-logging-1.4.1 google-cloud-audit-log-0.2.5 google-cloud-automl-2.13.1 google-cloud-batch-0.17.11 google-cloud-bigquery-3.17.2 google-cloud-bigquery-datatransfer-3.14.1 google-cloud-bigquery-storage-2.24.0 google-cloud-bigtable-2.23.0 google-cloud-build-3.23.1 google-cloud-compute-1.16.1 google-cloud-container-2.40.0 google-cloud-core-2.4.1 google-cloud-datacatalog-3.18.1 google-cloud-dataflow-client-0.8.8 google-cloud-dat aform-0.5.7 google-cloud-dataplex-1.12.1 google-cloud-dataproc-5.9.1 google-cloud-dataproc-metastore-1.15.1 google-cloud-dlp-3.15.1 google-cloud-kms-2.21.1 google-cloud-language-2.13.1 google-cloud-logging-3.9.0 google-cloud-memcache-1.9.1 google-cloud-monitoring-2.19.1 google-cloud-orchestration-airflow-1.11.1 google-cloud-os-login-2.14.1 google-cloud-pubsub-2.19.4 google-cloud-redis-2.15.1 google-cloud-resource-manager-1.12.1 google-cloud-run-0.10.3 google-cloud-secret-manager-2.18.1 google-cloud-spanner-3.42.0 google-cloud-speech-2.24.1 google-cloud-storage-2.14.0 google-cloud-storage-transfer-1.11.1 google-cloud-tasks-2.16.1 google-cloud-texttospeech-2.16.1 google-cloud-translate-3.15.1 google-cloud-videointelligence-2.13.1 google-cloud-vision-3.7.0 google-cloud-workflows-1.14.1 google-crc32c-1.5.0 google-resumable-media-2.7.0 graphql-core-3.2.3 graphviz-0.20.1 grpc-google-iam-v1-0.13.0 grpc-interceptor-0.15.4 grpcio-gcp-0.2.2 grpcio-status-1.60.1 grpcio-tools-1.60.1 gssapi-1.8. 3 h2-4.1.0 hdfs-2.7.3 hmsclient-0.1.1 hpack-4.0.0 httpcore-0.16.3 httplib2-0.22.0 httpx-0.23.3 humanize-4.9.0 hvac-2.1.0 hyperframe-6.0.1 ijson-3.2.3 imagesize-1.4.1 importlib-resources-5.13.0 impyla-0.19.0 influxdb-client-1.40.0 ipykernel-6.29.2 isodate-0.6.1 jaydebeapi-1.2.3 jmespath-0.10.0 js2py-0.74 jschema-to-python-1.2.3 json-merge-patch-0.2 jsondiff-2.0.0 jsonpatch-1.33 jsonpath-ng-1.6.1 jsonpickle-3.0.2 jsonpointer-2.4 jsonschema-path-0.3.2 junit-xml-1.9 jupyter-client-8.6.0 jupyter-core-5.7.1 kombu-5.3.5 krb5-0.5.1 kubernetes-29.0.0 kubernetes-asyncio-29.0.0 kylinpy-2.8.4 ldap3-2.9.1 loguru-0.7.2 looker-sdk-24.0.0 lxml-5.1.0 lz4-4.3.3 mmhash3-3.0.1 moto-5.0.1 mpmath-1.3.0 msal-1.26.0 msal-extensions-1.1.0 msrest-0.7.1 msrestazure-0.6.4 multi-key-dict-2.0.3 mypy-1.8.0 mypy-boto3-appflow-1.34.0 mypy-boto3-rds-1.34.30 mypy-boto3-redshift-data-1.34.0 mypy-boto3-s3-1.34.14 mysql-connector-python-8.3.0 mysqlclient-2.2.4 nbclient-0.9.0 nbformat-5.9.2 neo4j-5.17.0 nest-asyncio-1.6. 0 networkx-3.1 numpy-1.24.4 objsize-0.6.1 openai-1.12.0 openapi-schema-validator-0.6.2 openapi-spec-validator-0.7.1 openlineage-integration-common-1.8.0 openlineage-python-1.8.0 openlineage-sql-1.8.0 openpyxl-3.1.2 opensearch-py-2.4.2 opentelemetry-exporter-prometheus-0.43b0 opsgenie-sdk-2.1.5 oracledb-2.0.1 orjson-3.9.13 oss2-2.18.4 pandas-2.0.3 pandas-gbq-0.21.0 pandas-stubs-2.0.3.230814 papermill-2.5.0 paramiko-3.4.0 pathable-0.4.3 pbr-6.0.0 pdpyras-5.2.0 pgvector-0.2.5 pinecone-client-2.2.4 pinotdb-5.1.2 platformdirs-3.11.0 ply-3.11 plyvel-1.5.1 portalocker-2.8.2 presto-python-client-0.8.4 prometheus-client-0.19.0 proto-plus-1.23.0 psycopg2-binary-2.9.9 pure-sasl-0.6.2 py-partiql-parser-0.5.1 py4j-0.10.9.7 pyarrow-14.0.2 pyarrow-hotfix-0.6 pyathena-3.3.0 pycountry-23.12.11 pycryptodome-3.20.0 pydata-google-auth-1.8.2 pydot-1.4.2 pydruid-0.6.6 pyexasol-0.25.2 pygraphviz-1.11 pyhive-0.7.0 pyiceberg-0.5.1 pyjsparser-2.7.1 pykerberos-1.2.4 pymssql-2.2.11 pyodbc-5.1.0 pyopenssl-23.3. 0 pyparsing-3.1.1 pypsrp-0.8.1 pyspark-3.5.0 pyspnego-0.10.2 python-arango-7.9.1 python-dotenv-1.0.1 python-http-client-3.3.7 python-jenkins-1.8.2 python-jose-3.3.0 python-ldap-3.4.4 python-telegram-bot-20.2 python3-saml-1.16.0 pywinrm-0.4.3 pyzmq-25.1.2 qdrant-client-1.7.3 reactivex-4.0.4 redis-4.6.0 redshift-connector-2.0.918 regex-2023.12.25 requests-file-2.0.0 requests-kerberos-0.14.0 requests-ntlm-1.2.0 responses-0.24.1 rfc3986-1.5.0 s3fs-2023.12.2 s3transfer-0.10.0 sarif-om-1.0.4 scramp-1.4.4 scrapbook-0.5.0 sendgrid-6.11.0 sentry-sdk-1.40.3 setuptools-66.1.1 shapely-2.0.2 simple-salesforce-1.12.5 slack-sdk-3.26.2 smbprotocol-1.12.0 snowballstemmer-2.2.0 snowflake-connector-python-3.7.0 snowflake-sqlalchemy-1.5.1 sortedcontainers-2.4.0 sphinx-5.3.0 sphinx-airflow-theme-0.0.12 sphinx-argparse-0.4.0 sphinx-autoapi-2.1.1 sphinx-copybutton-0.5.2 sphinx-design-0.5.0 sphinx-jinja-2.0.2 sphinx-rtd-theme-2.0.0 sphinxcontrib-applehelp-1.0.4 sphinxcontrib-devhelp-1.0.2 sphinxcontrib-htm lhelp-2.0.1 sphinxcontrib-httpdomain-1.8.1 sphinxcontrib-jquery-4.1 sphinxcontrib-jsmath-1.0.1 sphinxcontrib-qthelp-1.0.3 sphinxcontrib-redoc-1.6.0 sphinxcontrib-serializinghtml-1.1.5 sphinxcontrib-spelling-8.0.0 spython-0.3.13 sqlalchemy-bigquery-1.9.0 sqlalchemy-drill-1.1.4 sqlalchemy-redshift-0.8.14 sqlalchemy-spanner-1.6.2 sshpubkeys-3.3.1 sshtunnel-0.4.0 starkbank-ecdsa-2.2.0 statsd-4.0.1 strictyaml-1.7.3 sympy-1.12 tableauserverclient-0.25 thrift-0.16.0 thrift-sasl-0.4.3 tornado-6.4 tqdm-4.66.2 trino-0.327.0 types-requests-2.31.0.6 types-urllib3-1.26.25.14 tzlocal-5.2 uritemplate-4.1.1 urllib3-1.26.18 validators-0.22.0 vertica-python-1.3.8 vine-5.1.0 watchtower-3.0.1 weaviate-client-3.26.2 xmlsec-1.3.13 xmltodict-0.13.0 yandexcloud-0.258.0 zeep-4.2.1 zenpy-2.0.46 zope.event-5.0 zope.interface-6.1 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
