potiuk opened a new issue, #44023: URL: https://github.com/apache/airflow/issues/44023
Currently we have an edge case that when we release a new version of provider that other providers depend on, the sdist build might have a problem with fiding the dependent packages in some PRS - when the PRs are modifying only some packages as they are not locally built. When we build sdist providers, we only build them in some PRs for only affected providers in this PR - and their dependencies. The sdist provider builds are spit into chunks, and the chunks are built independently in parallel (for speed) which means that in some cases the required new version of package will be missing. For example this happened in https://github.com/apache/airflow/pull/44018 (https://github.com/apache/airflow/actions/runs/11835585982) when modified standard provider, ``` × No solution found when resolving dependencies: ╰─▶ Because only apache-airflow-providers-common-sql<=1.19.0 is available and apache-airflow-providers-standard==0.0.1.dev0 depends on apache-airflow-providers-common-sql>=1.20.0.dev0, we can conclude that apache-airflow-providers-standard==0.0.1.dev0 cannot be used. And because only apache-airflow-providers-standard==0.0.1.dev0 is available and you require apache-airflow-providers-standard, we can conclude that your requirements are unsatisfiable. Traceback (most recent call last): File "/opt/airflow/scripts/in_container/install_airflow_and_providers.py", line 565, in <module> install_airflow_and_providers() File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1157, in __call__ return self.main(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/rich_click/rich_command.py", line 152, in main rv = self.invoke(ctx) File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/local/lib/python3.9/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) File "/opt/airflow/scripts/in_container/install_airflow_and_providers.py", line 541, in install_airflow_and_providers run_command(base_install_providers_cmd, github_actions=github_actions, check=True) File "/opt/airflow/scripts/in_container/in_container_utils.py", line 47, in run_command result = subprocess.run(cmd, **kwargs) File "/usr/local/lib/python3.9/subprocess.py", line 528, in run raise CalledProcessError(retcode, process.args, subprocess.CalledProcessError: Command '['/usr/local/bin/uv', 'pip', 'install', '--python', '/usr/local/bin/python', '/dist/apache_airflow-3.0.0.dev0.tar.gz', '/dist/apache_airflow_providers_standard-0.0.1.dev0.tar.gz', '/dist/apache_airflow-3.0.0.dev0.tar.gz']' returned non-zero exit status 1. ``` We should be a bit smarter here and detect (in each chunk separately) which providers are **additionally** needed to be built - in this case since standard depends on `comon.sql >= 1.20.0`, the sdist chunk installation should add common.sql to the list of installed providers in the chunk where `standard` is installed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
