potiuk edited a comment on issue #18805: URL: https://github.com/apache/airflow/issues/18805#issuecomment-937936240
The problem is apache-beam. Unfortunately the "reference/constraints" for Airflow are the "lowest-common-denomination" of all our dependencies including all providers (because we want to be sure all of them can be installed). The warning you see is very specific for snowflake - snowflake allows to be installed with pyarrow <5, but it will raise the warning - which is why we go this way. Unfortunately apache beam has `pyarrow>=0.15.1,<5.0.0` and this is really what prevents us from going "higher". Maybe you would like to open an issue for apache-beam to upgrade it ? However, this is also the nice thing about constraints - constraints do not PREVENT you from installing different versions of dependencies. The constraints we produce give our users three things: * they can be sure that when they use constraints during install, airfllow will install correclty without complaining about conflicting dependencies * they can be sure all our tests for Airflow and Providers PASSED for this set of constraints. * our "reference" images use exactly this set of constraints However, If you are not using apache-beam provider in the same installation, you are completely free to install higher version of pyarrow - there is nothing to prevent you from doing it. You can easily extend or customize the image and "bump" arrow to higher version by extending the image: (https://airflow.apache.org/docs/docker-stack/build.html#example-of-adding-pypi-package) or in case the package needs compiling (for pyarrow it is quite likely), you can customize the image (https://airflow.apache.org/docs/docker-stack/build.html#examples-of-image-customizing) or extend it adding build-essentials (but this will make your image much bigger): https://airflow.apache.org/docs/docker-stack/build.html#example-when-you-add-packages-requiring-compilation. But then it's a bit on your own to make sure that the providers you use are working fine with this pyarrow version. This is the beauty (and complexity) of using constraints mostly instead of "install_requirements" for upper-bounds of the dependencies. They constraints provide "golden" set of dependency versions, but if you have the need to go higher - we will not limit "hard" the upper-bounds, unless there is a very good reason for that which is somehow related to Airflow core. The nice thing about it that those constraints are updated automatically, so we do not have to do anything to bump it. As soon as apache-beam will change their limits (apache-beam is the only other package that require pyarrow) our CI will automatically upgrade pyarrow to the newest applicable version, run all the tests - and if all the test pass it will update constraints for the next released version of Airflow. Marking this as won't fix - because there is nothing we should do - we simply need to wait until apache-beam updates their limits. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
