potiuk commented on code in PR #24652: URL: https://github.com/apache/airflow/pull/24652#discussion_r1022068572
########## Dockerfile: ########## @@ -1234,8 +1234,10 @@ ARG ADDITIONAL_PYTHON_DEPS="" # https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates # * authlib, gcloud_aio_auth, adal are needed to generate constraints for PyPI packages and can be removed after we release # new google, azure providers +# * cloudpickle==2.0.0 is required by snowfalke-snowpark-python, which conflicts with new versions of apache beam, so +# we pin apache-beam also to the latest version which uses cloudpickle==2.0.0. # !!! MAKE SURE YOU SYNCHRONIZE THE LIST BETWEEN: Dockerfile, Dockerfile.ci, find_newer_dependencies.py -ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="dill<0.3.3 pyarrow>=6.0.0 protobuf<4.21.0 authlib>=1.0.0 gcloud_aio_auth>=4.0.0 adal>=1.2.7" +ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="dill<0.3.3 pyarrow>=6.0.0 protobuf<4.21.0 authlib>=1.0.0 gcloud_aio_auth>=4.0.0 adal>=1.2.7 cloudpickle==2.0.0 apache-beam==2.39.0" Review Comment: i think this is not good to `pin` the requirements to specific versions. if we really want to limit them, we should rather say <= or < here. Why do we have "apache-beam==2.39.0" and "cloudpickle==2.0.0" - seems a bit odd choice and also it will heavily limit our capabilities. I believe the ARM image failing is caused by this one and also we are waiting for the new version of apache-beam to be able to release Python 3.11. I think limiting those **just** to have snowpark decorator is pretty wrong. Is there any reason snowpark cannot work with higher (also future) versions of those? I think - to be perfectly honest, if that would be the consequence of accepting the snowpark, then possibly snowflake should release their own snowpark decorator as add-on to the existing provider? Limiting us to potentially fixed and vulnerable versions of dependencies is not something we should treat lightly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
