potiuk commented on PR #38531: URL: https://github.com/apache/airflow/pull/38531#issuecomment-2022576617
This is cool :). But :) Certainy we will need to keep backwards-compatibility option. I think this change needs a bit more: 1) separate use_cloud_pickle from use_dill (and failure when both are true) 2) turning dill into an optional extra (Similar to virtualenv in hatch_build.py 3) similarly adding cloudpickle into an optional extra rather than mandatory requirement 4) Everywhere where dill and cloudpickle are used we need to raise an exception explaining that this is an optional feature and how to enable it (by using `[dill], [cloudpickle]` extras respectively or installing the dill/cloudpickle dependencies 5) significant note notig that `dill is no longer automatically installed and that `[dill]` extra is needed to use dill Some more documentation explaining differences between the two (somewhere here : https://airflow.apache.org/docs/apache-airflow/stable/howto/operator/python.html#pythonvirtualenvoperator) This way we will: * stop depending on dill as mandatory requirement * will not start depending on cloudpickle as mandatory requirement * have an easy way for users to continue using dill (add `[dill]` extra - without them having to adapt touse cloudpickle that might introduce various incompatibilities -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
