uranusjr commented on issue #20974: URL: https://github.com/apache/airflow/issues/20974#issuecomment-1022023067
So I dug around and the problem is that some of the objects can only be serialised with `dill`, not `pickle`. `**context` works if the operator is set up with `use_dill=True`, and `use_dill=False` (the default) fails if those non-serialisable objects are requested, which does not seem unreasonable to me. What’s the suggestion here? There are two approaches I can think of: 1. Keep the current behaviour (`**context` only works with `use_dill`), and add a friendly message (perhaps also some documentation) when the user does this to reject the callable at DAG-parse time. `**context` will no longer work, and users not wishing to use dill will have to explicitly name the arguments (e.g. `def my_func(ds):` instead). 2. Automatically exclude non-serialisable objects from when `use_dill` is not set. This makes `**context` automatically work, but may introduce additional confusion when the user actually tries to access things in it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
