Re: [I] Can't pass in **context kwarg to the PythonVirtualenvOperator's python_callable [airflow]
potiuk commented on issue #20974: URL: https://github.com/apache/airflow/issues/20974#issuecomment-3042335832 BTW. Now you might start to understand, why we generalized things to "arguments must be serializable" when we set some conditions on what can be passed as parameters :D It's really hard problem to have a complete and comprehensive explanation on. Big thing is to understand that you have two completely different interpreters - possibly running with completely different Python and dependency versions and you need to have object from one interpreter passed to the other interpreter. In Airflow 3 it is easier - becuase - by definition - context has to be serialized to be passed through Task SDK - we are using fast_api with Pydantic to send context from the server to the task - but it also means that context is "poorer" in Airflow 3 and some things that worked in Context of Airflow 2 do not work any more in Airflow 3. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Can't pass in **context kwarg to the PythonVirtualenvOperator's python_callable [airflow]
potiuk commented on issue #20974: URL: https://github.com/apache/airflow/issues/20974#issuecomment-3042333007 I think also a lot depends on Python versions. I suspect that the `code()` message comes from either different Python version or different `dill` version used/installed. Serialization is hard. There are many border conditions that influence what is "serializable" and then "deserializable" in different environment. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Can't pass in **context kwarg to the PythonVirtualenvOperator's python_callable [airflow]
jaklan commented on issue #20974:
URL: https://github.com/apache/airflow/issues/20974#issuecomment-3041842447
A few more notes:
- ofc `serializer='dill'` instead of deprecated `use_dill=True`
- I haven't tested `cloudpickle` yet, but based on the above comment from
@potiuk - I believe it should be the recommended one instead of `dill` + there
should a note that it's recommended only when variables not serializable by
`pickle` are really needed
- when I tried to test `cloudpickle` - I got a weird error saying it wasn't
installed, although... it was:
`requirements=["apache-airflow[cloudpickle]==2.10.1"]`
```shell
[2025-07-06, 14:39:27 UTC] {process_utils.py:194} INFO - Successfully
installed Babel-2.17.0 Flask-Babel-2.0.0 Flask-JWT-Extended-4.7.1
Flask-Limiter-3.12 Flask-SQLAlchemy-2.5.1 Mako-1.3.10 PyYAML-6.0.2
WTForms-3.2.1 aiohappyeyeballs-2.6.1 aiohttp-3.12.13 aiosignal-1.4.0
alembic-1.16.2 anyio-4.9.0 apache-airflow-2.10.1
apache-airflow-providers-common-compat-1.7.2
apache-airflow-providers-common-io-1.6.1
apache-airflow-providers-common-sql-1.27.3 apache-airflow-providers-fab-1.5.3
apache-airflow-providers-ftp-3.13.1 apache-airflow-providers-http-5.3.2
apache-airflow-providers-imap-3.9.1 apache-airflow-providers-smtp-2.1.1
apache-airflow-providers-sqlite-4.1.1 apispec-6.8.2 argcomplete-3.6.2
asgiref-3.9.0 attrs-25.3.0 blinker-1.9.0 cachelib-0.13.0 certifi-2025.6.15
cffi-1.17.1 charset_normalizer-3.4.2 click-8.2.1 clickclick-20.10.2
cloudpickle-3.1.1 colorama-0.4.6 colorlog-6.9.0 configupdater-3.2
connexion-2.14.2 cron-descriptor-1.4.5 croniter-6.0.0 cryptography-45.0.5
deprecated-
1.2.18 dill-0.4.0 dnspython-2.7.0 email-validator-2.2.0 flask-2.2.5
flask-appbuilder-4.5.3 flask-caching-2.3.1 flask-login-0.6.3
flask-session-0.5.0 flask-wtf-1.2.2 frozenlist-1.7.0 fsspec-2025.5.1
google-re2-1.1.20240702 googleapis-common-protos-1.70.0 grpcio-1.73.1
gunicorn-23.0.0 h11-0.16.0 httpcore-1.0.9 httpx-0.28.1 idna-3.10
importlib_metadata-8.7.0 inflection-0.5.1 itsdangerous-2.2.0 jinja2-3.1.6
jmespath-1.0.1 jsonschema-4.24.0 jsonschema-specifications-2025.4.1
lazy-object-proxy-1.11.0 limits-5.4.0 linkify-it-py-2.0.3 lockfile-0.12.2
markdown-it-py-3.0.0 markupsafe-3.0.2 marshmallow-3.26.1
marshmallow-oneofschema-3.2.0 marshmallow-sqlalchemy-0.28.2
mdit-py-plugins-0.4.2 mdurl-0.1.2 methodtools-0.4.7 more-itertools-10.7.0
multidict-6.6.3 opentelemetry-api-1.34.1 opentelemetry-exporter-otlp-1.34.1
opentelemetry-exporter-otlp-proto-common-1.34.1
opentelemetry-exporter-otlp-proto-grpc-1.34.1
opentelemetry-exporter-otlp-proto-http-1.34.1 opentelemetry-proto-1.34.1
opentelemetry-
sdk-1.34.1 opentelemetry-semantic-conventions-0.55b1 ordered-set-4.1.0
packaging-25.0 pathspec-0.12.1 pendulum-3.1.0 pluggy-1.6.0 prison-0.2.1
propcache-0.3.2 protobuf-5.29.5 psutil-7.0.0 pycparser-2.22 pygments-2.19.2
pyjwt-2.10.1 python-daemon-3.1.2 python-dateutil-2.9.0.post0 python-nvd3-0.16.0
python-slugify-8.0.4 pytz-2025.2 referencing-0.36.2 requests-2.32.4
requests-toolbelt-1.0.0 rfc3339-validator-0.1.4 rich-13.9.4 rich-argparse-1.7.1
rpds-py-0.26.0 setproctitle-1.3.6 six-1.17.0 sniffio-1.3.1 sqlalchemy-1.4.54
sqlalchemy-jsonfield-1.0.2 sqlalchemy-utils-0.41.2 sqlparse-0.5.3
tabulate-0.9.0 tenacity-9.1.2 termcolor-3.1.0 text-unidecode-1.3
typing-extensions-4.14.1 tzdata-2025.2 uc-micro-py-1.0.3 unicodecsv-0.14.1
universal-pathlib-0.2.6 urllib3-2.5.0 werkzeug-2.2.3 wirerope-1.0.0
wrapt-1.17.2 yarl-1.20.1 zipp-3.23.0
[2025-07-06, 14:39:29 UTC] {python.py:513} INFO - Use 'cloudpickle' as
serializer.
[2025-07-06, 14:39:29 UTC] {python.py:368} ERROR - Unable to import
`cloudpickle` module. Please install it with: pip install
'apache-airflow[cloudpickle]'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [I] Can't pass in **context kwarg to the PythonVirtualenvOperator's python_callable [airflow]
jaklan commented on issue #20974:
URL: https://github.com/apache/airflow/issues/20974#issuecomment-3041781553
First round of tests (Airflow 2.10.1, locally using `airflow standalone`):
1. `**kwargs`:
```python
def dummy_callable(**kwargs):
print(kwargs)
```
a. `system_site_packages=False` + `use_dill=False`:
```shell
TypeError: cannot pickle 'module' object
```
> As expected, caused by i.a. `'macros': `
b. `system_site_packages=False` + `use_dill=True` +
`requirements=["apache-airflow=2.10.1", "dill==0.3.1.1"]`:
```shell
TypeError: code() argument 13 must be str, not int
```
> I expected that one to work, but well 🙃 No idea what `code()` is
Traceback
```shell
[2025-07-06, 09:48:14 UTC] {process_utils.py:186} INFO - Executing cmd:
/var/folders/rr/t205jkwj18l8gddtwjvzdtprgn/T/venv6oahkg8d/bin/python
/var/folders/rr/t205jkwj18l8gddtwjvzdtprgn/T/venv-call8v8a8wby/script.py
/var/folders/rr/t205jkwj18l8gddtwjvzdtprgn/T/venv-call8v8a8wby/script.in
/var/folders/rr/t205jkwj18l8gddtwjvzdtprgn/T/venv-call8v8a8wby/script.out
/var/folders/rr/t205jkwj18l8gddtwjvzdtprgn/T/venv-call8v8a8wby/string_args.txt
/var/folders/rr/t205jkwj18l8gddtwjvzdtprgn/T/venv-call8v8a8wby/termination.log
[2025-07-06, 09:48:14 UTC] {process_utils.py:190} INFO - Output:
[2025-07-06, 09:48:16 UTC] {process_utils.py:194} INFO - Traceback (most
recent call last):
[2025-07-06, 09:48:16 UTC] {process_utils.py:194} INFO - File
"/var/folders/rr/t205jkwj18l8gddtwjvzdtprgn/T/venv-call8v8a8wby/script.py",
line 29, in
[2025-07-06, 09:48:16 UTC] {process_utils.py:194} INFO - arg_dict =
dill.load(file)
[2025-07-06, 09:48:16 UTC] {process_utils.py:194} INFO -
^^^
[2025-07-06, 09:48:16 UTC] {process_utils.py:194} INFO - File
"/var/folders/rr/t205jkwj18l8gddtwjvzdtprgn/T/venv6oahkg8d/lib/python3.11/site-packages/dill/_dill.py",
line 270, in load
[2025-07-06, 09:48:16 UTC] {process_utils.py:194} INFO - return
Unpickler(file, ignore=ignore, **kwds).load()
[2025-07-06, 09:48:16 UTC] {process_utils.py:194} INFO -
^
[2025-07-06, 09:48:16 UTC] {process_utils.py:194} INFO - File
"/var/folders/rr/t205jkwj18l8gddtwjvzdtprgn/T/venv6oahkg8d/lib/python3.11/site-packages/dill/_dill.py",
line 472, in load
[2025-07-06, 09:48:16 UTC] {process_utils.py:194} INFO - obj =
StockUnpickler.load(self)
[2025-07-06, 09:48:16 UTC] {process_utils.py:194} INFO -
^
[2025-07-06, 09:48:16 UTC] {process_utils.py:194} INFO - TypeError:
code() argument 13 must be str, not int
```
2. `get_current_context()`:
```python
def dummy_callable():
from airflow.operators.python import get_current_context
context = get_current_context()
print(context)
```
a. `system_site_packages=False` + `use_dill=False` +
`requirements=["apache-airflow=2.10.1"]`:
```shell
airflow.exceptions.AirflowException: Current context was requested but
no context was found! Are you running within an airflow task?
```
b. `system_site_packages=False` + `use_dill=True` +
`requirements=["apache-airflow=2.10.1", "dill==0.3.1.1"]`:
```shell
airflow.exceptions.AirflowException: Current context was requested but
no context was found! Are you running within an airflow task?
```
> Based on the above discussion - both expected I guess
3. Individual context parameters:
```python
def dummy_callable(ds, dag, task, ti, params, ...):
print(ds)
print(dag)
...
```
a. `system_site_packages=False` + `use_dill=False`:
- what works: strings (e.g. `ds`), integers (e.g. `expanded_ti_count`),
dicts (e.g. `params`), lists (e.g. `inlets`), bools (e.g. `test_mode`)
- what doesn't work due to `TypeError: cannot pickle 'module' object`:
`dag`, `task`, `macros`, `dag_run`
- what doesn't work due to `TypeError: generate_report() missing 1
required positional argument`: `task_instance`, `ti`, `conn`, `var`
- what doesn't work due to `ModuleNotFoundError: No module named
'airflow'`: `conf`
- what doesn't work due to `ModuleNotFoundError: No module named
'lazy_object_proxy'`: `triggering_dataset_events`
- what doesn't work due to `ModuleNotFoundError: No module named
'pendulum'`: pendulum.DateTime (e.g. `logical_date`)
b. `system_site_packages=False` + `use_dill=False` +
`requirements=["apache-airflow==2.10.1", "pendulum==3.0.0",
"lazy-object-proxy==1.10.0"]`:
- what started
Re: [I] Can't pass in **context kwarg to the PythonVirtualenvOperator's python_callable [airflow]
potiuk commented on issue #20974: URL: https://github.com/apache/airflow/issues/20974#issuecomment-3038870068 > [@potiuk](https://github.com/potiuk) thanks for clarification! I will play a bit more with various combinations of options and try to conclude it somehow, just the last question for now - there are still no plans to make `dill` the default serializer (at least for `PythonVirtualOperator`), right? If anything, maybe `cloudpickle` (as being safer and more modern) but we also do not want to make it enabled by default as it has performance and - potentially - security implications. Serialization is generally "hard" problem and we have about 5 or 6 differnet serializations in Airflow (pickle, dill, cloudpickle, two or three implementation of "airflow serializer", Pydantic, and probably few more ), We did not manage to get to conclusion whether to unify them or keep different serializations for different purposes even, so I doubt we will be able to get some conclusions on changing default behaviour. If anything - I think - maybe a good approach should be to catch the errors that are raised when you **really** need serialization here and provide a bit more descriptive error - linking - YES - to the documentation you might write about it :) . That would be even BETTER contribution than just writing docs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Can't pass in **context kwarg to the PythonVirtualenvOperator's python_callable [airflow]
jaklan commented on issue #20974: URL: https://github.com/apache/airflow/issues/20974#issuecomment-3038861588 @potiuk thanks for clarification! I will play a bit more with various combinations of options and try to conclude it somehow, just the last question for now - there are still no plans to make `dill` the default serializer (at least for `PythonVirtualOperator`), right? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Can't pass in **context kwarg to the PythonVirtualenvOperator's python_callable [airflow]
potiuk commented on issue #20974: URL: https://github.com/apache/airflow/issues/20974#issuecomment-3038808105 Also there are few other opinions and comments above in the thread, if you can gather them together and come up with consistent proposal for documentation update - that would be an awesome contribution. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Can't pass in **context kwarg to the PythonVirtualenvOperator's python_callable [airflow]
potiuk commented on issue #20974: URL: https://github.com/apache/airflow/issues/20974#issuecomment-3038796259 Sure. Happy to tell you my opinion - but remember it's an opinion, not "ultimate truth" and best way is to look at the code and verify it. 1) I think some kwargs might require dill or other form of serialization - this is pretty normal that some object cannot be serialized - that's why dill, cloudpickle and other serializers exist 2) As far as I know - no and it has never been planned for it to work. But I might be wrong - maybe someone planned it or maybe someone (you?) can contribute it so that it works. It would require to pass the whole context and serialize it back/forth in the script that is executed by PythonVirtualenvOperator 3) yes it has always been a universal solution, but if some context param cannot be serialized directly, it might require dill or cloudpickle - depends on the parameter you are passing. I think that is actually explaines somewhere in the docs that the parameters that you pass to Python Virtualenv need to be serializable, context parameters are no exception. Feel free to convert it to documentation update that will make it easier for others to understand it. For me those are pretty natural answers simply by knowing how things work and how passing parameters and serialization works, but - again - I have a lot of assumptions in my head so I do not even feel the need of explaining it - so you are probably the best person in the world now to attempt to do it so that people like you could understand it in a glance. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Can't pass in **context kwarg to the PythonVirtualenvOperator's python_callable [airflow]
jaklan commented on issue #20974: URL: https://github.com/apache/airflow/issues/20974#issuecomment-3038786743 @potiuk sure, I just want to understand what's the current state of things first. Let's try to wrap it up: 1. `**kwargs`: a. Should we never use it for `PythonVirtualenvOperator` or is it okay to use if we set `use_dill` to `True`? Are there any pitfalls with `dill` except a need to have it installed? 2. `get_current_context()`: a. As I understand it still doesn't work inside `PythonVirtualenvOperator`? Or is there any scenario (== set of configs) when it works? 3. Passing individual context variables as callable params: a. Is it an universal solution or are there any exceptions (in another issue I saw sth about issues with `ti` variable)? And generally - what's the final recommendation? `**kwargs` + `dill`, individual context variables, anything else? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Can't pass in **context kwarg to the PythonVirtualenvOperator's python_callable [airflow]
potiuk commented on issue #20974: URL: https://github.com/apache/airflow/issues/20974#issuecomment-3038776607 > docs definitely need some clarification. Absolutely - and the best way to do it - especially if you understand it now better - is to open a PR updating it in the docs. This is usually as easy as "Suggest a change on this page" - and being someone bitten by it - judging fom all the discussion here, you are probably the best person now to know where to add it and how to described it in the words that people like you would immediately recognise it. And when you open PR it will still be reviewed and commented by maintainers, so even if you won't get it perfectly right, opening PR will actually make some maintainers to look at it and comment and help you correct any mistake. This is how it works here - a lot of documentation (we have > 3600 contributors) who had problem, saw imperfect documentation, got it explained and solved and they contributed documentation update so that other contributors can benefit from it. You are absolutely wolcome to propose such clarification. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [I] Can't pass in **context kwarg to the PythonVirtualenvOperator's python_callable [airflow]
jaklan commented on issue #20974: URL: https://github.com/apache/airflow/issues/20974#issuecomment-3038773503 I was just affected by this issue on Airflow 2.10.3 and the only error message I got was `TypeError: cannot pickle 'module' object`, without any hints related to `use_dill` or passing context variables explicitly, so I believe nothing changed here? Also I haven't noticed in docs any note about not using `**kwargs` and generally - about a proper way to deal with context in `PythonVirtualenvOperator`, this issue is the only point of reference. Even if the issue is not going to be solved - docs definitely need some clarification. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
