GitHub user Snaptonis edited a discussion: Serialization issues with external_python decorator
Hey everybody, I started changing my dags to use @task.external_python(). I am still running into issues and I don't know what to do anymore. A couple dags fail with: `PicklingError: Can't pickle <function get_connection at 0x7f105f947110>: it's not the same object as tasks.connections.get_connection`. Another fails with: `PicklingError: Can't pickle local object <function my_dag.<locals>.get_config at 0x7f105f94ba00>`. I have a mixture of normal tasks and external_python tasks. I think the the problems start when I have expanded tasks where the result (so the xcom) is used in another task but I'm not certain. Airflow Version: 3.3.0 If you need more infos I'm happy to provide. Edit: More info I will describe the flow for how the errors happen: First error: 1. task()s that get some connections from airflow 2. task.external_python() that reads something from database 3. task.external_python() thats expanded with partial args that takes the connection and the data read from the previous task 4. task.external_python() that takes a different connection and the return value from the previous task 5. This task fails but it manages to do what it is supposed to do. The function mentioned in the error is not part of the task. Second error: Similar here. 1. taks()s that read connections and params for dag (get_config) 2. task.external_python() expanded with partial taking the values returned from the previous tasks 3. task.external_python() taking a connection and the values returned from the previous tasks 4. This task fails. Notably the function mentioned in the error is not part of this task. GitHub link: https://github.com/apache/airflow/discussions/71281 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
