JCoder01 opened a new issue, #31104:
URL: https://github.com/apache/airflow/issues/31104
### Apache Airflow version
Other Airflow 2 version (please specify below)
### What happened
After task execution when `ti.schedule_downstream_tasks()` is executed, the
worker crashes with a `TypeError: can't pickle module objects` several levels
deep in `deepcopy`.
### What you think should happen instead
The worker should not crash
### How to reproduce
The following modified example task group reproduces the issue:
```python
from __future__ import annotations
import pendulum
from airflow.decorators import task, task_group
from airflow.models.dag import DAG
@task.kubernetes
def task_1(value: int) -> str:
"""Empty Task1"""
return f"[ Task1 {value} ]"
@task.kubernetes
def task_2(value: str) -> str:
"""Empty Task2"""
return f"[ Task2 {value} ]"
# Creating TaskGroups
@task_group
def task_group_function(value: int) -> None:
"""TaskGroup for grouping related Tasks"""
task_2(task_1(value))
# Executing Tasks and TaskGroups
with DAG(
dag_id="example_task_group_decorator",
start_date=pendulum.datetime(2021, 1, 1, tz="UTC"),
catchup=False,
tags=["example"],
) as dag:
current_task_group = task_group_function()
if __name__ == '__main__':
from airflow.models.taskinstance import TaskInstance
dr = dag.get_last_dagrun()
if dr is None:
dr = dag.create_dagrun(state='success',
run_id=f'manual_{pendulum.now()}')
run_id = dr.run_id
task = dag.get_task('task_group_function.task_1')
ti = TaskInstance(task, run_id=run_id, map_index=1)
ti.render_templates(ti.get_template_context())
task = ti.task
ti.schedule_downstream_tasks()
```
### Operating System
Debian bullseye
### Versions of Apache Airflow Providers
apache-airflow 2.5.3
apache-airflow-providers-cncf-kubernetes-6.1.0
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else
This appears to be related to the `pickling_library` attribute.
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
