MatrixManAtYrService opened a new issue, #23107:
URL: https://github.com/apache/airflow/issues/23107
### Apache Airflow version
2.3.0b1 (pre-release)
### What happened
This dag has a problem. The `name` kwarg is missing from one of the mapped
instances.
```python3
from datetime import datetime
from airflow import DAG
from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import (
KubernetesPodOperator,
)
from airflow.configuration import conf
namespace = conf.get("kubernetes", "NAMESPACE")
with DAG(
dag_id="kpo_mapped",
start_date=datetime(1970, 1, 1),
schedule_interval=None,
) as dag:
KubernetesPodOperator(
task_id="cowsay_static_named",
name="cowsay_statc",
namespace=namespace,
image="docker.io/rancher/cowsay",
cmds=["cowsay"],
arguments=["moo"],
)
KubernetesPodOperator.partial(
task_id="cowsay_mapped",
# name="cowsay_mapped", # required field missing
image="docker.io/rancher/cowsay",
namespace=namespace,
cmds=["cowsay"],
).expand(arguments=[["mooooove"], ["cow"], ["get out the way"]])
KubernetesPodOperator.partial(
task_id="cowsay_mapped_named",
name="cowsay_mapped",
namespace=namespace,
image="docker.io/rancher/cowsay",
cmds=["cowsay"],
).expand(arguments=[["mooooove"], ["cow"], ["get out the way"]])
```
If you omit that field in an unmapped task, you get a dag parse error, which
is appropriate. But omitting it from the mapped task gives you this runtime
error in the task logs:
```
[2022-04-20, 05:11:02 UTC] {standard_task_runner.py:52} INFO - Started
process 60 to run task
[2022-04-20, 05:11:02 UTC] {standard_task_runner.py:79} INFO - Running:
['airflow', 'tasks', 'run', 'kpo_mapped', 'cowsay_mapped',
'manual__2022-04-20T05:11:01+00:00', '--job-id', '12', '--raw', '--subdir',
'DAGS_FOLDER/dags/taskmap/kpo_mapped.py', '--cfg-path', '/tmp/tmp_g3sj496',
'--map-index', '0', '--error-file', '/tmp/tmp2_313wxj']
[2022-04-20, 05:11:02 UTC] {standard_task_runner.py:80} INFO - Job 12:
Subtask cowsay_mapped
[2022-04-20, 05:11:02 UTC] {task_command.py:369} INFO - Running
<TaskInstance: kpo_mapped.cowsay_mapped manual__2022-04-20T05:11:01+00:00
map_index=0 [running]> on host airflow-worker-65f9fd9d5b-vpgnk
[2022-04-20, 05:11:02 UTC] {taskinstance.py:1863} WARNING - We expected to
get frame set in local storage but it was not. Please report this as an issue
with full logs at https://github.com/apache/airflow/issues/new
Traceback (most recent call last):
File
"/usr/local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line
1440, in _run_raw_task
self._execute_task_with_callbacks(context, test_mode)
File
"/usr/local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line
1544, in _execute_task_with_callbacks
task_orig = self.render_templates(context=context)
File
"/usr/local/lib/python3.9/site-packages/airflow/models/taskinstance.py", line
2210, in render_templates
rendered_task = self.task.render_template_fields(context)
File
"/usr/local/lib/python3.9/site-packages/airflow/models/mappedoperator.py", line
722, in render_template_fields
unmapped_task = self.unmap(unmap_kwargs=kwargs)
File
"/usr/local/lib/python3.9/site-packages/airflow/models/mappedoperator.py", line
508, in unmap
op = self.operator_class(**unmap_kwargs, _airflow_from_mapped=True)
File
"/usr/local/lib/python3.9/site-packages/airflow/models/baseoperator.py", line
390, in apply_defaults
result = func(self, **kwargs, default_args=default_args)
File
"/usr/local/lib/python3.9/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py",
line 259, in __init__
self.name = self._set
_name(name)
File
"/usr/local/lib/python3.9/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py",
line 442, in _set_name
raise AirflowException("`name` is required unless `pod_template_file` or
`full_pod_spec` is set")
airflow.exceptions.AirflowException: `name` is required unless
`pod_template_file` or `full_pod_spec` is set
```
But rather than failing the task, Airflow just thinks that the task is still
running:
<img width="833" alt="Screen Shot 2022-04-19 at 11 13 47 PM"
src="https://user-images.githubusercontent.com/5834582/164156155-41986d3a-d171-4943-8443-a0fc3c542988.png";>
### What you think should happen instead
Ideally this error would be surfaced when the dag is first parsed. If
that's not possible, then it should fail the task completely (i.e. a red square
should show up in the grid view).
### How to reproduce
Run the dag above
### Operating System
ubuntu (microk8s)
### Versions of Apache Airflow Providers
apache-airflow-providers-cncf-kubernetes | 4.0.0
### Deployment
Astronomer
### Deployment details
Deployed via the astronomer airflow helm chart, values:
```
airflow:
airflowHome: /usr/local/airflow
defaultAirflowRepository: 172.28.11.191:30500/airflow
defaultAirflowTag: tb11c-inner-operator-expansion
env:
- name: AIRFLOW__CORE__DAGBAG_IMPORT_ERROR_TRACEBACK_DEPTH
value: '99'
executor: CeleryExecutor
gid: 50000
images:
airflow:
pullPolicy: Always
repository: 172.28.11.191:30500/airflow
flower:
pullPolicy: Always
pod_template:
pullPolicy: Always
logs:
persistence:
enabled: true
size: 2Gi
scheduler:
livenessProbe:
timeoutSeconds: 45
triggerer:
livenessProbe:
timeoutSeconds: 45
```
Image base: `quay.io/astronomer/ap-airflow-dev:main`
Airflow version: `2.3.0.dev20220414`
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
![https://user-images.githubusercontent.com/5834582/164156155-41986d3a-d171-4943-8443-a0fc3c542988.png"](https://user-images.githubusercontent.com/5834582/164156155-41986d3a-d171-4943-8443-a0fc3c542988.png"){kind=link}