[
https://issues.apache.org/jira/browse/AIRFLOW-6166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cabell Maddux updated AIRFLOW-6166:
-----------------------------------
Description:
Hi,
I'm relatively new to airflow and am testing out Airflow running with the
Kubernetes executor locally on Minikube based off of the CI configs in the repo
([https://github.com/apache/airflow/tree/master/scripts/ci/kubernetes/app]).
Important to note that airflow on minikube using image:
apache/airflow:master-python3.7-ci
Was able to get everything up and running smoothly, however thought default
behavior for xcom when no `do_xcom_push` argument is provided to a
`KubernetesPodOperator` was a bit confusing.
If a user doesn't explicitly create a result.json in the pod default
`XCOM_MOUNT_PATH` as part of the task pod, obviously the task will fail
([https://github.com/apache/airflow/blob/a36cfe049a2c5948b24fde7a878fe19cabede5f7/airflow/kubernetes/pod_launcher.py#L214)]
```
cat: can't open '/airflow/xcom/return.json': No such file or directory
```
Rather than the user having to explicitly create result.json as part of their
task, maybe more aligned with other operators to me to create
`\{XCOM_MOUNT_PATH}/result.json` when creating the xcom sidecar -
[https://github.com/apache/airflow/blob/a36cfe049a2c5948b24fde7a878fe19cabede5f7/airflow/kubernetes/pod_generator.py#L36].
Otherwise, maybe would be nice to document that user of KubernetesPodOperator
should be careful to explicitly deal with `do_xcom_push` to avoid a bit of
confusion.
Happy to open a PR if this is something that you think needs to be dealt with
either in code or via docs. Also found some older tests associated with this
([https://github.com/apache/airflow/blob/36f3bfb0619cc78698280f6ec3bc985f84e58343/tests/contrib/minikube/test_kubernetes_pod_operator.py#L321)|https://github.com/apache/airflow/blob/36f3bfb0619cc78698280f6ec3bc985f84e58343/tests/contrib/minikube/test_kubernetes_pod_operator.py#L321-]
If already in the works, no worries.
Thanks!
was:
Hi,
I'm relatively new to airflow and am testing out Airflow running with the
Kubernetes executor locally on Minikube based off of the CI configs in the repo
([https://github.com/apache/airflow/tree/master/scripts/ci/kubernetes/app]).
Was able to get everything up and running smoothly, however thought default
behavior for xcom when no `do_xcom_push` argument is provided to a
`KubernetesPodOperator` was a bit confusing.
If a user doesn't explicitly create a result.json in the pod default
`XCOM_MOUNT_PATH` as part of the task pod, obviously the task will fail
([https://github.com/apache/airflow/blob/a36cfe049a2c5948b24fde7a878fe19cabede5f7/airflow/kubernetes/pod_launcher.py#L214)]
```
cat: can't open '/airflow/xcom/return.json': No such file or directory
```
Rather than the user having to explicitly create result.json as part of their
task, maybe more aligned with other operators to me to create
`\{XCOM_MOUNT_PATH}/result.json` when creating the xcom sidecar -
[https://github.com/apache/airflow/blob/a36cfe049a2c5948b24fde7a878fe19cabede5f7/airflow/kubernetes/pod_generator.py#L36].
Otherwise, maybe would be nice to document that user of KubernetesPodOperator
should be careful to explicitly deal with `do_xcom_push` to avoid a bit of
confusion.
Happy to open a PR if this is something that you think needs to be dealt with
either in code or via docs. Also found some older tests associated with this
([https://github.com/apache/airflow/blob/36f3bfb0619cc78698280f6ec3bc985f84e58343/tests/contrib/minikube/test_kubernetes_pod_operator.py#L321)|https://github.com/apache/airflow/blob/36f3bfb0619cc78698280f6ec3bc985f84e58343/tests/contrib/minikube/test_kubernetes_pod_operator.py#L321-]
If already in the works, no worries.
Thanks!
Environment: Ubuntu 16.04, minikube version: v1.5.2,
> KubernetesPodOperator resolve default xcom read behavior or improve
> associated documentation
> --------------------------------------------------------------------------------------------
>
> Key: AIRFLOW-6166
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6166
> Project: Apache Airflow
> Issue Type: Improvement
> Components: documentation, executor-kubernetes, xcom
> Affects Versions: 1.10.7
> Environment: Ubuntu 16.04, minikube version: v1.5.2,
> Reporter: Cabell Maddux
> Assignee: Daniel Imberman
> Priority: Minor
>
> Hi,
> I'm relatively new to airflow and am testing out Airflow running with the
> Kubernetes executor locally on Minikube based off of the CI configs in the
> repo
> ([https://github.com/apache/airflow/tree/master/scripts/ci/kubernetes/app]).
> Important to note that airflow on minikube using image:
> apache/airflow:master-python3.7-ci
> Was able to get everything up and running smoothly, however thought default
> behavior for xcom when no `do_xcom_push` argument is provided to a
> `KubernetesPodOperator` was a bit confusing.
> If a user doesn't explicitly create a result.json in the pod default
> `XCOM_MOUNT_PATH` as part of the task pod, obviously the task will fail
> ([https://github.com/apache/airflow/blob/a36cfe049a2c5948b24fde7a878fe19cabede5f7/airflow/kubernetes/pod_launcher.py#L214)]
> ```
> cat: can't open '/airflow/xcom/return.json': No such file or directory
> ```
> Rather than the user having to explicitly create result.json as part of their
> task, maybe more aligned with other operators to me to create
> `\{XCOM_MOUNT_PATH}/result.json` when creating the xcom sidecar -
> [https://github.com/apache/airflow/blob/a36cfe049a2c5948b24fde7a878fe19cabede5f7/airflow/kubernetes/pod_generator.py#L36].
> Otherwise, maybe would be nice to document that user of KubernetesPodOperator
> should be careful to explicitly deal with `do_xcom_push` to avoid a bit of
> confusion.
> Happy to open a PR if this is something that you think needs to be dealt with
> either in code or via docs. Also found some older tests associated with this
> ([https://github.com/apache/airflow/blob/36f3bfb0619cc78698280f6ec3bc985f84e58343/tests/contrib/minikube/test_kubernetes_pod_operator.py#L321)|https://github.com/apache/airflow/blob/36f3bfb0619cc78698280f6ec3bc985f84e58343/tests/contrib/minikube/test_kubernetes_pod_operator.py#L321-]
> If already in the works, no worries.
> Thanks!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
