Owen-CH-Leung commented on issue #31183: URL: https://github.com/apache/airflow/issues/31183#issuecomment-1555908290
https://github.com/apache/airflow/blob/main/airflow/providers/cncf/kubernetes/operators/spark_kubernetes.py#L96 I think the execute() method should return the object created by `hook.create_custom_object()` in order for xcom to work and sensor to poke the operator and get its logs. Also : https://github.com/apache/airflow/blob/main/airflow/providers/cncf/kubernetes/operators/spark_kubernetes.py#L103-L115 these few lines are trying to populate spark driver log directly into `SparkKubernetesOperator`, but I think the role to fetch spark driver logs should be done by sensor ? sounds duplicated to me -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
