joshuayeung opened a new issue, #23114:
URL: https://github.com/apache/airflow/issues/23114
### Apache Airflow Provider(s)
cncf-kubernetes
### Versions of Apache Airflow Providers
apache-airflow-providers-cncf-kubernetes==3.0.0
### Apache Airflow version
2.2.5 (latest released)
### Operating System
Debian GNU/Linux 10 (buster)
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### What happened
When using `SparkKubernetesSensor` with `attach_log=True`, it cannot get the
log correctly with the below error:
``` [2022-04-20, 08:42:04 UTC] {spark_kubernetes.py:95} WARNING - Could not
read logs for pod spark-pi-0.4753748373914717-1-driver. It may have been
disposed.
Make sure timeToLiveSeconds is set on your SparkApplication spec.
underlying exception: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'Audit-Id':
'29ac5abb-452d-4411-a420-8d74155e187d', 'Cache-Control': 'no-cache, private',
'Content-Type': 'application/json', 'Date': 'Wed, 20 Apr 2022 08:42:04 GMT',
'Content-Length': '259'})
HTTP response body:
b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"a
container name must be specified for pod spark-pi-0.4753748373914717-1-driver,
choose one of: [istio-init istio-proxy
spark-kubernetes-driver]","reason":"BadRequest","code":400}\n'
```
It is because no container is specified when calling kubernetes
hook.get_pod_logs
https://github.com/apache/airflow/blob/501a3c3fbefbcc0d6071a00eb101110fc4733e08/airflow/providers/cncf/kubernetes/sensors/spark_kubernetes.py#L85
### What you think should happen instead
It should get the log of container `spark-kubernetes-driver`
### How to reproduce
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
