hanna-liashchuk opened a new pull request, #26560: URL: https://github.com/apache/airflow/pull/26560
closes: ISSUE-18468 and ISSUE-23114 In case if a SparkApplications has a sidecar container, SparkKubernetesSensor fails to retrieve logs for the application because the container name is not specified. According to [this](https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/pkg/config/constants.go#L306) constants, Spark driver container always named "spark-kubernetes-driver". Error message look like this: ``` [2022-09-20, 17:06:41 EEST] {{spark_kubernetes.py:89}} WARNING - Could not read logs for pod XXXX. It may have been disposed. Make sure timeToLiveSeconds is set on your SparkApplication spec. underlying exception: (400) Reason: Bad Request HTTP response headers: HTTPHeaderDict({'Audit-Id': '86d533f0-a61e-4126-a4fa-a11d03c40ec0', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'Date': 'Tue, 20 Sep 2022 14:06:41 GMT', 'Content-Length': '268'}) HTTP response body: b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"a container name must be specified for pod XXXX, choose one of: [spark-kubernetes-driver minio-sidekick]","reason":"BadRequest","code":400}\n' [2022-09-20, 17:06:41 EEST] {{spark_kubernetes.py:115}} INFO - Spark application ended successfully ``` --- **^ Add meaningful description above** Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information. In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed. In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
