jpatel-fpmx opened a new issue #16909:
URL: https://github.com/apache/airflow/issues/16909
<!--
Welcome to Apache Airflow! For a smooth issue process, try to answer the
following questions.
Don't worry if they're not all applicable; just try to include what you can
:-)
If you need to include code snippets or logs, please put them in fenced code
blocks. If they're super-long, please use the details tag like
<details><summary>super-long log</summary> lots of stuff </details>
Please delete these comment blocks before submitting the issue.
-->
**Apache Airflow version**: v2.1.1
**Kubernetes version (if you are using kubernetes)** (use `kubectl
version`):-
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.2",
GitCommit:"092fbfbf53427de67cac1e9fa54aaa09a28371d7", GitTreeState:"clean",
BuildDate:"2021-06-16T12:52:14Z", GoVersion:"go1.16.5", Compiler:"gc",
Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"19+",
GitVersion:"v1.19.8-eks-96780e",
GitCommit:"96780e1b30acbf0a52c38b6030d7853e575bcdf3", GitTreeState:"clean",
BuildDate:"2021-03-10T21:32:29Z", GoVersion:"go1.15.8", Compiler:"gc",
Platform:"linux/amd64"}
**Environment**: Development
- **Cloud provider or hardware configuration**: AWS
- **OS** (e.g. from /etc/os-release):
- **Kernel** (e.g. `uname -a`):
- **Install tools**:
- **Others**:
**What happened**:
I am unable to connect to my Kubernetes cluster from Airflow while using
SparkKubernetesOperator. I have hosted Airflow and Spark-operator on EKS. I
have created a connection on Airflow to connect to kubernetes cluster by using
"in cluster configuration".
I am just running the sample application just to check the execution of
spark on Kubernetes through Airflow. Please find below the sample application
link:-
https://github.com/apache/airflow/blob/main/airflow/providers/cncf/kubernetes/example_dags/example_spark_kubernetes.py
<!-- (please include exact error messages if you can) -->
HTTP response body:
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"sparkapplications.sparkoperator.k8s.io
is forbidden: User \"system:serviceaccount:***:***-cluster\" cannot create
resource \"sparkapplications\" in API group \"sparkoperator.k8s.io\" in the
namespace
\"spark-operator\"","reason":"Forbidden","details":{"group":"sparkoperator.k8s.io","kind":"sparkapplications"},"code":403}
**What you expected to happen**:
Kubernetes Airflow should schedule and run spark job using spark operator.
<!-- What do you think went wrong? -->
The Airflow Kubernetes connection is not working.
**How to reproduce it**:
1. Deploy Spark operator using helm on Kubernetes cluster.
2. Deploy Airflow using helm on Kubernetes cluster.
3. Deploy below application by writing Airflow DAG:-
https://github.com/apache/airflow/blob/main/airflow/providers/cncf/kubernetes/example_dags/example_spark_kubernetes.py
**Anything else we need to know**:
<!--
How often does this problem occur? Once? Every time etc? Everytime
HTTP response body:
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"sparkapplications.sparkoperator.k8s.io
is forbidden: User \"system:serviceaccount:***:***-cluster\" cannot create
resource \"sparkapplications\" in API group \"sparkoperator.k8s.io\" in the
namespace
\"spark-operator\"","reason":"Forbidden","details":{"group":"sparkoperator.k8s.io","kind":"sparkapplications"},"code":403}
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
