[
https://issues.apache.org/jira/browse/YUNIKORN-1961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17783242#comment-17783242
]
Yu-Lin Chen commented on YUNIKORN-1961:
---------------------------------------
Hi [~mitdesai],
The spark-submit sequentially create driver/executor pods. However, from
YuniKorn’s perspective, these are two independent pods without any inherent
connection. If user information is not set in the pod, the YuniKorn admission
controller simply adds the user information based on the authenticated user
interacting with the API server.
In your wordcount example, the authenticated user who requests the
driver/executor pod is not matched. The users accounts are:
# Create Driver Pod with your user account([email protected])
# Create Executor Pod with your service account (spark)
We can config spark-submit to make sure the user-info consistent.
Below is the valid solutioins from my test:
# Set up spark-submit to request the driver pod using ‘spark’ service account,
so driver/executor have same user-info.
ex:
a. Add ‘spark’ to kubectl context,
b. Change current context to spark or config 'spark.kubernetes.context' in
spark-submit
# Set up spark-submit to request executor pod with ‘[email protected]’ user
account, so driver/executor have same user-info. The configs are:
a. Add 'spark.kubernetes.authenticate.driver.clientCertFile' in spark-submit
b. Add spark.kubernetes.authenticate.driver.clientKeyFile in spark-submit
c. remove spark.kubernetes.authenticate.driver.serviceAccountName from
spark-submit
(The driver pod will request executor with ‘[email protected]’ account.)
# Set up spark-submit to add user-info annotation in driver/executor directly.
ex:
{code:java}
--conf
spark.kubernetes.driver.annotation.yunikorn\.apache\.org/user\.info="{\"user\":
\"system:serviceaccount:spark-ns:spark\",\"groups\":
[\"system:serviceaccounts\",\"system:serviceaccounts:spark-ns\",\"system:authenticated\"]}"
\
--conf
spark.kubernetes.executor.annotation.yunikorn\.apache\.org/user\.info="{\"user\":
\"system:serviceaccount:spark-ns:spark\",\"groups\":
[\"system:serviceaccounts\",\"system:serviceaccounts:spark-ns\",\"system:authenticated\"]}"
\ {code}
(But it is required to add your accounts to
'admissionController.accessControl.systemUsers'. I don’t think this is a good
solution.[2] ex:
{code:java}
admissionController.accessControl.systemUsers:
"^system:serviceaccount:kube-system:,[email protected],system:serviceaccount:mdesai:spark"
{code}
)
You can find spark-submit configuration in below link [3].
In my opinion, this issue was caused by the spark application side. if we can
configure spark-submit to ensure user-info consistent, there should be no need
for code changes in YuniKorn.
Reference Link:
[1]
[https://stackoverflow.com/questions/42170380/how-to-add-users-to-kubernetes-kubectl]
[2]
[https://yunikorn.apache.org/docs/next/user_guide/service_config/#admissioncontrolleraccesscontrolsystemusers]
[3]
[https://spark.apache.org/docs/3.3.1/running-on-kubernetes.html#configuration]
> User Info annotation incorrect for executor pods
> ------------------------------------------------
>
> Key: YUNIKORN-1961
> URL: https://issues.apache.org/jira/browse/YUNIKORN-1961
> Project: Apache YuniKorn
> Issue Type: Bug
> Components: shim - kubernetes
> Reporter: Mit Desai
> Assignee: Yu-Lin Chen
> Priority: Major
>
> UserInfo annotation is getting populated correctly for driver pods. But for
> the executor pods, _spark_ service account information is populated. Ideally,
> this should be same as the driver pod where we have the information of the
> user who originally submitted the job.
> Here is the example of pod describe when running a worcount job with 2
> executors:
> {noformat}
> Name: md-wc-94ec368a6d3b30cd-driver
> Namespace: mdesai
> Priority: 0
> Service Account: spark
> Node: *** node1 ***
> Start Time: Wed, 06 Sep 2023 18:21:59 -0700
> Labels: spark-app-selector=spark-c9315eb5d3bf4552b30c77090a328249
> spark-role=driver
> Annotations: otherAnnotations: omited
> yunikorn.apache.org/allow-preemption: true
> yunikorn.apache.org/user.info:
> {"user":"[email protected]","groups":["group1","group2","...
> Status: Running {noformat}
>
> {noformat}
> Name: javawordcount-ca2f0f8a6d3b64e6-exec-1
> Namespace: mdesai
> Priority: 0
> Service Account: spark
> Node: *** node2 ***
> Start Time: Wed, 06 Sep 2023 18:22:06 -0700
> Labels: spark-app-selector=spark-c9315eb5d3bf4552b30c77090a328249
> spark-exec-id=1
> spark-exec-resourceprofile-id=0
> spark-role=executor
> Annotations: otherAnnotations: omited
> yunikorn.apache.org/allow-preemption: true
> yunikorn.apache.org/user.info:
>
> {"user":"system:serviceaccount:mdesai:spark","groups":["system:serviceaccounts","system:serviceaccounts:mdesai","system:authenticated"]}
> {noformat}
>
> {noformat}
> Name: javawordcount-ca2f0f8a6d3b64e6-exec-2
> Namespace: mdesai
> Priority: 0
> Service Account: spark
> Node: *** node3 ***
> Start Time: Wed, 06 Sep 2023 18:22:07 -0700
> Labels: spark-app-selector=spark-c9315eb5d3bf4552b30c77090a328249
> spark-exec-id=2
> spark-exec-resourceprofile-id=0
> spark-role=executor
> Annotations: otherAnnotations: omited
> yunikorn.apache.org/allow-preemption: true
> yunikorn.apache.org/user.info:
>
> {"user":"system:serviceaccount:mdesai:spark","groups":["system:serviceaccounts","system:serviceaccounts:mdesai","system:authenticated"]}
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
