[
https://issues.apache.org/jira/browse/SPARK-27373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16841506#comment-16841506
]
Thomas Graves edited comment on SPARK-27373 at 5/16/19 4:21 PM:
----------------------------------------------------------------
for the kubernetes side, it has 2 options for requesting containers: 1) pod
templates, 2) through normal spark and spark.kubernetes configs
For adding in the spark resource support, we can take the spark configs
spark.\{driver/executor}.resource.\{resourceName}.count and combine this with a
new config for the vendor name like
spark.\{driver/executor}.resource.\{resourceName}.vendor to match the device
plugin support from k8s (
[https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/)]
and add that to the PodBuilder.
We could make the vendor config kubernetes specific, but I'm thinking we leave
it generic and then just state its only supported on kubernetes right now.
Depending on the setup, I could see this being useful for say YARN since yarn
support attributes and vendor could be an attribute
spark already has functionality to override and add certain things in the pod
templates so we can use similar functionality with the resources. So we can
support both the pod templates and the configs the same way.
was (Author: tgraves):
for the kubernetes side, it has 2 options for requesting containers: 1) pod
templates, 2) through normal spark and spark.kubernetes configs
For adding in the spark resource support, we can take the spark configs
spark.\{driver/executor}.resource.\{resourceName}.count and combine this with a
new config for the vendor name like
spark.\{driver/executor}.resource.\{resourceName}.vendor to match the device
plugin support from k8s (
[https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/)]
and add that to the PodBuilder.
spark already has functionality to override and add certain things in the pod
templates so we can use similar functionality with the resources. So we can
support both the pod templates and the configs the same way.
> Design: Kubernetes support for GPU-aware scheduling
> ---------------------------------------------------
>
> Key: SPARK-27373
> URL: https://issues.apache.org/jira/browse/SPARK-27373
> Project: Spark
> Issue Type: Sub-task
> Components: Kubernetes
> Affects Versions: 3.0.0
> Reporter: Xiangrui Meng
> Assignee: Thomas Graves
> Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
