[
https://issues.apache.org/jira/browse/SPARK-29415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16964360#comment-16964360
]
Thomas Graves edited comment on SPARK-29415 at 10/31/19 8:03 PM:
-----------------------------------------------------------------
More details:
TaskResourceRequest - this supports taking a resourceName and an amount (Double
for fractional resources). It only supports cpus (spark.task.cpus) and
accelerator resource types (spark.*.resource.[resourceName].*. So user can
specify cpus and resources like GPU's and FPGAS. The accerator type resources
match what we already have for configs in the acceralator aware scheduling.
[https://github.com/apache/spark/blob/master/docs/configuration.md#custom-resource-scheduling-and-configuration-overview]
ExecutorResourceRequest - this supports specifying the requirements for the
executors. It supports all the configs needed for accelerator aware scheduling
- {{spark.\{executor/driver}.resource.\{resourceName}.\{amount,
discoveryScript, vendor} as well as heap memory, overhead memory, pyspark
memory, and cores. In order to support memory types we added a "units"
parameter into ExecutorResourceRequest. the other parameters resourceName,
vendor, discoveryScript, amount all match the accelerator aware scheduling
parameters.}}
ResourceProfile - this class takes in the executor and task requirement and
holds them to be used by other components. For instance, we have to pass the
executor resources into the cluster managers so it can ask for the proper
containers. The requests have to also be passed into the executors when
launched so they use the correct discovery Script. The task requirements are
used by the scheduler to assign tasks to proper containers. {{ We also have a
ResourceProfile object that has an accessor to get the default ResourceProfile.
This is the profile generated from the configs the user passes in when the
spark application is submitted. So it will have --executor-cores, memory,
overhead memory, pyspark memory, accelerator resources the user all specified
via --confs or properties file on submit. The default profile will be used in
a lot of places since the user may never specify another ResourceProfile and
want an easy way to access it.}}
was (Author: tgraves):
More details:
TaskResourceRequest - this supports taking a resourceName and an amount (Double
for fractional resources). It only supports cpus (spark.task.cpus) and
accelerator resource types (spark.*.resource.[resourceName].*. So user can
specify cpus and resources like GPU's and FPGAS. The accerator type resources
match what we already have for configs in the acceralator aware scheduling.
[https://github.com/apache/spark/blob/master/docs/configuration.md#custom-resource-scheduling-and-configuration-overview]
ExecutorResourceRequest - this supports specifying the requirements for the
executors. It supports all the configs needed for accelerator aware scheduling
- {{spark.\{executor/driver}.resource.\{resourceName}.\{amount,
discoveryScript, vendor} as well as heap memory, overhead memory, pyspark
memory, and cores. In order to support memory types we added a "units"
parameter into ExecutorResourceRequest. the other parameters resourceName,
vendor, discoveryScript, amount all match the accelerator aware scheduling
parameters.}}
ResourceProfile - this class takes in the executor and task requirement and
holds them to be used by other components. For instance, we have to pass the
executor resources into the cluster managers so it can ask for the proper
containers. The requests have to also be passed into the executors when
launched so they use the correct discovery Script. The task requirements are
used by the scheduler to assign tasks to proper containers. {{ We also have a
ResourceProfile object that has an accessor to get the default ResourceProfile.
This is the profile generated from the configs the user passes in when the
spark application is submitted. So it will have --executor-cores, memory,
overhead memory, pyspark memory, accelerator resources the user all specified
via --confs or properties file on submit.}}
> Stage Level Sched: Add base ResourceProfile and Request classes
> ---------------------------------------------------------------
>
> Key: SPARK-29415
> URL: https://issues.apache.org/jira/browse/SPARK-29415
> Project: Spark
> Issue Type: Story
> Components: Spark Core
> Affects Versions: 3.0.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Priority: Major
>
> this is just to add initial ResourceProfile, ExecutorResourceRequest and
> taskResourceRequest classes that are used by the other parts of the code.
> Initially we will have them private until we have other pieces in place.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
