Hi
As I was going through Spark 3 config params, I noticed following group of
params. I could not understand what are they for. Can anyone please point
me in the right direction?
spark.driver.resource.{resourceName}.amount 0 Amount of a particular
resource type to use on the driver. If this is used, you must also specify
the spark.driver.resource.{resourceName}.discoveryScript for the driver to
find the resource on startup. 3.0.0
spark.driver.resource.{resourceName}.discoveryScript None A script for the
driver to run to discover a particular resource type. This should write to
STDOUT a JSON string in the format of the ResourceInformation class. This
has a name and an array of addresses. For a client-submitted driver,
discovery script must assign different resource addresses to this driver
comparing to other drivers on the same host. 3.0.0
spark.driver.resource.{resourceName}.vendor None Vendor of the resources to
use for the driver. This option is currently only supported on Kubernetes
and is actually both the vendor and domain following the Kubernetes device
plugin naming convention. (e.g. For GPUs on Kubernetes this config would be
set to nvidia.com or amd.com) 3.0.0
spark.resources.discoveryPlugin
org.apache.spark.resource.ResourceDiscoveryScriptPlugin Comma-separated
list of class names implementing
org.apache.spark.api.resource.ResourceDiscoveryPlugin to load into the
application. This is for advanced users to replace the resource discovery
class with a custom implementation. Spark will try each class specified
until one of them returns the resource information for that resource. It
tries the discovery script last if none of the plugins return information
for that resource. 3.0.0
--
Best Regards,
Ayan Guha
