hi, I have configured the GPU scheduling for spark-3.0.0 on yarn following the official document ,but the job seems not runing with GPU . Do I need to modify my code to invoke CUDA ? Is there any tutorial can be shared ?
running logs: ... 2020-06-13 10:58:01,938 INFO spark.SparkContext: Running Spark version 3.0.0-preview2 2020-06-13 10:58:04,101 INFO resource.ResourceUtils: ============================================================== 2020-06-13 10:58:04,105 INFO resource.ResourceUtils: Resources for spark.driver: gpu -> [name: gpu, addresses: 0] spark-default.conf: ... spark.executor.resource.gpu.amount 1 spark.worker.resource.gpu.amount 1 spark.driver.resource.gpu.amount 1 spark.driver.resource.gpu.discoveryScript /usr/local/spark-3.0.0/examples/src/main/scripts/getGpusResources.sh spark.worker.resource.gpu.discoveryScript /usr/local/spark-3.0.0/examples/src/main/scripts/getGpusResources.sh nodemanager log: ... 2020-06-13 10:55:07,702 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin.ResourcePluginManager: Found Resource plugins from configuration: [yarn.io/gpu] 2020-06-13 10:55:07,745 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin.gpu.GpuDiscoverer: Trying to discover GPU information ... 2020-06-13 10:55:10,601 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.resourceplugin.gpu.GpuDiscoverer: Discovered GPU information: === GPUs in the system === Driver Version:440.82 ProductName=GeForce GTX 950M, MinorNumber=0, TotalMemory=2004MiB, Utilization=2.0% Thanks charles -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org