wbo4958 commented on PR #43323:
URL: https://github.com/apache/spark/pull/43323#issuecomment-1756559085
## With dynamic allocation enabled.
``` bash
./bin/spark-shell --master k8s://$K8S_SERVER \
--conf spark.kubernetes.container.image=spark:pr_k8s \
--conf spark.kubernetes.context=minikube \
--conf spark.kubernetes.namespace=spark-demo \
--verbose \
--num-executors=1 \
--conf spark.executor.cores=4 \
--conf spark.task.cpus=1 \
--conf spark.dynamicAllocation.enabled=true
```
The above command enables the dynamic allocation and the max executors
required is set to 1 in order to test.
### TaskResourceProfile without any specific executor request information
Test code,
``` scala
import org.apache.spark.resource.{ResourceProfileBuilder,
TaskResourceRequests}
val rdd = sc.range(0, 100, 1, 4)
var rdd1 = rdd.repartition(3)
val treqs = new TaskResourceRequests().cpus(3)
val rp = new ResourceProfileBuilder().require(treqs).build
rdd1 = rdd1.withResources(rp)
rdd1.collect()
```
The rp refers to the TaskResourceProfile without any specific executor
request information, thus the executor information will utilize the default
values from Default ResourceProfile (executor.cores=4).
The above code will require an extra executor which will have the same
`executor.cores/memory` as the default ResourceProfile.
### Different executor request information
``` scala
import org.apache.spark.resource.{ExecutorResourceRequests,
ResourceProfileBuilder, TaskResourceRequests}
val rdd = sc.range(0, 100, 1, 4)
var rdd1 = rdd.repartition(3)
val ereqs = new ExecutorResourceRequests().cores(6);
val treqs = new TaskResourceRequests().cpus(5)
val rp = new ResourceProfileBuilder().require(ereqs).require(treqs).build
rdd1 = rdd1.withResources(rp)
rdd1.collect()
```
we can see the "Executor ID = 2" has the 6 CPU cores.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
