PHILO-HE commented on code in PR #10172:
URL:
https://github.com/apache/incubator-gluten/pull/10172#discussion_r2209925165
##########
gluten-substrait/src/main/scala/org/apache/spark/sql/execution/GlutenAutoAdjustStageResourceProfile.scala:
##########
@@ -156,17 +158,25 @@ object GlutenAutoAdjustStageResourceProfile extends
Logging {
def applyNewResourceProfileIfPossible(
plan: SparkPlan,
rp: ResourceProfile,
- rpManager: ResourceProfileManager): SparkPlan = {
+ rpManager: ResourceProfileManager,
+ sparkConf: SparkConf): SparkPlan = {
+ // Reflects resource changes in some configurations that will be passed to
the native side.
+ val coresPerExecutor =
rp.getExecutorCores.getOrElse(sparkConf.get(EXECUTOR_CORES))
+ val coresPerTask = rp.getTaskCpus.getOrElse(sparkConf.get(CPUS_PER_TASK))
+ val taskSlots = coresPerExecutor / coresPerTask
+
SQLConf.get.setConfString(GlutenCoreConfig.NUM_TASK_SLOTS_PER_EXECUTOR.key,
taskSlots.toString)
+ val offHeapSize =
rp.executorResources.get(ResourceProfile.OFFHEAP_MEM).map(_.amount).get
+ SQLConf.get.setConfString(
+ GlutenCoreConfig.COLUMNAR_OFFHEAP_SIZE_IN_BYTES.key,
+ offHeapSize.toString)
+ SQLConf.get.setConfString(
+ GlutenCoreConfig.COLUMNAR_TASK_OFFHEAP_SIZE_IN_BYTES.key,
+ (offHeapSize / taskSlots).toString)
Review Comment:
Yes, we should consider this situation. Just updated. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
