Re: [PR] [CORE] Decrease offheap memory size in resource profile for whole stage fallback case [incubator-gluten]
PHILO-HE commented on code in PR #8911: URL: https://github.com/apache/incubator-gluten/pull/8911#discussion_r2059510861 ## gluten-substrait/src/main/scala/org/apache/spark/sql/execution/GlutenAutoAdjustStageResourceProfile.scala: ## @@ -88,6 +88,11 @@ case class GlutenAutoAdjustStageResourceProfile(glutenConf: GlutenConfig, spark: val newExecutorMemory = new ExecutorResourceRequest(ResourceProfile.MEMORY, newMemoryAmount.toLong) executorResource.put(ResourceProfile.MEMORY, newExecutorMemory) + + val newExecutorOffheap = +new ExecutorResourceRequest(ResourceProfile.OFFHEAP_MEM, offheapRequest.get.amount / 10) Review Comment: @zhouyuan, the on-heap memory is increased by specifying a larger value for `ResourceProfile.MEMORY` prior to the off-heap adjustment. If I have missed something, please let me know. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [CORE] Decrease offheap memory size in resource profile for whole stage fallback case [incubator-gluten]
zhouyuan commented on code in PR #8911: URL: https://github.com/apache/incubator-gluten/pull/8911#discussion_r2057912493 ## gluten-substrait/src/main/scala/org/apache/spark/sql/execution/GlutenAutoAdjustStageResourceProfile.scala: ## @@ -88,6 +88,11 @@ case class GlutenAutoAdjustStageResourceProfile(glutenConf: GlutenConfig, spark: val newExecutorMemory = new ExecutorResourceRequest(ResourceProfile.MEMORY, newMemoryAmount.toLong) executorResource.put(ResourceProfile.MEMORY, newExecutorMemory) + + val newExecutorOffheap = +new ExecutorResourceRequest(ResourceProfile.OFFHEAP_MEM, offheapRequest.get.amount / 10) Review Comment: @PHILO-HE Thanks for improving on this! just to confirm, I can see the code logic is to reduce the offheap memory size to 1/10, but on-heap memory size is not increased, is this intended? Thanks, -yuan -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [CORE] Decrease offheap memory size in resource profile for whole stage fallback case [incubator-gluten]
jackylee-ch commented on code in PR #8911: URL: https://github.com/apache/incubator-gluten/pull/8911#discussion_r2030763118 ## gluten-substrait/src/main/scala/org/apache/spark/sql/execution/GlutenAutoAdjustStageResourceProfile.scala: ## @@ -88,6 +88,11 @@ case class GlutenAutoAdjustStageResourceProfile(glutenConf: GlutenConfig, spark: val newExecutorMemory = new ExecutorResourceRequest(ResourceProfile.MEMORY, newMemoryAmount.toLong) executorResource.put(ResourceProfile.MEMORY, newExecutorMemory) + + val newExecutorOffheap = +new ExecutorResourceRequest(ResourceProfile.OFFHEAP_MEM, offheapRequest.get.amount / 10) Review Comment: This would work only for Spark 3.5.4 or higher as https://github.com/apache/spark/pull/48963 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [CORE] Decrease offheap memory size in resource profile for whole stage fallback case [incubator-gluten]
jackylee-ch merged PR #8911: URL: https://github.com/apache/incubator-gluten/pull/8911 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [CORE] Decrease offheap memory size in resource profile for whole stage fallback case [incubator-gluten]
PHILO-HE commented on code in PR #8911: URL: https://github.com/apache/incubator-gluten/pull/8911#discussion_r2034296838 ## gluten-substrait/src/main/scala/org/apache/spark/sql/execution/GlutenAutoAdjustStageResourceProfile.scala: ## @@ -88,6 +88,11 @@ case class GlutenAutoAdjustStageResourceProfile(glutenConf: GlutenConfig, spark: val newExecutorMemory = new ExecutorResourceRequest(ResourceProfile.MEMORY, newMemoryAmount.toLong) executorResource.put(ResourceProfile.MEMORY, newExecutorMemory) + + val newExecutorOffheap = +new ExecutorResourceRequest(ResourceProfile.OFFHEAP_MEM, offheapRequest.get.amount / 10) Review Comment: @JkSelf, let's first use this empirical fixed setting and wait for feedback from user side. In theory, user don't need to configure too much offheap memory in stage fallback case. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [CORE] Decrease offheap memory size in resource profile for whole stage fallback case [incubator-gluten]
JkSelf commented on code in PR #8911: URL: https://github.com/apache/incubator-gluten/pull/8911#discussion_r2034291415 ## gluten-substrait/src/main/scala/org/apache/spark/sql/execution/GlutenAutoAdjustStageResourceProfile.scala: ## @@ -88,6 +88,11 @@ case class GlutenAutoAdjustStageResourceProfile(glutenConf: GlutenConfig, spark: val newExecutorMemory = new ExecutorResourceRequest(ResourceProfile.MEMORY, newMemoryAmount.toLong) executorResource.put(ResourceProfile.MEMORY, newExecutorMemory) + + val newExecutorOffheap = +new ExecutorResourceRequest(ResourceProfile.OFFHEAP_MEM, offheapRequest.get.amount / 10) Review Comment: @PHILO-HE Should we consider making this setting configurable? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [CORE] Decrease offheap memory size in resource profile for whole stage fallback case [incubator-gluten]
github-actions[bot] commented on PR #8911: URL: https://github.com/apache/incubator-gluten/pull/8911#issuecomment-2782315299 Run Gluten Clickhouse CI on x86 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [CORE] Decrease offheap memory size in resource profile for whole stage fallback case [incubator-gluten]
zjuwangg commented on code in PR #8911: URL: https://github.com/apache/incubator-gluten/pull/8911#discussion_r2030562526 ## gluten-substrait/src/main/scala/org/apache/spark/sql/execution/GlutenAutoAdjustStageResourceProfile.scala: ## @@ -88,6 +88,11 @@ case class GlutenAutoAdjustStageResourceProfile(glutenConf: GlutenConfig, spark: val newExecutorMemory = new ExecutorResourceRequest(ResourceProfile.MEMORY, newMemoryAmount.toLong) executorResource.put(ResourceProfile.MEMORY, newExecutorMemory) + + val newExecutorOffheap = +new ExecutorResourceRequest(ResourceProfile.OFFHEAP_MEM, offheapRequest.get.amount / 10) Review Comment: LGTM in whole stage fallback case! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [CORE] Decrease offheap memory size in resource profile for whole stage fallback case [incubator-gluten]
PHILO-HE commented on code in PR #8911: URL: https://github.com/apache/incubator-gluten/pull/8911#discussion_r2024107576 ## gluten-substrait/src/main/scala/org/apache/spark/sql/execution/GlutenAutoAdjustStageResourceProfile.scala: ## @@ -88,6 +88,11 @@ case class GlutenAutoAdjustStageResourceProfile(glutenConf: GlutenConfig, spark: val newExecutorMemory = new ExecutorResourceRequest(ResourceProfile.MEMORY, newMemoryAmount.toLong) executorResource.put(ResourceProfile.MEMORY, newExecutorMemory) + + val newExecutorOffheap = +new ExecutorResourceRequest(ResourceProfile.OFFHEAP_MEM, offheapRequest.get.amount / 10) Review Comment: @zjuwangg, could you take a look? This is just an empirical setting. And I am not sure which setting is better. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [CORE] Decrease offheap memory size in resource profile for whole stage fallback case [incubator-gluten]
github-actions[bot] commented on PR #8911: URL: https://github.com/apache/incubator-gluten/pull/8911#issuecomment-2771393535 Run Gluten Clickhouse CI on x86 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [CORE] Decrease offheap memory size in resource profile for whole stage fallback case [incubator-gluten]
github-actions[bot] commented on PR #8911: URL: https://github.com/apache/incubator-gluten/pull/8911#issuecomment-2700644150 Run Gluten Clickhouse CI on x86 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [CORE] Decrease offheap memory size in resource profile for whole stage fallback case [incubator-gluten]
github-actions[bot] commented on PR #8911:
URL:
https://github.com/apache/incubator-gluten/pull/8911#issuecomment-2700643424
Thanks for opening a pull request!
Could you open an issue for this pull request on Github Issues?
https://github.com/apache/incubator-gluten/issues
Then could you also rename ***commit message*** and ***pull request title***
in the following format?
[GLUTEN-${ISSUES_ID}][COMPONENT]feat/fix: ${detailed message}
See also:
* [Other pull requests](https://github.com/apache/incubator-gluten/pulls/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
