Re: [PR] [GLUTEN-10915][VL] Fix dynamic offheap sizing feature by setting default offheap size [incubator-gluten]
zhouyuan commented on PR #10916: URL: https://github.com/apache/incubator-gluten/pull/10916#issuecomment-3435492753 @zhli1142015 @wForget Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-10915][VL] Fix dynamic offheap sizing feature by setting default offheap size [incubator-gluten]
zhouyuan merged PR #10916: URL: https://github.com/apache/incubator-gluten/pull/10916 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-10915][VL] Fix dynamic offheap sizing feature by setting default offheap size [incubator-gluten]
github-actions[bot] commented on PR #10916: URL: https://github.com/apache/incubator-gluten/pull/10916#issuecomment-3431797481 Run Gluten Clickhouse CI on x86 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-10915][VL] Fix dynamic offheap sizing feature by setting default offheap size [incubator-gluten]
zhouyuan commented on code in PR #10916:
URL:
https://github.com/apache/incubator-gluten/pull/10916#discussion_r2451581104
##
gluten-core/src/main/scala/org/apache/gluten/GlutenPlugin.scala:
##
@@ -118,6 +118,7 @@ private object GlutenDriverPlugin extends Logging {
// 1GB default
1024 * 1024 * 1024
}
+conf.set(GlutenCoreConfig.SPARK_OFFHEAP_SIZE_KEY, "0")
Review Comment:
@wForget for now if dynamic offheap sizing feature is enabled, the spark
off-heap settings will be override(ignored). We have already documented a bit
on this behavior in below doc.
https://github.com/apache/incubator-gluten/blob/main/docs/developers/VeloxDynamicSizingOffheap.md#dynamic-off-heap-sizing
I should add WARN log here if off-heap related settings are detected, so
users may pay more attention on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-10915][VL] Fix dynamic offheap sizing feature by setting default offheap size [incubator-gluten]
wForget commented on code in PR #10916:
URL:
https://github.com/apache/incubator-gluten/pull/10916#discussion_r2450289817
##
gluten-core/src/main/scala/org/apache/gluten/GlutenPlugin.scala:
##
@@ -118,6 +118,7 @@ private object GlutenDriverPlugin extends Logging {
// 1GB default
1024 * 1024 * 1024
}
+conf.set(GlutenCoreConfig.SPARK_OFFHEAP_SIZE_KEY, "0")
Review Comment:
> When DYNAMIC_OFFHEAP_SIZING_ENABLED is enabled, the memory used by Velox
is tracked under the on-heap category by the Spark memory manager, so Ithink
there is no need to enable off-heap memory.
Thanks, that makes sense to me. @zhouyuan Can we point this out in the
documentation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-10915][VL] Fix dynamic offheap sizing feature by setting default offheap size [incubator-gluten]
zhli1142015 commented on code in PR #10916:
URL:
https://github.com/apache/incubator-gluten/pull/10916#discussion_r2450279792
##
gluten-core/src/main/scala/org/apache/gluten/GlutenPlugin.scala:
##
@@ -118,6 +118,7 @@ private object GlutenDriverPlugin extends Logging {
// 1GB default
1024 * 1024 * 1024
}
+conf.set(GlutenCoreConfig.SPARK_OFFHEAP_SIZE_KEY, "0")
Review Comment:
When DYNAMIC_OFFHEAP_SIZING_ENABLED is enabled, the memory used by Velox is
tracked under the on-heap category by the Spark memory manager, so Ithink there
is no need to enable off-heap memory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-10915][VL] Fix dynamic offheap sizing feature by setting default offheap size [incubator-gluten]
wForget commented on code in PR #10916:
URL:
https://github.com/apache/incubator-gluten/pull/10916#discussion_r2450266549
##
gluten-core/src/main/scala/org/apache/gluten/GlutenPlugin.scala:
##
@@ -118,6 +118,7 @@ private object GlutenDriverPlugin extends Logging {
// 1GB default
1024 * 1024 * 1024
}
+conf.set(GlutenCoreConfig.SPARK_OFFHEAP_SIZE_KEY, "0")
Review Comment:
Should we allow both `DYNAMIC_OFFHEAP_SIZING_ENABLED` and
`SPARK_OFFHEAP_ENABLED_KEY` to be enabled at the same time?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-10915][VL] Fix dynamic offheap sizing feature by setting default offheap size [incubator-gluten]
github-actions[bot] commented on PR #10916: URL: https://github.com/apache/incubator-gluten/pull/10916#issuecomment-3427670944 Run Gluten Clickhouse CI on x86 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-10915][VL] Fix dynamic offheap sizing feature by setting default offheap size [incubator-gluten]
github-actions[bot] commented on PR #10916: URL: https://github.com/apache/incubator-gluten/pull/10916#issuecomment-3427564605 Run Gluten Clickhouse CI on x86 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-10915][VL] Fix dynamic offheap sizing feature by setting default offheap size [incubator-gluten]
github-actions[bot] commented on PR #10916: URL: https://github.com/apache/incubator-gluten/pull/10916#issuecomment-3425993295 Run Gluten Clickhouse CI on x86 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-10915][VL] Fix dynamic offheap sizing feature by setting default offheap size [incubator-gluten]
zhouyuan commented on code in PR #10916:
URL:
https://github.com/apache/incubator-gluten/pull/10916#discussion_r2447552115
##
gluten-core/src/main/scala/org/apache/gluten/GlutenPlugin.scala:
##
@@ -118,6 +118,7 @@ private object GlutenDriverPlugin extends Logging {
// 1GB default
1024 * 1024 * 1024
}
+conf.set(GlutenCoreConfig.SPARK_OFFHEAP_SIZE_KEY, "0")
Review Comment:
@zhli1142015 this config is required during gluten startup, previously if
this is set in spark.conf then gluten will fail to start
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-10915][VL] Fix dynamic offheap sizing feature by setting default offheap size [incubator-gluten]
zhouyuan commented on code in PR #10916:
URL:
https://github.com/apache/incubator-gluten/pull/10916#discussion_r2447552115
##
gluten-core/src/main/scala/org/apache/gluten/GlutenPlugin.scala:
##
@@ -118,6 +118,7 @@ private object GlutenDriverPlugin extends Logging {
// 1GB default
1024 * 1024 * 1024
}
+conf.set(GlutenCoreConfig.SPARK_OFFHEAP_SIZE_KEY, "0")
Review Comment:
@zhli1142015 this config is required during gluten loading native libs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-10915][VL] Fix dynamic offheap sizing feature by setting default offheap size [incubator-gluten]
github-actions[bot] commented on PR #10916: URL: https://github.com/apache/incubator-gluten/pull/10916#issuecomment-3425715875 Run Gluten Clickhouse CI on x86 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-10915][VL] Fix dynamic offheap sizing feature by setting default offheap size [incubator-gluten]
github-actions[bot] commented on PR #10916: URL: https://github.com/apache/incubator-gluten/pull/10916#issuecomment-3425408302 Run Gluten Clickhouse CI on x86 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-10915][VL] Fix dynamic offheap sizing feature by setting default offheap size [incubator-gluten]
wangyum commented on code in PR #10916:
URL:
https://github.com/apache/incubator-gluten/pull/10916#discussion_r2446858280
##
backends-velox/src/test/scala/org/apache/gluten/execution/DynamicOffHeapSizingSuite.scala:
##
@@ -70,4 +70,38 @@ class DynamicOffHeapSizingSuite extends
VeloxWholeStageTransformerSuite {
runAndCompare(query)
}
}
+
+ test("Dynamic off-heap sizing with setting offheap") {
+withSQLConf(
+ GlutenCoreConfig.SPARK_OFFHEAP_ENABLED_KEY -> "false") {
+ if (DynamicOffHeapSizingMemoryTarget.isJava9OrLater()) {
Review Comment:
```
Error: Failed to execute goal
com.diffplug.spotless:spotless-maven-plugin:2.27.2:check (spotless-check) on
project backends-velox: The following files had format violations:
Error:
src/test/scala/org/apache/gluten/execution/DynamicOffHeapSizingSuite.scala
Error: @@ -72,8 +72,7 @@
Error: ??}
Error:
Error: ??test("Dynamic?off-heap?sizing?with?setting?offheap")?{
Error: -withSQLConf(
Error:
-??GlutenCoreConfig.SPARK_OFFHEAP_ENABLED_KEY?->?"false")?{
Error:
+withSQLConf(GlutenCoreConfig.SPARK_OFFHEAP_ENABLED_KEY?->?"false")?{
Error:
??if?(DynamicOffHeapSizingMemoryTarget.isJava9OrLater())?{
Error: val?query?=
Error: ??"""
Error: Run 'mvn spotless:apply' to fix these violations.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-10915][VL] Fix dynamic offheap sizing feature by setting default offheap size [incubator-gluten]
github-actions[bot] commented on PR #10916: URL: https://github.com/apache/incubator-gluten/pull/10916#issuecomment-3423908669 Run Gluten Clickhouse CI on x86 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-10915][VL] Fix dynamic offheap sizing feature by setting default offheap size [incubator-gluten]
github-actions[bot] commented on PR #10916: URL: https://github.com/apache/incubator-gluten/pull/10916#issuecomment-3423728490 Run Gluten Clickhouse CI on x86 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-10915][VL] Fix dynamic offheap sizing feature by setting default offheap size [incubator-gluten]
github-actions[bot] commented on PR #10916: URL: https://github.com/apache/incubator-gluten/pull/10916#issuecomment-3423595348 Run Gluten Clickhouse CI on x86 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-10915][VL] Fix dynamic offheap sizing feature by setting default offheap size [incubator-gluten]
github-actions[bot] commented on PR #10916: URL: https://github.com/apache/incubator-gluten/pull/10916#issuecomment-3423569084 Run Gluten Clickhouse CI on x86 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-10915][VL] Fix dynamic offheap sizing feature by setting default offheap size [incubator-gluten]
github-actions[bot] commented on PR #10916: URL: https://github.com/apache/incubator-gluten/pull/10916#issuecomment-3423552305 Run Gluten Clickhouse CI on x86 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-10915][VL] Fix dynamic offheap sizing feature by setting default offheap size [incubator-gluten]
github-actions[bot] commented on PR #10916: URL: https://github.com/apache/incubator-gluten/pull/10916#issuecomment-3423539128 Run Gluten Clickhouse CI on x86 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-10915][VL] Fix dynamic offheap sizing feature by setting default offheap size [incubator-gluten]
github-actions[bot] commented on PR #10916: URL: https://github.com/apache/incubator-gluten/pull/10916#issuecomment-3423259542 Run Gluten Clickhouse CI on x86 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-10915][VL] Fix dynamic offheap sizing feature by setting default offheap size [incubator-gluten]
github-actions[bot] commented on PR #10916: URL: https://github.com/apache/incubator-gluten/pull/10916#issuecomment-3422970478 Run Gluten Clickhouse CI on x86 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
