ulysses-you commented on code in PR #5439:
URL: https://github.com/apache/incubator-gluten/pull/5439#discussion_r1596197989
##########
shims/common/src/main/scala/org/apache/gluten/GlutenConfig.scala:
##########
@@ -1821,4 +1832,42 @@ object GlutenConfig {
.internal()
.booleanConf
.createWithDefault(true)
+
+ val DYNAMIC_OFFHEAP_SIZING_ENABLED =
+ buildConf(GlutenConfig.GLUTEN_DYNAMIC_OFFHEAP_SIZING_ENABLED)
+ .internal()
+ .doc(
+ "Experimental: When set to true, the offheap config
(spark.memory.offHeap.size) will " +
+ "be ignored and instead we will consider onheap and offheap memory
in combination, " +
+ "both counting towards the executor memory config
(spark.executor.memory). We will " +
+ "make use of JVM APIs to determine how much onheap memory is use,
alongside tracking " +
+ "offheap allocations made by Gluten. We will then proceed to
enforcing a total memory " +
+ "quota, calculated by the sum of what memory is committed and in use
in the Java " +
+ "heap. Since the calculation of the total quota happens as offheap
allocation happens " +
+ "and not as JVM heap memory is allocated, it is possible that we can
oversubscribe " +
+ "memory. Additionally, note that this change is experimental and may
have performance " +
+ "implications.")
+ .booleanConf
+ .createWithDefault(false)
+
+ val DYNAMIC_OFFHEAP_SIZING_MEMORY_FRACTION =
+ buildConf(GlutenConfig.GLUTEN_DYNAMIC_OFFHEAP_SIZING_MEMORY_FRACTION)
+ .internal()
+ .doc(
+ "Experimental: Determines the memory fraction used to determine the
total " +
+ "memory available for offheap and onheap allocations when the
dynamic offheap " +
+ "sizing feature is enabled. The default is set to match
spark.executor.memoryFraction.")
+ .doubleConf
+ .checkValue(v => v >= 0 && v <= 1, "offheap sizing memory fraction must
between [0, 1]")
+ .createWithDefault(0.6)
+
+ val COLUMNAR_ONHEAP_SIZE_IN_BYTES =
+ buildConf(GlutenConfig.GLUTEN_ONHEAP_SIZE_IN_BYTES_KEY)
Review Comment:
do we still need this config since we have
`DYNAMIC_OFFHEAP_SIZING_MEMORY_FRACTION` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
