wenwj0 commented on issue #7605: URL: https://github.com/apache/incubator-gluten/issues/7605#issuecomment-2426101699
> Why are you using the option spark.gluten.memory.dynamic.offHeap.sizing.enabled? @wenwj0 In our scenario, we have various configurations for different executor memory. I try to use this property because it seems that I can use the existing configurations without the need to set extra off-heap memory. > I believe your case is because on-heap occupies the remaining 2.2 GiB - 388 MiB, which could be considered by-design of the feature. But it would be great if you can share more background information about your use case. From my impression only a few users are using the option. I agree with you. MAX_MEMORY_IN_BYTES refers to off-heap memory, hence (size + usedOffHeapBytesNow + usedOnHeapBytes > MAX_MEMORY_IN_BYTES) => (8MiB + 388MiB + 1927MiB = 2323MiB > MAX_MEMORY_IN_BYTES(2GiB) If the MAX_MEMORY_IN_BYTES is onheap memory or onheap+offheap memory, maybe it works. @zhztheplayer -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
