andygrove commented on issue #5116: URL: https://github.com/apache/datafusion-comet/issues/5116#issuecomment-5850482098
One correction to the premise: stage reversion from #4519 is off by default. `spark.comet.exec.transitionRevert.enabled` defaults to false. When it's on, it only reverts a stage with more than `spark.comet.exec.transitionRevert.maxTransitions` columnar-to-row transitions, which defaults to 2. So stages come out all-native or all-JVM only with the revert on and `maxTransitions=0`. Otherwise a stage can still mix the two. A native profile would also need `spark.executor.memoryOverhead` headroom, not just off-heap memory, because the native memory that no pool tracks comes out of the overhead. The "Configuring Executor Memory Overhead" section of the memory tuning guide covers this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
