andygrove opened a new issue, #6189: URL: https://github.com/apache/datafusion-comet/issues/6189
### Describe the bug `CometDriverPlugin.warnIfExecutorMemoryOverheadUnset` warns whenever `spark.executor.memoryOverhead` is not set ([Plugins.scala#L165-L180](https://github.com/apache/datafusion-comet/blob/67803a7a422c44de07af1e5d25c1dbeae8df68d4/spark/src/main/scala/org/apache/spark/Plugins.scala#L165-L180)). For large executors, the tuning guide recommends raising `spark.executor.memoryOverheadFactor` instead ([tuning.md#L177-L184](https://github.com/apache/datafusion-comet/blob/67803a7a422c44de07af1e5d25c1dbeae8df68d4/docs/source/user-guide/latest/tuning.md?plain=1#L177-L184)). Users who follow that advice are still told to set `spark.executor.memoryOverhead`. The warning also fires in local mode, where there is no container to size. The native memory usage log already skips local mode. ### Steps to reproduce Start an application with the Comet plugin, off-heap memory enabled, `spark.executor.memoryOverheadFactor=0.2` and no `spark.executor.memoryOverhead`. The driver logs "spark.executor.memoryOverhead is not set". ### Expected behavior No warning when the overhead is configured through `spark.executor.memoryOverheadFactor` or `spark.kubernetes.memoryOverheadFactor`, or when the master is local. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
