andygrove opened a new issue, #6188: URL: https://github.com/apache/datafusion-comet/issues/6188
### Describe the bug The executor's native memory usage log (#6162) warns when untracked native memory plus Spark's off-heap usage exceeds `spark.memory.offHeap.size` plus the memory overhead. `CometExecIterator.executorMemoryOverhead` takes the overhead from `spark.executor.memoryOverhead`. If that is unset, it uses `spark.executor.memoryOverheadFactor` (default 0.1) of the executor memory, with a minimum of `spark.executor.minMemoryOverhead` ([CometExecIterator.scala#L465-L483](https://github.com/apache/datafusion-comet/blob/67803a7a422c44de07af1e5d25c1dbeae8df68d4/spark/src/main/scala/org/apache/comet/CometExecIterator.scala#L465-L483)). On Kubernetes, Spark does not size the executor pod this way when `spark.executor.memoryOverheadFactor` is unset. `BasicExecutorFeatureStep` falls back to `spark.kubernetes.memoryOverheadFactor` ([BasicExecutorFeatureStep.scala#L66-L70](https://github.com/apache/spark/blob/v4.1.1/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicExecutorFeatureStep.scala#L66-L70)). For PySpark and SparkR applications in cluster mode, `BasicDriverFeatureStep` sets that factor to 0.4 and passes it on to the executors ([BasicDriverFeatureStep.scala#L50-L63](https://github.com/apache/spark/blob/v4.1.1/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/BasicDriverFeatureStep.scala#L50-L63)). For those applications Comet's limit is too low, and the executor warns that the cluster manager may kill it while it is still inside its pod. A user-set `spark.kubernetes.memoryOverheadFactor` is ignored in the same way. The tuning guide documents the 0.4 default ([tuning.md#L163-L165](https://github.com/apache/datafusion-comet/blob/67803a7a422c44de07af1e5d25c1dbeae8df68d4/docs/source/user-guide/latest/tuning.md?plain=1#L163-L165)), yet also says the warning sizes the overhead "as Spark sizes the default container" ([tuning.md#L234-L237](https://github.com/apache/datafusion-comet/blob/67803a7a422c44de07af1e5d25c1dbeae8df68d4/docs/source/user-guide/latest/tuning.md?plain=1#L234-L237)). A standalone cluster has no container limit, but it gets the same warning text. Only local mode is excluded. ### Steps to reproduce Submit a PySpark application to Kubernetes in cluster mode with `spark.executor.memory=8g`, `spark.memory.offHeap.enabled=true`, `spark.memory.offHeap.size=4g` and no overhead settings. The pod gets 3276 MiB of overhead (0.4 x 8 GiB), but the log compares against 819 MiB (0.1 x 8 GiB). ### Expected behavior The overhead is resolved the same way the cluster manager resolves it. Standalone clusters either get no container warning or a differently worded one. ### Additional context Python applications also add `spark.executor.pyspark.memory` to the pod. The log leaves it out, which only makes the warning fire earlier. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
