andygrove opened a new pull request, #6197: URL: https://github.com/apache/datafusion-comet/pull/6197
## Which issue does this PR close? No issue was filed. This is a small documentation correction found while checking what `CometDriverPlugin` still does after #6054. ## Rationale for this change The "Comet SQL Plugin" section of the contributor guide's plugin overview makes two claims that are no longer true: - "The plugin is loaded on the Spark driver and does not provide executor-side plugins." `CometPlugin` has provided `CometExecutorPlugin` since #4734, which uses it to shut down the native tokio runtime when an executor stops. - "The plugin will update the current `SparkConf` with the extra configuration provided by Comet, such as executor memory configuration." #6054 removed the `spark.executor.memoryOverhead` adjustment, and the plugin no longer changes any executor memory setting. ## What changes are included in this PR? The section now describes what the plugin does today: - `CometDriverPlugin` runs before any `SparkSession` exists, which is what lets it set static configuration. - It sets `spark.comet.version`, then stops with a warning unless `spark.memory.offHeap.enabled` or `spark.comet.exec.onHeap.enabled` is `true`. - Otherwise it registers the session extension, installs Comet's cache serializer when the in-memory cache is enabled, registers Comet's metrics when `spark.comet.metrics.enabled=true`, and logs warnings for problem settings such as an unset `spark.executor.memoryOverhead`. - It does not change executor memory settings, and the section links to the tuning guide for sizing them. - Both components shut down the native tokio runtime when the driver or an executor stops. - The session extension can be registered without the plugin, which is how most of Comet's test suites and the Spark SQL tests enable Comet, so the driver plugin's steps do not run for them. ## How are these changes tested? This is a documentation-only change. Each statement was checked against `Plugins.scala`. `prettier --check` passes on the page, and a Sphinx build of `docs/source` produces no warnings from it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
