comphead commented on code in PR #6698:
URL: https://github.com/apache/datafusion-comet/pull/6698#discussion_r4197472493
##########
docs/source/user-guide/latest/tuning/memory.md:
##########
@@ -282,3 +282,20 @@ use more than `N` times this value on shared local disks.
If the limit is reache
fail and the query errors out. Raise this on workloads with large
sort/aggregate/shuffle spills, or
lower it to protect executors on shared disks, remembering that the total
across an executor is a
multiple of this value.
+
+## Compressing Spill Files
+
+Native sorts, aggregations, and sort-merge joins spill through DataFusion,
which writes spill files
Review Comment:
Added. When DataFusion 55.1's `NestedLoopJoinExec` falls back to spilling,
it writes both sides through `SpillManager` with the session's
`spill_compression`, so the new codec covers it.
##########
spark/src/main/scala/org/apache/comet/CometConf.scala:
##########
@@ -1259,10 +1259,23 @@ object CometConf extends ShimCometConf {
.doc(
"Development and testing configuration option to allow DataFusion
configs set in " +
"Spark configuration settings starting with
`spark.comet.datafusion.` to be passed " +
- "into native execution.")
+ "into native execution. Configs listed in " +
+ "`spark.comet.exec.allowedDataFusionConfigs` are passed even when
this is false.")
.booleanConf
.createWithDefault(false)
+ val COMET_ALLOWED_DATAFUSION_CONFIGS: ConfigEntry[Seq[String]] =
Review Comment:
Done in 16fb61d17c, as andygrove also suggested.
`spark.comet.exec.spill.compression.codec` is translated in
`prepare_datafusion_session_context` the same way
`COMET_PARQUET_ROW_FILTER_PUSHDOWN_ENABLED` is, and its resolved value is in
the list in `serializeCometSQLConfs`. I used `lz4`, `zstd` and `none` rather
than DataFusion's value names, so the accepted values don't depend on
DataFusion's `SpillCompression`, and `checkValues` rejects a bad value with a
clear message instead of a panic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
