andygrove commented on code in PR #6943:
URL: https://github.com/apache/datafusion-comet/pull/6943#discussion_r4264460567
##########
docs/source/user-guide/latest/tuning/shuffle.md:
##########
@@ -115,6 +115,12 @@ both Comet's native shuffle and the JVM columnar shuffle
writer. Supported value
useful when shuffle I/O or network bandwidth is the bottleneck. When ZSTD is
selected, the level is
controlled by `spark.comet.shuffle.compression.zstd.level` (default `1`).
+With zstd, Comet also stores the offsets of string, binary, array and map
columns as lengths before
Review Comment:
Good catch. It now says this applies to native shuffle and to JVM shuffle
blocks without dictionary-encoded columns, and that JVM blocks where
`spark.comet.shuffle.jvm.preferDictionary.ratio` dictionary-encodes a column,
and Celeborn blocks, keep their offsets (1e0879139b).
##########
spark/src/main/scala/org/apache/comet/CometConf.scala:
##########
@@ -823,6 +823,17 @@ object CometConf extends ShimCometConf {
.intConf
.createWithDefault(1)
+ val COMET_SHUFFLE_COMPRESSION_OFFSETS_AS_LENGTHS: ConfigEntry[Boolean] =
+ conf("spark.comet.shuffle.compression.offsetsAsLengths.enabled")
+ .category(CATEGORY_SHUFFLE)
+ .doc(
+ "Whether Comet shuffle stores the offsets of string, binary, array and
map columns as " +
Review Comment:
Done in 1e0879139b. The description now says it does not apply to Celeborn
shuffle or to JVM shuffle blocks with dictionary-encoded columns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
