viirya commented on code in PR #6943:
URL: https://github.com/apache/datafusion-comet/pull/6943#discussion_r4267186198
##########
docs/source/user-guide/latest/tuning/shuffle.md:
##########
@@ -115,6 +115,12 @@ both Comet's native shuffle and the JVM columnar shuffle
writer. Supported value
useful when shuffle I/O or network bandwidth is the bottleneck. When ZSTD is
selected, the level is
controlled by `spark.comet.shuffle.compression.zstd.level` (default `1`).
+With zstd, Comet also stores the offsets of string, binary, array and map
columns as lengths before
Review Comment:
Thanks, this reads well now.
##########
spark/src/main/scala/org/apache/comet/CometConf.scala:
##########
@@ -823,6 +823,17 @@ object CometConf extends ShimCometConf {
.intConf
.createWithDefault(1)
+ val COMET_SHUFFLE_COMPRESSION_OFFSETS_AS_LENGTHS: ConfigEntry[Boolean] =
+ conf("spark.comet.shuffle.compression.offsetsAsLengths.enabled")
+ .category(CATEGORY_SHUFFLE)
+ .doc(
+ "Whether Comet shuffle stores the offsets of string, binary, array and
map columns as " +
Review Comment:
Thanks, looks good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
