andygrove commented on code in PR #4791:
URL: https://github.com/apache/datafusion-comet/pull/4791#discussion_r3532489072
##########
spark/src/main/scala/org/apache/comet/CometConf.scala:
##########
@@ -368,6 +368,26 @@ object CometConf extends ShimCometConf {
.booleanConf
.createWithDefault(false)
+ val COMET_AGG_USE_LARGE_DATATYPES: ConfigEntry[Boolean] =
+ conf(s"$COMET_EXEC_CONFIG_PREFIX.aggregation.useLargeDataTypes")
+ .category(CATEGORY_EXEC)
+ .doc(
+ "When true, Comet wraps Utf8/Binary group-by expressions inside a Cast
to " +
+ "LargeUtf8/LargeBinary immediately before each native HashAggregate,
and wraps the " +
+ "aggregate's output in a Projection that casts those columns back to
Utf8/Binary. " +
+ "This promotes DataFusion's per-task group-key byte buffer from
`i32` offsets " +
+ "(2 GiB hard cap) to `i64` offsets, removing the `offset overflow,
buffer size > " +
+ "2147483647` failure that can hit CUBE / GROUPING SETS /
COUNT(DISTINCT) workloads " +
+ "where a single partition accumulates more than 2 GiB of distinct
string keys. " +
+ "Scan, shuffle, and the JVM FFI boundary remain Utf8/Binary, so
neither downstream " +
+ "Spark nor Comet's shuffle (which does not support LargeUtf8) is
affected. The cast " +
+ "is an offset-width promotion and only rebuilds the offset buffer
(the value bytes " +
+ "are shared), so per-batch overhead is O(rows) not O(bytes).
Defaults to false " +
+ "because the cap is only reachable for very large per-partition
group cardinalities; " +
+ "enable it when you see the offset-overflow error.")
+ .booleanConf
+ .createWithDefault(true)
Review Comment:
The PR description says that this is an experimental feature and is disabled
by default. Is it intentional that it is enabled by default here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
