andygrove commented on code in PR #4658:
URL: https://github.com/apache/datafusion-comet/pull/4658#discussion_r3716480660
##########
spark/src/main/scala/org/apache/comet/CometConf.scala:
##########
@@ -121,6 +121,16 @@ object CometConf extends ShimCometConf {
.booleanConf
.createWithDefault(true)
+ val COMET_ICEBERG_WRITE_SPLIT_OPERATOR_ENABLED: ConfigEntry[Boolean] =
+ conf("spark.comet.write.iceberg.splitOperator.enabled")
+ .category(CATEGORY_TESTING)
+ .doc(
+ "Whether to rewrite Iceberg V2 writes from Spark's combined V2
write/commit operator " +
+ "into Comet's two-operator shape: a file writer exec (inside AQE)
and a committer " +
+ "(outside AQE).")
+ .booleanConf
+ .createWithDefault(true)
Review Comment:
Note that this config is temporarily enabled by default to allow Iceberg
tests suites to run against the new feature. We'll need to set this to false
before merging.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
