jaceklaskowski commented on code in PR #1007:
URL: https://github.com/apache/datafusion-comet/pull/1007#discussion_r1807929649
##########
docs/source/user-guide/configs.md:
##########
@@ -50,6 +50,7 @@ Comet provides the following configuration settings.
| spark.comet.exec.localLimit.enabled | Whether to enable localLimit by
default. | true |
| spark.comet.exec.memoryFraction | The fraction of memory from Comet memory
overhead that the native memory manager can use for execution. The purpose of
this config is to set aside memory for untracked data structures, as well as
imprecise size estimation during memory acquisition. Default value is 0.7. |
0.7 |
| spark.comet.exec.project.enabled | Whether to enable project by default. |
true |
+| spark.comet.exec.replaceSortMergeJoin | Experimental feature to force Spark
to replace SortMergeJoin with ShuffledHashJoin for improved performance. See
tuning guide for more information regarding stability of this feature. | false |
Review Comment:
Can we add a link to the tuning guide? 🙏
##########
common/src/main/scala/org/apache/comet/CometConf.scala:
##########
@@ -263,6 +263,14 @@ object CometConf extends ShimCometConf {
.booleanConf
.createWithDefault(false)
+ val COMET_REPLACE_SMJ: ConfigEntry[Boolean] =
+ conf(s"$COMET_EXEC_CONFIG_PREFIX.replaceSortMergeJoin")
+ .doc("Experimental feature to force Spark to replace SortMergeJoin with
ShuffledHashJoin " +
+ "for improved performance. See tuning guide for more information
regarding stability of " +
Review Comment:
Can we add a link to the tuning guide?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
