viirya commented on code in PR #832:
URL: https://github.com/apache/datafusion-comet/pull/832#discussion_r1718506539
##########
common/src/main/scala/org/apache/comet/CometConf.scala:
##########
@@ -84,15 +84,33 @@ object CometConf extends ShimCometConf {
.booleanConf
.createWithDefault(sys.env.getOrElse("ENABLE_COMET", "true").toBoolean)
- val COMET_SCAN_ENABLED: ConfigEntry[Boolean] =
conf("spark.comet.scan.enabled")
+ val COMET_NATIVE_SCAN_ENABLED: ConfigEntry[Boolean] =
conf("spark.comet.scan.enabled")
.doc(
- "Whether to enable Comet scan. When this is turned on, Spark will use
Comet to read " +
- "Parquet data source. Note that to enable native vectorized execution,
both this " +
- "config and 'spark.comet.exec.enabled' need to be enabled. By default,
this config " +
- "is true.")
+ "Whether to enable native scans. When this is turned on, Spark will use
Comet to " +
+ "read supported data sources (currently only Parquet is supported
natively). Note " +
+ "that to enable native vectorized execution, both this config and " +
+ "'spark.comet.exec.enabled' need to be enabled. By default, this
config is true.")
.booleanConf
.createWithDefault(true)
+ val COMET_CONVERT_FROM_PARQUET_ENABLED: ConfigEntry[Boolean] =
+ conf("spark.comet.convert.parquet.enabled")
+ .doc(
+ "When enabled, data from Parquet v1 and v2 scans will be converted to
Arrow format. Note " +
+ "that to enable native vectorized execution, both this config and " +
+ "'spark.comet.exec.enabled' need to be enabled.")
+ .booleanConf
+ .createWithDefault(false)
Review Comment:
This might be confusing to `COMET_NATIVE_SCAN_ENABLED`. We probably can
mention in the doc that this doesn't use Comet native scan but Spark parquet
reader and convert it to Comet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org
