comphead commented on code in PR #7082:
URL: https://github.com/apache/datafusion-comet/pull/7082#discussion_r4338820996
##########
spark/src/main/scala/org/apache/comet/CometConf.scala:
##########
@@ -184,6 +184,19 @@ object CometConf extends ShimCometConf {
.booleanConf
.createWithDefault(false)
+ val COMET_PARQUET_SHARED_METADATA_CACHE_ENABLED: ConfigEntry[Boolean] =
+ conf("spark.comet.parquet.metadataCache.sharedWithinStage.enabled")
+ .category(CATEGORY_PARQUET)
+ .doc(
+ "When enabled, the tasks of one Spark stage that run on the same
executor share the " +
+ "native Parquet scan's cache of file footers and page indexes, so a
file split " +
+ "across many tasks has its footer fetched and parsed once per
executor instead of " +
Review Comment:
"once per executor" is the best case, since tasks that start together each
miss, as the new `CometExecSuite` test notes. This text becomes the
`configs.md` entry of a default-on setting, so could it also say what the cache
holds? Each stage keeps up to 50 MiB of decoded footers and page indexes
outside Spark's memory accounting, and up to two such caches outlive their
tasks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
