yihua commented on code in PR #19912:
URL: https://github.com/apache/hudi/pull/19912#discussion_r4077715186
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieIncrementalFileIndex.scala:
##########
@@ -71,6 +71,20 @@ class HoodieIncrementalFileIndex(override val spark:
SparkSession,
}).map(fileStatus => fileStatus.getPath.toString).toArray
}
+ private val reportMaxFileSize: Boolean = options.getOrElse(
+ DataSourceReadOptions.INCREMENTAL_REPORT_MAX_FILE_SIZE.key,
+
DataSourceReadOptions.INCREMENTAL_REPORT_MAX_FILE_SIZE.defaultValue.toString
+ ).toBoolean
+
+ override def sizeInBytes: Long = {
Review Comment:
Was the `sizeInBytes == 0` observed with #19898 applied? On current master
the prune rule still runs `filterFileSlices` against this index and fills
`cachedAllInputFileSlices` with the whole table, so the planner sees the full
table size rather than 0; it drops to 0 once #19898 skips the rule. If so it
would help to say in the description that this is the companion to #19898 and
land them together.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
