cshuo commented on issue #20480: URL: https://github.com/apache/hudi/issues/20480#issuecomment-6516999374
> feel like the pruning is still valid if the end boundary of the query is the latest. Thanks for the suggestion. I think reading through the latest commit is necessary, but not sufficient on its own. We also need to consider the start boundary, whether `read.commits.limit` restricts the commits consumed in a streaming round, and how `skip_compaction`, `skip_clustering`, and `skip_insertoverwrite` affect the selected timeline and file slices relative to the current partition statistics. My preference is to conservatively disable partition stats pruning for incremental reads for now. Other pruning mechanisms, such as partition-predicate pruning, col stats prunning are still available. If users report a performance issue, we can revisit this and enable partition stats pruning for cases where it is safe. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
