Alexey Kudinkin created HUDI-5384:
-------------------------------------
Summary: Make sure predicates are appropriately pushed down to
HoodieFileIndex when lazy listing
Key: HUDI-5384
URL: https://issues.apache.org/jira/browse/HUDI-5384
Project: Apache Hudi
Issue Type: Bug
Components: spark, spark-sql
Reporter: Alexey Kudinkin
Assignee: Alexey Kudinkin
Fix For: 0.13.0
After introduction of lazy-listing capability in HUDI-4812, it exposed an issue
in Spark's design, where predicates are pushed-down into generic FileIndex
implementations only during the execution phase.
This poses following issues:
# HoodieFileIndex isn't listing the table until `listFiles` method is invoked
# Listing would actually be performed only during actual execution in
`FileSourceScanExac` node
# Since listing isn't performed until the actual execution, table statistics
are initialized w/ bogus values (of 1 byte) and Cost-based Optimizations (CBO)
will be taking incorrect decisions based on that
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
