Linhong Liu created SPARK-60652:
-----------------------------------
Summary: Refresh every matching file index in a cached plan when
refreshing a path
Key: SPARK-60652
URL: https://issues.apache.org/jira/browse/SPARK-60652
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Linhong Liu
refreshByPath uses a short-circuiting plan.exists traversal while refreshing
file indexes as a side effect. A cached union or join reading multiple
locations under one refreshed parent can therefore refresh only the first
matching file index and rebuild from stale listings in the others. Visit all
matching reads before rebuilding the cache. Deduplicate by file-index object
identity within each cached entry so shared indexes refresh once while distinct
equal indexes are refreshed independently. Preserve unrelated-path and
time-travel exclusions. Regression coverage should add files outside Spark
writes, then verify complete refreshed results, shared and distinct index
refresh counts, unrelated paths, and multiple cached entries.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
