Baymine opened a new pull request, #66134:
URL: https://github.com/apache/doris/pull/66134
### What problem does this PR solve?
Issue Number: close #66133
Problem Summary: When Hive table files are written out-of-band (e.g. by an
external job that only replaces data files without issuing an HMS DDL/event
notification), the HMS table's in-memory `updateTime` on the FE never
advances.
The FE SQL cache (`NereidsSqlCacheManager` / `SqlCacheContext`) keys cache
validity on `HMSExternalTable.getUpdateTime()`, so a stale `updateTime` lets
the
cache keep serving result sets computed from the old files, returning
incorrect
data to queries until the table's metadata is next refreshed.
Fix: in `HiveScanNode.getFileSplitByPartitions(...)`, while iterating the
file
status list of each partition, bump `hmsTable.updateTime` to the maximum file
`modificationTime` seen when that mtime is greater than the current
`updateTime`. This mirrors the semantics already used for the count-pushdown
path (which also reads file mtimes) and ensures the SQL cache key advances
the
first time a query observes a fresher file, so subsequent queries recompute.
This is strictly an improvement over the pre-patch state (where `updateTime`
never advanced from file scans at all). Concurrent `getFileSplitByPartitions`
calls are a racy read-modify-write on `updateTime`, but the worst case is
that
two racing scans both observe the same (or a slightly older) mtime and only
one
wins the `setUpdateTime` — the value still only moves forward, so this can
never
make the cache *more* stale than before; at most it delays invalidation by
one
scan.
### Release note
When Hive data files are replaced out-of-band (without an HMS DDL event), the
FE SQL cache now invalidates correctly because the table's `updateTime` is
bumped from the freshest file modification time observed during the scan.
### Check List (For Author)
- Test: Unit Test
- Added
`HiveScanNodeTest.testGetFileSplitBumpsUpdateTimeWhenFileMtimeNewer`
and `testGetFileSplitDoesNotBumpUpdateTimeWhenFileMtimeNotNewer`. Both
drive the real `getFileSplitByPartitions` (via reflection) against a
mocked `HMSExternalTable` and `HiveExternalMetaCache` returning a
crafted
`FileCacheValue`, and assert via Mockito verification that
`setUpdateTime` is called / not called. Full class: 10 tests, 0
failures,
0 errors.
- FE build (`./build.sh --fe`) green; checkstyle 0 violations.
- Behavior changed: Yes — `HMSExternalTable.updateTime` may now advance
during
a scan to the max file mtime; this only affects SQL cache invalidation
timing
for Hive external tables and cannot make the cache more stale.
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
