924060929 opened a new pull request, #67996:
URL: https://github.com/apache/doris/pull/67996
### What problem does this PR solve?
Issue Number: N/A
Problem Summary:
Paimon tables whose schema changes can be queried in Doris before the table
is
externally dropped and recreated with a different schema by Spark/Paimon
with the
same name. After `REFRESH TABLE`, Doris still serves data of the OLD table
(frozen
schema/snapshot) until the SDK cache TTL expires, or fails with missing old
schema
files after the latest snapshot advances.
Root cause: the Paimon SDK's `CatalogFactory.createCatalog` wraps every
catalog in a
`CachingCatalog` by default, which caches `Table` objects with schema and
snapshot
state frozen at load time, and only exposes a per-table
`invalidateTable(Identifier)`
API. Doris-side `REFRESH TABLE`/`REFRESH DATABASE`/`REFRESH CATALOG` only
flush the
Doris-owned caches (latest snapshot, schema-at-snapshot memo, partition
view); the SDK
`CachingCatalog` keeps returning the stale `Table`, and the SDK exposes no
db/catalog-level eviction for the connector to reach.
Fix: move the paimon Table cache from the Paimon SDK into Doris's own cache
framework.
- `PaimonCatalogFactory.buildCatalogOptions` now always emits
`cache-enabled=false`,
so the SDK `CachingCatalog` is never built (an explicit
`paimon.cache-enabled`
setting no longer re-enables it).
- New `PaimonTableCache` caches the loaded paimon `Table` in the connector
`CatalogMetaCache`, scoped per table, so every `REFRESH` line
(table/database/
catalog) invalidates it together with the other Doris-owned caches. It
still
attaches the SDK snapshot/stat caches and a shared per-catalog manifest
SegmentsCache, preserving the scan-time performance layer the
`CachingCatalog`
used to provide.
- `CatalogBackedPaimonCatalogOps.getTable` routes through the new cache; the
connector builds the `Table` fresh from the bare catalog on a miss.
### Release note
Fix stale reads after an external drop/recreate of a same-name Paimon table:
Doris-side `REFRESH TABLE` now fully invalidates the Paimon SDK table cache.
### Check List (For Author)
- Test: fe-connector-paimon unit tests (579, incl. new PaimonTableCacheTest
and
updated catalog-options snapshots) and fe-connector-cache unit tests (141)
pass
- Behavior changed: Yes (Paimon SDK CachingCatalog is disabled; Doris's own
registry-backed Table cache replaces it)
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
