Gabriel39 opened a new pull request, #68305:
URL: https://github.com/apache/doris/pull/68305
### What problem does this PR solve?
Repeated Lance metadata reads call `describeTable` before opening the
Dataset, even when the catalog already shares a Lance metadata Session. This
repeats filesystem discovery or REST requests and serializes callers on the
namespace lock during query planning.
Cache immutable table URIs and normalized access options for both filesystem
and REST catalogs. Cache hits bypass the namespace lock, and concurrent misses
for the same table share one load. Dataset opens and snapshot selection still
run for every read.
- Add `lance.table_access_cache_ttl_seconds` (default `60`; `0` disables
caching), with at most 10,000 entries per catalog client generation. Reads do
not extend the TTL.
- Bound entries containing vended storage options by `expires_at_millis`,
reserving 30 seconds before credential expiry. Missing, malformed, expired or
nearly expired credential deadlines disable caching for that response.
- Table, database and catalog metadata invalidation retire the access cache,
including in-flight loads. Table/database refresh conservatively clears all
access entries because Doris refresh names may be mapped names. It does not
rotate the native Session.
- Keep index inspection/admission and current index-job locator validation
on an uncached path so they verify the current target.
### Release note
Reduce repeated Lance query-planning work by caching table access resolution
for filesystem and REST catalogs, with credential-aware expiry and explicit
refresh invalidation.
### Validation
- Added regression tests that first failed on the original implementation:
two reads caused two `describeTable` calls for both filesystem and REST
catalogs.
- 60 FE tests passed (0 failures/errors/skips): access-cache expiry,
credential handling, concurrent loading/invalidation, catalog lifecycle,
property validation, filesystem/REST catalogs, and metadata-cache routing.
- FE Checkstyle passed with zero violations.
- No end-to-end latency benchmark was run.
### Check List (For Author)
- Test
- [x] Unit Test
- Behavior changed:
- [x] Yes. Table URI/access-option changes can remain cached until TTL
expiry or explicit refresh; Dataset versions are not cached here. Set
`lance.table_access_cache_ttl_seconds=0` to retain per-read resolution.
- Does this need documentation?
- [x] Yes. The Lance Catalog property reference should include the
optional TTL property and its refresh/credential semantics described above.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
