Gabriel39 opened a new pull request, #70216: URL: https://github.com/apache/doris/pull/70216
### What problem does this PR solve? Problem Summary: Fluss scans repeatedly export session variables, fetch table metadata and partition lists, and rebuild tail-key suppression state. The shared Arrow importer also performs avoidable per-value work for null maps, contiguous strings/binary values and UTC timestamps. **Original author: @fresh-borzoni (Anton Borisov).** This PR ports the following four commits from [fresh-borzoni/doris, `fluss-native-reader`](https://github.com/fresh-borzoni/doris/tree/fluss-native-reader), preserving their authorship and order: | Source commit | Optimization | | --- | --- | | [654852f941cd](https://github.com/fresh-borzoni/doris/commit/654852f941cd860e12c974c5305ec0b92b7d4ea0) | Memoize connector session properties per execution; cache variable visibility; obtain path partition keys through a dedicated SPI method. Reset execution state when reusing a prepared statement. | | [41863f6bcaf9](https://github.com/fresh-borzoni/doris/commit/41863f6bcaf9bbac75d873661451eb74f827db8e) | Append Arrow null maps by batch, copy eligible contiguous string/binary buffers in bulk, and use civil-calendar arithmetic for UTC timestamp conversion. | | [5b32769278ad](https://github.com/fresh-borzoni/doris/commit/5b32769278ad0939bf6d041790e01f1621c65a16) | Cache table information, pruning partition lists and snapshot-specific lake properties across statements; reuse immutable table information when opening JNI scanners. | | [431296821ef0](https://github.com/fresh-borzoni/doris/commit/431296821ef068eaf0325228d794fcd3aec5bb3a) | Reuse versioned tail keys across scans; add integer hash/sorted-walk and composite packed-key suppression paths to reduce generic predicate overhead. | Adaptations for current master: - Keep the existing JNI Fluss reader. The intervening native-reader implementation on the source branch is outside these four commits. - Preserve master's timestamp range/date validation and existing equality-delete hash index. Unsupported key shapes keep the generic predicate path. - Bump the connector SPI API to 30 and update its recorded public surface. - Bound metadata caches, coalesce concurrent misses and discard older in-flight loads after FE `REFRESH`. Isolate cached lake properties by table/schema identity, snapshot, projection and session context. - Send planned table/schema identities to JNI scanners and reload mismatched metadata. Use a memory-tracked BE LRU for versioned suppression tails (64 MiB, at most 256 retained entries); older FE requests retain scan-local caching. Cache keys include client configuration, table/schema identity, key types, timezone and tail bounds. Leases keep key storage alive while readers use it. ### Local validation and measurements | Measurement | Baseline | With optimization | Scope | | --- | --- | --- | --- | | UTC calendar conversion, 8,388,608 values | 804.017 ms | 163.772 ms (**4.91x**) | Extracted conversion kernel with the actual DateTimeV2 representation; single-threaded x86-64, Clang 20 `-O3`, reused UTC timezone, two warmups and seven alternating trials, reported medians. | | 100 table-info lookups against an embedded Fluss cluster | 100 RPCs, 539–549 ms | 1 RPC, 4.8–9.6 ms | Two local JDK 17 runs, one coordinator/tablet server, ten warmup requests and a 600-second TTL. Includes the first cache miss; invalidation forces the next RPC. | These measurements cover conversion and metadata lookup overhead. They do not establish end-to-end query speedups. The Arrow result applies to the shared Arrow importer; this PR retains master's JNI Fluss reader. Validation completed: - 187 Java tests passed: 30 FE, 6 SPI surface, 123 Fluss connector and 28 JNI scanner/cache tests. Connector and scanner tests include embedded Fluss clusters, cache expiry/disable/invalidation, refresh races and schema/identity changes. - The affected C++ translation units compiled. An additional 4,000 randomized sorted-key cases passed under ASAN/UBSAN against an independent set-membership oracle. UTC conversion matched 1,000,007 randomized and boundary inputs. - FE Checkstyle, clang-format 16, header hygiene and changed-line clang-tidy checks passed. - Added `test_fluss_scan_caches.groovy`: repeated cached/live metadata reads, union versus Fluss-only results, composite/partitioned/timestamp fixtures, three timezones, `REFRESH` and a self-join. The suite compiled; external SQL execution requires the Fluss lake regression environment and was not run locally. - 70 BE tests passed under ASAN (33 Arrow importer tests and 37 Fluss union-reader tests), using a focused executable built from this checkout's production libraries and repository test main/support. The installed Lance/Paimon Rust archives both define `rust_eh_personality`, so this local executable needed a linker duplicate-symbol workaround. The normal full BE link remains for CI. ### Release note Reduce connector scan setup, shared Arrow import and Fluss union-read overhead. Fluss catalog properties `fluss.metadata.cache.ttl.sec` and `fluss.partition.cache.ttl.sec` default to 10 and 5 seconds respectively; non-positive values disable the corresponding cache. FE `REFRESH` invalidates affected metadata caches. Connector plugins must be rebuilt for SPI API version 30. ### Check List (For Author) - Test - [x] Regression test (added and compiled; external execution pending CI) - [x] Unit Test - [x] Manual test (methodology and scope above) - Behavior changed: - [x] Yes. Short metadata TTLs are enabled by default, and connector SPI API version is 30. - Does this need documentation? - [x] Yes. The new TTL defaults, opt-out and refresh behavior are described above and in the cache source contracts; user-facing catalog documentation should include these properties. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
