XiaoHongbo-Hope opened a new pull request, #1223: URL: https://github.com/apache/paimon-rust/pull/1223
## Summary Opt-in process-local sharing of parsed Parquet metadata across independently created catalogs/read contexts. Default remains unchanged. - Add `file-format.metadata-cache.shared` (default `false`). - Share a weakly registered Parquet cache slot for equal budgets within a process. - Keep ROW metadata, sidecar sizes and BLOB indexes context-local. - Avoid blocking on the registry and bypass metadata inherited from another PID. - Cover reuse across 40 readers, budget/lifetime isolation, ROW isolation and invalid options. This is not cross-process memory sharing or a disk block-cache change. ## Experimental evidence An earlier prototype, not this rebased head, was tested on a storage-only mixed JPEG workload: 64 spawned workers, 180-second window, same snapshot and wheel with sharing toggled, 100-GiB disk cache with 256-KiB blocks. | Warm run | frames/s | Peak anonymous GiB | | --- | ---: | ---: | | Independent metadata | 211.82 | 252.18 | | Shared metadata 1 | 220.14 | 199.53 | | Shared metadata 2 | 224.57 | 201.76 | The final independent repeat failed on catalog-endpoint DNS resolution, so this is not a complete two-pair acceptance result. Completed arms matched 1,024 common-prefix frame checks and had no OOM. Timed arms perform different work; the memory delta is not a direct measurement of duplicate footer bytes. ## Draft / remaining validation - Focused tests on this head are running; prototype tests passed previously. - Repeat end-to-end measurements on this head: main now reserves a separate ROW budget. - Review storage/provider namespace and authorization boundaries for broader reuse. - Add real fork/concurrent-construction coverage; simulated PID mismatch is not a fork test. - Repeat the missing baseline before claiming complete throughput acceptance. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
