sundapeng opened a new pull request, #1014: URL: https://github.com/apache/paimon-rust/pull/1014
### Purpose A REST catalog can put OSS cache endpoints next to the storage endpoint in a table token, for example an OSS accelerator for metadata and a cache cluster for data files. Today paimon-rust can only use `fs.oss.endpoint`, so every request, including writes, listings and files that are rewritten in place, goes to whatever that endpoint is. This lets FileIO choose the endpoint per request, with the same rules as the Java and Python clients: reads and file status of immutable Paimon files may use a cache target, everything else stays on the storage endpoint. ### Brief change log - New options, read from the merged catalog and token options: | Option | Default | Meaning | |---|---|---| | `io-cache.enabled` | `false` | Switch; a value in the catalog options wins over the token | | `io-cache.origin.endpoint` | `fs.oss.endpoint` | Endpoint for requests that do not use a target | | `io-cache.policy` | none | Operations that may use a target: `read`, `meta` (file status), or `none` | | `io-cache.whitelist` | all types | File types that may use a target: `meta`, `data`, `bucket-index`, `global-index`, `file-index`, or `*` | | `io-cache.targets` | none | Target names, `[a-z][a-z0-9-]*` | | `io-cache.target.<name>.endpoint` | none | Target URL; `https` when the scheme is omitted | | `io-cache.target.<name>.path-style-access` | `false` | Path-style URLs for the target | | `io-cache.endpoint` | none | A single target named `default`, ignored when `io-cache.targets` is set | | `io-cache.routes` | first target with an endpoint | `types=target;...`; the first rule listing the file type picks the target, no match means origin | - Only reads and file status of files that Paimon writes once under a unique name can use a target: manifests, statistics, indexes, and data files with the `data-` / `changelog-` prefixes or the table's `data-file.prefix` / `changelog-file.prefix`. Existence checks, writes, listing, deletes, renames, copies, files rewritten in place (`LATEST`, tags, temp files), sequentially numbered metadata (`snapshot-N`, `schema-N`, `changelog/changelog-N`), unknown names and files outside the table's bucket always use origin. - A target is used like any other endpoint: the same retry settings apply and its errors are returned, not retried on origin, so each cache endpoint is expected to be highly available. - One OpenDAL operator per (bucket, endpoint); the local block cache keys stay the same whichever endpoint served the bytes. - `fs.oss.path-style-access` enables path-style URLs for OSS, and `dlf.oss-endpoint` on the client turns routing off. Related: apache/paimon#10350 (Java), apache/paimon#10351 (Python) implement the same rules and share the routing test vectors. ### Tests - `io::cache_routing`: 65 routing cases and 35 file-name cases from shared JSON vectors (`testdata/io_cache`), plus option parsing, multi-target routing, `io-cache.enabled` precedence and the `dlf.oss-endpoint` override. - `io::file_io` routing tests against local OSS stand-ins: only reads and status reach a target, routes split file types across two targets, other buckets, copies, writes and listings stay on origin, target errors are not retried on origin, a target "not found" is final, local cache keys are shared. - REST token FileIO: table files routed from a vended token, catalog `io-cache.enabled=false` wins, `dlf.oss-endpoint` disables routing; snapshot commits and discovery never use a target. - `cargo test -p paimon --lib`: 3577 passed. `cargo fmt --all -- --check` and `cargo clippy --all-targets --workspace --features fulltext,vortex -- -D warnings`. - Local end-to-end against two or three mock OSS endpoints, with real writes, reads, time travel, partitions and expiration, every request checked against the same rules: single target, two targets and routing off pass, with the same split of requests as the Java and Python clients. ### API and Format No API or format change; new FileIO options only. ### Documentation `docs/src/getting-started.md`: OSS cache routing options and `fs.oss.path-style-access`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
