sundapeng opened a new pull request, #10350: URL: https://github.com/apache/paimon/pull/10350
### Purpose A REST catalog can vend read caches next to the storage endpoint with the table token, for example an OSS accelerator for metadata and a cache cluster in front of the bucket for data files. Today the only way to use one is to replace `fs.oss.endpoint`, so every request of a table goes through the cache, including writes, listings, deletes and files rewritten in place, and a table can use one cache only. This lets `JindoFileIO` pick the endpoint of each request. The decision lives in `IoCacheRouting` (paimon-common), so other FileIOs can share it. With `io-cache.enabled=true` and at least one target: - Only reads and status lookups, as allowed by `io-cache.policy` (`read`, `meta`), go to a target, and only for files Paimon names with a UUID and never rewrites: manifests, manifest lists, index manifests, manifest sidecars and statistics; data and changelog files (`data-` / `changelog-`, or the table's `data-file.prefix` / `changelog-file.prefix`); bucket, global and file indexes. - Snapshots, schemas, `changelog/changelog-N`, hint files, tags, consumers, temp files, unknown names (for example migrated Hive files) and paths outside the table's bucket stay at origin, as do `exists`, writes, listings, deletes, renames and copies. - A target is used like origin: its errors are thrown as they are and the SDK retries are unchanged, so each cache endpoint is expected to be highly available. - Without `io-cache.enabled`, with `dlf.oss-endpoint` set, or in the JindoCache mode (namespace RPC address), nothing changes. | Option | Default | Meaning | |---|---|---| | `io-cache.enabled` | `false` | Switch; an explicit value in the catalog options wins over the token | | `io-cache.origin.endpoint` | `fs.oss.endpoint` | Endpoint for every request that does not use a target | | `io-cache.policy` | none | Operations that may use a target: `read`, `meta` (status and size) | | `io-cache.whitelist` | `*` | File types that may use a target: `meta`, `data`, `bucket-index`, `global-index`, `file-index`, or `*` | | `io-cache.targets` | none | Target names, `[a-z][a-z0-9-]*` | | `io-cache.target.<name>.endpoint` | none | Target URL, `https` when the scheme is omitted | | `io-cache.target.<name>.path-style-access` | `false` | Path-style addressing, for an IP or a host without wildcard DNS | | `io-cache.target.<name>.region` | origin's | Signing region | | `io-cache.endpoint` | none | A single target named `default`; ignored when `io-cache.targets` is set | | `io-cache.routes` | first target with an endpoint takes all types | `types=target;...`, e.g. `meta=accel;data,global-index=cluster`; the first rule listing the type wins, no match means origin | Other changes: - `RESTTokenFileIO` passes the table location (`io-cache.table-location`) so that paths in other buckets stay at origin. - `JindoFileIO` keeps one file system per target and bucket; origin drops the `fs.oss.dlf-cache.*` JindoSDK options so it never talks to a cache. - `FileType.unwrapTempFileName` is public for the sequential-metadata check. Related: PY_PR (Python) and apache/paimon-rust#RUST_PR (Rust) implement the same rules. The three share the routing test vectors in `paimon-common/src/test/resources/io-cache`. ### Tests - `IoCacheRoutingTest`: 65 routing cases and 35 file-name cases from the shared vectors, plus option parsing, target settings and the bucket check; 109 passed. - `JindoIoCacheRoutingTest`: the option set of each target (endpoint, https, path-style, region), requests on their target or origin, JindoCache precedence, and target errors thrown without a retry on origin; 9 passed. `RESTTokenFileIOTest` (11), `JindoFileIOTest` (5), `FileTypeTest` (18) and `TestJindoUserAgent` (5) pass. - Local end-to-end against two or three mock OSS endpoints, with real Paimon writes, reads, time travel, partitions and expiration, every request checked against the same rules: single target, two targets and routing off pass for Java, Python and Rust, with the same split of requests between origin and targets. ### API and Format No. ### Documentation No. The options are vended by the REST catalog with the table token. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
