sundapeng opened a new pull request, #10351: URL: https://github.com/apache/paimon/pull/10351
### Purpose Some deployments put read caches next to the OSS endpoint and vend them with the table token, for example an OSS accelerator or a cache cluster in front of the bucket. Today the only way to use one is to replace `fs.oss.endpoint`, which sends every request through the cache: writes, listings, deletes and files that are rewritten in place. This lets pypaimon choose the endpoint per request, with the same rules as the Java client in #10350. When the token (or the catalog options) carries `io-cache.enabled=true`, `RESTTokenFileIO` builds one FileIO for the origin endpoint and one per target, and routes each call: - Only reads and metadata lookups go to a target, as allowed by `io-cache.policy` (`read`, `meta`), and only for files Paimon names with a UUID and never rewrites: manifests, manifest lists, index manifests, statistics and manifest sidecars; data and changelog files (also `data-file.prefix` / `changelog-file.prefix`); bucket, global and file indexes. Snapshots, schemas, hint files, tags, consumers, temp files and unknown names stay at the origin, as do `exists`, writes, listings, deletes, renames, copies and presigned URLs. Paths outside the table's bucket stay at the origin. - Targets: `io-cache.endpoint` is a single target; `io-cache.targets` with `io-cache.target.<name>.endpoint` / `path-style-access` / `region` declares several, and `io-cache.routes` (`types=target;...`, first match wins, e.g. `meta=accel;data,global-index=cluster`) sends each file type to one of them. Without routes the first target with an endpoint takes every type. `io-cache.whitelist` filters file types before the routes. - `io-cache.origin.endpoint` is the origin (default `fs.oss.endpoint`). `dlf.oss-endpoint` still overrides everything, and an explicit `io-cache.enabled` in the catalog options wins over the token, as in Java. - Endpoints are URLs. pyarrow gets the scheme separately and supports path-style addressing (`fs.oss.second.level.domain.enable`); pyjindo gets `fs.oss.https.enable`. - A target is used like the origin: its errors surface as they are, with the usual SDK retries, so each cache endpoint is expected to be highly available. Readers keep working through `FileIO.filesystem`, which routes each pyarrow call the same way. Without `io-cache.enabled` nothing changes. Related: #10350 (Java), apache/paimon-rust#RUST_PR (Rust). The three clients share the routing test vectors. ### Tests - `io_cache_routing_test.py`: 65 shared routing cases, 35 file-name cases and option parsing; 117 passed. - `io_cache_file_io_test.py`: every routing case through each FileIO and pyarrow filesystem call, readers and the local block cache on top, target errors, target creation; 205 passed. - `rest_token_file_io_test.py`: origin and target FileIO options, `io-cache.enabled` precedence, `dlf.oss-endpoint`; `file_io_test.py`: URL endpoints and path-style addressing for pyarrow. - Related suites (file type, file IO, OSS, Jindo, PVFS, REST user agent, caching, REST token): 567 passed, 33 skipped. flake8 is clean. - Local end-to-end against two or three mock OSS endpoints, with real writes, reads, time travel, partitions and expiration, every request checked against the same rules: single target, two targets and routing off pass, with the same split of requests as the Java and Rust clients. ### API and Format No. ### Documentation No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
