JunRuiLee commented on PR #686:
URL: https://github.com/apache/paimon-rust/pull/686#issuecomment-5212393100
> How about object store?
Good point. The initial version removed the 512 KiB prefetch
unconditionally, which could add serialized HTTP round trips for object stores.
I have revised the change so the policy is selected by a storage
capability:
- HDFS, local filesystem, memory, and custom filesystem readers use exact
footer, metadata, and page-index range reads.
- OSS, S3, COS, AzDLS, OBS, and GCS keep the existing 512 KiB metadata
prefetch.
The capability is determined in the storage/IO layer and propagated
through `FileRead`, rather than inferred from the URI scheme. I also added
tests covering both the exact-read path and the retained object-store prefetch
path.
We have only benchmarked HDFS so far, so preserving the existing
object-store behavior is intentional until we have object-store measurements.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
