lucasfang opened a new issue, #331: URL: https://github.com/apache/paimon-cpp/issues/331
## Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon-cpp/issues) and found nothing similar. ## Motivation The read path opens a data file with a length it already knows: the manifest carries the file size, and both `DataFileReaderFactory::Open` and `PrefetchFileBatchReaderImpl` pass a `FileStatus(path, size)` into `FileSystem::Open`. But `JindoFileSystem` only implements `Open(const std::string& path)`, and the base `FileSystem::Open(const FileStatus&)` default just forwards to it, so the known length is dropped. On the Jindo/OSS side, opening a reader without a length makes the store issue its own `getFileStatus` request to resolve the size before it can serve reads. That is one extra full-latency RPC to the object store on the critical path of every file open — per data file, per reader — even though the caller already had the answer. For scans over many small files this open-time round-trip is a measurable and entirely avoidable part of read latency. The arrow-based `ObjectStoreFileSystem` already avoids this: its `Open(const FileStatus&)` validates the length and hands it straight to the input stream, skipping the `HeadObject` it would otherwise issue. The Jindo backend has no equivalent fast path. ## Solution Plumb the trusted length from `FileStatus` down to the Jindo store so open can skip its own `getFileStatus`, mirroring what `ObjectStoreFileSystem` already does. - `JindoFileSystem` overrides `Open(const FileStatus&)`, validates the length is non-negative, and routes both `Open` overloads through a new protected `OpenReader(path, std::optional<int64_t> file_length)` hook. `Open(path)` passes `std::nullopt` and keeps the original behavior. - When a length is present, `OpenReader` calls a new vendored `JdoFileSystem::openReader(path, file_length, result)` overload that sets `JDO_OPEN_OPTS_HAS_GET_FILE_STATUS=true` and `JDO_OPEN_OPTS_FILE_LENGTH=<len>` on the open options. The `HAS_GET_FILE_STATUS` hint is the gate that actually suppresses the store's `getFileStatus`; `FILE_LENGTH` alone has no effect. The benefit is automatic for the existing read path, since it already passes a `FileStatus` with the manifest size. ## Anything else? - Scope: the hint is honored by the cloud object-store open path (`JdcOpenCall`); local and JFS opens ignore these options, so their behavior is unchanged. - The length is trusted, not re-validated at open: a stale or wrong length surfaces later as a short or failed read rather than at open time. In the read path the length comes from the manifest for the file being read, so it is authoritative, and a negative length is rejected up front. - No new public API and no storage format or protocol change: `FileSystem::Open(const FileStatus&)` already existed, this only implements it in the Jindo subclass; the new `openReader` overload lives in vendored `third_party/`. ## Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
