dossett opened a new pull request, #3678: URL: https://github.com/apache/parquet-java/pull/3678
Metadata-aware callers may already know a Parquet file's exact length from a catalog, manifest, or prior listing. `HadoopInputFile.fromPath(Path, Configuration)` currently performs `getFileStatus()` unconditionally, discarding that knowledge and potentially adding a remote metadata request before the file can be read. This adds a length-aware factory that trusts the caller-provided value and passes it to Hadoop's `openFile()` API. The API is format-neutral, but it enables table formats and other metadata-aware readers to avoid the status lookup when they can guarantee the length. Existing path- and status-based construction retain their current behavior. This follows up on the known-length optimization discussed in #3395. The change also raises the Hadoop baseline from 3.3.0 to 3.3.5, where `FS_OPTION_OPENFILE_LENGTH` was standardized and supported by S3A. Summary: - Add `HadoopInputFile.fromPath(Path, long, Configuration)`. - Preserve `FileStatus` as the preferred open hint when one is already available. - Otherwise pass the known length through `FS_OPTION_OPENFILE_LENGTH`. - Reject negative lengths and document that the supplied length is trusted without validation. Testing: - `./mvnw -pl parquet-hadoop -Dtest=TestHadoopOpenFile test` - `./mvnw -q -pl parquet-hadoop -DskipTests spotless:check` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
