XiaoHongbo-Hope opened a new pull request, #10152: URL: https://github.com/apache/paimon/pull/10152
### Purpose Paimon data-file metadata already contains the immutable file size, but the Python Parquet reader discarded it. PyArrow therefore queried the file length again when opening each Parquet file. With the Jindo `PyFileSystemHandler`, the length lookup is implemented as a seek from the end and can add an object-store metadata request per cold file open. This change: - forwards `DataFileMeta.file_size` to `FormatPyArrowReader`; - supplies the size to `ParquetFileFormat.make_fragment` on PyArrow 7+; - lets the Jindo handler use registered Paimon file sizes for end-relative seeks and unbounded reads; - bounds the Jindo known-size cache and includes the size in the Parquet metadata-cache key; and - keeps the PyArrow 6 code path compatible. A real Jindo read of the same Parquet object changed both file-length operations from `seek(0, SEEK_END)` to absolute seeks at the manifest size. The corresponding OSS trace contained the two data GETs and no metadata request. ### Tests - `PYTHONPATH=paimon-python python -m pytest -q paimon-python/pypaimon/tests/jindo_file_system_test.py paimon-python/pypaimon/tests/parquet_metadata_cache_test.py` (PyArrow 19.0.1: 31 passed, 19 skipped) - same test command with PyArrow 6.0.1 (31 passed, 19 skipped) - relevant Parquet/reader suite excluding optional Lance tests (137 passed, 4 skipped, 9 deselected) - `flake8` on all changed Python files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
