plusplusjiajia opened a new pull request, #10817:
URL: https://github.com/apache/paimon/pull/10817
### Purpose
PyArrow implements `PythonFile.read_at(nbytes, offset)` as `seek(offset)`
plus `read(nbytes)` on the wrapped file object, so `JindoInputFile.read_at`,
which calls Jindo's `pread`, is never reached. Every Parquet column chunk and
manifest range read goes through Jindo's sequential stream, which reads ahead a
whole buffer.
Measured on Alibaba Cloud OSS with pyjindosdk 6.10.4, bytes taken from the
OSS access logs:
| Read | PyArrow S3 | Jindo, master | Jindo, this PR |
|---|---|---|---|
| 40 positional reads of 12 KB | 0.49 MB | 58.46 MB | 0.49 MB |
| 3 of 22 columns, 16 key lookups | 38.11 MB | 50.56 MB | 38.11 MB |
| 22 of 22 columns, 16 key lookups | 376.36 MB | 380.34 MB in 106 GETs |
376.45 MB in 106 GETs |
Calling pyjindo's `pread` directly returns exactly the requested bytes.
Jindo's read-ahead options do not change the amplification of seek plus read.
`JindoInputFile` now tracks its own position:
- `seek` only moves the logical position.
- A read that continues the previous one keeps the stream and its read-ahead.
- A read after a jump of at most `fs.oss.jindo.pread-max-size` (default 4
MiB) fetches exactly its range with `pread`.
- Larger reads keep the stream, which downloads big ranges in parallel.
Full-row reads issue the same 106 parallel GETs and take the same time as
before.
Setting `fs.oss.jindo.pread-max-size` to 0 restores the previous behavior.
The option is consumed by pypaimon and not forwarded to Jindo.
### Tests
- New tests in `pypaimon/tests/jindo_file_system_test.py`:
- a read after a seek fetches exactly its range with `pread`;
- a contiguous read after it switches back to the stream;
- sequential reads from the start keep the stream;
- reads are clamped to the known file size;
- `SEEK_CUR` works;
- reads above the threshold, and a threshold of 0, keep the stream;
- PyArrow `PythonFile.read_at` reaches `pread`;
- the option is not forwarded to Jindo, and the handler passes it to input
files.
- `test_known_size_avoids_backend_seek_from_end` is updated for the lazy
seek.
- `jindo_file_system_test.py`, `video_format_test.py`,
`caching_file_io_test.py` and `rest/rest_token_file_io_test.py` pass (138
passed).
- The table above was measured twice against OSS with identical bytes.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
