mrdrivingduck opened a new pull request, #189:
URL: https://github.com/apache/paimon-cpp/pull/189
I am not sure whether introducing a new `FileSystem` API is acceptable
upstream, but...
## Summary
This PR adds an `Open(path, file_size)` path to the object store file system
API for callers that already have trusted file metadata.
Paimon split reads already carry the data-file size from manifest metadata.
Passing that size through avoids a per-file object metadata request, while
retaining `Open(path)` as the default path for callers without a known size.
This is a generic object store optimization. My WIP OSS(v2) filesystem uses
the supplied size to skip its per-file `HeadObject` request, and the S3
filesystem can use the same known-size opening path.
## Benchmark
The benchmark exercises My WIP OSS(v2) filesystem, using the same real
Paimon table throughout:
- Location:
`oss://paimon-cpp-fs/paimon-cpp-demo-20260710-220136/demo.db/events`
- 96 ORC data files, about 4.27 MiB each
- Result validation: `count(*) = 786,435`, `sum(quantity) = 39,713,738`
The only difference is how data files are opened:
- Baseline: `Open(path)`, which requires one `HeadObject` request per file.
- Optimized: `Open(path, file_size)`, which uses the file size already
present in the Paimon manifest and skips that request.
| Workload | Concurrency / runs | `Open(path)` | `Open(path, size)` |
Improvement |
| --- | --- | ---: | ---: | ---: |
| Create 96 input streams without reading data | Single-threaded, sequential
| 1019.788 ms | 0.120 ms | ~8,500× faster; saves 99.988% |
| Open every file and read its first byte | Single-threaded, 5 alternating
runs | 1.667 s | 1.111 s | 33.3% faster |
| DuckDB full-table aggregate | `threads=8`, 5 independent processes | 1.837
s | 1.315 s | 28.4% faster |
| DuckDB full-table aggregate | `threads=1`, 5 independent processes | 3.182
s | 2.557 s | 19.7% faster |
The largest gain occurs when many known-size files are opened sequentially
with little or no reading, which isolates the cost of the per-file metadata
request. In an end-to-end DuckDB scan, file reads, decoding, and concurrency
reduce the relative impact, but the aggregate query still shows a stable
19.7%–28.4% mean latency reduction on this 96-file table.
## End-to-end query
```sql
SET threads = <1 or 8>;
ATTACH oss://paimon-cpp-fs/paimon-cpp-demo-20260710-220136/
AS paimon_demo (TYPE paimon, READ_ONLY);
SELECT count(*) AS row_count, sum(quantity) AS quantity_sum
FROM paimon_demo.demo.events;
```
Raw timings from five independent processes:
| Mode | `Open(path)` | `Open(path, size)` | Median improvement |
| --- | --- | --- | --- |
| `threads=8` | 2.478, 1.859, 1.696, 1.896, 1.256 s | 1.355, 1.324, 1.254,
1.328, 1.314 s | 1.859 → 1.324 s, 28.8% faster |
| `threads=1` | 3.619, 3.255, 2.682, 3.740, 2.614 s | 2.508, 2.695, 2.615,
2.462, 2.503 s | 3.255 → 2.508 s, 23.0% faster |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
