lxy-9602 commented on code in PR #277:
URL: https://github.com/apache/paimon-cpp/pull/277#discussion_r3920852718
##########
include/paimon/table/source/plan.h:
##########
@@ -34,5 +35,13 @@ class PAIMON_EXPORT Plan {
virtual const std::vector<std::shared_ptr<Split>>& Splits() const = 0;
/// Snapshot id of this plan, return `std::nullopt` if the table is empty.
virtual std::optional<int64_t> SnapshotId() const = 0;
+ /// Immutable snapshot metadata used to build this plan.
+ ///
+ /// The returned view can be injected into a later batch scan to reuse the
already resolved and
+ /// parsed snapshot. Only non-streaming, non-real-time latest-snapshot
plans publish a reusable
+ /// view; other plans return `nullptr`.
+ virtual std::shared_ptr<const SnapshotReadView> GetSnapshotReadView()
const {
+ return nullptr;
+ }
Review Comment:
Thanks for the work. I am not yet convinced that the performance benefit
justifies the new public API and ABI impact.
A caller can already avoid resolving `LATEST` by specifying
`scan.snapshot-id`, and can avoid schema metadata I/O through
`SetTableSchema()`. Compared with that approach, `SnapshotReadView` mainly
avoids:
- loading and deserializing the snapshot JSON;
- repeatedly deserializing the schema JSON passed through `SetTableSchema()`;
- some associated object construction.
The existing Paimon cache does not cache parsed snapshot or schema objects.
However, when the underlying filesystem already caches metadata and small
files, the remaining benefit may mostly be JSON parsing and object
construction. It is unclear whether that is significant compared with manifest
planning and data access.
Could we provide a benchmark for the intended point-lookup workload,
comparing:
1. normal latest-snapshot scan;
2. `scan.snapshot-id` + `SetTableSchema()` + the existing cache;
3. `SnapshotReadView` + the same cache?
It would be helpful to report planning latency (P50/P99), metadata
I/O/listing counts, and ideally end-to-end QPS. In particular, the comparison
between cases 2 and 3 would isolate the incremental benefit that requires this
new API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
