lxy-9602 commented on code in PR #269:
URL: https://github.com/apache/paimon-cpp/pull/269#discussion_r3914787003
##########
src/paimon/core/realtime/realtime_primary_key_writer.cpp:
##########
@@ -245,15 +235,14 @@ Result<CommitIncrement>
RealtimePrimaryKeyWriter::PrepareCommit(bool wait_compac
return increment;
}
-Status RealtimePrimaryKeyWriter::FlushSegment(const
std::shared_ptr<RealtimeSegmentHandle>& segment,
- const OffsetRange&
sealed_offsets) {
+Status RealtimePrimaryKeyWriter::FlushSegment(
+ const std::shared_ptr<RealtimeSegmentHandle>& segment) {
PAIMON_ASSIGN_OR_RAISE(std::vector<std::unique_ptr<BatchReader>> readers,
realtime_store_->CreateCommitReaders(segment));
Review Comment:
I agree that validating the raw row count would provide useful defense in
depth. However, there is no simple reliable check at this layer with the
current streaming reader contract.
Unlike append, PK commit readers are multiple one-shot streams consumed
through MOR. The row count after MOR is not comparable because deduplication
may reduce it. Checking before MOR would require wrapping every raw reader with
shared counting state, detecting completion across all reader EOFs, and
propagating a mismatch before the rolling file writer completes. Pre-reading
would instead require buffering or reopening the streams.
Since `CreateCommitReaders` already requires plugins to expose every sealed
row exactly once, I will add a TODO/follow-up for cardinality validation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
