zhuxiangyi commented on code in PR #9657:
URL: https://github.com/apache/paimon/pull/9657#discussion_r4044387481
##########
paimon-core/src/main/java/org/apache/paimon/operation/AbstractFileStoreScan.java:
##########
@@ -319,13 +320,26 @@ public Plan plan() {
manifestsResult.allManifests.stream()
.mapToLong(f -> f.numAddedFiles() -
f.numDeletedFiles())
.sum();
+ // for DELTA and CHANGELOG scan modes the result contains both ADD
and DELETE entries,
+ // only ADD entries will actually be read, so size and record
count only count them
+ long resultedTableFilesSize = 0L;
+ long resultedRecordCount = 0L;
+ for (ManifestEntry entry : result) {
+ if (entry.kind() == FileKind.ADD) {
+ resultedTableFilesSize += entry.file().fileSize();
Review Comment:
Thanks — you are right about readChanges(), which reads the DELETE entries
as before files.
Moved the size/record fold from AbstractFileStoreScan.plan() into
SnapshotReaderImpl, after the read path is chosen: read() counts ADD only (also
for a DELTA plan), readChanges() counts ADD + DELETE, readIncrementalDiff()
counts the ADD entries of both snapshots. ScanMetrics.reportResultedFiles()
carries it; gauge names are unchanged, so Spark/Flink/docs are untouched.
resultedTableFiles still counts every entry, as before.
Added ScanResultedFilesMetricsTest covering each path through the real
reader and stream scan, including the deletion-only change and the overwrite
path of FollowUpScanner for both append and primary key tables. On the previous
commit its four change-read/diff cases fail and the four ADD-only cases pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
