singhpk234 commented on code in PR #4517:
URL: https://github.com/apache/iceberg/pull/4517#discussion_r858238644
##########
spark/v3.2/spark/src/main/java/org/apache/iceberg/spark/source/SparkMicroBatchStream.java:
##########
@@ -111,7 +112,10 @@ public Offset latestOffset() {
}
Snapshot latestSnapshot = table.currentSnapshot();
- return new StreamingOffset(latestSnapshot.snapshotId(),
Iterables.size(latestSnapshot.addedFiles()), false);
+ return new StreamingOffset(latestSnapshot.snapshotId(),
+
Long.parseLong(latestSnapshot.summary().getOrDefault(SnapshotSummary.ADDED_FILES_PROP,
Review Comment:
I see, I didn't use `PropertyUtil.propertyAsLong` as to provide the default,
I will have to iterate the addedFiles iterator to compute it's size (implying
materilizing this), presently this util doesn't supports sending it a supplier
so that it can be computed only when key is not present. what do you recommend
here ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
