difin commented on code in PR #6707:
URL: https://github.com/apache/hive/pull/6707#discussion_r3909143246
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/IcebergTableUtil.java:
##########
@@ -804,32 +923,128 @@ public static TransformSpec getTransformSpec(Table
table, String transformName,
return spec;
}
- public static <T> List<T> readColStats(Table table, Long snapshotId,
Predicate<BlobMetadata> filter) {
- List<T> colStats = Lists.newArrayList();
+ /**
+ * Whether the stored column statistics still describe the table: the
current snapshot owns them,
+ * or only row-preserving commits (compaction) separate it from the snapshot
that does. Derived
+ * from the table metadata, so it holds for the writes of every engine.
+ */
+ public static boolean colStatsAccurate(Table table, Snapshot snapshot,
Configuration conf) {
+ return colStatsAccurate(table, snapshot, isPartitionStats(table, conf));
+ }
- String statsPath = IcebergTableUtil.getColStatsPath(table, snapshotId);
- if (statsPath == null) {
- LOG.warn("Column stats file not found for snapshot: {}", snapshotId);
- return colStats;
+ /** The same, judging the granularity the caller goes on to read. */
+ static boolean colStatsAccurate(Table table, Snapshot snapshot, boolean
partitionLevel) {
+ return getColStatsFile(table, snapshot.snapshotId(), partitionLevel) !=
null;
+ }
+
+ /**
+ * The partitions written since a snapshot. Null when the walk cannot answer
- an expired
+ * snapshot broke the chain, or the bound was reached - and every partition
then counts as
+ * changed. The bound is on manifests read, not on snapshots walked.
+ */
+ static Set<String> partitionsChangedSince(Table table, Snapshot snapshot,
long sinceSnapshotId,
+ Configuration conf, boolean capped) {
+ // the walk reads manifests, and every column of a query asks the same
question of them
+ String cacheKey = MODIFIED_PARTITIONS_PREFIX + table.name() + '.' +
snapshot.snapshotId() + '.' +
+ sinceSnapshotId + '.' + capped;
Review Comment:
Should this be converted to `String.format()` for better readability?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
