huan233usc commented on code in PR #18657:
URL: https://github.com/apache/iceberg/pull/18657#discussion_r4396157343
##########
spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/source/SparkScan.java:
##########
@@ -234,12 +235,14 @@ protected Statistics estimateStatistics(Snapshot
snapshot) {
Map<Integer, List<BlobMetadata>> groupedByField =
metadataList.stream()
+ .filter(metadata -> !metadata.fields().isEmpty())
.collect(
Collectors.groupingBy(
metadata -> metadata.fields().get(0),
Collectors.toList()));
+ Schema snapshotSchema = SnapshotUtil.schemaFor(table,
snapshot.snapshotId());
for (Map.Entry<Integer, List<BlobMetadata>> entry :
groupedByField.entrySet()) {
- String colName = table.schema().findColumnName(entry.getKey());
+ String colName = snapshotSchema.findColumnName(entry.getKey());
Review Comment:
Does the snapshot schema still match the column names exposed by this scan?
Schema-only renames keep the same snapshot, but Spark matches stats by name. If
`a` has NDV 100 and `b` has NDV 1, swapping their names reports `a=100, b=1`
instead of `a=1, b=100`. Could we resolve IDs against the scan schema, skip
missing IDs, and add a test that checks the mapped stats?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
