amogh-jahagirdar commented on code in PR #18925:
URL: https://github.com/apache/iceberg/pull/18925#discussion_r4565053152
##########
core/src/main/java/org/apache/iceberg/TrackedFileAdapters.java:
##########
@@ -583,7 +597,189 @@ private void wrapInternal(DataFile newFile) {
newFile.content());
this.file = newFile;
- this.stats = hasContentStats(newFile) ? statsWrapper.wrap(newFile) :
null;
+ this.stats = wrapStats(fileWithPartitionSourceStats());
+ }
+
+ private ContentStats wrapStats(DataFile dataFile) {
+ return hasContentStats(dataFile) ? statsWrapper.wrap(dataFile) : null;
+ }
+
+ private DataFile fileWithPartitionSourceStats() {
+ PartitionSpec spec = specsById.get(file.specId());
+ Preconditions.checkArgument(
+ spec != null, "Cannot find partition spec for spec ID: %s",
file.specId());
+
+ Set<Integer> partitionSourceIds = Sets.newHashSet();
+ Set<Integer> floatingPointSourceIds = Sets.newHashSet();
+ for (PartitionField field : spec.fields()) {
Review Comment:
We could cache the partition source IDs and field positions per spec ID
instead of going through the spec's fields for every file, but it adds state to
the adapter, and it may not be worth it since that's O(spec width) per file and
specs are usually only a few fields wide.
##########
core/src/main/java/org/apache/iceberg/TrackedFileAdapters.java:
##########
@@ -583,7 +597,189 @@ private void wrapInternal(DataFile newFile) {
newFile.content());
this.file = newFile;
- this.stats = hasContentStats(newFile) ? statsWrapper.wrap(newFile) :
null;
+ this.stats = wrapStats(fileWithPartitionSourceStats());
+ }
+
+ private ContentStats wrapStats(DataFile dataFile) {
+ return hasContentStats(dataFile) ? statsWrapper.wrap(dataFile) : null;
+ }
+
+ private DataFile fileWithPartitionSourceStats() {
+ PartitionSpec spec = specsById.get(file.specId());
+ Preconditions.checkArgument(
+ spec != null, "Cannot find partition spec for spec ID: %s",
file.specId());
+
+ Set<Integer> partitionSourceIds = Sets.newHashSet();
+ Set<Integer> floatingPointSourceIds = Sets.newHashSet();
+ for (PartitionField field : spec.fields()) {
+ if (field.transform().isVoid()) {
+ continue;
+ }
+
+ Preconditions.checkArgument(
+ field.transform().preservesOrder(),
+ "Cannot convert data file %s: partition field does not preserve
order: %s",
+ file.location(),
+ field);
+ Type sourceType = tableSchema.findType(field.sourceId());
+ Preconditions.checkArgument(
+ sourceType != null,
+ "Cannot convert data file %s: partition source column was dropped:
%s",
+ file.location(),
+ field);
+
+ partitionSourceIds.add(field.sourceId());
+ if (StatsUtil.isFloatingPoint(sourceType)) {
+ floatingPointSourceIds.add(field.sourceId());
+ }
+ }
+
+ if (partitionSourceIds.isEmpty()) {
+ return file;
+ }
+
+ Map<Integer, Long> valueCounts =
+ fillFromPartition(file.valueCounts(), partitionSourceIds, id ->
file.recordCount());
+ Map<Integer, Long> nullValueCounts =
+ fillFromPartition(file.nullValueCounts(), partitionSourceIds,
this::nullValueCount);
+ Map<Integer, Long> nanValueCounts =
+ fillFromPartition(file.nanValueCounts(), floatingPointSourceIds,
this::nanValueCount);
+ Map<Integer, ByteBuffer> lowerBounds =
+ fillFromPartition(file.lowerBounds(), partitionSourceIds, id ->
boundsFor(id).lower());
+ Map<Integer, ByteBuffer> upperBounds =
+ fillFromPartition(file.upperBounds(), partitionSourceIds, id ->
boundsFor(id).upper());
Review Comment:
We could compute a column's bounds once instead of separately for the lower
and upper views, but it makes this less readable, and it may not be worth it
since it only applies to partition source columns and each computation is a
loop over a few partition fields.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]