amogh-jahagirdar commented on code in PR #18925:
URL: https://github.com/apache/iceberg/pull/18925#discussion_r4565018394


##########
core/src/main/java/org/apache/iceberg/TrackedFileAdapters.java:
##########
@@ -583,7 +597,189 @@ private void wrapInternal(DataFile newFile) {
           newFile.content());
 
       this.file = newFile;
-      this.stats = hasContentStats(newFile) ? statsWrapper.wrap(newFile) : 
null;
+      this.stats = wrapStats(fileWithPartitionSourceStats());
+    }
+
+    private ContentStats wrapStats(DataFile dataFile) {
+      return hasContentStats(dataFile) ? statsWrapper.wrap(dataFile) : null;
+    }
+
+    private DataFile fileWithPartitionSourceStats() {
+      PartitionSpec spec = specsById.get(file.specId());
+      Preconditions.checkArgument(
+          spec != null, "Cannot find partition spec for spec ID: %s", 
file.specId());
+
+      Set<Integer> partitionSourceIds = Sets.newHashSet();
+      Set<Integer> floatingPointSourceIds = Sets.newHashSet();
+      for (PartitionField field : spec.fields()) {
+        if (field.transform().isVoid()) {
+          continue;
+        }
+
+        Preconditions.checkArgument(
+            field.transform().preservesOrder(),
+            "Cannot convert data file %s: partition field does not preserve 
order: %s",
+            file.location(),
+            field);
+        Type sourceType = tableSchema.findType(field.sourceId());
+        Preconditions.checkArgument(
+            sourceType != null,

Review Comment:
   Users can drop columns that are still referenced by historical partition 
specs. After the drop, those specs resolve the source column's type as 
`unknown`, so for files written under them we can't fill the source column 
bounds from the partition value. For identity and truncate the partition 
value's type is `unknown` too. For year, month, day and hour the partition type 
is fixed, but the source could be a date or any of the timestamp types, and 
their bounds are encoded in different units. 
   Ideally we'd get the source column's type from an older schema, but expire 
snapshots can technically remove every schema that has it while files under 
that spec are still live. Arguably that's an issue in expire snapshots, because 
it keeps a spec while removing the schemas that spec still references fields 
for.
   
   Right now i just fail this case to prevent producing metadata that would 
lead to an inability to reconstruct the tuple but I think we'll need to address 
this.
   
   To address this we could:
   - Fix expire snapshots to keep the schemas a live spec relies on, and pass 
the table's schemas by ID into `TrackedFileAdapters.forDataFile` along with the 
specs, so the adapter can take the source type from a schema that has the 
column.
   - Store the partition value directly as a stats-only `partition-value` 
field. This works for any transform, since its type is the partition field's 
result type.
   - Least preferable: require users to rewrite these specific files and their 
equality deletes, so that no live file is written under a spec with a dropped 
source column.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to