amogh-jahagirdar commented on code in PR #18925:
URL: https://github.com/apache/iceberg/pull/18925#discussion_r4565018394
##########
core/src/main/java/org/apache/iceberg/TrackedFileAdapters.java:
##########
@@ -583,7 +597,189 @@ private void wrapInternal(DataFile newFile) {
newFile.content());
this.file = newFile;
- this.stats = hasContentStats(newFile) ? statsWrapper.wrap(newFile) :
null;
+ this.stats = wrapStats(fileWithPartitionSourceStats());
+ }
+
+ private ContentStats wrapStats(DataFile dataFile) {
+ return hasContentStats(dataFile) ? statsWrapper.wrap(dataFile) : null;
+ }
+
+ private DataFile fileWithPartitionSourceStats() {
+ PartitionSpec spec = specsById.get(file.specId());
+ Preconditions.checkArgument(
+ spec != null, "Cannot find partition spec for spec ID: %s",
file.specId());
+
+ Set<Integer> partitionSourceIds = Sets.newHashSet();
+ Set<Integer> floatingPointSourceIds = Sets.newHashSet();
+ for (PartitionField field : spec.fields()) {
+ if (field.transform().isVoid()) {
+ continue;
+ }
+
+ Preconditions.checkArgument(
+ field.transform().preservesOrder(),
+ "Cannot convert data file %s: partition field does not preserve
order: %s",
+ file.location(),
+ field);
+ Type sourceType = tableSchema.findType(field.sourceId());
+ Preconditions.checkArgument(
+ sourceType != null,
Review Comment:
Users can drop columns that are still referenced by historical partition
specs. After the drop, those specs resolve the source column's type as
`unknown`, so for files written under them we can't fill the source column
bounds from the partition value. For identity and truncate the partition
value's type is `unknown` too. For year, month, day and hour the partition type
is fixed, but the source could be a date or any of the timestamp types, and
their bounds are encoded in different units.
Ideally we'd get the source column's type from an older schema, but expire
snapshots can technically remove every schema that has it while files under
that spec are still live. Arguably that's an issue in expire snapshots, because
it keeps a spec while removing the schemas that spec relies on.
Right now i just fail this case to prevent producing metadata that would
lead to an inability to reconstruct the tuple but I think we'll need to address
this.
To address this we could:
- Fix expire snapshots to keep the schemas a live spec relies on, and pass
the table's schemas by ID into `TrackedFileAdapters.forDataFile` along with the
specs, so the adapter can take the source type from a schema that has the
column.
- Store the partition value directly as a stats-only `partition-value`
field. This works for any transform, since its type is the partition field's
result type.
- Least preferable: require users to rewrite these specific files and their
equality deletes, so that no live file is written under a spec with a dropped
source column.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]