amogh-jahagirdar commented on code in PR #18925:
URL: https://github.com/apache/iceberg/pull/18925#discussion_r4565018381
##########
core/src/main/java/org/apache/iceberg/TrackedFileAdapters.java:
##########
@@ -583,7 +597,189 @@ private void wrapInternal(DataFile newFile) {
newFile.content());
this.file = newFile;
- this.stats = hasContentStats(newFile) ? statsWrapper.wrap(newFile) :
null;
+ this.stats = wrapStats(fileWithPartitionSourceStats());
+ }
+
+ private ContentStats wrapStats(DataFile dataFile) {
+ return hasContentStats(dataFile) ? statsWrapper.wrap(dataFile) : null;
+ }
+
+ private DataFile fileWithPartitionSourceStats() {
+ PartitionSpec spec = specsById.get(file.specId());
+ Preconditions.checkArgument(
+ spec != null, "Cannot find partition spec for spec ID: %s",
file.specId());
+
+ Set<Integer> partitionSourceIds = Sets.newHashSet();
+ Set<Integer> floatingPointSourceIds = Sets.newHashSet();
+ for (PartitionField field : spec.fields()) {
+ if (field.transform().isVoid()) {
+ continue;
+ }
+
+ Preconditions.checkArgument(
+ field.transform().preservesOrder(),
Review Comment:
Once we support stats-only fields, a `partition-value` field can store the
bucket output directly. Its type is the partition field's result type, so we
don't need the source type. The adapter would take the table's stats fields
alongside the specs and fill the `partition-value` field from the partition
tuple. `MapBackedContentStats` would also need to resolve the stats-only field
types, since right now it only looks up types in the table schema.
##########
core/src/main/java/org/apache/iceberg/TrackedFileAdapters.java:
##########
@@ -583,7 +597,189 @@ private void wrapInternal(DataFile newFile) {
newFile.content());
this.file = newFile;
- this.stats = hasContentStats(newFile) ? statsWrapper.wrap(newFile) :
null;
+ this.stats = wrapStats(fileWithPartitionSourceStats());
+ }
+
+ private ContentStats wrapStats(DataFile dataFile) {
+ return hasContentStats(dataFile) ? statsWrapper.wrap(dataFile) : null;
+ }
+
+ private DataFile fileWithPartitionSourceStats() {
+ PartitionSpec spec = specsById.get(file.specId());
+ Preconditions.checkArgument(
+ spec != null, "Cannot find partition spec for spec ID: %s",
file.specId());
+
+ Set<Integer> partitionSourceIds = Sets.newHashSet();
+ Set<Integer> floatingPointSourceIds = Sets.newHashSet();
+ for (PartitionField field : spec.fields()) {
+ if (field.transform().isVoid()) {
+ continue;
+ }
+
+ Preconditions.checkArgument(
+ field.transform().preservesOrder(),
+ "Cannot convert data file %s: partition field does not preserve
order: %s",
+ file.location(),
+ field);
+ Type sourceType = tableSchema.findType(field.sourceId());
+ Preconditions.checkArgument(
+ sourceType != null,
Review Comment:
Users can drop columns that are still referenced by historical partition
specs. After the drop, those specs resolve the source column's type as
`unknown`, so for files written under them we can't fill the source column
bounds from the partition value. For identity and truncate the partition
value's type is `unknown` too. For year, month, day and hour the partition type
is fixed, but the source could be a date or any of the timestamp types, and
their bounds are encoded in different units. Ideally we'd get the source
column's type from an older schema, but expire snapshots can technically remove
every schema that has it while files under that spec are still live. Arguably
that's an issue in expire snapshots, because it keeps a spec while removing the
schemas that spec relies on.
To address this we could:
- Fix expire snapshots to keep the schemas a live spec relies on, and pass
the table's schemas by ID into `TrackedFileAdapters.forDataFile` along with the
specs, so the adapter can take the source type from the latest schema that has
the column.
- Store the partition value directly as a stats-only `partition-value`
field. This works for any transform, since its type is the partition field's
result type.
- Least preferable: require users to rewrite these specific files and their
equality deletes, so that no live file is written under a spec with a dropped
source column.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]