badalprasadsingh commented on code in PR #2487:
URL: https://github.com/apache/iceberg-go/pull/2487#discussion_r4577494633


##########
table/arrow_utils.go:
##########
@@ -1276,7 +1280,12 @@ func (a *arrowProjectionVisitor) Struct(st 
iceberg.StructType, structArr arrow.A
                        dt := a.typeToArrowType(field.Type)
                        alloc := compute.GetAllocator(a.ctx)
 
+                       constant, hasConstant := a.partitionConstants[field.ID]
                        switch {
+                       case hasConstant && constant != nil:
+                               arr = defaultToArray(constant, field.Type, dt, 
structArr.Len(), alloc)

Review Comment:
   p1: manifests keep the partition type the file was written with, after 
promoting a float identity column to double, this builds a float32 array under 
a float64 field and "ToArrowTable" panics (mismatch data type float32 vs 
float64)
   
   better: can we widen the value to "field.Type" first? int to long already 
works



##########
table/arrow_scanner.go:
##########
@@ -3069,10 +3111,10 @@ func (as *arrowScan) recordsFromTask(ctx 
context.Context, task tblutils.Enumerat
                        return 
array.NewRecordBatch(invariants.outputArrowSchema, r.Columns(), r.NumRows()), 
nil
                }
 
-               return ToRequestedSchema(ctx, as.projectedSchema, readSchema, 
r, SchemaOptions{
+               return projectToRequestedSchema(ctx, as.projectedSchema, 
readSchema, r, SchemaOptions{
                        UseLargeTypes:   as.useLargeTypes,
                        TableProperties: invariants.tableProperties,
-               })
+               }, nil, partitionConstants)
        })
 
        err = as.processRecordsWithPlans(ctx, task, iceSchema, rowFilter, rdr, 
colIndices, pipeline, posSource, sink, filterPlans)

Review Comment:
   a P0: filters still see this column as missing -> on the issue's table, v2 
Delete(region = 'west' AND id = 1) removes id 1 from east; mor deletes, 
overwrite and incremental scans also go wrong
   
   can the file's partition value be folded into its filter (also 
"producePosDeletesFromTask" and row-group pruning)?



##########
table/arrow_scanner.go:
##########
@@ -2836,6 +2836,47 @@ func pruningFilterHasMissingInitialDefault(
        return false, nil
 }
 
+// identityPartitionConstants returns the data file's identity partition
+// values keyed by source field ID. A writer may leave an identity partition
+// column out of its data files, as Delta and Hive-style layouts do; the spec's
+// column projection rule 1 reads such a field from the partition value.
+func identityPartitionConstants(meta Metadata, file iceberg.DataFile) 
map[int]any {
+       if meta == nil || file == nil {
+               return nil
+       }
+       spec := meta.PartitionSpecByID(int(file.SpecID()))

Review Comment:
   nothing tests that this uses the file's spec rather than the current one, or 
that the partition value wins over initial-default (a null partition value 
stays null)
   
   can we add a file written before a spec change and a field with an 
initial-default?



##########
table/arrow_scanner.go:
##########
@@ -3053,6 +3094,7 @@ func (as *arrowScan) recordsFromTask(ctx context.Context, 
task tblutils.Enumerat
                pipeline = append(pipeline, filterFunc)
        }
 
+       partitionConstants := identityPartitionConstants(as.metadata, 
task.Value.File)

Review Comment:
   p0: when an eq-delete keyed on this column applies to the file, the step at 
lin3032 projects it first, without these constants, the column reads null, the 
delete never matches, and a cow delete rewrites the rows into partition null
   
   better: can that step use projectToRequestedSchema with partitionConstants 
too?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to