nssalian commented on code in PR #2637:
URL: https://github.com/apache/iceberg-go/pull/2637#discussion_r4632666594
##########
table/internal/partition_aggregate.go:
##########
@@ -139,15 +140,33 @@ func CoercePartition(values map[int]any, partitionType
*iceberg.StructType) map[
coerced := make(map[int]any, partitionType.NumFields())
for i := range partitionType.NumFields() {
- id := partitionType.Field(i).ID
- if value, ok := values[id]; ok {
- coerced[id] = value
+ field := partitionType.Field(i)
+ if value, ok := values[field.ID]; ok {
+ coerced[field.ID] = widenPartitionValue(value,
field.Type)
}
}
return coerced
}
+// widenPartitionValue applies the exact int to long and float to double
+// promotions, so values from manifests written before an identity source was
+// promoted aggregate with values written after it.
+func widenPartitionValue(value any, typ iceberg.Type) any {
+ switch v := value.(type) {
+ case int32:
+ if _, ok := typ.(iceberg.Int64Type); ok {
+ return int64(v)
+ }
+ case float32:
+ if _, ok := typ.(iceberg.Float64Type); ok {
+ return float64(v)
+ }
Review Comment:
+1, I can reproduce this on this head. After promoting an identity partition
source from float to double and appending rows before and after,
`tbl.Inspect().Files()`, `Entries()` and `AllEntries()` fail with "cannot
append scalar of type float32 to builder for type float64", while
`Partitions()` now works. Those tables build partition values through
`PartitionStructBuilder`, which falls back to `PartitionValueScalar`
(`table/internal/partition_struct_builder.go:168`), and that function has no
float case. The position deletes table uses the same builder.
Reusing the helper from this PR fixes all of them. I tried this locally and
all three tables return both rows:
```go
func PartitionValueScalar(value any, typ iceberg.Type, arrowType
arrow.DataType) (scalar.Scalar, error) {
value = widenPartitionValue(value, typ)
switch typ.(type) {
```
For the end-to-end test you asked for, a float to double case that appends
one row before and one after the promotion and reads both `Files()` and
`Partitions()` would cover this fix and the one in the PR. It fails on main
today.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]