0z5a opened a new issue, #2603:
URL: https://github.com/apache/iceberg-go/issues/2603

   ### Apache Iceberg version
   
   main (development)
   
   ### Please describe the bug 🐞
   
   `YearTransform.ToHumanString` formats the year with `strconv.Itoa`, so years 
before 1000 come out without leading zeros. A row with `ts = 0935-07-12` lands 
in `ts_year=935/`, and the common sentinel date `0001-01-01` lands in 
`ts_year=1/`. The month, day and hour transforms in the same package already 
pad to four digits (`0935-07`, `0935-07-12`, `0935-07-12-00`), so only the year 
value is off.
   
   Java (`TransformUtil.humanYear` uses `String.format("%04d", ...)`) and 
PyIceberg (`to_human_year` uses `{:0=4d}`) both produce `0935` and `0001`. As a 
result, iceberg-go writes year-partitioned files for such rows into a different 
directory than Java/Spark or PyIceberg would for the same partition value, and 
the human-readable value differs from what the other implementations show.
   
   Tested on main at 7cf76392bbf28bff03bccd1917efebfd5824e4bd (Go 1.27.1, macOS 
arm64).
   
   Repro: save this as `year_partition_path_test.go` in the repository root and 
run `go test -v -run TestYearPartitionPath .`
   
   ```go
   package iceberg_test
   
   import (
        "fmt"
        "testing"
        "time"
   
        "github.com/apache/iceberg-go"
   )
   
   type rec []any
   
   func (r rec) Size() int            { return len(r) }
   func (r rec) Get(pos int) any      { return r[pos] }
   func (r rec) Set(pos int, val any) { r[pos] = val }
   
   func TestYearPartitionPath(t *testing.T) {
        schema := iceberg.NewSchema(0,
                iceberg.NestedField{ID: 1, Name: "ts", Type: 
iceberg.PrimitiveTypes.Timestamp, Required: true})
        spec := iceberg.NewPartitionSpecID(1,
                iceberg.PartitionField{SourceIDs: []int{1}, FieldID: 1000, 
Transform: iceberg.YearTransform{}, Name: "ts_year"},
                iceberg.PartitionField{SourceIDs: []int{1}, FieldID: 1001, 
Transform: iceberg.MonthTransform{}, Name: "ts_month"},
                iceberg.PartitionField{SourceIDs: []int{1}, FieldID: 1002, 
Transform: iceberg.DayTransform{}, Name: "ts_day"},
                iceberg.PartitionField{SourceIDs: []int{1}, FieldID: 1003, 
Transform: iceberg.HourTransform{}, Name: "ts_hour"},
        )
        for _, tm := range []time.Time{
                time.Date(2017, 12, 1, 18, 0, 0, 0, time.UTC),
                time.Date(935, 7, 12, 0, 0, 0, 0, time.UTC),
                time.Date(1, 1, 1, 0, 0, 0, 0, time.UTC),
        } {
                lit := iceberg.NewLiteral(iceberg.Timestamp(tm.UnixMicro()))
                r := rec{}
                for _, f := range spec.Fields() {
                        r = append(r, 
f.Transform.Apply(iceberg.Optional[iceberg.Literal]{Valid: true, Val: 
lit}).Val.Any())
                }
                fmt.Println(tm.UnixMicro(), []any(r), spec.PartitionToPath(r, 
schema))
        }
   }
   ```
   
   Output on main:
   
   ```
   === RUN   TestYearPartitionPath
   1512151200000000 [47 575 17501 420042] 
ts_year=2017/ts_month=2017-12/ts_day=2017-12-01/ts_hour=2017-12-01-18
   -32644857600000000 [-1035 -12414 -377834 -9068016] 
ts_year=935/ts_month=0935-07/ts_day=0935-07-12/ts_hour=0935-07-12-00
   -62135596800000000 [-1969 -23628 -719162 -17259888] 
ts_year=1/ts_month=0001-01/ts_day=0001-01-01/ts_hour=0001-01-01-00
   --- PASS: TestYearPartitionPath (0.00s)
   ```
   
   PyIceberg 0.12.0 with the same schema, spec and timestamps 
(`spec.partition_to_path(Record(*vals), schema)`) computes the same partition 
ordinals but different year paths:
   
   ```
   1512151200000000 [47, 575, 17501, 420042] 
ts_year=2017/ts_month=2017-12/ts_day=2017-12-01/ts_hour=2017-12-01-18
   -32644857600000000 [-1035, -12414, -377834, -9068016] 
ts_year=0935/ts_month=0935-07/ts_day=0935-07-12/ts_hour=0935-07-12-00
   -62135596800000000 [-1969, -23628, -719162, -17259888] 
ts_year=0001/ts_month=0001-01/ts_day=0001-01-01/ts_hour=0001-01-01-00
   ```
   
   The 2017 row matches in both, and so do the month, day and hour values for 
every row. The only difference is the year value for years before 1000. The 
same thing shows up directly with 
`iceberg.YearTransform{}.ToHumanString(iceberg.PrimitiveTypes.Int32, 
int32(-1035))`, which returns `"935"`.
   
   The cause is `YearTransform.ToHumanString` in `transforms.go`, which returns 
`strconv.Itoa(int(v) + epochTM.Year())`. The fix is to format years below 1000 
with `%04d`, as Java does, and keep the `strconv.Itoa` path for four-digit 
years. I'd like to work on this and have a small fix and a test case ready to 
open as a PR once someone confirms.
   
   This issue was drafted with Claude Code (claude-opus-5-5); the results above 
are from the runs described.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to