0z5a opened a new issue, #2603:
URL: https://github.com/apache/iceberg-go/issues/2603
### Apache Iceberg version
main (development)
### Please describe the bug 🐞
`YearTransform.ToHumanString` formats the year with `strconv.Itoa`, so years
before 1000 come out without leading zeros. A row with `ts = 0935-07-12` lands
in `ts_year=935/`, and the common sentinel date `0001-01-01` lands in
`ts_year=1/`. The month, day and hour transforms in the same package already
pad to four digits (`0935-07`, `0935-07-12`, `0935-07-12-00`), so only the year
value is off.
Java (`TransformUtil.humanYear` uses `String.format("%04d", ...)`) and
PyIceberg (`to_human_year` uses `{:0=4d}`) both produce `0935` and `0001`. As a
result, iceberg-go writes year-partitioned files for such rows into a different
directory than Java/Spark or PyIceberg would for the same partition value, and
the human-readable value differs from what the other implementations show.
Tested on main at 7cf76392bbf28bff03bccd1917efebfd5824e4bd (Go 1.27.1, macOS
arm64).
Repro: save this as `year_partition_path_test.go` in the repository root and
run `go test -v -run TestYearPartitionPath .`
```go
package iceberg_test
import (
"fmt"
"testing"
"time"
"github.com/apache/iceberg-go"
)
type rec []any
func (r rec) Size() int { return len(r) }
func (r rec) Get(pos int) any { return r[pos] }
func (r rec) Set(pos int, val any) { r[pos] = val }
func TestYearPartitionPath(t *testing.T) {
schema := iceberg.NewSchema(0,
iceberg.NestedField{ID: 1, Name: "ts", Type:
iceberg.PrimitiveTypes.Timestamp, Required: true})
spec := iceberg.NewPartitionSpecID(1,
iceberg.PartitionField{SourceIDs: []int{1}, FieldID: 1000,
Transform: iceberg.YearTransform{}, Name: "ts_year"},
iceberg.PartitionField{SourceIDs: []int{1}, FieldID: 1001,
Transform: iceberg.MonthTransform{}, Name: "ts_month"},
iceberg.PartitionField{SourceIDs: []int{1}, FieldID: 1002,
Transform: iceberg.DayTransform{}, Name: "ts_day"},
iceberg.PartitionField{SourceIDs: []int{1}, FieldID: 1003,
Transform: iceberg.HourTransform{}, Name: "ts_hour"},
)
for _, tm := range []time.Time{
time.Date(2017, 12, 1, 18, 0, 0, 0, time.UTC),
time.Date(935, 7, 12, 0, 0, 0, 0, time.UTC),
time.Date(1, 1, 1, 0, 0, 0, 0, time.UTC),
} {
lit := iceberg.NewLiteral(iceberg.Timestamp(tm.UnixMicro()))
r := rec{}
for _, f := range spec.Fields() {
r = append(r,
f.Transform.Apply(iceberg.Optional[iceberg.Literal]{Valid: true, Val:
lit}).Val.Any())
}
fmt.Println(tm.UnixMicro(), []any(r), spec.PartitionToPath(r,
schema))
}
}
```
Output on main:
```
=== RUN TestYearPartitionPath
1512151200000000 [47 575 17501 420042]
ts_year=2017/ts_month=2017-12/ts_day=2017-12-01/ts_hour=2017-12-01-18
-32644857600000000 [-1035 -12414 -377834 -9068016]
ts_year=935/ts_month=0935-07/ts_day=0935-07-12/ts_hour=0935-07-12-00
-62135596800000000 [-1969 -23628 -719162 -17259888]
ts_year=1/ts_month=0001-01/ts_day=0001-01-01/ts_hour=0001-01-01-00
--- PASS: TestYearPartitionPath (0.00s)
```
PyIceberg 0.12.0 with the same schema, spec and timestamps
(`spec.partition_to_path(Record(*vals), schema)`) computes the same partition
ordinals but different year paths:
```
1512151200000000 [47, 575, 17501, 420042]
ts_year=2017/ts_month=2017-12/ts_day=2017-12-01/ts_hour=2017-12-01-18
-32644857600000000 [-1035, -12414, -377834, -9068016]
ts_year=0935/ts_month=0935-07/ts_day=0935-07-12/ts_hour=0935-07-12-00
-62135596800000000 [-1969, -23628, -719162, -17259888]
ts_year=0001/ts_month=0001-01/ts_day=0001-01-01/ts_hour=0001-01-01-00
```
The 2017 row matches in both, and so do the month, day and hour values for
every row. The only difference is the year value for years before 1000. The
same thing shows up directly with
`iceberg.YearTransform{}.ToHumanString(iceberg.PrimitiveTypes.Int32,
int32(-1035))`, which returns `"935"`.
The cause is `YearTransform.ToHumanString` in `transforms.go`, which returns
`strconv.Itoa(int(v) + epochTM.Year())`. The fix is to format years below 1000
with `%04d`, as Java does, and keep the `strconv.Itoa` path for four-digit
years. I'd like to work on this and have a small fix and a test case ready to
open as a PR once someone confirms.
This issue was drafted with Claude Code (claude-opus-5-5); the results above
are from the runs described.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]