hcrosse opened a new issue, #2674: URL: https://github.com/apache/iceberg-go/issues/2674
### Apache Iceberg version main (development) ### Please describe the bug 🐞 `AddDataFilesByPath` accepts Parquet files storing a `timestamp` or `timestamptz` column as `TIMESTAMP(MILLIS)`, but records bounds and infers the `day` partition as if the values were microseconds. Filters on that column then silently return no rows. Projection is correct. Such files come from writers like parquet-go, which [uses `TIMESTAMP(MILLIS)`](https://github.com/parquet-go/parquet-go/blob/90bee42a64b670cb55f054a8b26e72381b825611/schema.go#L117) for a field tagged `timestamp` without a precision. I reproduced this on `main` and on v0.7.0, where the method is named `AddFiles`, with Go 1.26.8 on macOS arm64. The program below adds one file with three rows on 2024-01-01. Its bounds fall on 1970-01-20, the partition is day 19 instead of 19723, and `PlanFiles` returns no files for either filter. Reads fail even when the manifest has no bounds for `ts`. In separate tests, row group statistics, dictionary and bloom filters each skipped matching rows. A filter that none of them pruned failed with `invalid: referenced field %!s(<nil>) was timestamp[ms], but should have been timestamp[us]`. These checks read the file itself, so they would also affect MILLIS files that PyIceberg's `add_files` adds with correct bounds. https://github.com/apache/iceberg/issues/18173 reports the same statistics problem in Java, and https://github.com/apache/iceberg/pull/18321 proposes converting the statistics to microseconds. Equality deletes on `ts` fail too. In a separate test, I added a file with the same three rows to an unpartitioned table and used `CommitEqualityDeletes` to delete the row at 00:00:01.5. A scan with no filter still returned that row without an error, whether or not the file had statistics. The same delete on a microsecond file removed the row. **Cause** `convertToIceberg` maps milliseconds to the microsecond types, so the schema check passes: https://github.com/apache/iceberg-go/blob/74b684cf085355bfddba0d2c74743bacd5d2eae3/table/arrow_utils.go#L400-L423 `DataFileStatsFromMeta` copies raw min and max values because `wrapStatsForType` has no timestamp case: https://github.com/apache/iceberg-go/blob/74b684cf085355bfddba0d2c74743bacd5d2eae3/table/internal/parquet_files.go#L1985-L2004 https://github.com/apache/iceberg-go/blob/74b684cf085355bfddba0d2c74743bacd5d2eae3/table/internal/parquet_files.go#L2037-L2064 Row group pruning uses `parquetStatsBounds`, which converts decimals stored as integers since #2009 but passes timestamp statistics through unchanged: https://github.com/apache/iceberg-go/blob/74b684cf085355bfddba0d2c74743bacd5d2eae3/table/evaluators.go#L985-L996 **Reproduction** Save this as `go.mod` and the program as `main.go`, then run `go mod tidy && go run .`: ```go module repro go 1.26.8 require ( github.com/apache/arrow-go/v18 v18.8.0 github.com/apache/iceberg-go v0.7.1-0.20261113211822-8d13e317c79f github.com/uptrace/bun/driver/sqliteshim v1.3.0 ) ``` ```go package main import ( "context" "encoding/binary" "fmt" "os" "path/filepath" "time" "github.com/apache/arrow-go/v18/arrow" "github.com/apache/arrow-go/v18/arrow/array" "github.com/apache/arrow-go/v18/arrow/memory" "github.com/apache/arrow-go/v18/parquet" "github.com/apache/arrow-go/v18/parquet/pqarrow" "github.com/apache/iceberg-go" "github.com/apache/iceberg-go/catalog" sqlcat "github.com/apache/iceberg-go/catalog/sql" "github.com/apache/iceberg-go/table" "github.com/uptrace/bun/driver/sqliteshim" ) // The unit the Parquet file stores `ts` in. The table field is `timestamp` (microseconds). const unit = arrow.Millisecond var instants = []time.Time{ time.Date(2024, 1, 1, 0, 0, 0, 0, time.UTC), time.Date(2024, 1, 1, 0, 0, 1, 500_000_000, time.UTC), time.Date(2024, 1, 1, 12, 0, 0, 0, time.UTC), } func check(name, want, got string) { status := "OK " if want != got { status = "MISMATCH" } fmt.Printf("%s %-42s expected=%-46s actual=%s\n", status, name, want, got) } func must[T any](v T, err error) T { if err != nil { panic(err) } return v } func main() { ctx := context.Background() dir := must(os.MkdirTemp("", "repro")) defer os.RemoveAll(dir) cat := must(catalog.Load(ctx, "default", iceberg.Properties{ "type": "sql", "uri": "file://" + filepath.Join(dir, "catalog.db"), sqlcat.DriverKey: sqliteshim.ShimName, sqlcat.DialectKey: string(sqlcat.SQLite), "warehouse": "file://" + dir, })) must(0, cat.CreateNamespace(ctx, table.Identifier{"db"}, nil)) spec := iceberg.NewPartitionSpec(iceberg.PartitionField{ SourceIDs: []int{2}, FieldID: 1000, Name: "ts_day", Transform: iceberg.DayTransform{}}) tbl := must(cat.CreateTable(ctx, table.Identifier{"db", "t"}, iceberg.NewSchema(0, iceberg.NestedField{ID: 1, Name: "id", Type: iceberg.PrimitiveTypes.Int64}, iceberg.NestedField{ID: 2, Name: "ts", Type: iceberg.PrimitiveTypes.Timestamp}), catalog.WithPartitionSpec(&spec), catalog.WithProperties(iceberg.Properties{"format-version": "2"}))) // One row group, statistics on, field IDs set, ts stored as TIMESTAMP(unit). fid := func(id string) arrow.Metadata { return arrow.MetadataFrom(map[string]string{"PARQUET:field_id": id}) } sc := arrow.NewSchema([]arrow.Field{ {Name: "id", Type: arrow.PrimitiveTypes.Int64, Nullable: true, Metadata: fid("1")}, {Name: "ts", Type: &arrow.TimestampType{Unit: unit}, Nullable: true, Metadata: fid("2")}}, nil) ids := array.NewInt64Builder(memory.DefaultAllocator) tss := array.NewTimestampBuilder(memory.DefaultAllocator, &arrow.TimestampType{Unit: unit}) for i, t := range instants { ids.Append(int64(i)) tss.Append(arrow.Timestamp(t.UnixNano() / int64(unit.Multiplier()))) } rec := array.NewRecordBatch(sc, []arrow.Array{ids.NewArray(), tss.NewArray()}, int64(len(instants))) path := filepath.Join(dir, "data.parquet") f := must(os.Create(path)) must(0, pqarrow.WriteTable(array.NewTableFromRecords(sc, []arrow.RecordBatch{rec}), f, 100, parquet.NewWriterProperties(parquet.WithStats(true)), pqarrow.DefaultWriterProps())) f.Close() tx := tbl.NewTransaction() must(0, tx.AddDataFilesByPath(ctx, []string{path}, nil, false)) tbl = must(tx.Commit(ctx)) fmt.Printf("Parquet file stores ts as TIMESTAMP(%v); table field is timestamp (us)\n", unit) tasks := must(tbl.Scan().PlanFiles(ctx)) df := tasks[0].File fmtTS := func(b []byte) string { return time.UnixMicro(int64(binary.LittleEndian.Uint64(b))).UTC().Format(time.RFC3339) } check("manifest lower bound of ts", instants[0].Format(time.RFC3339), fmtTS(df.LowerBoundValues()[2])) check("manifest upper bound of ts", instants[2].Format(time.RFC3339), fmtTS(df.UpperBoundValues()[2])) day := fmt.Sprint(df.Partition()[1000]) check("partition ts_day (days since epoch)", fmt.Sprint(instants[0].Unix()/86400), day) scan := func(name, wantRows string, wantFiles int, opts ...table.ScanOption) { s := tbl.Scan(append(opts, table.WithSelectedFields("id", "ts"))...) files := must(s.PlanFiles(ctx)) out, err := s.ToArrowTable(ctx) if err != nil { check(name, wantRows, "error: "+err.Error()) return } var got []string for _, ch := range out.Column(1).Data().Chunks() { for _, v := range ch.(*array.Timestamp).TimestampValues() { got = append(got, v.ToTime(arrow.Microsecond).Format("15:04:05.000")) } } check(name+": rows", wantRows, fmt.Sprintf("%d %v", out.NumRows(), got)) check(name+": PlanFiles", fmt.Sprint(wantFiles), fmt.Sprint(len(files))) } all := "3 [00:00:00.000 00:00:01.500 12:00:00.000]" scan("no filter", all, 1) eq := iceberg.EqualTo(iceberg.Reference("ts"), iceberg.Timestamp(instants[1].UnixMicro())) scan("ts == 2024-01-01T00:00:01.5Z", "1 [00:00:01.500]", 1, table.WithRowFilter(eq)) ge := iceberg.GreaterThanEqual(iceberg.Reference("ts"), iceberg.Timestamp(instants[0].UnixMicro())) scan("ts >= 2024-01-01T00:00:00Z", all, 1, table.WithRowFilter(ge)) } ``` Output: ```text $ go mod tidy && go run . # iceberg-go 8d13e317c79f36f38f52a16b2f81f1d966f1fc27 (main), go1.26.8 darwin/arm64 Parquet file stores ts as TIMESTAMP(ms); table field is timestamp (us) MISMATCH manifest lower bound of ts expected=2024-01-01T00:00:00Z actual=1970-01-20T17:21:07Z MISMATCH manifest upper bound of ts expected=2024-01-01T12:00:00Z actual=1970-01-20T17:21:50Z MISMATCH partition ts_day (days since epoch) expected=19723 actual=19 OK no filter: rows expected=3 [00:00:00.000 00:00:01.500 12:00:00.000] actual=3 [00:00:00.000 00:00:01.500 12:00:00.000] OK no filter: PlanFiles expected=1 actual=1 MISMATCH ts == 2024-01-01T00:00:01.5Z: rows expected=1 [00:00:01.500] actual=0 [] MISMATCH ts == 2024-01-01T00:00:01.5Z: PlanFiles expected=1 actual=0 MISMATCH ts >= 2024-01-01T00:00:00Z: rows expected=3 [00:00:00.000 00:00:01.500 12:00:00.000] actual=0 [] MISMATCH ts >= 2024-01-01T00:00:00Z: PlanFiles expected=1 actual=0 ``` **Expected behavior** Filters on `ts` return the matching rows whether or not the file has statistics, a dictionary or a bloom filter. Equality deletes on `ts` remove the matching rows. `AddDataFilesByPath` records manifest bounds in microseconds and infers partition values from those bounds. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
