hcrosse opened a new issue, #2674:
URL: https://github.com/apache/iceberg-go/issues/2674

   ### Apache Iceberg version
   
   main (development)
   
   ### Please describe the bug 🐞
   
   `AddDataFilesByPath` accepts Parquet files storing a `timestamp` or 
`timestamptz` column as `TIMESTAMP(MILLIS)`, but records bounds and infers the 
`day` partition as if the values were microseconds. Filters on that column then 
silently return no rows. Projection is correct. Such files come from writers 
like parquet-go, which [uses 
`TIMESTAMP(MILLIS)`](https://github.com/parquet-go/parquet-go/blob/90bee42a64b670cb55f054a8b26e72381b825611/schema.go#L117)
 for a field tagged `timestamp` without a precision.
   
   I reproduced this on `main` and on v0.7.0, where the method is named 
`AddFiles`, with Go 1.26.8 on macOS arm64.
   
   The program below adds one file with three rows on 2024-01-01. Its bounds 
fall on 1970-01-20, the partition is day 19 instead of 19723, and `PlanFiles` 
returns no files for either filter.
   
   Reads fail even when the manifest has no bounds for `ts`. In separate tests, 
row group statistics, dictionary and bloom filters each skipped matching rows. 
A filter that none of them pruned failed with `invalid: referenced field 
%!s(<nil>) was timestamp[ms], but should have been timestamp[us]`. These checks 
read the file itself, so they would also affect MILLIS files that PyIceberg's 
`add_files` adds with correct bounds.
   
   https://github.com/apache/iceberg/issues/18173 reports the same statistics 
problem in Java, and https://github.com/apache/iceberg/pull/18321 proposes 
converting the statistics to microseconds.
   
   Equality deletes on `ts` fail too. In a separate test, I added a file with 
the same three rows to an unpartitioned table and used `CommitEqualityDeletes` 
to delete the row at 00:00:01.5. A scan with no filter still returned that row 
without an error, whether or not the file had statistics. The same delete on a 
microsecond file removed the row.
   
   **Cause**
   
   `convertToIceberg` maps milliseconds to the microsecond types, so the schema 
check passes:
   
   
https://github.com/apache/iceberg-go/blob/74b684cf085355bfddba0d2c74743bacd5d2eae3/table/arrow_utils.go#L400-L423
   
   `DataFileStatsFromMeta` copies raw min and max values because 
`wrapStatsForType` has no timestamp case:
   
   
https://github.com/apache/iceberg-go/blob/74b684cf085355bfddba0d2c74743bacd5d2eae3/table/internal/parquet_files.go#L1985-L2004
   
   
https://github.com/apache/iceberg-go/blob/74b684cf085355bfddba0d2c74743bacd5d2eae3/table/internal/parquet_files.go#L2037-L2064
   
   Row group pruning uses `parquetStatsBounds`, which converts decimals stored 
as integers since #2009 but passes timestamp statistics through unchanged:
   
   
https://github.com/apache/iceberg-go/blob/74b684cf085355bfddba0d2c74743bacd5d2eae3/table/evaluators.go#L985-L996
   
   **Reproduction**
   
   Save this as `go.mod` and the program as `main.go`, then run `go mod tidy && 
go run .`:
   
   ```go
   module repro
   
   go 1.26.8
   
   require (
        github.com/apache/arrow-go/v18 v18.8.0
        github.com/apache/iceberg-go v0.7.1-0.20261113211822-8d13e317c79f
        github.com/uptrace/bun/driver/sqliteshim v1.3.0
   )
   ```
   
   ```go
   package main
   
   import (
        "context"
        "encoding/binary"
        "fmt"
        "os"
        "path/filepath"
        "time"
   
        "github.com/apache/arrow-go/v18/arrow"
        "github.com/apache/arrow-go/v18/arrow/array"
        "github.com/apache/arrow-go/v18/arrow/memory"
        "github.com/apache/arrow-go/v18/parquet"
        "github.com/apache/arrow-go/v18/parquet/pqarrow"
        "github.com/apache/iceberg-go"
        "github.com/apache/iceberg-go/catalog"
        sqlcat "github.com/apache/iceberg-go/catalog/sql"
        "github.com/apache/iceberg-go/table"
        "github.com/uptrace/bun/driver/sqliteshim"
   )
   
   // The unit the Parquet file stores `ts` in. The table field is `timestamp` 
(microseconds).
   const unit = arrow.Millisecond
   
   var instants = []time.Time{
        time.Date(2024, 1, 1, 0, 0, 0, 0, time.UTC),
        time.Date(2024, 1, 1, 0, 0, 1, 500_000_000, time.UTC),
        time.Date(2024, 1, 1, 12, 0, 0, 0, time.UTC),
   }
   
   func check(name, want, got string) {
        status := "OK      "
        if want != got {
                status = "MISMATCH"
        }
        fmt.Printf("%s %-42s expected=%-46s actual=%s\n", status, name, want, 
got)
   }
   
   func must[T any](v T, err error) T {
        if err != nil {
                panic(err)
        }
   
        return v
   }
   
   func main() {
        ctx := context.Background()
        dir := must(os.MkdirTemp("", "repro"))
        defer os.RemoveAll(dir)
   
        cat := must(catalog.Load(ctx, "default", iceberg.Properties{
                "type": "sql", "uri": "file://" + filepath.Join(dir, 
"catalog.db"),
                sqlcat.DriverKey: sqliteshim.ShimName, sqlcat.DialectKey: 
string(sqlcat.SQLite),
                "warehouse": "file://" + dir,
        }))
        must(0, cat.CreateNamespace(ctx, table.Identifier{"db"}, nil))
        spec := iceberg.NewPartitionSpec(iceberg.PartitionField{
                SourceIDs: []int{2}, FieldID: 1000, Name: "ts_day", Transform: 
iceberg.DayTransform{}})
        tbl := must(cat.CreateTable(ctx, table.Identifier{"db", "t"}, 
iceberg.NewSchema(0,
                iceberg.NestedField{ID: 1, Name: "id", Type: 
iceberg.PrimitiveTypes.Int64},
                iceberg.NestedField{ID: 2, Name: "ts", Type: 
iceberg.PrimitiveTypes.Timestamp}),
                catalog.WithPartitionSpec(&spec),
                catalog.WithProperties(iceberg.Properties{"format-version": 
"2"})))
   
        // One row group, statistics on, field IDs set, ts stored as 
TIMESTAMP(unit).
        fid := func(id string) arrow.Metadata { return 
arrow.MetadataFrom(map[string]string{"PARQUET:field_id": id}) }
        sc := arrow.NewSchema([]arrow.Field{
                {Name: "id", Type: arrow.PrimitiveTypes.Int64, Nullable: true, 
Metadata: fid("1")},
                {Name: "ts", Type: &arrow.TimestampType{Unit: unit}, Nullable: 
true, Metadata: fid("2")}}, nil)
        ids := array.NewInt64Builder(memory.DefaultAllocator)
        tss := array.NewTimestampBuilder(memory.DefaultAllocator, 
&arrow.TimestampType{Unit: unit})
        for i, t := range instants {
                ids.Append(int64(i))
                tss.Append(arrow.Timestamp(t.UnixNano() / 
int64(unit.Multiplier())))
        }
        rec := array.NewRecordBatch(sc, []arrow.Array{ids.NewArray(), 
tss.NewArray()}, int64(len(instants)))
        path := filepath.Join(dir, "data.parquet")
        f := must(os.Create(path))
        must(0, pqarrow.WriteTable(array.NewTableFromRecords(sc, 
[]arrow.RecordBatch{rec}), f, 100,
                parquet.NewWriterProperties(parquet.WithStats(true)), 
pqarrow.DefaultWriterProps()))
        f.Close()
   
        tx := tbl.NewTransaction()
        must(0, tx.AddDataFilesByPath(ctx, []string{path}, nil, false))
        tbl = must(tx.Commit(ctx))
   
        fmt.Printf("Parquet file stores ts as TIMESTAMP(%v); table field is 
timestamp (us)\n", unit)
        tasks := must(tbl.Scan().PlanFiles(ctx))
        df := tasks[0].File
        fmtTS := func(b []byte) string {
                return 
time.UnixMicro(int64(binary.LittleEndian.Uint64(b))).UTC().Format(time.RFC3339)
        }
        check("manifest lower bound of ts", instants[0].Format(time.RFC3339), 
fmtTS(df.LowerBoundValues()[2]))
        check("manifest upper bound of ts", instants[2].Format(time.RFC3339), 
fmtTS(df.UpperBoundValues()[2]))
        day := fmt.Sprint(df.Partition()[1000])
        check("partition ts_day (days since epoch)", 
fmt.Sprint(instants[0].Unix()/86400), day)
   
        scan := func(name, wantRows string, wantFiles int, opts 
...table.ScanOption) {
                s := tbl.Scan(append(opts, table.WithSelectedFields("id", 
"ts"))...)
                files := must(s.PlanFiles(ctx))
                out, err := s.ToArrowTable(ctx)
                if err != nil {
                        check(name, wantRows, "error: "+err.Error())
                        return
                }
                var got []string
                for _, ch := range out.Column(1).Data().Chunks() {
                        for _, v := range 
ch.(*array.Timestamp).TimestampValues() {
                                got = append(got, 
v.ToTime(arrow.Microsecond).Format("15:04:05.000"))
                        }
                }
                check(name+": rows", wantRows, fmt.Sprintf("%d %v", 
out.NumRows(), got))
                check(name+": PlanFiles", fmt.Sprint(wantFiles), 
fmt.Sprint(len(files)))
        }
        all := "3 [00:00:00.000 00:00:01.500 12:00:00.000]"
        scan("no filter", all, 1)
        eq := iceberg.EqualTo(iceberg.Reference("ts"), 
iceberg.Timestamp(instants[1].UnixMicro()))
        scan("ts == 2024-01-01T00:00:01.5Z", "1 [00:00:01.500]", 1, 
table.WithRowFilter(eq))
        ge := iceberg.GreaterThanEqual(iceberg.Reference("ts"), 
iceberg.Timestamp(instants[0].UnixMicro()))
        scan("ts >= 2024-01-01T00:00:00Z", all, 1, table.WithRowFilter(ge))
   }
   ```
   
   Output:
   
   ```text
   $ go mod tidy && go run .   # iceberg-go 
8d13e317c79f36f38f52a16b2f81f1d966f1fc27 (main), go1.26.8 darwin/arm64
   Parquet file stores ts as TIMESTAMP(ms); table field is timestamp (us)
   MISMATCH manifest lower bound of ts                 
expected=2024-01-01T00:00:00Z                           
actual=1970-01-20T17:21:07Z
   MISMATCH manifest upper bound of ts                 
expected=2024-01-01T12:00:00Z                           
actual=1970-01-20T17:21:50Z
   MISMATCH partition ts_day (days since epoch)        expected=19723           
                               actual=19
   OK       no filter: rows                            expected=3 [00:00:00.000 
00:00:01.500 12:00:00.000]     actual=3 [00:00:00.000 00:00:01.500 12:00:00.000]
   OK       no filter: PlanFiles                       expected=1               
                               actual=1
   MISMATCH ts == 2024-01-01T00:00:01.5Z: rows         expected=1 
[00:00:01.500]                               actual=0 []
   MISMATCH ts == 2024-01-01T00:00:01.5Z: PlanFiles    expected=1               
                               actual=0
   MISMATCH ts >= 2024-01-01T00:00:00Z: rows           expected=3 [00:00:00.000 
00:00:01.500 12:00:00.000]     actual=0 []
   MISMATCH ts >= 2024-01-01T00:00:00Z: PlanFiles      expected=1               
                               actual=0
   ```
   
   **Expected behavior**
   
   Filters on `ts` return the matching rows whether or not the file has 
statistics, a dictionary or a bloom filter. Equality deletes on `ts` remove the 
matching rows. `AddDataFilesByPath` records manifest bounds in microseconds and 
infers partition values from those bounds.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to