JingsongLi commented on PR #10193: URL: https://github.com/apache/paimon/pull/10193#issuecomment-5951370510
[P1] Normalize nested TIMESTAMP(0) values before enabling Avro writes The new `s` branch in `PyarrowFieldParser.to_avro_type` also applies recursively to valid `ROW<ts TIMESTAMP(0)>` and `ARRAY<TIMESTAMP(0)>` table columns. Those values do not round-trip on a non-UTC host. `LocalFileIO.write_avro` attaches UTC only to top-level naive datetimes; nested datetimes remain naive and fastavro converts them using the host's timezone. I reproduced this with an actual Paimon Avro table, writing/committing/reopening it under `TZ=Asia/Shanghai`. With identical `2024-01-02 03:04:05` values in a top-level TIMESTAMP(0), a ROW child and an ARRAY element, the top-level value remains correct but both nested values read as `2024-01-01 19:04:05`. Epoch second -1 similarly becomes -28801 in the nested fields. Java's Paimon Avro reader also reads the shifted values, confirming they are persisted incorrectly rather than merely displayed differently by Python. The UTC-host control and canonical UTC TIMESTAMP_LTZ(0) control round-trip correctly. Nulls are preserved. On the base code, the `s` schema is rejected before writing. The analogous nested ms/us timezone bug already exists in the shared helper; this finding specifically concerns the newly accepted second-precision columns changing from explicit rejection to successful writes with corrupted timestamps. Please recursively normalize naive datetime values inside ROW/ARRAY structures before handing them to fastavro, and add an actual-table non-UTC timezone round-trip test for these newly supported types. The current test covers only a top-level datetime, which is already normalized. Validation: 39 Python data-type/nested-table tests and 18 Java Avro tests pass. Additional persisted-table controls cover negative epochs, nulls, ROW/ARRAY fields, naive/LTZ timestamps, and Shanghai/UTC hosts; Java reads and projection controls verify the Python-written files. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
