morningman opened a new issue, #67447: URL: https://github.com/apache/doris/issues/67447
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version master (`ddbaaab1388`) ### What's Wrong? Doris writes year-zero `DATETIME` values to Parquet correctly, but cannot read its own file back — the values come back `NULL`. `be/src/core/data_type_serde/parquet_timestamp.h` gates every Parquet timestamp read at year 1: ```cpp inline constexpr int64_t MIN_DORIS_TIMESTAMP_MICROS = -62135596800000000LL; // 0001-01-01 inline constexpr int64_t MAX_DORIS_TIMESTAMP_MICROS = 253402300799999999LL; // 9999-12-31 inline Status validate_parquet_timestamp_micros(int64_t timestamp_micros) { if (timestamp_micros < MIN_DORIS_TIMESTAMP_MICROS || timestamp_micros > MAX_DORIS_TIMESTAMP_MICROS) { return Status::DataQualityError( "Parquet timestamp is outside the Doris 0001-9999 range: micros={}", timestamp_micros); } return Status::OK(); } ``` But the DATETIME type itself starts at `0000-01-01`, whose micros value is `-62167219200000000` — below that bound. So the reader is stricter than the type it materialises into. ### What You Expected? A `DATETIME` value that Doris accepts, stores and exports should read back from Doris's own Parquet file. ### How to Reproduce? ```sql CREATE TABLE dt0 (id INT, ts DATETIME(6)) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1 PROPERTIES('replication_num'='1'); INSERT INTO dt0 VALUES (1,'0000-01-01 12:34:56'),(2,'0000-03-01 00:00:00'), (3,'1969-12-31 23:59:59'),(4,'2024-01-01 12:00:00'); -- the type stores them fine SELECT id, CAST(ts AS STRING) FROM dt0 ORDER BY id; -- 1 0000-01-01 12:34:56.000000 -- 2 0000-03-01 00:00:00.000000 -- 3 1969-12-31 23:59:59.000000 -- 4 2024-01-01 12:00:00.000000 SELECT * FROM dt0 ORDER BY id INTO OUTFILE 'file:///tmp/dtz/p_' FORMAT AS PARQUET; ``` Read the file back with the `local()` TVF: ``` id ts_read 1 NULL <-- lost 2 NULL <-- lost 3 1969-12-31 23:59:59.000000 4 2024-01-01 12:00:00.000000 ``` Note `0000-03-01` is also lost, so this is a plain range check, not a calendar edge case. ### Anything Else? **The write side is correct**: Spark reads the same year-zero timestamps out of a Doris-written Iceberg table without trouble (verified with `apache/spark:4.0.0` against the Iceberg REST catalog — Spark returned `0000-01-01 12:34:56` for the row Doris returned `NULL` for). So only Doris's reader rejects them. The constant name and the error text both say `0001-9999`, so the bound may be deliberate. If so, the asymmetry is still worth resolving in one direction or the other: - widen the reader to `0000-01-01` (`-62167219200000000`) to match the DATETIME range, or - reject year-zero timestamps on the write path too, so Doris never produces a file it cannot read. Found while running Iceberg regression suites for #67366. ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
