Doris-Breakwater commented on issue #66486: URL: https://github.com/apache/doris/issues/66486#issuecomment-5190501963
Initial triage: this looks like a **4.1.0 reader compatibility issue that is probably already fixed**, while the proposed statistics write-sanitization issue is not yet demonstrated by the supplied evidence. Verified from the 4.1.0 code: - `__internal_schema.column_statistics.min` and `max` are `VARCHAR(65533)`. The ANALYZE SQL explicitly casts `MIN()`/`MAX()` to `STRING` before persisting them. Therefore, a value such as `0000-00-00 00:00:00` in either field is stored as text; a plain scan of those two fields should not call the BE date SerDe. - FE converts those strings back according to the source column type in `ColumnStatistic.fromResultRow()`. An `AnalysisException` while parsing a date min/max is caught and degraded to an unbounded min/max. This is an FE statistics-deserialization path, not the BE storage-reader initialization path shown in the report. - In 4.1.0, date/datetime `from_olap_string()` failures are propagated through zone-map deserialization and can abort segment/rowset reader initialization with the reported `parse date or datetime fail` status. [#63035](https://github.com/apache/doris/pull/63035), backported to branch 4.1 by [#63073](https://github.com/apache/doris/pull/63073), changed DATE/DATETIME/DATEV2/DATETIMEV2 storage-string parse failures to a safe type default. Its code comment specifically covers invalid/zero-date placeholders in zone maps. The backport is absent from tag 4.1.0 and present in 4.1.1 and later 4.1 tags. - [#63010](https://github.com/apache/doris/pull/63010), also present starting with 4.1.1, prevents the optimizer from requesting column statistics for system tables such as tables in `__internal_schema`. Consequently, the current stack is consistent with the known BE storage-string/zone-map issue, but it does not prove that the text in `column_statistics.min/max` itself makes that table unreadable. The failing storage field could instead be a DATE/DATETIME zone-map or default-value entry. The truncated stack is not sufficient to distinguish those paths. Suggested next steps: 1. Clean the already affected statistics state, then reproduce on 4.1.1 or a newer 4.1 release. If the reader failure disappears, this can likely be closed as fixed by #63073 (with #63010 also avoiding recursive/system-table CBO lookup). 2. If it still reproduces, please provide: - a minimal schema/load/ANALYZE/query SQL reproduction, including the exact failing SQL; - exact FE and BE build versions/commit hashes; - the complete BE error and stack, especially frames mentioning `ZoneMap::from_proto`, `from_zonemap_string`, `from_fe_string`, or `DefaultValueColumnIterator`; - the query ID plus tablet, rowset, and segment IDs from the BE log; - `allow_zero_date` from every BE and whether it changed after the data was written; - the business column type and load method/options that admitted the zero date; - evidence identifying whether `0000-00-00 00:00:00` is only in the VARCHAR `min`/`max` fields or also in a DATE/DATETIME storage zone map/default. 3. If a current 4.1 build reads the table successfully but still persists an unusable min/max statistic, treat that as a separate statistics-correctness issue. A regression should cover importing the value, running ANALYZE, querying `column_statistics`, and loading the resulting CBO statistic. Write-side sanitization should be based on the source column type; silently ignoring arbitrary parse failures in the general BE reader would be too broad. For mitigation, prefer dropping statistics for only the affected business table once the internal table is readable. Truncating `column_statistics` removes statistics for every table and should remain a last-resort recovery step. Breakwater-GitHub-Analysis-Slot: slot_a2bcc2b14fc1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
