jackylee-ch commented on PR #25694: URL: https://github.com/apache/datafusion/pull/25694#issuecomment-6302219750
I've updated the description; does it answer your questions? Your guess is right: a clear planning error instead of a panic or a wrong answer. In short: 1. Scenario: a table provider's scan reports statistics for the wrong number of columns. In anvai-labs/proximaDB#76, the scan fell back to `Statistics::default()`, which has no column entries; the description has two more cases. 2. Today: rules and operators read column statistics by position, and nothing checks the count when a plan reports them, so a mismatch surfaces wherever they are first read. DataFusion 54's `AggregateExec` indexes its input's column statistics, so after upgrading, proximaDB's aggregate queries panicked there (`index out of bounds: the len is 0 but the index is 0`), with nothing pointing at the scan. With too many entries, `min`/`max` can return another column's values. 3. Proposed: check the count whenever statistics are reported for a plan node, and fail planning with an error naming that node, e.g. `MyExec returned 1 column statistics, expected 2 (one per field of its schema)`. Queries over such a scan then fail at planning, even `SELECT *`, which works today; the upgrade guide shows the fix. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
