StealthEyeLLC opened a new pull request, #23405: URL: https://github.com/apache/datafusion/pull/23405
## Which issue does this PR close? Closes #23127. ## Rationale for this change GROUP BY over dictionary-encoded columns could emit more distinct group values than the input dictionary key type can represent. The input dictionary key type bounds each input dictionary array, but not necessarily the aggregate output cardinality. For example, Dictionary(UInt8, Utf8) input batches can each be valid while the aggregate result contains more than 256 distinct groups. Re-encoding emitted group values with UInt8 keys can then fail with DictionaryKeyOverflowError. ## What changes are included in this PR? This keeps the original group schema for row conversion and interning, but uses a widened dictionary key type for emitted group-value arrays. The aggregate output schema uses the same widened dictionary type so emitted arrays remain compatible with RecordBatch construction. This also adds regression coverage for: - emitting 257 distinct Dictionary(UInt8, Utf8) group values - aggregate output schema compatibility for widened dictionary group keys ## Are these changes tested? Yes. ```bash cargo test -p datafusion-physical-plan dict_uint8_utf8_257_groups_emit_promotes_key_type -- --nocapture cargo test -p datafusion-physical-plan aggregate_output_schema_widens_dictionary_group_key_type -- --nocapture cargo test -p datafusion-physical-plan group_values -- --nocapture cargo test -p datafusion-physical-plan -- --nocapture ./dev/rust_lint.sh ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
