StealthEyeLLC commented on PR #23405: URL: https://github.com/apache/datafusion/pull/23405#issuecomment-4938296305
Thanks for the review, @Rich-T-kid. I reworked the implementation around dynamic, monotonic key promotion rather than widening every dictionary key to UInt64. The current behavior is: * emission begins with the planned dictionary key type * the key grows one width only when Arrow reports DictionaryKeyOverflowError * UInt8 remains UInt8 through 256 distinct values, and 257 values promote it to UInt16 * once promoted, the key type does not shrink on later emissions * top-level dictionary values are materialized for row interning so aggregate stages can accept promoted dictionary keys * partial aggregate batches remain schema-stable across repartition/coalescing by being sliced at the planned key capacity and compactly re-encoded * final, ordered-final, and legacy streams update their advertised schema when runtime output promotes * spill IPC uses a stable internal dictionary width I added coverage for the promotion boundary, signed keys, no-shrink behavior, partial-to-final aggregation, the legacy path, and a repartitioned SQL regression. The current source passes the targeted Rust tests, SQLLogicTest, formatting, and clippy with warnings denied. The Apache workflow runs are still waiting for maintainer approval. Could you please take another look and approve the workflows when convenient? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
