Rich-T-kid opened a new pull request, #27448: URL: https://github.com/apache/datafusion/pull/27448
## Which issue does this PR close?\n\n- Part of #20935.\n\n## Rationale for this change\n\nencode and decode currently materialize dictionary-encoded binary input and transform repeated values once per row. Low-cardinality inputs can instead transform each referenced dictionary value once and reuse the keys.\n\nThis is a draft performance experiment. It is gated on the existing hosted TPC-H, TPC-DS, and partitioned ClickBench suites and will be closed if they do not show an attributable speedup.\n\n## What changes are included in this PR?\n\n- Preserve dictionary encoding through encode and decode coercion.\n- Transform referenced dictionary values once and reuse input keys.\n- Ignore invalid unreferenced decode values while retaining errors for referenced invalid values.\n- Compact oversized retained dictionaries before transformation.\n- Add unit and SQL logic coverage for dictionary return types, nulls, invalid values, and compaction.\n\nNo benchmark files or custom benchmark q ueries are added or changed.\n\n## What is the testing strategy?\n\n- cargo test -p datafusion-functions encoding::inner::tests --lib\n- cargo test -p datafusion-sqllogictest --test sqllogictests -- encoding\n- cargo clippy -p datafusion-functions --lib --tests -- -D warnings\n- cargo fmt --all -- --check\n- Full repository pre-push hook\n- Hosted TPC-H, TPC-DS, and partitioned ClickBench runs on the exact PR head\n\n## Are there any user-facing changes?\n\nLogical values are unchanged. Dictionary input now returns dictionary output with the same key type. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
