Rich-T-kid opened a new pull request, #27448:
URL: https://github.com/apache/datafusion/pull/27448

   ## Which issue does this PR close?\n\n- Part of #20935.\n\n## Rationale for 
this change\n\nencode and decode currently materialize dictionary-encoded 
binary input and transform repeated values once per row. Low-cardinality inputs 
can instead transform each referenced dictionary value once and reuse the 
keys.\n\nThis is a draft performance experiment. It is gated on the existing 
hosted TPC-H, TPC-DS, and partitioned ClickBench suites and will be closed if 
they do not show an attributable speedup.\n\n## What changes are included in 
this PR?\n\n- Preserve dictionary encoding through encode and decode 
coercion.\n- Transform referenced dictionary values once and reuse input 
keys.\n- Ignore invalid unreferenced decode values while retaining errors for 
referenced invalid values.\n- Compact oversized retained dictionaries before 
transformation.\n- Add unit and SQL logic coverage for dictionary return types, 
nulls, invalid values, and compaction.\n\nNo benchmark files or custom 
benchmark q
 ueries are added or changed.\n\n## What is the testing strategy?\n\n- cargo 
test -p datafusion-functions encoding::inner::tests --lib\n- cargo test -p 
datafusion-sqllogictest --test sqllogictests -- encoding\n- cargo clippy -p 
datafusion-functions --lib --tests -- -D warnings\n- cargo fmt --all -- 
--check\n- Full repository pre-push hook\n- Hosted TPC-H, TPC-DS, and 
partitioned ClickBench runs on the exact PR head\n\n## Are there any 
user-facing changes?\n\nLogical values are unchanged. Dictionary input now 
returns dictionary output with the same key type.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to