Dandandan opened a new pull request, #24558:
URL: https://github.com/apache/datafusion/pull/24558
## Which issue does this PR close?
None; split out of #24456 so the default change can be reviewed on its own.
## Rationale for this change
`JoinSelection` runs a hash join as `CollectLeft` -- gathering one side into
a single
partition -- when that side's estimated size stays under
`datafusion.optimizer.hash_join_single_partition_threshold`, and
repartitions both inputs
otherwise. That bound has been 1MB.
1MB is small for the choice it governs. Collecting a build side of a few MB
copies it
once; the alternative hashes and exchanges *both* inputs and then builds one
hash table
per partition. Raising the threshold to 4MB lets joins against a small side
skip that
exchange, which is the common shape of a fact table joined to a dimension
table.
The row-count threshold (`hash_join_single_partition_threshold_rows`, 128k)
is unchanged,
so this only affects plans where a byte estimate is available.
## What changes are included in this PR?
- `hash_join_single_partition_threshold` defaults to 4MB instead of 1MB, and
the
generated `configs.md` row follows.
- The two sqllogictest files that print or restore the value are updated.
- The `join_selection` tests build their inputs from the threshold but
asserted the byte
sizes the 1MB default happened to produce (`8192` / `2097152`). They now
derive those
expectations from the configured threshold, so the assertions stay correct
wherever the
default sits.
## Are these changes tested?
Yes, by existing coverage: the full sqllogictest suite (504 files) passes,
as do the
`datafusion`, `datafusion-common` and `datafusion-physical-optimizer` unit
and
integration tests. No new test is added -- the change is a default, and the
tests that
pin plan shapes around it already exist.
## Are there any user-facing changes?
Yes: a changed default, so a query whose build side is between 1MB and 4MB
now plans as
`CollectLeft` where it previously repartitioned. `configs.md` documents the
new value,
and anyone relying on the old behaviour can set the option back to `1048576`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
