[
https://issues.apache.org/jira/browse/IMPALA-13260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Csaba Ringhofer updated IMPALA-13260:
-------------------------------------
Description:
Currently NULL keys are hashed to a single value and sent to a single fragment
instance in partitioned joins. This can cause data skew if the number of NULL
keys is large.
If a NULL key guarantees that no row is matched on the build side, then columns
from build side will be all NULL and it doesn't matter which fragment instance
processes the row.
Always sending rows with NULL key to a local fragment instance would both
reduce data skew and make the shuffle cheaper (no compression/network). If
mt_dop>0 then to completely avoid data these rows would need to be spread
evenly among the local fragment instances.
One caveat is that sending NULL keys locally would "weaken" the partitioning of
the fragment, so it is no longer "partitioned by col", but "partitioned by col
(with the exception of NULLs)". For example if the outer join is followed by a
grouping aggregation that uses the same key, then a shuffle is still needed as
the aggregation needs all NULL keys in the same fragment instance.
was:
Currently NULL keys are hashed to a single value and sent to a single fragment
instance in partitioned joins. This can cause data skew if the number of NULL
keys is large.
If a NULL key guarantees that no row is matched on the build side, then columns
from build side will be all NULL and it doesn't matter which fragment instance
processes the row.
Always sending rows with NULL key to a local fragment instance would both
reduce data skew and make the shuffle cheaper (no compression/network). If
mt_dop>0 then to completely avoid data these rows would need to be spread
evenly among the local fragment instances.
> Exchange on the probe side of outer joins could send NULL keys to local target
> ------------------------------------------------------------------------------
>
> Key: IMPALA-13260
> URL: https://issues.apache.org/jira/browse/IMPALA-13260
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Reporter: Csaba Ringhofer
> Priority: Major
>
> Currently NULL keys are hashed to a single value and sent to a single
> fragment instance in partitioned joins. This can cause data skew if the
> number of NULL keys is large.
> If a NULL key guarantees that no row is matched on the build side, then
> columns from build side will be all NULL and it doesn't matter which fragment
> instance processes the row.
> Always sending rows with NULL key to a local fragment instance would both
> reduce data skew and make the shuffle cheaper (no compression/network). If
> mt_dop>0 then to completely avoid data these rows would need to be spread
> evenly among the local fragment instances.
> One caveat is that sending NULL keys locally would "weaken" the partitioning
> of the fragment, so it is no longer "partitioned by col", but "partitioned by
> col (with the exception of NULLs)". For example if the outer join is followed
> by a grouping aggregation that uses the same key, then a shuffle is still
> needed as the aggregation needs all NULL keys in the same fragment instance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
