Zoltán Borók-Nagy created IMPALA-15672:
------------------------------------------
Summary: Broadcast join cost truncates
broadcast_to_partition_factor * sqrt(dop) to a long, under-costing broadcasts
at MT_DOP > 1
Key: IMPALA-15672
URL: https://issues.apache.org/jira/browse/IMPALA-15672
Project: IMPALA
Issue Type: Bug
Components: Frontend
Reporter: Zoltán Borók-Nagy
With mt_dop > 1 and USE_DOP_FOR_COSTING, DistributedPlanner multiplies the hash
table build cost of a broadcast join by broadcast_to_partition_factor *
sqrt(dop), to model that one thread per node builds the broadcast table while a
partitioned join builds with dop threads (master 1a26e1f778):
{code:java}
hashTblBuildCost *= (long) (ctx_.getQueryOptions().broadcast_to_partition_factor
* Math.max(1.0, Math.sqrt(actual_dop)));
{code}
The factor is cast to long before the multiplication. With the default factor 1
and MT_DOP=8, sqrt(8) = 2.83 becomes 2, so the build cost term is 29% lower
than the cost model intends. A product below 1 (e.g. a factor below 1 at low
dop) becomes 0, which makes the build free.
*Fix:* multiply in double and round the product, e.g.
Math.round(hashTblBuildCost * factor * sqrt(dop)).
Related data from TPC-DS SF1000 on 8 executors (MT_DOP=8, PLANNER=CALCITE): a
shared broadcast build really is single-threaded per host (IMPALA-15631), so
the sqrt model is generous. Sweeping BROADCAST_TO_PARTITION_FACTOR on the 30
queries whose plans it changes gave -16.3%, -18.1% and -18.8% for 1.5, 2 and 4;
with the truncation, 4 x floor(sqrt(8)) = 8 = dop. On top of the fixes in
IMPALA-?, IMPALA-? and IMPALA-15631, factor 4 was worth another -2.2% (q29,
q15, q6, q38, q87). The fix alone was not measured; it is equivalent to a
factor of about 1.4.
Prototype: https://gerrit.cloudera.org/#/c/25086/
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
