[
https://issues.apache.org/jira/browse/HIVE-30226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-30226:
----------------------------------
Description:
A task with several map joins should build their hash tables at the same
time: each map join's dummy parent starts an asynchronous hash table load when
it is initialized. But the record
processors initialize the dummy operators one by one, and each initialization
waits for its own load to finish before the next dummy starts. The loads
therefore run one after the other. In
TPC-DS q35, Reducer 3 started building the catalog_sales hash table only
after the web_sales one was done.
*Fix*
* Operator#initializeDeferringAsyncWaits starts the asynchronous
initialization of every operator first and waits for them only afterwards.
* MapRecordProcessor and ReduceRecordProcessor initialize their dummy
operators with it, so the hash tables of a task's map joins load in parallel.
*Measured*
Cluster (TPC-DS 10 TB Parquet, 10 x r5d, same session). The run also had
HIVE-30224 and HIVE-30225 on, so these figures are for the three together:
* -135.4 s over 26 queries.
* q35 38.6 -> 19.8 s, q78 -24.7 s, q75 -21.9 s.
> Tez: a task with several map joins loads their hash tables one after the
> other
> -------------------------------------------------------------------------------
>
> Key: HIVE-30226
> URL: https://issues.apache.org/jira/browse/HIVE-30226
> Project: Hive
> Issue Type: Improvement
> Reporter: Denys Kuzmenko
> Priority: Major
>
> A task with several map joins should build their hash tables at the same
> time: each map join's dummy parent starts an asynchronous hash table load
> when it is initialized. But the record
> processors initialize the dummy operators one by one, and each
> initialization waits for its own load to finish before the next dummy starts.
> The loads therefore run one after the other. In
> TPC-DS q35, Reducer 3 started building the catalog_sales hash table only
> after the web_sales one was done.
>
>
>
>
> *Fix*
>
>
> * Operator#initializeDeferringAsyncWaits starts the asynchronous
> initialization of every operator first and waits for them only afterwards.
>
> * MapRecordProcessor and ReduceRecordProcessor initialize their dummy
> operators with it, so the hash tables of a task's map joins load in parallel.
>
>
>
>
> *Measured*
>
>
> Cluster (TPC-DS 10 TB Parquet, 10 x r5d, same session). The run also had
> HIVE-30224 and HIVE-30225 on, so these figures are for the three together:
>
> * -135.4 s over 26 queries.
>
>
> * q35 38.6 -> 19.8 s, q78 -24.7 s, q75 -21.9 s.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
