Joe McDonnell created IMPALA-15750:
--------------------------------------
Summary: Use the Parquet page index to split row groups into small
granules to combat skew
Key: IMPALA-15750
URL: https://issues.apache.org/jira/browse/IMPALA-15750
Project: IMPALA
Issue Type: Improvement
Components: Backend
Affects Versions: Impala 5.1.0
Reporter: Joe McDonnell
Parquet scanners operate on a whole row group as the unit of granularity. When
there are few files and the files have a single row group each, the parallelism
is limited. If this is known at planning time, the planner will use min(number
of scan ranges, mt_dop) threads per instance. At runtime, this can happen if a
runtime filter eliminates a large number of files. We may be running with
mt_dop=12, but if there are only two surviving files, then only two threads
will be busy. This can cause skew up the entire plan tree for mt_dop > 0,
because the fragment instance is doing both the scan and the subsequent
joins/aggregations/etc.
We want to be able to split below the row group level to allow multiple threads
to work on a single row group. The Parquet page index provides the offsets of
data pages within each column. This provides a way to split a row group into
smaller granules of N rows each and know which data pages belong to each N-row
chunk. We should use this to reduce skew and achieve higher parallelism.
>From a design point of view, we want to avoid changing the IO profile too much
>(i.e. we don't want to end up with many small IOs). Ideally, we could share
>decompression for overlapping pages. We also need to think about when the
>splitting starts to take place. We may want to wait until the queue of scan
>ranges is empty before splitting things apart.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
