[
https://issues.apache.org/jira/browse/HIVE-30182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-30182:
----------------------------------
Description:
`ParquetFilterPredicateConverter.toFilterPredicate` returns null (no row-group
filter at all) when any leaf of the search argument cannot be built. A leaf
whose literal is a runtime value (DynamicValue, e.g. a semijoin's min/max) that
is not available when the reader builds its filter is such a leaf: BETWEEN/IN
get an empty literal list from SearchArgumentImpl.getLiteralList and
FilterPredicateLeafBuilder throws "Not able to build 'between' operation filter
with [] which needs two literals"; a single-literal leaf throws
NoDynamicValuesException. The static leaves of the same conjunction are lost
with it, and every failure logs an ERROR with a stack trace: 45,927 of them in
one TPC-DS 10 TB run, all from LLAP IO-Elevator threads reading
semijoin-filtered Parquet scans.
Fix: such a leaf filters nothing. The search argument pushes NOT down to the
leaves, so dropping a leaf from a conjunction only weakens the filter; a
disjunction containing it still filters nothing.
Not covered here: the runtime values are not reachable from the LLAP IO threads
at all, so a semijoin's min/max never prunes Parquet row groups on that path
(separate issue).
was:
LlapDecider raises the minimum reducers of an auto-reduced vertex to
hive.tez.llap.min.reducer.per.executor (0.33) x
the cluster's executors (adjustAutoParallelism), so auto-reduce does not shrink
a reducer to a handful of tasks on a
large cluster. It applies the floor only to a ReduceWork: a reduce-side merge
join is a MergeJoinWork, whose main and
merged reduce works feed one vertex through their edges, and is skipped.
TPC-DS q51 at 10 TB on 24 LLAP daemons (288 executors): the FULL OUTER merge
join vertex ran 13 tasks while its input
reducers and the PTF after it ran 253; the floor would have kept it at 96 or
more.
Fix: apply the floor to the merge join's main work and each merged reduce work
(their edge properties are the ones
the vertex's edges use). TestLlapDecider covers it.
> Parquet: a leaf without its runtime value drops the whole
> ---------------------------------------------------------
>
> Key: HIVE-30182
> URL: https://issues.apache.org/jira/browse/HIVE-30182
> Project: Hive
> Issue Type: Sub-task
> Components: Parquet, Query Processor
> Reporter: Denys Kuzmenko
> Priority: Major
>
> `ParquetFilterPredicateConverter.toFilterPredicate` returns null (no
> row-group filter at all) when any leaf of the search argument cannot be
> built. A leaf whose literal is a runtime value (DynamicValue, e.g. a
> semijoin's min/max) that is not available when the reader builds its filter
> is such a leaf: BETWEEN/IN get an empty literal list from
> SearchArgumentImpl.getLiteralList and FilterPredicateLeafBuilder throws "Not
> able to build 'between' operation filter with [] which needs two literals"; a
> single-literal leaf throws NoDynamicValuesException. The static leaves of the
> same conjunction are lost with it, and every failure logs an ERROR with a
> stack trace: 45,927 of them in one TPC-DS 10 TB run, all from LLAP
> IO-Elevator threads reading semijoin-filtered Parquet scans.
> Fix: such a leaf filters nothing. The search argument pushes NOT down to the
> leaves, so dropping a leaf from a conjunction only weakens the filter; a
> disjunction containing it still filters nothing.
> Not covered here: the runtime values are not reachable from the LLAP IO
> threads at all, so a semijoin's min/max never prunes Parquet row groups on
> that path (separate issue).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
