[
https://issues.apache.org/jira/browse/IMPALA-3561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-3561:
----------------------------------
Moving with the rest of fine-grained metadata loading. I'm not sure how
impactful this is for table-level stats, so maybe we just want to close it.
> Planner should request statistics for relevant columns not entire table
> -----------------------------------------------------------------------
>
> Key: IMPALA-3561
> URL: https://issues.apache.org/jira/browse/IMPALA-3561
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
> Affects Versions: Impala 2.6.0
> Reporter: Mostafa Mokhtar
> Priority: Minor
> Labels: catalog-server, performance, planner, scalability
>
> When generating a plan the planner request statistics for all columns
> referenced by all tables which is not necessary, this creates load on the
> catalog and bloats the meta data cache memory.
> Stats are only required for columns involved in :
> * Join
> * Aggregations
> * Filters
> The query below will fetch statistics for all 22 columns/partitions from
> store_sales and which is un-necessary as only ss_item_sk and ss_promo_sk are
> needed.
> {code}
> select
> count(*)
> from
> store_sales,
> item
> where
> ss_item_sk = i_item_sk
> group by ss_promo_sk;
> {code}
> If this issues is addressed it should reduce the meta data cache memory by an
> order of magnitude.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
