[
https://issues.apache.org/jira/browse/IMPALA-3563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Armstrong updated IMPALA-3563:
----------------------------------
External issue ID: (was: IMPALA-3563)
Parent: (was: IMPALA-2649)
Issue Type: Bug (was: Sub-task)
> Evaluate using global opposed to per partition statistics
> ---------------------------------------------------------
>
> Key: IMPALA-3563
> URL: https://issues.apache.org/jira/browse/IMPALA-3563
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Affects Versions: Impala 2.5.0
> Reporter: Mostafa Mokhtar
> Priority: Major
> Labels: catalog-server, planner, scalability
>
> Impala and other SQL on Hadoop solutions use per partition statistics which
> creates a metadata scalability problem which I reckon outweighs benefits of
> having more accurate statistics.
> This is the proposal is for a partitioned table :
> * "Compute statistics" computes and stores per partition HLL same as before
> * Catalog merges the HLL(s) for all partitions and stores/persists global
> statistics
> * Impalad(s) never request per partition statics only global stats
> * The only time the catalog needs to read the per partition HLL is when
> regenerating the global stats as part of adding/removing partitions
> In other words during planning the partitioned table looks very similar to a
> non-partitioned table.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
