[
https://issues.apache.org/jira/browse/IMPALA-14879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Boglarka Egyed reassigned IMPALA-14879:
---------------------------------------
Assignee: Mihaly Szjatinya
> Support incremental stats in Iceberg tables with Puffin files
> -------------------------------------------------------------
>
> Key: IMPALA-14879
> URL: https://issues.apache.org/jira/browse/IMPALA-14879
> Project: IMPALA
> Issue Type: Epic
> Reporter: Csaba Ringhofer
> Assignee: Mihaly Szjatinya
> Priority: Major
> Labels: iceberg
>
> Currently Impala always reads the whole table during COMPUTE STATS (unless
> table sample is used).
> It is possible to provide efficient incremental implementation in Iceberg (as
> long as deletes are ignored).
> - COMPUTE STATS should save stats in Puffin (e.g. Theta sketches for NDV)
> for the snapshot where the stats were computed
> - subsequent COMPUTE STATS only need to read new files and merge the results
> with the one in the snapshot
> This would be both easier to use and more efficient than partition based
> COMPUTE INCREMENTAL STATS for Hive tables.
> One issue to solve is how to store statistics not handled by Iceberg, e.g.
> max/avg length of strings.
> The following design doc has detailed discussion of stats usage of different
> engines in Iceberg, but did not come to a conclusion:
> https://docs.google.com/document/d/1H9uYt53Q1_CcOXOfLcr0hXRxvqflg_k_xeVorMLrWbM/edit?tab=t.0#heading=h.hg9vbmr09jb8
> I see 3 ways to solve this:
> 1. wait till Iceberg has a solution for all of Impala's stats
> 2. write non-Icerberg compatible stats in an Impala specific key in Puffin
> 3. keep storing missing stats in HMS, but improve them to provide incremental
> support (e.g. store snapshot id)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
