[
https://issues.apache.org/jira/browse/LENS-280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14319953#comment-14319953
]
Amareshwari Sriramadasu commented on LENS-280:
----------------------------------------------
Here is the current status - with respect to latest partition and partition
lookups in storage table resolver
Latest partition :
- Its implemented as another partition in table
- Dim tables are directly queried on latest part
- latest date is calculated by getting latest of all
- there is only one latest amongst all update periods
StorateTableResolver :
* Prune facts with unsupported storage
* For each fact, for each range queried
** If partition lookup is on least interval of fact, if partition exists it is
included, if not it will fail
** If partition lookup is on higher granular interval
*** If partition does not exist, look for finer granular
*** If partition exists, if close of books feature is enabled (by specifying
lens.cube.query.process.time.partition.column), iterates over all partitions
and finds if look ahead number of process time partitions are registered
**** if not, add finer granular partitions in lookahead process time period
If we want to do all the above without looking up metastore, we will have to
cache the following details :
For each fact, for each storage ,for each update period, for each time
dimension,
we will have to cache first partition, latest partition and holes. We might
have to cache partitions for which look ahead process time partitions are not
registered, for each non-process time dimension.
> Reduce number of metastore lookups during cube query rewriting
> --------------------------------------------------------------
>
> Key: LENS-280
> URL: https://issues.apache.org/jira/browse/LENS-280
> Project: Apache Lens
> Issue Type: Improvement
> Components: cube
> Reporter: Amareshwari Sriramadasu
> Fix For: 2.1
>
>
> Right now, StorageTableResolver does metastore look up for finding number of
> partitions for each time value in the range provided.
> This issue is to look at reducing the number of lookups.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
