[
https://issues.apache.org/jira/browse/KYLIN-3138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16306407#comment-16306407
]
Ruslan Dautkhanov commented on KYLIN-3138:
------------------------------------------
Thank you for sharing that insight on KYLIN-2727 - that's a very cool feature
too.
I understand that it might be a good amount of refactor to implement on-demand
cuboid builds.
Although I think it might be only solution for a handful number of dimensions.
{quote}as the build may take a long time to finish{quote}
In out case (for our datasets) we estimate it would take 30-120 seconds for
each cuboid build which is not terrible.
Assuming there is a "hot" and ready to fire Spark backend application that has
the dataset cached and executors available for new cuboid builds..
The good part is that after each cuboid build, data will be always returned
fast for that combination of dimensions.
> cuboids on-demand build
> -----------------------
>
> Key: KYLIN-3138
> URL: https://issues.apache.org/jira/browse/KYLIN-3138
> Project: Kylin
> Issue Type: New Feature
> Components: General, Job Engine, Query Engine, Spark Engine
> Affects Versions: v2.2.0, v2.3.0
> Reporter: Ruslan Dautkhanov
> Assignee: Shaofeng SHI
> Priority: Critical
>
> We just started using Kylin and quite like it so far.
> Although some of the datasets we have are quite wide to even consider for
> OLAP cubing.
> Unless those cuboids will be built on-demand.
> I know some commercial non-open source products do this successfully.
> This idea is to build a cuboid only when a user actually needs it.
> So for example, our BI dashboards does a certain rollup, so then a SQL
> query hits Kylin backend. Kylin realizes it hasn't built that particular
> cuboid just yet,
> so immediately starts building it. Users has to wait a bit longer first time
> it request that combination of dimensions. But all other requests or requests
> of other users will be fast from that point on.
> Kylin (or any other OLAP solution) wouldn't be feasible to use on very wide
> datasets
> unless this on-demand functionality is implemented. For example, some
> datasets we have have 100-200 dimensions. And we don't know up front rollups
> users would want to do.
> Suggesting to have a new dimension build rule "lazy / on-demand". All
> previous rules apply. This new rule type would mean, a cuboid for a
> particular set of dimensions wouldn't be built up-front if it's marked as
> "lazy / on-demand".
> Thoughts / ideas?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
