Thanks Billy, below is my answer for your questions: Is there a configuration item on Cube to tell which snapshot would be used? >> When user design the cube, there will be a configuration in the 'Advance >> Setting' tab to let user configure to use global snapshot or not, but user >> cannot change the snapshot reference after build(Maybe we can add it in the >> future, it is easy to implement but not sure it make sense or not).
Could the segment level snapshot be refreshed independently? >> Very good question. I think we can add a new button "refresh lookup table" >> at cube level, so that user can refresh snapshots for cube/segments >> independently. Could we show the snapshot size, build time, storage path on the GUI? >> Yes, this is in the design "Management" section, user can view all snapshots >> for each lookup table, and detail information for each snapshot, and which >> cube/segments use this snapshot At 2018-04-01 23:12:37, "Billy Liu" <billy...@apache.org> wrote: >Hi Magang, > >Very good proposal. I have one few questions here. In item 2, you >mentioned global level snapshot and segment level snapshot. This is >similar scenario with handling slowly change dimensions. Is there a >configuration item on Cube to tell which snapshot would be used? Could >the segment level snapshot be refreshed independently? Could we show >the snapshot size, build time, storage path on the GUI? > >With Warm regards > >Billy Liu > > >2018-04-01 18:19 GMT+08:00 ShaoFeng Shi <shaofeng...@apache.org>: >> Thank you, Gang! Looking forward to seeing this feature in Kylin. >> >> 2018-04-01 17:35 GMT+08:00 Ma Gang <mg4w...@163.com>: >> >>> Sure ShaoFeng, I add a storageType field in the SnapshotDesc to support >>> different materialized storages. >>> At 2018-04-01 09:57:23, "ShaoFeng Shi" <shaofeng...@apache.org> wrote: >>> >Thank you Ma Gang; This is a good proposal. Externalizing the lookup >>> >snapshots will reduce the burden of Kylin query servers. >>> > >>> >My only comment is, does this implementation support extension? You know >>> >since Kylin 1.5, Kylin has the plug-in architecture, HBase is one >>> >implementation for the storage. Kylin core modules don't directly depend >>> on >>> >HBase anymore; so please take this into consideration when you implement >>> >it. >>> > >>> >2018-03-30 17:57 GMT+08:00 magang <mg4w...@163.com>: >>> > >>> >> Hi all, >>> >> >>> >> There are two limitations for current lookup table design: >>> >> >>> >> 1. lookup table size is limited, because table snapshot need to be >>> cached >>> >> in >>> >> Kylin server, too large snapshot table will break Kylin server, also the >>> >> snapshot building may take very long time when it is too large. >>> >> 2. each segment has its own lookup table snapshot, but some users may >>> need >>> >> a >>> >> global snapshot table, which means when the global table is updated, the >>> >> query for all segments need to reflect the change. >>> >> >>> >> To resolve the above limitations, I have created ticket: >>> >> https://issues.apache.org/jira/browse/KYLIN-3221 >>> >> <https://issues.apache.org/jira/browse/KYLIN-3221> , and put initial >>> new >>> >> design doc there, any comments and suggestions are welcome. >>> >> >>> >> -- >>> >> Sent from: http://apache-kylin.74782.x6.nabble.com/ >>> >> >>> > >>> > >>> > >>> >-- >>> >Best regards, >>> > >>> >Shaofeng Shi 史少锋 >>> >> >> >> >> -- >> Best regards, >> >> Shaofeng Shi 史少锋
