Thank you Ma Gang; This is a good proposal. Externalizing the lookup snapshots will reduce the burden of Kylin query servers.
My only comment is, does this implementation support extension? You know since Kylin 1.5, Kylin has the plug-in architecture, HBase is one implementation for the storage. Kylin core modules don't directly depend on HBase anymore; so please take this into consideration when you implement it. 2018-03-30 17:57 GMT+08:00 magang <mg4w...@163.com>: > Hi all, > > There are two limitations for current lookup table design: > > 1. lookup table size is limited, because table snapshot need to be cached > in > Kylin server, too large snapshot table will break Kylin server, also the > snapshot building may take very long time when it is too large. > 2. each segment has its own lookup table snapshot, but some users may need > a > global snapshot table, which means when the global table is updated, the > query for all segments need to reflect the change. > > To resolve the above limitations, I have created ticket: > https://issues.apache.org/jira/browse/KYLIN-3221 > <https://issues.apache.org/jira/browse/KYLIN-3221> , and put initial new > design doc there, any comments and suggestions are welcome. > > -- > Sent from: http://apache-kylin.74782.x6.nabble.com/ > -- Best regards, Shaofeng Shi 史少锋
