I created a JIRA for this https://issues.apache.org/jira/browse/KYLIN-599
Further discussion goes there. :-) On Tue, Feb 10, 2015 at 10:15 AM, vipul jhawar <[email protected]> wrote: > Hi Xu > > Could we just have Kafka and use its consumers alone instead of storm as > well in the setup. If you could provide some more details on how a kafka + > storm would be a better fit as it also introduces more complexity in the > system instead of the simple log Q which we can get with kafka alone. > > Thanks > > On Mon, Feb 9, 2015 at 8:55 PM, 蒋旭 <[email protected]> wrote: > > > Hi Guys, > > > > I write a simple proposal of real time support in kylin as below. Please > > help to review! > > > > > > 1. Kafka + storm will build inverted index in memory. These index will > be > > inserted into hbase by batch (e.g. every 5 minutes). > > > > > > 2. The inverted index in hbase will keep the short term data (e.g. 7 > > days). These index will be converted into data cube by batch (e.g. every > 7 > > day). > > > > > > 3. The data cube in hbase will keep the long term data. > > > > > > 4. Query engine will decide to use inverted index or data cube in hbase > by > > time range. In future, the query engine can also use the in-memory > inverted > > index in storm that can reduce the data latency from minutes to seconds. > > > > > > Thanks > > Jiang Xu >
