Hi Gang,

The real-time OLAP capability is pretty cool; I have a couple of questions

1) How to bridge the real-time cube with a cube built from Hive? You know,
in Kylin the source type is marked at the table level, which means a table
is either a Hive table, a JDBC table or a streaming table.  To implement
the lambda architecture, how to composite the batch cube with the real-time
cube (with the same table)? This seems not mentioned in the design doc.
2) How it be together with the as-is NRT (near real-time) solution
introduced in v1.6? Many users are building cube directly from Kafka,
though they are in the mini or micro batches. Can the new streaming
solution work together with the NRT cube? E.g, if I don't need to do ETL in
Hive, can I use the batch job to fetch data from Kafka, and use
the streaming real-time receivers together?
3) Does the "Build engine" of the real-time solution follow the plug-in
architecture, so that it can support non-HBase storage? As you know we're
implementing the parquet storage. Can this solution support other storages
without much rework?

Thanks for raising this discussion.

Ma Gang <mg4w...@163.com> 于2018年10月31日周三 上午9:57写道:

> Jira ticket has been created, and the related design doc is attached in
> the ticket: https://issues.apache.org/jira/browse/KYLIN-3654
> 在 2018-10-30 21:40:34,"ShaoFeng Shi" <shaofeng...@apache.org> 写道:
> >Hi Gang,
> >
> >The design doc is still missing; can you upload it to somewhere and then
> >provide a link?
> >
> >Ma Gang <mg4w...@163.com> 于2018年10月30日周二 下午8:35写道:
> >
> >> Resend the design doc, not sure why the attachment is removed in the
> >> previous mail.
> >>
> >> At 2018-10-30 15:24:01, "Ma Gang" <mg4w...@163.com> wrote:
> >>
> >> Hi all,
> >>
> >> eBay Kylin team has developed a new Kylin streaming solution, the basic
> >> idea is to build a streaming cluster to ingest data from streaming
> >> source(Kafka), and provide query for real-time data, the data
> preparation
> >> latency is milliseconds, which means the data is queryable almost when
> it
> >> is ingested, attach is the architecture design doc.
> >> We would like to contribute the feature to community, please let us know
> >> if you have any concern.
> >>
> >> Thanks,
> >> Gang(Allen) Ma
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >
> >
> >--
> >Best regards,
> >
> >Shaofeng Shi 史少锋

Best regards,

Shaofeng Shi 史少锋

Reply via email to