Hi ShaoFeng,
Sorry I reply the wrong email, copy my reply here for further discussion :)Very 
good questions, please see my comments start with [Gang]:1) How to bridge the 
real-time cube with a cube built from Hive? You know,
in Kylin the source type is marked at the table level, which means a table
is either a Hive table, a JDBC table or a streaming table.  To implement
the lambda architecture, how to composite the batch cube with the real-time
cube (with the same table)? This seems not mentioned in the design doc.[Gang] 
>> there is a sourceType field in TableDesc to indicate the source type, I just 
add new types for the table that has more than 1 source, for example: 
ID_KAFKA_HIVE=21, means the table source can be both Kafka and Hive.

2) How it be together with the as-is NRT (near real-time) solution
introduced in v1.6? Many users are building cube directly from Kafka,
though they are in the mini or micro batches. Can the new streaming
solution work together with the NRT cube? E.g, if I don't need to do ETL in
Hive, can I use the batch job to fetch data from Kafka, and use
the streaming real-time receivers together?[Gang] >>The new streaming solution 
is totally new, it works separately with the current streaming solution, there 
is no conflict with the NRT solution, so they can run together in the same 
Kylin platform, but currently they cannot work together as you said.
3) Does the "Build engine" of the real-time solution follow the plug-in 
architecture, so that it can support non-HBase storage? As you know we're 
implementing the parquet storage. Can this solution support other storages 
without much rework?[Gang] >>Yes, the "Build engine" follows the plug-in 
architecture, so it is easy to support non-HBase storage. In eBay, we just use 
InMemCubing, so currently we only have InMemCubing algorithm, but I think it is 
easy to extend to support LayerCubing.




At 2018-10-31 15:31:01, "ShaoFeng Shi" <shaofeng...@apache.org> wrote:
>Hi Gang,
>
>The real-time OLAP capability is pretty cool; I have a couple of questions
>here:
>
>1) How to bridge the real-time cube with a cube built from Hive? You know,
>in Kylin the source type is marked at the table level, which means a table
>is either a Hive table, a JDBC table or a streaming table.  To implement
>the lambda architecture, how to composite the batch cube with the real-time
>cube (with the same table)? This seems not mentioned in the design doc.
>2) How it be together with the as-is NRT (near real-time) solution
>introduced in v1.6? Many users are building cube directly from Kafka,
>though they are in the mini or micro batches. Can the new streaming
>solution work together with the NRT cube? E.g, if I don't need to do ETL in
>Hive, can I use the batch job to fetch data from Kafka, and use
>the streaming real-time receivers together?
>3) Does the "Build engine" of the real-time solution follow the plug-in
>architecture, so that it can support non-HBase storage? As you know we're
>implementing the parquet storage. Can this solution support other storages
>without much rework?
>
>Thanks for raising this discussion.
>
>Ma Gang <mg4w...@163.com> 于2018年10月31日周三 上午9:57写道:
>
>> Jira ticket has been created, and the related design doc is attached in
>> the ticket: https://issues.apache.org/jira/browse/KYLIN-3654
>>
>>
>> 在 2018-10-30 21:40:34,"ShaoFeng Shi" <shaofeng...@apache.org> 写道:
>> >Hi Gang,
>> >
>> >The design doc is still missing; can you upload it to somewhere and then
>> >provide a link?
>> >
>> >Ma Gang <mg4w...@163.com> 于2018年10月30日周二 下午8:35写道:
>> >
>> >> Resend the design doc, not sure why the attachment is removed in the
>> >> previous mail.
>> >>
>> >> At 2018-10-30 15:24:01, "Ma Gang" <mg4w...@163.com> wrote:
>> >>
>> >> Hi all,
>> >>
>> >> eBay Kylin team has developed a new Kylin streaming solution, the basic
>> >> idea is to build a streaming cluster to ingest data from streaming
>> >> source(Kafka), and provide query for real-time data, the data
>> preparation
>> >> latency is milliseconds, which means the data is queryable almost when
>> it
>> >> is ingested, attach is the architecture design doc.
>> >> We would like to contribute the feature to community, please let us know
>> >> if you have any concern.
>> >>
>> >> Thanks,
>> >> Gang(Allen) Ma
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >
>> >
>> >--
>> >Best regards,
>> >
>> >Shaofeng Shi 史少锋
>>
>
>
>-- 
>Best regards,
>
>Shaofeng Shi 史少锋

Reply via email to