Re: Re: Re: [DISCUSS] New Kylin Streaming Solution From eBay

2018-11-03 Thread Billy Liu
> >> >c) log collection > > >> > > > >> >2. As I know, ebay’s New Kylin Streaming Solution use replica Set > > to > > >> ensure that income messages wouldn’t lost if some processes lost. I > > think > > >> replica set is a

Re: Re: Re: [DISCUSS] New Kylin Streaming Solution From eBay

2018-11-02 Thread ShaoFeng Shi
ost. I > think > >> replica set is a set of kafka cosumer processes which is responsible for > >> ingest message and build base cuboid in memory. Could you please show me > >> some detail about how replica Set provide HA guarantee? How to configure > >> it? A link / pape

Re:Re: Re: [DISCUSS] New Kylin Streaming Solution From eBay

2018-11-01 Thread Ma Gang
al/replication/). >> > >> >3. How to add or remove node of replica Set in production env? How >> to monitor the health/pressure of replica Set cluster ? >> > >> >4. Does all measure are supported in ebay’s New Kylin Streaming >> Solution? What about c

Re: [DISCUSS] New Kylin Streaming Solution From eBay

2018-11-01 Thread Xiaoxiang Yu
> > > >Best wishes, >Xiaoxiang Yu > > >发件人: Ma Gang >答复: "dev@kylin.apache.org" >日期: 2018年10月30日 星期二 15:24 >收件人: "dev@kylin.apache.org" >主题: [DISCUSS] New Kylin Streaming So

Re: Re: [DISCUSS] New Kylin Streaming Solution From eBay

2018-11-01 Thread ShaoFeng Shi
in ebay’s New Kylin Streaming > Solution? What about count distinct(bitmap)? > > > >5. It seems ebay’s New Kylin Streaming Solution use a custom > columnar storage, why not use a open source mature columnar storage > solution ? Have your ever compare the performance of your

Re:Re: [DISCUSS] New Kylin Streaming Solution From eBay

2018-10-31 Thread Ma Gang
ylin Streaming Solution use a custom columnar >storage, why not use a open source mature columnar storage solution ? Have >your ever compare the performance of your custom columnar storage to open >source columnar storage solution ? > > > >------------ >Best wishes, >

Re: [DISCUSS] New Kylin Streaming Solution From eBay

2018-10-31 Thread Xiaoxiang Yu
t;dev@kylin.apache.org" 日期: 2018年10月30日 星期二 15:24 收件人: "dev@kylin.apache.org" 主题: [DISCUSS] New Kylin Streaming Solution From eBay Hi all, eBay Kylin team has developed a new Kylin streaming solution, the basic idea is to build a streaming cluster to ingest data from streaming source(

Re:Re: Re: [DISCUSS] New Kylin Streaming Solution From eBay

2018-10-31 Thread Ma Gang
Hi ShaoFeng, Sorry I reply the wrong email, copy my reply here for further discussion :)Very good questions, please see my comments start with [Gang]:1) How to bridge the real-time cube with a cube built from Hive? You know, in Kylin the source type is marked at the table level, which means a tab

Re: Re: [DISCUSS] New Kylin Streaming Solution From eBay

2018-10-31 Thread ShaoFeng Shi
Hi Gang, The real-time OLAP capability is pretty cool; I have a couple of questions here: 1) How to bridge the real-time cube with a cube built from Hive? You know, in Kylin the source type is marked at the table level, which means a table is either a Hive table, a JDBC table or a streaming table

Re:Re: [DISCUSS] New Kylin Streaming Solution From eBay

2018-10-30 Thread Ma Gang
Jira ticket has been created, and the related design doc is attached in the ticket: https://issues.apache.org/jira/browse/KYLIN-3654 在 2018-10-30 21:40:34,"ShaoFeng Shi" 写道: >Hi Gang, > >The design doc is still missing; can you upload it to somewhere and then >provide a link? > >Ma Gang 于2018年

Re: [DISCUSS] New Kylin Streaming Solution From eBay

2018-10-30 Thread ShaoFeng Shi
Hi Gang, The design doc is still missing; can you upload it to somewhere and then provide a link? Ma Gang 于2018年10月30日周二 下午8:35写道: > Resend the design doc, not sure why the attachment is removed in the > previous mail. > > At 2018-10-30 15:24:01, "Ma Gang" wrote: > > Hi all, > > eBay Kylin tea

Re:[DISCUSS] New Kylin Streaming Solution From eBay

2018-10-30 Thread Ma Gang
Resend the design doc, not sure why the attachment is removed in the previous mail. At 2018-10-30 15:24:01, "Ma Gang" wrote: Hi all, eBay Kylin team has developed a new Kylin streaming solution, the basic idea is to build a streaming cluster to ingest data from streaming source(Kafka), and

[DISCUSS] New Kylin Streaming Solution From eBay

2018-10-30 Thread Ma Gang
Hi all, eBay Kylin team has developed a new Kylin streaming solution, the basic idea is to build a streaming cluster to ingest data from streaming source(Kafka), and provide query for real-time data, the data preparation latency is milliseconds, which means the data is queryable almost when it