Hi gang, I am so glad to know that eBay has a solution for realtime olap on 
kylin. I have some small question:

1.      Is it possible to use Yarn as cluster manager for index task. 
Coordinator process will set up them at specificed period. Yarn will manage :

a)       retry these task if some failed

b)       resource allocation

c)       log collection

2.      As I know, ebay’s New Kylin Streaming Solution use replica Set to 
ensure that income messages wouldn’t lost if some processes  lost. I think 
replica set is a set of kafka cosumer processes which is responsible for ingest 
message and build base cuboid in memory. Could you please show me some detail 
about how replica Set provide HA guarantee? How to configure it? A link / paper 
is OK.  I found one but I don’t know if it same meaning for your replica Set.

a)       [Mongodb replication](https://docs.mongodb.com/manual/replication/).

3.      How to add or remove node of replica Set in production env? How to 
monitor the health/pressure of replica Set cluster ?

4.      Does all measure are supported in ebay’s New Kylin Streaming Solution? 
What about count distinct(bitmap)?

5.      It seems ebay’s New Kylin Streaming Solution use a custom columnar 
storage, why not use a open source mature columnar storage  solution ? Have 
your ever compare the performance of your custom columnar storage to open 
source columnar storage  solution ?

Best wishes,
Xiaoxiang Yu

发件人: Ma Gang <mg4w...@163.com>
答复: "dev@kylin.apache.org" <dev@kylin.apache.org>
日期: 2018年10月30日 星期二 15:24
收件人: "dev@kylin.apache.org" <dev@kylin.apache.org>
主题: [DISCUSS] New Kylin Streaming Solution From eBay

Hi all,

eBay Kylin team has developed a new Kylin streaming solution, the basic idea is 
to build a streaming cluster to ingest data from streaming source(Kafka), and 
provide query for real-time data, the data preparation latency is milliseconds, 
which means the data is queryable almost when it is ingested, attach is the 
architecture design doc.
We would like to contribute the feature to community, please let us know if you 
have any concern.

Gang(Allen) Ma

Reply via email to