Hi ShaoFeng,
For streaming ingest/query performance, there is a doc: 
https://drive.google.com/file/d/1GSBMpRuVQRmr8Ev2BWvssfMd-Rck9vsH/view?ths=true 
, it is also in the design doc's 'performance' section attached in the jira: 
https://issues.apache.org/jira/browse/KYLIN-3654
For stability, it is very stable in our environment, but currently it is not 
widely used in eBay, so it is hard to say.
I will start to merge code to master branch, it may take some time because our 
current version is Kylin 2.1.0, hope it can be done before Nov.30, but I cannot 
guarantee it, there is lots of other works to do.

At 2018-11-01 15:08:12, "ShaoFeng Shi" <shaofeng...@apache.org> wrote:
>Hi Gang,
>
>Thank you for the information, that is helpful for understanding the
>overall design and implementation.
>
>Do you have some statistical information, like performance, throughput,
>stability, etc.? Besides, what's the plan of contributing it to the
>community? Thanks!
>
>
>Ma Gang <mg4w...@163.com> 于2018年11月1日周四 下午2:45写道:
>
>> Thanks Xiaoxiang,
>> Very good questions! Please see my comments started with [Gang]:
>>
>>
>> 1.      Is it possible to use Yarn as cluster manager for index task.
>> Coordinator process will set up them at specificed period.
>> [Gang] I think it is possible, but in current design,  the indexing task
>> is designed as long running task, it also can provide query service, this
>> makes the whole system very simple and efficiency, I don't think we need to
>> stop/start indexing task time by time. But use yarn to manage the resource
>> is possible, we need to redesign the existing coordinator, to make it easy
>> to deploy to Yarn, Kubernetes, etc. Hope this can be done after
>> contribution to community.
>>
>> 2.      As I know, ebay’s New Kylin Streaming Solution use replica Set to
>> ensure that income messages wouldn’t lost if some processes  lost. I think
>> replica set is a set of kafka cosumer processes which is responsible for
>> ingest message and build base cuboid in memory. Could you please show me
>> some detail about how replica Set provide HA guarantee? How to configure
>> it? A link / paper is OK.  I found one but I don’t know if it same meaning
>> for your replica Set.
>>
>>
>> [Gang] Yes, it is similar as the MongoDB replication, but currently we
>> don't replicate data from Primary node, just assign the same Kafka
>> topic/partitions to the receivers in a ReplicaSet, all receivers in a
>> ReplicaSet will consume data from Kafka, so if one receiver is down, other
>> receivers in the ReplicaSet are still consuming the same Kafka data, so the
>> consume/query will not be impact. And We don't guarantee that the receivers
>> in a ReplicaSet have the same consuming rate, but we can guarantee that the
>> user can view data consistently by stick to the query to one receiver for
>> one cube.
>> The HA implementation is a little bit naive, but simple and worked. Maybe
>> in the future, we can do HA by replication to support other streaming
>> sources that don't support multiple consumers and don't have persistent
>> store.
>>
>> 3.      How to add or remove node of replica Set in production env? How to
>> monitor the health/pressure of replica Set cluster ?
>> [Gang] Currently we have UI/restful api to let admin to add/remove node
>> to/from a ReplicaSet, and have a simple ui to let admin monitor the health,
>> consuming rate for each receiver/cube. Also all metrics are collected using
>> yammer metrics framework, it is easy to exposed to other monitor system.
>>
>> 4.      Does all measure are supported in ebay’s New Kylin Streaming
>> Solution? What about count distinct(bitmap)?
>> [Gang] Most measures are supported, but precise count distinct(bitmap) is
>> not support in case that the distinct dimension is not int type. As you
>> know, to support precise count distinct for not-int type dimension, it
>> needs to build global dictionary, it is not possible in the streaming env.
>>
>>
>> 5.      It seems ebay’s New Kylin Streaming Solution use a custom columnar
>> storage, why not use a open source mature columnar storage  solution ? Have
>> your ever compare the performance of your custom columnar storage to open
>> source columnar storage  solution ?
>>
>> [Gang] Most open source columnar format like Parquet, ORC are designed to
>> use in Hadoop env, the streaming data are in local disk, so I didn't
>> consider them at the beginning. It is not very hard to define columnar
>> format to store Kylin specific data, use a customize columnar storage, you
>> can use mmap file to scan data, add row-level invert index for all
>> dimensions, so I think the performance will be better compared to using
>> common columnar format. I didn't compare the performance, but the storage
>> engine is pluggable, you may contribute a parquet storage if you are
>> interesting.
>>
>>
>>
>>
>>
>>
>> At 2018-11-01 12:42:25, "Xiaoxiang Yu" <xiaoxiang...@kyligence.io> wrote:
>> >Hi gang, I am so glad to know that eBay has a solution for realtime olap
>> on kylin. I have some small question:
>> >
>> >
>> >1.      Is it possible to use Yarn as cluster manager for index task.
>> Coordinator process will set up them at specificed period. Yarn will manage
>> :
>> >
>> >a)       retry these task if some failed
>> >
>> >b)       resource allocation
>> >
>> >c)       log collection
>> >
>> >2.      As I know, ebay’s New Kylin Streaming Solution use replica Set to
>> ensure that income messages wouldn’t lost if some processes  lost. I think
>> replica set is a set of kafka cosumer processes which is responsible for
>> ingest message and build base cuboid in memory. Could you please show me
>> some detail about how replica Set provide HA guarantee? How to configure
>> it? A link / paper is OK.  I found one but I don’t know if it same meaning
>> for your replica Set.
>> >
>> >a)       [Mongodb replication](
>> https://docs.mongodb.com/manual/replication/).
>> >
>> >3.      How to add or remove node of replica Set in production env? How
>> to monitor the health/pressure of replica Set cluster ?
>> >
>> >4.      Does all measure are supported in ebay’s New Kylin Streaming
>> Solution? What about count distinct(bitmap)?
>> >
>> >5.      It seems ebay’s New Kylin Streaming Solution use a custom
>> columnar storage, why not use a open source mature columnar storage
>> solution ? Have your ever compare the performance of your custom columnar
>> storage to open source columnar storage  solution ?
>> >
>> >
>> >
>> >----------------
>> >Best wishes,
>> >Xiaoxiang Yu
>> >
>> >
>> >发件人: Ma Gang <mg4w...@163.com>
>> >答复: "dev@kylin.apache.org" <dev@kylin.apache.org>
>> >日期: 2018年10月30日 星期二 15:24
>> >收件人: "dev@kylin.apache.org" <dev@kylin.apache.org>
>> >主题: [DISCUSS] New Kylin Streaming Solution From eBay
>> >
>> >Hi all,
>> >
>> >eBay Kylin team has developed a new Kylin streaming solution, the basic
>> idea is to build a streaming cluster to ingest data from streaming
>> source(Kafka), and provide query for real-time data, the data preparation
>> latency is milliseconds, which means the data is queryable almost when it
>> is ingested, attach is the architecture design doc.
>> >We would like to contribute the feature to community, please let us know
>> if you have any concern.
>> >
>> >Thanks,
>> >Gang(Allen) Ma
>> >
>> >
>> >
>> >
>> >
>>
>
>
>-- 
>Best regards,
>
>Shaofeng Shi 史少锋

Reply via email to