[
https://issues.apache.org/jira/browse/KYLIN-4806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xingxing Di updated KYLIN-4806:
-------------------------------
Description:
Currently kylin manually maintains stream receiver replicasets. Under this
design, when we have many cubes, we cannot achieve a perfect balance between
resource isolation and maintenance costs:
1. Maintain a large number of replicasets to maintain resource isolation,
which will cause a lot of maintenance costs
2. Maintain less replicasets to reduce maintenance costs, so one replicaset
should be used for multiple cubes, resource isolation will be a problem. eg:
One cube’s source data(Kafka topic) has grown abnormally, which may cause the
streaming receiver process unstable.
In the context of cloud native, kylin should add native support for kubernetes,
and then we can easily achieve *cube level streaming receiver cluster* (a set
of replicatset ) by define the necessary parameters such as parallelism, cpu,
memory, etc.
was:
Currently kylin manually maintains stream receiver replicasets. Under this
design, when we have many cubes, we cannot achieve a perfect balance between
resource isolation and maintenance costs:
1. Maintain a large number of replicasets to maintain resource isolation, which
will cause a lot of maintenance costs
2. Maintain less replicasets to reduce maintenance costs, so one replicaset
should be used for multiple cubes, resource isolation will be a problem. eg:
One cube’s source data(Kafka topic) has grown abnormally, which may cause the
streaming receiver process unstable.
In the context of cloud native, kylin should add native support for kubernetes,
and then we can easily achieve *cube level streaming receiver cluster* (a set
of replicatset ) by define the necessary parameters such as parallelism, cpu,
memory, etc.
> Native kubernetes support for kylin streaming receivers
> -------------------------------------------------------
>
> Key: KYLIN-4806
> URL: https://issues.apache.org/jira/browse/KYLIN-4806
> Project: Kylin
> Issue Type: New Feature
> Components: Real-time Streaming
> Reporter: Xingxing Di
> Priority: Major
>
> Currently kylin manually maintains stream receiver replicasets. Under this
> design, when we have many cubes, we cannot achieve a perfect balance between
> resource isolation and maintenance costs:
> 1. Maintain a large number of replicasets to maintain resource isolation,
> which will cause a lot of maintenance costs
> 2. Maintain less replicasets to reduce maintenance costs, so one replicaset
> should be used for multiple cubes, resource isolation will be a problem. eg:
> One cube’s source data(Kafka topic) has grown abnormally, which may cause the
> streaming receiver process unstable.
> In the context of cloud native, kylin should add native support for
> kubernetes, and then we can easily achieve *cube level streaming receiver
> cluster* (a set of replicatset ) by define the necessary parameters such as
> parallelism, cpu, memory, etc.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
