[
https://issues.apache.org/jira/browse/SLIDER-1222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jaeboo Jeong updated SLIDER-1222:
---------------------------------
Attachment: SLIDER-1222.01.patch
> During decreasing the number of components, we need to release the components
> of specific nodes.
> ------------------------------------------------------------------------------------------------
>
> Key: SLIDER-1222
> URL: https://issues.apache.org/jira/browse/SLIDER-1222
> Project: Slider
> Issue Type: Improvement
> Components: agent, core
> Affects Versions: Slider 0.81
> Reporter: Jaeboo Jeong
> Attachments: SLIDER-1222.01.patch
>
>
> If we want to create a shard cluster, we need index numbers to split shards.
> Currently we can use app_container_tag for shard index number.
> And I also can add replication nodes to increases data availability.
> For example, If we want to 5 shards and 2 replication cluster, shard index
> numbers are like this.
> total desired components = 10
> shard_index = app_container_tag % 5
> replication = 0 if app_container_tag < 5 else 1
> However using app_container_tag is a problem during decreasing the number of
> components.
> Because we don’t know that allocated components are for shard or not.
> So shard component can be released even if we want to decrease replication
> component.
> For a shard cluster, I added the following functionality.
> - write allocated component’s shard and replication informations in zookeeper
> - during releasing component, replication component is released first
> - add command to change shard and replication configuration
> And we need to add the following configurations and script functions.
> - Application Configuration
> {code}
> "global": {
> "create.default.zookeeper.node": "true",
> "site.global.zookeeper.quorum": "${ZK_HOST}",
> "site.global.zookeeper.port": "2181",
> "site.global.zookeeper.znode.parent": "${DEFAULT_ZK_PATH}",
> "site.global.COMPONENT_NAME.shard": "10",
> "site.global.COMPONENT_NAME.replication": "2"
> }
> {code}
> - Resource Specification
> {code}
> "components": {
> "COMPONENT_NAME": {
> "yarn.role.priority": "1",
> "yarn.component.instances": "10",
> "yarn.vcores": "2",
> "yarn.memory": "8192",
> "yarn.component.placement.policy": "32",
> "yarn.placement.escalate.seconds": "60"
> }
> {code}
> - Placement Policy
> {code}
> DEFAULT_SHARD_REPLICATION = 32
> ANTI_AFFINITY_SHARD_REPLICATION = 64
> {code}
> - add allocation function in start() function of App Package
> {code}
> import container_tag
> ...
> def start(self, env):
> import params
> env.set_params(params)
> shard, replication =
> container_tag.allocate_container_tag(params.zk_server, params.zk_parent,
> 'COMPONENT_NAME', params.container_id)
> ...
> {code}
> - make argument data for container_tag.allocate_container_tag() in params.py
> {code}
> config = Script.get_config()
> global_conf = config['configurations']['global']
> zk_quorum = global_conf['zookeeper.quorum']
> zk_port = global_conf['zookeeper.port']
> zk_server = ','.join([h + ':' + str(nopq_zk_port) for h in
> nopq_zk_quorum.split(',')])
> zk_parent = global_conf['zookeeper.znode.parent']
> container_id = global_conf['app_container_id']
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
