[
https://issues.apache.org/jira/browse/STORM-2693?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16139904#comment-16139904
]
Jungtaek Lim commented on STORM-2693:
-------------------------------------
[~danny0405]
Just curious, could you try out pacemaker to see it helps on your case?
> Topology submission or kill takes too much time when topologies grow to a few
> hundred
> -------------------------------------------------------------------------------------
>
> Key: STORM-2693
> URL: https://issues.apache.org/jira/browse/STORM-2693
> Project: Apache Storm
> Issue Type: Improvement
> Components: storm-core
> Affects Versions: 0.9.6, 1.0.2, 1.1.0, 1.0.3
> Reporter: Yuzhao Chen
> Fix For: 1.1.0
>
> Attachments: 2FA30CD8-AF15-4352-992D-A67BD724E7FB.png,
> D4A30D40-25D5-4ACF-9A96-252EBA9E6EF6.png
>
>
> Now for a storm cluster with 40 hosts [with 32 cores/128G memory] and
> hundreds of topologies, nimbus submission and killing will take about minutes
> to finish. For example, for a cluster with 300 hundred of topologies，it will
> take about 8 minutes to submit a topology, this affect our efficiency
> seriously.
> So, i check out the nimbus code and find two factor that will effect nimbus
> submission/killing time for a scheduling round:
> * read existing-assignments from zookeeper for every topology [will take
> about 4 seconds for a 300 topologies cluster]
> * read all the workers heartbeats and update the state to nimbus cache [will
> take about 30 seconds for a 300 topologies cluster]
> the key here is that Storm now use zookeeper to collect heartbeats [not RPC],
> and also keep physical plan [assignments] using zookeeper which can be
> totally local in nimbus.
> So, i think we should make some changes to storm's heartbeats and assignments
> management.
> For assignment promotion:
> 1. nimbus will put the assignments in memory also cache a copy to
> local disk
> 2. when restart or HA leader trigger nimbus will recover
> assignments from disk
> 3. nimbus will tell supervisor its assignment every time scheduling
> 4. supervisor will sync assignments at fixed time
> For heartbeats promotion:
> 1. workers will report executors ok or wrong to supervisor at fixed time
> 2. supervisor will report workers heartbeats to nimbus at fixed time
> 3. if supervisor die, it will tell nimbus through runtime hook
> or let nimbus find it through aware supervisor if survive
> 4. let supervisor decide if worker is running ok or invalid , supervisor will
> tell nimbus which executors of every topology are ok
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
