In storm a slot is just a port, but in order to be able to use it effectively
you need to treat it like a slot in the old hadoop. At least until
https://issues.apache.org/jira/browse/STORM-893 is released. On our clusters
we typically set it up so there is one slot per core and about 3.5 GB of
memory. This may seem like a lot of slots compared to the CPU in a cluster but
for the most part it is fine and gives us about 60% CPU utilization across our
multi-tenant clusters. There are a number of situations where this ratio is
just way off and they need much more CPU, for that we use the isolation feature
in the multi-tenant scheduler so the end user can have a decent idea of about
how much memory/CPU they will have access to, and can set the number of hosts
they want.
This is still not perfect, especially for bolts or spouts that are outliers
compared to the rest of the topology. We have put in some hacks to work around
this, but STORM-893 with resource aware scheduling is the long term solution to
this problem.
How does this over-subscription of the CPU impact the performance of the
topologies? Well there have been a number of cases where we have had to talk
to someone doing something truly CPU intensive and ask them to move to
isolation so it does not impact other topologies. After we have scheduling
based off of resources requested, we really want to move to have true isolation
between workers using cgroups, and along with that have a feedback loop to
nimbus and the scheduler so that if your topology is using more resources we
can reschedule if needed and either increase the size of the ask + update the
cgroup, or move load to other boxes if needed.
- Bobby
On Tuesday, September 8, 2015 6:51 AM, Renkai <[email protected]> wrote:
+1 want to know more about this,too
在 15/9/8 15:25，“#ZHANG SHUHAO#”<[email protected]> 写入:
>Hi,
>
>When we launch a supervisor node with storm.yaml, we can configure slots port
>(default is 4). Based on my understanding, this is nothing but a threshold to
>prevent too-many concurrent running workers (maximum is 4 by default then).
>However, my experiment results shows that number of slots does affect
>performance non-trivially.
>
>In Hadoop earlier version, slots reserve resources. Does storm slot has the
>similar meaning?
>Can someone help me to clarify this?
>
>Thanks.
>
>Tony.
