Re: Details about the Storm Scheduler

2016-08-25 Thread Roshan Naik
AM To: "dev@storm.apache.org<mailto:dev@storm.apache.org>" <dev@storm.apache.org<mailto:dev@storm.apache.org>>, Bobby Evans <ev...@yahoo-inc.com<mailto:ev...@yahoo-inc.com>> Cc: Roshan Naik <ros...@hortonworks.com<mailto:ros...@hortonworks.com>>

Re: Details about the Storm Scheduler

2016-08-24 Thread Nathan Leung
Also your bolt may be pending on a call to an external resource (e.g. DB) and thus not consuming much CPU despite a relatively high usage. On Wed, Aug 24, 2016 at 10:09 AM, Bobby Evans wrote: > But CGroups is restricting the actual CPU usage and scheduling is taking

Re: Details about the Storm Scheduler

2016-08-24 Thread Bobby Evans
But CGroups is restricting the actual CPU usage and scheduling is taking the CPU usage into account so as to not overload a box.  You can use the latency to guess how much CPU is being used, but that only works for a single threaded bolt/spout.  Not all bolts/spouts are single threaded.  Think

Re: Details about the Storm Scheduler

2016-08-23 Thread Roshan Naik
On 8/22/16, 6:55 AM, "Bobby Evans" wrote: >Getting the CPU used for a worker is simple, but getting the CPU used for >individual components is not so simple/almost impossible for >multi-threaded bolts/spouts. The current scheduling assumes that the CPU >for all

Re: Details about the Storm Scheduler

2016-08-22 Thread Bobby Evans
That is something that we have been thinking about for a while (elasticity in a topology).  There are a lot of obstacles to overcome, beyond just the scheduler.  1) The metrics feedback loop is far from ideal in being able to automatically detect a bottleneck.  Capacity kind of works, but the

Re: Details about the Storm Scheduler

2016-08-21 Thread Abhishek Agarwal
Auto scaling doesn't exist right now. And yes adding it will definitely add a lot of value On Aug 21, 2016 6:38 PM, "Aakash Khochare" wrote: Greetings, I am Aakash Khochare. I am currently a Masters student in Indian Institute of Science, Bangalore. I know that

Details about the Storm Scheduler

2016-08-21 Thread Aakash Khochare
Greetings, I am Aakash Khochare. I am currently a Masters student in Indian Institute of Science, Bangalore. I know that the number of worker processes to be used by a topology and the number of executors can be changed using storm rebalance. However, is there any facility that does this work