Re: Resource Aware Scheduler Kills Nimbus

[Wuyang Zhang]

Hi Alessio,

Thank you for your reply. I think the storm community can consider to
introduce some automatic resource analysis mechanisms to set up those
values. The resource-aware scheduling algorithm itself is great, while it
may not perform well without accurately setting the task resource demands.

Cheers,
Wuyang
ᐧ

On Mon, Feb 19, 2018 at 11:43 AM, Alessio Pagliari 
wrote:

> Hi Wuyang,
>
> This is a nice question, I didn’t find anything that computes it
> automatically, yet. By now you have to test your topology tuning the
> resource requirements until you don’t find pleasing values.
>
> Cheers,
>
> Alessio
>
> On 10 Feb 2018, at 21:42, Wuyang Zhang  wrote:
>
> Hi Jerry,
>
> I am not familiar with the pull operation on the Storm website... Can you
> do it when you get a chance?
>
> Also, I wonder how can I accurately specify the resource requirement,
> especially the cpu demand, for each task?
>
> Best,
> Wuyang
> ᐧ
>
> On Sat, Feb 10, 2018 at 3:21 PM, Jerry Peng 
> wrote:
>
>> Yup, an error in the documentation
>>
>> Feel free to submit a pull request to fix the documentation.
>>
>> Best,
>>
>> Jerry
>>
>> On Sat, Feb 10, 2018 at 1:42 PM, Wuyang Zhang 
>> wrote:
>>
>>> Ok, I find it works by adding
>>> storm.scheduler: org.apache.storm.scheduler.res
>>> ource.ResourceAwareScheduler
>>>
>>> instead of adding
>>>
>>> storm.scheduler: 
>>> “org.apache.storm.scheduler.resource.ResourceAwareScheduler”
>>>
>>>
>>>
>>> suggested by the website..
>>>
>>> ᐧ
>>>
>>> On Sat, Feb 10, 2018 at 2:22 PM, Wuyang Zhang >> > wrote:
>>>
 When I redo the procedure, here is the more log

 *2018-02-10 14:10:32.791 o.a.s.d.nimbus main [INFO] Using custom
 scheduler: “org.apache.storm.scheduler.re
 source.ResourceAwareScheduler”*
 *2018-02-10 14:10:32.793 o.a.s.d.nimbus main [ERROR] Error on
 initialization of server service-handler*
 *java.lang.ClassNotFoundException: “org.apache.storm.scheduler.re
 source.ResourceAwareScheduler”*
 * at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
 ~[?:1.8.0_151]*
 * at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
 ~[?:1.8.0_151]*
 * at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
 ~[?:1.8.0_151]*
 * at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
 ~[?:1.8.0_151]*
 * at java.lang.Class.forName0(Native Method) ~[?:1.8.0_151]*
 * at java.lang.Class.forName(Class.java:264) ~[?:1.8.0_151]*
 * at org.apache.storm.util$new_instance.invoke(util.clj:1027)
 ~[storm-core-1.1.1.jar:1.1.1]*
 * at org.apache.storm.daemon.nimbus$mk_scheduler.invoke(nimbus.cl
 j:127) ~[storm-core-1.1.1.jar:1.1.1]*
 * at org.apache.storm.daemon.nimbus$nimbus_data.invoke(nimbus.clj:215)
 ~[storm-core-1.1.1.jar:1.1.1]*
 * at
 org.apache.storm.daemon.nimbus$fn__11007$exec_fn__1370__auto11008.invoke(nimbus.clj:2451)
 ~[storm-core-1.1.1.jar:1.1.1]*
 * at clojure.lang.AFn.applyToHelper(AFn.java:156)
 ~[clojure-1.7.0.jar:?]*
 * at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.7.0.jar:?]*
 * at clojure.core$apply.invoke(core.clj:630) ~[clojure-1.7.0.jar:?]*
 * at
 org.apache.storm.daemon.nimbus$fn__11007$service_handler__11040.doInvoke(nimbus.clj:2448)
 ~[storm-core-1.1.1.jar:1.1.1]*
 * at clojure.lang.RestFn.invoke(RestFn.java:421) ~[clojure-1.7.0.jar:?]*
 * at
 org.apache.storm.daemon.nimbus$launch_server_BANG_.invoke(nimbus.clj:2536)
 ~[storm-core-1.1.1.jar:1.1.1]*
 * at org.apache.storm.daemon.nimbus$_launch.invoke(nimbus.clj:2569)
 ~[storm-core-1.1.1.jar:1.1.1]*
 * at org.apache.storm.daemon.nimbus$_main.invoke(nimbus.clj:2592)
 ~[storm-core-1.1.1.jar:1.1.1]*
 * at clojure.lang.AFn.applyToHelper(AFn.java:152)
 ~[clojure-1.7.0.jar:?]*
 * at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.7.0.jar:?]*
 * at org.apache.storm.daemon.nimbus.main(Unknown Source)
 ~[storm-core-1.1.1.jar:1.1.1]*
 *2018-02-10 14:10:32.806 o.a.s.util main [ERROR] Halting process:
 ("Error on initialization")*
 *java.lang.RuntimeException: ("Error on initialization")*
 * at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341)
 ~[storm-core-1.1.1.jar:1.1.1]*
 * at clojure.lang.RestFn.invoke(RestFn.java:423) ~[clojure-1.7.0.jar:?]*
 * at
 org.apache.storm.daemon.nimbus$fn__11007$service_handler__11040.doInvoke(nimbus.clj:2448)
 ~[storm-core-1.1.1.jar:1.1.1]*
 * at clojure.lang.RestFn.invoke(RestFn.java:421) ~[clojure-1.7.0.jar:?]*
 * at
 org.apache.storm.daemon.nimbus$launch_server_BANG_.invoke(nimbus.clj:2536)
 ~[storm-core-1.1.1.jar:1.1.1]*
 * at org.apache.storm.daemon.nimbus$_launch.invoke(nimbus.clj:2569)
 ~[storm-core-1.1.1.jar:1.1.1]*
 * at org.apache.storm.daemon.nimbus$_main.invoke(nimbus.clj:2592)
 ~[storm-core-1.1.1.jar:1.1.

Re: Resource Aware Scheduler Kills Nimbus

[Alessio Pagliari]

Hi Wuyang,

This is a nice question, I didn’t find anything that computes it automatically, 
yet. By now you have to test your topology tuning the resource requirements 
until you don’t find pleasing values.

Cheers,

Alessio

> On 10 Feb 2018, at 21:42, Wuyang Zhang  wrote:
> 
> Hi Jerry, 
> 
> I am not familiar with the pull operation on the Storm website... Can you do 
> it when you get a chance?
> 
> Also, I wonder how can I accurately specify the resource requirement, 
> especially the cpu demand, for each task?
> 
> Best, 
> Wuyang
> ᐧ
> 
> On Sat, Feb 10, 2018 at 3:21 PM, Jerry Peng  > wrote:
> Yup, an error in the documentation
> 
> Feel free to submit a pull request to fix the documentation.
> 
> Best,
> 
> Jerry
> 
> On Sat, Feb 10, 2018 at 1:42 PM, Wuyang Zhang  > wrote:
> Ok, I find it works by adding 
> storm.scheduler: org.apache.storm.scheduler.resource.ResourceAwareScheduler
> 
> instead of adding 
> 
> storm.scheduler: “org.apache.storm.scheduler.re 
> source.ResourceAwareScheduler”
> 
> 
> suggested by the website..
> 
> ᐧ
> 
> On Sat, Feb 10, 2018 at 2:22 PM, Wuyang Zhang  > wrote:
> When I redo the procedure, here is the more log
> 
> 2018-02-10 14:10:32.791 o.a.s.d.nimbus main [INFO] Using custom scheduler: 
> “org.apache.storm.scheduler.re 
> source.ResourceAwareScheduler”
> 2018-02-10 14:10:32.793 o.a.s.d.nimbus main [ERROR] Error on initialization 
> of server service-handler
> java.lang.ClassNotFoundException: “org.apache.storm.scheduler.re 
> source.ResourceAwareScheduler”
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:381) 
> ~[?:1.8.0_151]
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_151]
>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) 
> ~[?:1.8.0_151]
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_151]
>   at java.lang.Class.forName0(Native Method) ~[?:1.8.0_151]
>   at java.lang.Class.forName(Class.java:264) ~[?:1.8.0_151]
>   at org.apache.storm.util$new_instance.invoke(util.clj:1027) 
> ~[storm-core-1.1.1.jar:1.1.1]
>   at org.apache.storm.daemon.nimbus$mk_scheduler.invoke(nimbus.cl 
> j:127) ~[storm-core-1.1.1.jar:1.1.1]
>   at org.apache.storm.daemon.nimbus$nimbus_data.invoke(nimbus.clj:215) 
> ~[storm-core-1.1.1.jar:1.1.1]
>   at 
> org.apache.storm.daemon.nimbus$fn__11007$exec_fn__1370__auto11008.invoke(nimbus.clj:2451)
>  ~[storm-core-1.1.1.jar:1.1.1]
>   at clojure.lang.AFn.applyToHelper(AFn.java:156) ~[clojure-1.7.0.jar:?]
>   at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.7.0.jar:?]
>   at clojure.core$apply.invoke(core.clj:630) ~[clojure-1.7.0.jar:?]
>   at 
> org.apache.storm.daemon.nimbus$fn__11007$service_handler__11040.doInvoke(nimbus.clj:2448)
>  ~[storm-core-1.1.1.jar:1.1.1]
>   at clojure.lang.RestFn.invoke(RestFn.java:421) ~[clojure-1.7.0.jar:?]
>   at 
> org.apache.storm.daemon.nimbus$launch_server_BANG_.invoke(nimbus.clj:2536) 
> ~[storm-core-1.1.1.jar:1.1.1]
>   at org.apache.storm.daemon.nimbus$_launch.invoke(nimbus.clj:2569) 
> ~[storm-core-1.1.1.jar:1.1.1]
>   at org.apache.storm.daemon.nimbus$_main.invoke(nimbus.clj:2592) 
> ~[storm-core-1.1.1.jar:1.1.1]
>   at clojure.lang.AFn.applyToHelper(AFn.java:152) ~[clojure-1.7.0.jar:?]
>   at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.7.0.jar:?]
>   at org.apache.storm.daemon.nimbus.main(Unknown Source) 
> ~[storm-core-1.1.1.jar:1.1.1]
> 2018-02-10 14:10:32.806 o.a.s.util main [ERROR] Halting process: ("Error on 
> initialization")
> java.lang.RuntimeException: ("Error on initialization")
>   at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341) 
> ~[storm-core-1.1.1.jar:1.1.1]
>   at clojure.lang.RestFn.invoke(RestFn.java:423) ~[clojure-1.7.0.jar:?]
>   at 
> org.apache.storm.daemon.nimbus$fn__11007$service_handler__11040.doInvoke(nimbus.clj:2448)
>  ~[storm-core-1.1.1.jar:1.1.1]
>   at clojure.lang.RestFn.invoke(RestFn.java:421) ~[clojure-1.7.0.jar:?]
>   at 
> org.apache.storm.daemon.nimbus$launch_server_BANG_.invoke(nimbus.clj:2536) 
> ~[storm-core-1.1.1.jar:1.1.1]
>   at org.apache.storm.daemon.nimbus$_launch.invoke(nimbus.clj:2569) 
> ~[storm-core-1.1.1.jar:1.1.1]
>   at org.apache.storm.daemon.nimbus$_main.invoke(nimbus.clj:2592) 
> ~[storm-core-1.1.1.jar:1.1.1]
>   at clojure.lang.AFn.applyToHelper(AFn.java:152) ~[clojure-1.7.0.jar:?]
>   at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.7.0.jar:?]
>   at org.apache.storm.daemon.nimbus.main(Unknown Source) 
> ~[storm-core-1.1.1.jar:1.1.1]
> 
> I download a binary version of  Storm 1.1.1 from 
> http://www.apache.org/dyn/closer.lua/storm/apache-storm-1.1.1/apache-storm-1.1.1.

Re: Resource Aware Scheduler Kills Nimbus

[Wuyang Zhang]

Hi Jerry,

I am not familiar with the pull operation on the Storm website... Can you
do it when you get a chance?

Also, I wonder how can I accurately specify the resource requirement,
especially the cpu demand, for each task?

Best,
Wuyang
ᐧ

On Sat, Feb 10, 2018 at 3:21 PM, Jerry Peng 
wrote:

> Yup, an error in the documentation
>
> Feel free to submit a pull request to fix the documentation.
>
> Best,
>
> Jerry
>
> On Sat, Feb 10, 2018 at 1:42 PM, Wuyang Zhang 
> wrote:
>
>> Ok, I find it works by adding
>> storm.scheduler: org.apache.storm.scheduler.res
>> ource.ResourceAwareScheduler
>>
>> instead of adding
>>
>> storm.scheduler: “org.apache.storm.scheduler.resource.ResourceAwareScheduler”
>>
>> suggested by the website..
>>
>> ᐧ
>>
>> On Sat, Feb 10, 2018 at 2:22 PM, Wuyang Zhang 
>> wrote:
>>
>>> When I redo the procedure, here is the more log
>>>
>>> *2018-02-10 14:10:32.791 o.a.s.d.nimbus main [INFO] Using custom
>>> scheduler: “org.apache.storm.scheduler.re
>>> source.ResourceAwareScheduler”*
>>> *2018-02-10 14:10:32.793 o.a.s.d.nimbus main [ERROR] Error on
>>> initialization of server service-handler*
>>> *java.lang.ClassNotFoundException: “org.apache.storm.scheduler.re
>>> source.ResourceAwareScheduler”*
>>> * at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>>> ~[?:1.8.0_151]*
>>> * at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>>> ~[?:1.8.0_151]*
>>> * at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
>>> ~[?:1.8.0_151]*
>>> * at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>>> ~[?:1.8.0_151]*
>>> * at java.lang.Class.forName0(Native Method) ~[?:1.8.0_151]*
>>> * at java.lang.Class.forName(Class.java:264) ~[?:1.8.0_151]*
>>> * at org.apache.storm.util$new_instance.invoke(util.clj:1027)
>>> ~[storm-core-1.1.1.jar:1.1.1]*
>>> * at org.apache.storm.daemon.nimbus$mk_scheduler.invoke(nimbus.cl
>>> j:127) ~[storm-core-1.1.1.jar:1.1.1]*
>>> * at org.apache.storm.daemon.nimbus$nimbus_data.invoke(nimbus.clj:215)
>>> ~[storm-core-1.1.1.jar:1.1.1]*
>>> * at
>>> org.apache.storm.daemon.nimbus$fn__11007$exec_fn__1370__auto11008.invoke(nimbus.clj:2451)
>>> ~[storm-core-1.1.1.jar:1.1.1]*
>>> * at clojure.lang.AFn.applyToHelper(AFn.java:156) ~[clojure-1.7.0.jar:?]*
>>> * at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.7.0.jar:?]*
>>> * at clojure.core$apply.invoke(core.clj:630) ~[clojure-1.7.0.jar:?]*
>>> * at
>>> org.apache.storm.daemon.nimbus$fn__11007$service_handler__11040.doInvoke(nimbus.clj:2448)
>>> ~[storm-core-1.1.1.jar:1.1.1]*
>>> * at clojure.lang.RestFn.invoke(RestFn.java:421) ~[clojure-1.7.0.jar:?]*
>>> * at
>>> org.apache.storm.daemon.nimbus$launch_server_BANG_.invoke(nimbus.clj:2536)
>>> ~[storm-core-1.1.1.jar:1.1.1]*
>>> * at org.apache.storm.daemon.nimbus$_launch.invoke(nimbus.clj:2569)
>>> ~[storm-core-1.1.1.jar:1.1.1]*
>>> * at org.apache.storm.daemon.nimbus$_main.invoke(nimbus.clj:2592)
>>> ~[storm-core-1.1.1.jar:1.1.1]*
>>> * at clojure.lang.AFn.applyToHelper(AFn.java:152) ~[clojure-1.7.0.jar:?]*
>>> * at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.7.0.jar:?]*
>>> * at org.apache.storm.daemon.nimbus.main(Unknown Source)
>>> ~[storm-core-1.1.1.jar:1.1.1]*
>>> *2018-02-10 14:10:32.806 o.a.s.util main [ERROR] Halting process:
>>> ("Error on initialization")*
>>> *java.lang.RuntimeException: ("Error on initialization")*
>>> * at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341)
>>> ~[storm-core-1.1.1.jar:1.1.1]*
>>> * at clojure.lang.RestFn.invoke(RestFn.java:423) ~[clojure-1.7.0.jar:?]*
>>> * at
>>> org.apache.storm.daemon.nimbus$fn__11007$service_handler__11040.doInvoke(nimbus.clj:2448)
>>> ~[storm-core-1.1.1.jar:1.1.1]*
>>> * at clojure.lang.RestFn.invoke(RestFn.java:421) ~[clojure-1.7.0.jar:?]*
>>> * at
>>> org.apache.storm.daemon.nimbus$launch_server_BANG_.invoke(nimbus.clj:2536)
>>> ~[storm-core-1.1.1.jar:1.1.1]*
>>> * at org.apache.storm.daemon.nimbus$_launch.invoke(nimbus.clj:2569)
>>> ~[storm-core-1.1.1.jar:1.1.1]*
>>> * at org.apache.storm.daemon.nimbus$_main.invoke(nimbus.clj:2592)
>>> ~[storm-core-1.1.1.jar:1.1.1]*
>>> * at clojure.lang.AFn.applyToHelper(AFn.java:152) ~[clojure-1.7.0.jar:?]*
>>> * at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.7.0.jar:?]*
>>> * at org.apache.storm.daemon.nimbus.main(Unknown Source)
>>> ~[storm-core-1.1.1.jar:1.1.1]*
>>>
>>> I download a binary version of  Storm 1.1.1 from
>>> http://www.apache.org/dyn/closer.lua/storm/apache-storm
>>> -1.1.1/apache-storm-1.1.1.tar.gz.
>>> ᐧ
>>>
>>> On Sat, Feb 10, 2018 at 2:02 PM, Wuyang Zhang >> > wrote:
>>>
 Hi Jerry,

 I just retry to use the resource aware scheduler in a single node with
 4 slots without any further settings. The scheduler still kills the nimbus
 process. Please find nimbus.log and storm.yaml attached.

 Without add the line

 storm.scheduler: 
 “org.apache.storm.scheduler.re

Re: Resource Aware Scheduler Kills Nimbus

[Jerry Peng]

Yup, an error in the documentation

Feel free to submit a pull request to fix the documentation.

Best,

Jerry

On Sat, Feb 10, 2018 at 1:42 PM, Wuyang Zhang 
wrote:

> Ok, I find it works by adding
> storm.scheduler: org.apache.storm.scheduler.resource.
> ResourceAwareScheduler
>
> instead of adding
>
> storm.scheduler: “org.apache.storm.scheduler.resource.ResourceAwareScheduler”
>
> suggested by the website..
>
> ᐧ
>
> On Sat, Feb 10, 2018 at 2:22 PM, Wuyang Zhang 
> wrote:
>
>> When I redo the procedure, here is the more log
>>
>> *2018-02-10 14:10:32.791 o.a.s.d.nimbus main [INFO] Using custom
>> scheduler: “org.apache.storm.scheduler.re
>> source.ResourceAwareScheduler”*
>> *2018-02-10 14:10:32.793 o.a.s.d.nimbus main [ERROR] Error on
>> initialization of server service-handler*
>> *java.lang.ClassNotFoundException: “org.apache.storm.scheduler.re
>> source.ResourceAwareScheduler”*
>> * at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
>> ~[?:1.8.0_151]*
>> * at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_151]*
>> * at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
>> ~[?:1.8.0_151]*
>> * at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_151]*
>> * at java.lang.Class.forName0(Native Method) ~[?:1.8.0_151]*
>> * at java.lang.Class.forName(Class.java:264) ~[?:1.8.0_151]*
>> * at org.apache.storm.util$new_instance.invoke(util.clj:1027)
>> ~[storm-core-1.1.1.jar:1.1.1]*
>> * at org.apache.storm.daemon.nimbus$mk_scheduler.invoke(nimbus.clj:127)
>> ~[storm-core-1.1.1.jar:1.1.1]*
>> * at org.apache.storm.daemon.nimbus$nimbus_data.invoke(nimbus.clj:215)
>> ~[storm-core-1.1.1.jar:1.1.1]*
>> * at
>> org.apache.storm.daemon.nimbus$fn__11007$exec_fn__1370__auto11008.invoke(nimbus.clj:2451)
>> ~[storm-core-1.1.1.jar:1.1.1]*
>> * at clojure.lang.AFn.applyToHelper(AFn.java:156) ~[clojure-1.7.0.jar:?]*
>> * at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.7.0.jar:?]*
>> * at clojure.core$apply.invoke(core.clj:630) ~[clojure-1.7.0.jar:?]*
>> * at
>> org.apache.storm.daemon.nimbus$fn__11007$service_handler__11040.doInvoke(nimbus.clj:2448)
>> ~[storm-core-1.1.1.jar:1.1.1]*
>> * at clojure.lang.RestFn.invoke(RestFn.java:421) ~[clojure-1.7.0.jar:?]*
>> * at
>> org.apache.storm.daemon.nimbus$launch_server_BANG_.invoke(nimbus.clj:2536)
>> ~[storm-core-1.1.1.jar:1.1.1]*
>> * at org.apache.storm.daemon.nimbus$_launch.invoke(nimbus.clj:2569)
>> ~[storm-core-1.1.1.jar:1.1.1]*
>> * at org.apache.storm.daemon.nimbus$_main.invoke(nimbus.clj:2592)
>> ~[storm-core-1.1.1.jar:1.1.1]*
>> * at clojure.lang.AFn.applyToHelper(AFn.java:152) ~[clojure-1.7.0.jar:?]*
>> * at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.7.0.jar:?]*
>> * at org.apache.storm.daemon.nimbus.main(Unknown Source)
>> ~[storm-core-1.1.1.jar:1.1.1]*
>> *2018-02-10 14:10:32.806 o.a.s.util main [ERROR] Halting process: ("Error
>> on initialization")*
>> *java.lang.RuntimeException: ("Error on initialization")*
>> * at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341)
>> ~[storm-core-1.1.1.jar:1.1.1]*
>> * at clojure.lang.RestFn.invoke(RestFn.java:423) ~[clojure-1.7.0.jar:?]*
>> * at
>> org.apache.storm.daemon.nimbus$fn__11007$service_handler__11040.doInvoke(nimbus.clj:2448)
>> ~[storm-core-1.1.1.jar:1.1.1]*
>> * at clojure.lang.RestFn.invoke(RestFn.java:421) ~[clojure-1.7.0.jar:?]*
>> * at
>> org.apache.storm.daemon.nimbus$launch_server_BANG_.invoke(nimbus.clj:2536)
>> ~[storm-core-1.1.1.jar:1.1.1]*
>> * at org.apache.storm.daemon.nimbus$_launch.invoke(nimbus.clj:2569)
>> ~[storm-core-1.1.1.jar:1.1.1]*
>> * at org.apache.storm.daemon.nimbus$_main.invoke(nimbus.clj:2592)
>> ~[storm-core-1.1.1.jar:1.1.1]*
>> * at clojure.lang.AFn.applyToHelper(AFn.java:152) ~[clojure-1.7.0.jar:?]*
>> * at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.7.0.jar:?]*
>> * at org.apache.storm.daemon.nimbus.main(Unknown Source)
>> ~[storm-core-1.1.1.jar:1.1.1]*
>>
>> I download a binary version of  Storm 1.1.1 from
>> http://www.apache.org/dyn/closer.lua/storm/apache-storm
>> -1.1.1/apache-storm-1.1.1.tar.gz.
>> ᐧ
>>
>> On Sat, Feb 10, 2018 at 2:02 PM, Wuyang Zhang 
>> wrote:
>>
>>> Hi Jerry,
>>>
>>> I just retry to use the resource aware scheduler in a single node with 4
>>> slots without any further settings. The scheduler still kills the nimbus
>>> process. Please find nimbus.log and storm.yaml attached.
>>>
>>> Without add the line
>>>
>>> storm.scheduler: 
>>> “org.apache.storm.scheduler.resource.ResourceAwareScheduler”
>>>
>>> in storm.yaml, everything works fine.
>>>
>>> I also tried
>>>
>>> conf.setTopologyStrategy(org.apache.storm.scheduler.resource.strategies.scheduling.DefaultResourceAwareStrategy.class);
>>>
>>> The cluster can run, but I cannot see any log related to
>>> DefaultResourceAware. It seems like that it doesn't really run.
>>>
>>> Please take a look at the log and see if I miss anything to enable them
>

Re: Resource Aware Scheduler Kills Nimbus

[Wuyang Zhang]

Ok, I find it works by adding
storm.scheduler: org.apache.storm.scheduler.resource.ResourceAwareScheduler

instead of adding

storm.scheduler: “org.apache.storm.scheduler.resource.ResourceAwareScheduler”

suggested by the website..

ᐧ

On Sat, Feb 10, 2018 at 2:22 PM, Wuyang Zhang 
wrote:

> When I redo the procedure, here is the more log
>
> *2018-02-10 14:10:32.791 o.a.s.d.nimbus main [INFO] Using custom
> scheduler: “org.apache.storm.scheduler.resource.ResourceAwareScheduler”*
> *2018-02-10 14:10:32.793 o.a.s.d.nimbus main [ERROR] Error on
> initialization of server service-handler*
> *java.lang.ClassNotFoundException:
> “org.apache.storm.scheduler.resource.ResourceAwareScheduler”*
> * at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> ~[?:1.8.0_151]*
> * at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_151]*
> * at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
> ~[?:1.8.0_151]*
> * at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_151]*
> * at java.lang.Class.forName0(Native Method) ~[?:1.8.0_151]*
> * at java.lang.Class.forName(Class.java:264) ~[?:1.8.0_151]*
> * at org.apache.storm.util$new_instance.invoke(util.clj:1027)
> ~[storm-core-1.1.1.jar:1.1.1]*
> * at org.apache.storm.daemon.nimbus$mk_scheduler.invoke(nimbus.clj:127)
> ~[storm-core-1.1.1.jar:1.1.1]*
> * at org.apache.storm.daemon.nimbus$nimbus_data.invoke(nimbus.clj:215)
> ~[storm-core-1.1.1.jar:1.1.1]*
> * at
> org.apache.storm.daemon.nimbus$fn__11007$exec_fn__1370__auto11008.invoke(nimbus.clj:2451)
> ~[storm-core-1.1.1.jar:1.1.1]*
> * at clojure.lang.AFn.applyToHelper(AFn.java:156) ~[clojure-1.7.0.jar:?]*
> * at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.7.0.jar:?]*
> * at clojure.core$apply.invoke(core.clj:630) ~[clojure-1.7.0.jar:?]*
> * at
> org.apache.storm.daemon.nimbus$fn__11007$service_handler__11040.doInvoke(nimbus.clj:2448)
> ~[storm-core-1.1.1.jar:1.1.1]*
> * at clojure.lang.RestFn.invoke(RestFn.java:421) ~[clojure-1.7.0.jar:?]*
> * at
> org.apache.storm.daemon.nimbus$launch_server_BANG_.invoke(nimbus.clj:2536)
> ~[storm-core-1.1.1.jar:1.1.1]*
> * at org.apache.storm.daemon.nimbus$_launch.invoke(nimbus.clj:2569)
> ~[storm-core-1.1.1.jar:1.1.1]*
> * at org.apache.storm.daemon.nimbus$_main.invoke(nimbus.clj:2592)
> ~[storm-core-1.1.1.jar:1.1.1]*
> * at clojure.lang.AFn.applyToHelper(AFn.java:152) ~[clojure-1.7.0.jar:?]*
> * at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.7.0.jar:?]*
> * at org.apache.storm.daemon.nimbus.main(Unknown Source)
> ~[storm-core-1.1.1.jar:1.1.1]*
> *2018-02-10 14:10:32.806 o.a.s.util main [ERROR] Halting process: ("Error
> on initialization")*
> *java.lang.RuntimeException: ("Error on initialization")*
> * at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341)
> ~[storm-core-1.1.1.jar:1.1.1]*
> * at clojure.lang.RestFn.invoke(RestFn.java:423) ~[clojure-1.7.0.jar:?]*
> * at
> org.apache.storm.daemon.nimbus$fn__11007$service_handler__11040.doInvoke(nimbus.clj:2448)
> ~[storm-core-1.1.1.jar:1.1.1]*
> * at clojure.lang.RestFn.invoke(RestFn.java:421) ~[clojure-1.7.0.jar:?]*
> * at
> org.apache.storm.daemon.nimbus$launch_server_BANG_.invoke(nimbus.clj:2536)
> ~[storm-core-1.1.1.jar:1.1.1]*
> * at org.apache.storm.daemon.nimbus$_launch.invoke(nimbus.clj:2569)
> ~[storm-core-1.1.1.jar:1.1.1]*
> * at org.apache.storm.daemon.nimbus$_main.invoke(nimbus.clj:2592)
> ~[storm-core-1.1.1.jar:1.1.1]*
> * at clojure.lang.AFn.applyToHelper(AFn.java:152) ~[clojure-1.7.0.jar:?]*
> * at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.7.0.jar:?]*
> * at org.apache.storm.daemon.nimbus.main(Unknown Source)
> ~[storm-core-1.1.1.jar:1.1.1]*
>
> I download a binary version of  Storm 1.1.1 from http://www.apache.org/
> dyn/closer.lua/storm/apache-storm-1.1.1/apache-storm-1.1.1.tar.gz.
> ᐧ
>
> On Sat, Feb 10, 2018 at 2:02 PM, Wuyang Zhang 
> wrote:
>
>> Hi Jerry,
>>
>> I just retry to use the resource aware scheduler in a single node with 4
>> slots without any further settings. The scheduler still kills the nimbus
>> process. Please find nimbus.log and storm.yaml attached.
>>
>> Without add the line
>>
>> storm.scheduler: “org.apache.storm.scheduler.resource.ResourceAwareScheduler”
>>
>> in storm.yaml, everything works fine.
>>
>> I also tried
>>
>> conf.setTopologyStrategy(org.apache.storm.scheduler.resource.strategies.scheduling.DefaultResourceAwareStrategy.class);
>>
>> The cluster can run, but I cannot see any log related to
>> DefaultResourceAware. It seems like that it doesn't really run.
>>
>> Please take a look at the log and see if I miss anything to enable them
>> to work?
>>
>> Best,
>> Wuyang
>> ᐧ
>>
>> On Sat, Feb 10, 2018 at 12:23 PM, Zhang Wuyang 
>> wrote:
>>
>>> Hi Jerry,
>>>
>>> Thank you for the reply.
>>>
>>> Please find the storm.yaml below:
>>>
>>> ### These MUST be filled in for a storm configuration
>>>  storm.zookeeper.servers:
>>>  - "10.10.21.1"
>>>  - "10.10.21.2"
>>>  - "10.10.21.3"
>>>

Re: Resource Aware Scheduler Kills Nimbus

[Wuyang Zhang]

When I redo the procedure, here is the more log

*2018-02-10 14:10:32.791 o.a.s.d.nimbus main [INFO] Using custom scheduler:
“org.apache.storm.scheduler.resource.ResourceAwareScheduler”*
*2018-02-10 14:10:32.793 o.a.s.d.nimbus main [ERROR] Error on
initialization of server service-handler*
*java.lang.ClassNotFoundException:
“org.apache.storm.scheduler.resource.ResourceAwareScheduler”*
* at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
~[?:1.8.0_151]*
* at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_151]*
* at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
~[?:1.8.0_151]*
* at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_151]*
* at java.lang.Class.forName0(Native Method) ~[?:1.8.0_151]*
* at java.lang.Class.forName(Class.java:264) ~[?:1.8.0_151]*
* at org.apache.storm.util$new_instance.invoke(util.clj:1027)
~[storm-core-1.1.1.jar:1.1.1]*
* at org.apache.storm.daemon.nimbus$mk_scheduler.invoke(nimbus.clj:127)
~[storm-core-1.1.1.jar:1.1.1]*
* at org.apache.storm.daemon.nimbus$nimbus_data.invoke(nimbus.clj:215)
~[storm-core-1.1.1.jar:1.1.1]*
* at
org.apache.storm.daemon.nimbus$fn__11007$exec_fn__1370__auto11008.invoke(nimbus.clj:2451)
~[storm-core-1.1.1.jar:1.1.1]*
* at clojure.lang.AFn.applyToHelper(AFn.java:156) ~[clojure-1.7.0.jar:?]*
* at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.7.0.jar:?]*
* at clojure.core$apply.invoke(core.clj:630) ~[clojure-1.7.0.jar:?]*
* at
org.apache.storm.daemon.nimbus$fn__11007$service_handler__11040.doInvoke(nimbus.clj:2448)
~[storm-core-1.1.1.jar:1.1.1]*
* at clojure.lang.RestFn.invoke(RestFn.java:421) ~[clojure-1.7.0.jar:?]*
* at
org.apache.storm.daemon.nimbus$launch_server_BANG_.invoke(nimbus.clj:2536)
~[storm-core-1.1.1.jar:1.1.1]*
* at org.apache.storm.daemon.nimbus$_launch.invoke(nimbus.clj:2569)
~[storm-core-1.1.1.jar:1.1.1]*
* at org.apache.storm.daemon.nimbus$_main.invoke(nimbus.clj:2592)
~[storm-core-1.1.1.jar:1.1.1]*
* at clojure.lang.AFn.applyToHelper(AFn.java:152) ~[clojure-1.7.0.jar:?]*
* at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.7.0.jar:?]*
* at org.apache.storm.daemon.nimbus.main(Unknown Source)
~[storm-core-1.1.1.jar:1.1.1]*
*2018-02-10 14:10:32.806 o.a.s.util main [ERROR] Halting process: ("Error
on initialization")*
*java.lang.RuntimeException: ("Error on initialization")*
* at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341)
~[storm-core-1.1.1.jar:1.1.1]*
* at clojure.lang.RestFn.invoke(RestFn.java:423) ~[clojure-1.7.0.jar:?]*
* at
org.apache.storm.daemon.nimbus$fn__11007$service_handler__11040.doInvoke(nimbus.clj:2448)
~[storm-core-1.1.1.jar:1.1.1]*
* at clojure.lang.RestFn.invoke(RestFn.java:421) ~[clojure-1.7.0.jar:?]*
* at
org.apache.storm.daemon.nimbus$launch_server_BANG_.invoke(nimbus.clj:2536)
~[storm-core-1.1.1.jar:1.1.1]*
* at org.apache.storm.daemon.nimbus$_launch.invoke(nimbus.clj:2569)
~[storm-core-1.1.1.jar:1.1.1]*
* at org.apache.storm.daemon.nimbus$_main.invoke(nimbus.clj:2592)
~[storm-core-1.1.1.jar:1.1.1]*
* at clojure.lang.AFn.applyToHelper(AFn.java:152) ~[clojure-1.7.0.jar:?]*
* at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.7.0.jar:?]*
* at org.apache.storm.daemon.nimbus.main(Unknown Source)
~[storm-core-1.1.1.jar:1.1.1]*

I download a binary version of  Storm 1.1.1 from
http://www.apache.org/dyn/closer.lua/storm/apache-storm-1.1.1/apache-storm-1.1.1.tar.gz
.
ᐧ

On Sat, Feb 10, 2018 at 2:02 PM, Wuyang Zhang 
wrote:

> Hi Jerry,
>
> I just retry to use the resource aware scheduler in a single node with 4
> slots without any further settings. The scheduler still kills the nimbus
> process. Please find nimbus.log and storm.yaml attached.
>
> Without add the line
>
> storm.scheduler: “org.apache.storm.scheduler.resource.ResourceAwareScheduler”
>
> in storm.yaml, everything works fine.
>
> I also tried
>
> conf.setTopologyStrategy(org.apache.storm.scheduler.resource.strategies.scheduling.DefaultResourceAwareStrategy.class);
>
> The cluster can run, but I cannot see any log related to
> DefaultResourceAware. It seems like that it doesn't really run.
>
> Please take a look at the log and see if I miss anything to enable them to
> work?
>
> Best,
> Wuyang
> ᐧ
>
> On Sat, Feb 10, 2018 at 12:23 PM, Zhang Wuyang 
> wrote:
>
>> Hi Jerry,
>>
>> Thank you for the reply.
>>
>> Please find the storm.yaml below:
>>
>> ### These MUST be filled in for a storm configuration
>>  storm.zookeeper.servers:
>>  - "10.10.21.1"
>>  - "10.10.21.2"
>>  - "10.10.21.3"
>>  - "10.10.21.4"
>>  - "10.10.21.5"
>>  - "10.10.21.6"
>>  - "10.10.21.7"
>>
>>  nimbus.seeds: ["10.10.21.1"]
>>
>>  storm.zookeeper.port: 2181
>> ## These may optionally be filled in:
>>
>>  worker.heap.memory.mb: 8192
>>
>>  supervisor.scheduler.meta:
>> tag: 10.10.20.1
>>
>> #storm.scheduler: storm.EmptyScheduler
>>  storm.scheduler: “org.apache.storm.scheduler.re
>> source.ResourceAwareScheduler”
>>  supervisor.memory.capacity.mb: 2048.0
>>  su

Re: Resource Aware Scheduler Kills Nimbus

[Wuyang Zhang]

Hi Jerry,

I just retry to use the resource aware scheduler in a single node with 4
slots without any further settings. The scheduler still kills the nimbus
process. Please find nimbus.log and storm.yaml attached.

Without add the line

storm.scheduler: “org.apache.storm.scheduler.resource.ResourceAwareScheduler”

in storm.yaml, everything works fine.

I also tried

conf.setTopologyStrategy(org.apache.storm.scheduler.resource.strategies.scheduling.DefaultResourceAwareStrategy.class);

The cluster can run, but I cannot see any log related to
DefaultResourceAware. It seems like that it doesn't really run.

Please take a look at the log and see if I miss anything to enable them to
work?

Best,
Wuyang
ᐧ

On Sat, Feb 10, 2018 at 12:23 PM, Zhang Wuyang 
wrote:

> Hi Jerry,
>
> Thank you for the reply.
>
> Please find the storm.yaml below:
>
> ### These MUST be filled in for a storm configuration
>  storm.zookeeper.servers:
>  - "10.10.21.1"
>  - "10.10.21.2"
>  - "10.10.21.3"
>  - "10.10.21.4"
>  - "10.10.21.5"
>  - "10.10.21.6"
>  - "10.10.21.7"
>
>  nimbus.seeds: ["10.10.21.1"]
>
>  storm.zookeeper.port: 2181
> ## These may optionally be filled in:
>
>  worker.heap.memory.mb: 8192
>
>  supervisor.scheduler.meta:
> tag: 10.10.20.1
>
> #storm.scheduler: storm.EmptyScheduler
>  storm.scheduler: “org.apache.storm.scheduler.resource.
> ResourceAwareScheduler”
>  supervisor.memory.capacity.mb: 2048.0
>  supervisor.cpu.capacity: 4000.0
>  supervisor.slots.ports:
>  - 6700
>  - 6701
>  - 6702
>  - 6703
>
>  worker.childopts: "-Xmx%HEAP-MEM%m -Xms6000m -XX:+PrintGCDetails
> -Xloggc:artifacts/gc.log -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps
> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1M
> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=artifacts/heapdump"
>
> On Sat, Feb 10, 2018 at 12:16 PM, Jerry Peng 
> wrote:
>
>> Wuyang,
>>
>> Can you provide your whole storm.yaml ?
>>
>> Best,
>>
>> Jerry
>>
>> On Fri, Feb 9, 2018 at 9:44 PM, Wuyang Zhang 
>> wrote:
>>
>>> Hi,
>>>
>>> I have the difficulty with the resource aware scheduler from the version
>>> 1.1.1.
>>>
>>> As suggested by the website, I add the following lines to storm.yaml:
>>>
>>>  *storm.scheduler: “org.apache.storm.scheduler.re
>>> source.ResourceAwareScheduler”*
>>> * supervisor.memory.capacity.mb: 2048.0*
>>> * supervisor.cpu.capacity: 400.0*
>>>
>>> Afterwards,  I rerun the storm and the nimbus process has been killed.
>>> The last log I found from nimbus.log is:
>>> o.a.s.d.nimbus main [INFO] Using custom scheduler: “
>>> org.apache.storm.scheduler.resource.ResourceAwareScheduler”.
>>>
>>> I suppose something goes wrong when it runs the scheduler.
>>>
>>> Do I miss anything to enable this scheduler?
>>>
>>> Best,
>>> Wuyang
>>> ᐧ
>>>
>>
>>
>


nimbus.log
Description: Binary data


storm.yaml
Description: Binary data

Re: Resource Aware Scheduler Kills Nimbus

[Zhang Wuyang]

Hi Jerry,

Thank you for the reply.

Please find the storm.yaml below:

### These MUST be filled in for a storm configuration
 storm.zookeeper.servers:
 - "10.10.21.1"
 - "10.10.21.2"
 - "10.10.21.3"
 - "10.10.21.4"
 - "10.10.21.5"
 - "10.10.21.6"
 - "10.10.21.7"

 nimbus.seeds: ["10.10.21.1"]

 storm.zookeeper.port: 2181
## These may optionally be filled in:

 worker.heap.memory.mb: 8192

 supervisor.scheduler.meta:
tag: 10.10.20.1

#storm.scheduler: storm.EmptyScheduler
 storm.scheduler:
“org.apache.storm.scheduler.resource.ResourceAwareScheduler”
 supervisor.memory.capacity.mb: 2048.0
 supervisor.cpu.capacity: 4000.0
 supervisor.slots.ports:
 - 6700
 - 6701
 - 6702
 - 6703

 worker.childopts: "-Xmx%HEAP-MEM%m -Xms6000m -XX:+PrintGCDetails
-Xloggc:artifacts/gc.log -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps
-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1M
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=artifacts/heapdump"

On Sat, Feb 10, 2018 at 12:16 PM, Jerry Peng 
wrote:

> Wuyang,
>
> Can you provide your whole storm.yaml ?
>
> Best,
>
> Jerry
>
> On Fri, Feb 9, 2018 at 9:44 PM, Wuyang Zhang 
> wrote:
>
>> Hi,
>>
>> I have the difficulty with the resource aware scheduler from the version
>> 1.1.1.
>>
>> As suggested by the website, I add the following lines to storm.yaml:
>>
>>  *storm.scheduler: “org.apache.storm.scheduler.re
>> source.ResourceAwareScheduler”*
>> * supervisor.memory.capacity.mb: 2048.0*
>> * supervisor.cpu.capacity: 400.0*
>>
>> Afterwards,  I rerun the storm and the nimbus process has been killed.
>> The last log I found from nimbus.log is:
>> o.a.s.d.nimbus main [INFO] Using custom scheduler: “
>> org.apache.storm.scheduler.resource.ResourceAwareScheduler”.
>>
>> I suppose something goes wrong when it runs the scheduler.
>>
>> Do I miss anything to enable this scheduler?
>>
>> Best,
>> Wuyang
>> ᐧ
>>
>
>

Re: Resource Aware Scheduler Kills Nimbus

[Jerry Peng]

Wuyang,

Can you provide your whole storm.yaml ?

Best,

Jerry

On Fri, Feb 9, 2018 at 9:44 PM, Wuyang Zhang 
wrote:

> Hi,
>
> I have the difficulty with the resource aware scheduler from the version
> 1.1.1.
>
> As suggested by the website, I add the following lines to storm.yaml:
>
>  *storm.scheduler:
> “org.apache.storm.scheduler.resource.ResourceAwareScheduler”*
> * supervisor.memory.capacity.mb: 2048.0*
> * supervisor.cpu.capacity: 400.0*
>
> Afterwards,  I rerun the storm and the nimbus process has been killed.
> The last log I found from nimbus.log is:
> o.a.s.d.nimbus main [INFO] Using custom scheduler:
> “org.apache.storm.scheduler.resource.ResourceAwareScheduler”.
>
> I suppose something goes wrong when it runs the scheduler.
>
> Do I miss anything to enable this scheduler?
>
> Best,
> Wuyang
> ᐧ
>

Resource Aware Scheduler Kills Nimbus

[Wuyang Zhang]

Hi,

I have the difficulty with the resource aware scheduler from the version
1.1.1.

As suggested by the website, I add the following lines to storm.yaml:

 *storm.scheduler:
“org.apache.storm.scheduler.resource.ResourceAwareScheduler”*
* supervisor.memory.capacity.mb: 2048.0*
* supervisor.cpu.capacity: 400.0*

Afterwards,  I rerun the storm and the nimbus process has been killed.
The last log I found from nimbus.log is:
o.a.s.d.nimbus main [INFO] Using custom scheduler:
“org.apache.storm.scheduler.resource.ResourceAwareScheduler”.

I suppose something goes wrong when it runs the scheduler.

Do I miss anything to enable this scheduler?

Best,
Wuyang
ᐧ