Hi Alessio, Thank you for your reply. I think the storm community can consider to introduce some automatic resource analysis mechanisms to set up those values. The resource-aware scheduling algorithm itself is great, while it may not perform well without accurately setting the task resource demands.
Cheers, Wuyang ᐧ On Mon, Feb 19, 2018 at 11:43 AM, Alessio Pagliari <[email protected]> wrote: > Hi Wuyang, > > This is a nice question, I didn’t find anything that computes it > automatically, yet. By now you have to test your topology tuning the > resource requirements until you don’t find pleasing values. > > Cheers, > > Alessio > > On 10 Feb 2018, at 21:42, Wuyang Zhang <[email protected]> wrote: > > Hi Jerry, > > I am not familiar with the pull operation on the Storm website... Can you > do it when you get a chance? > > Also, I wonder how can I accurately specify the resource requirement, > especially the cpu demand, for each task? > > Best, > Wuyang > ᐧ > > On Sat, Feb 10, 2018 at 3:21 PM, Jerry Peng <[email protected]> > wrote: > >> Yup, an error in the documentation >> >> Feel free to submit a pull request to fix the documentation. >> >> Best, >> >> Jerry >> >> On Sat, Feb 10, 2018 at 1:42 PM, Wuyang Zhang <[email protected]> >> wrote: >> >>> Ok, I find it works by adding >>> storm.scheduler: org.apache.storm.scheduler.res >>> ource.ResourceAwareScheduler >>> >>> instead of adding >>> >>> storm.scheduler: >>> “org.apache.storm.scheduler.resource.ResourceAwareScheduler” >>> >>> >>> >>> suggested by the website.. >>> >>> ᐧ >>> >>> On Sat, Feb 10, 2018 at 2:22 PM, Wuyang Zhang <[email protected] >>> > wrote: >>> >>>> When I redo the procedure, here is the more log >>>> >>>> *2018-02-10 14:10:32.791 o.a.s.d.nimbus main [INFO] Using custom >>>> scheduler: “org.apache.storm.scheduler.re >>>> <http://org.apache.storm.scheduler.re/>source.ResourceAwareScheduler”* >>>> *2018-02-10 14:10:32.793 o.a.s.d.nimbus main [ERROR] Error on >>>> initialization of server service-handler* >>>> *java.lang.ClassNotFoundException: “org.apache.storm.scheduler.re >>>> <http://org.apache.storm.scheduler.re/>source.ResourceAwareScheduler”* >>>> * at java.net.URLClassLoader.findClass(URLClassLoader.java:381) >>>> ~[?:1.8.0_151]* >>>> * at java.lang.ClassLoader.loadClass(ClassLoader.java:424) >>>> ~[?:1.8.0_151]* >>>> * at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) >>>> ~[?:1.8.0_151]* >>>> * at java.lang.ClassLoader.loadClass(ClassLoader.java:357) >>>> ~[?:1.8.0_151]* >>>> * at java.lang.Class.forName0(Native Method) ~[?:1.8.0_151]* >>>> * at java.lang.Class.forName(Class.java:264) ~[?:1.8.0_151]* >>>> * at org.apache.storm.util$new_instance.invoke(util.clj:1027) >>>> ~[storm-core-1.1.1.jar:1.1.1]* >>>> * at org.apache.storm.daemon.nimbus$mk_scheduler.invoke(nimbus.cl >>>> <http://nimbus.cl/>j:127) ~[storm-core-1.1.1.jar:1.1.1]* >>>> * at org.apache.storm.daemon.nimbus$nimbus_data.invoke(nimbus.clj:215) >>>> ~[storm-core-1.1.1.jar:1.1.1]* >>>> * at >>>> org.apache.storm.daemon.nimbus$fn__11007$exec_fn__1370__auto____11008.invoke(nimbus.clj:2451) >>>> ~[storm-core-1.1.1.jar:1.1.1]* >>>> * at clojure.lang.AFn.applyToHelper(AFn.java:156) >>>> ~[clojure-1.7.0.jar:?]* >>>> * at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.7.0.jar:?]* >>>> * at clojure.core$apply.invoke(core.clj:630) ~[clojure-1.7.0.jar:?]* >>>> * at >>>> org.apache.storm.daemon.nimbus$fn__11007$service_handler__11040.doInvoke(nimbus.clj:2448) >>>> ~[storm-core-1.1.1.jar:1.1.1]* >>>> * at clojure.lang.RestFn.invoke(RestFn.java:421) ~[clojure-1.7.0.jar:?]* >>>> * at >>>> org.apache.storm.daemon.nimbus$launch_server_BANG_.invoke(nimbus.clj:2536) >>>> ~[storm-core-1.1.1.jar:1.1.1]* >>>> * at org.apache.storm.daemon.nimbus$_launch.invoke(nimbus.clj:2569) >>>> ~[storm-core-1.1.1.jar:1.1.1]* >>>> * at org.apache.storm.daemon.nimbus$_main.invoke(nimbus.clj:2592) >>>> ~[storm-core-1.1.1.jar:1.1.1]* >>>> * at clojure.lang.AFn.applyToHelper(AFn.java:152) >>>> ~[clojure-1.7.0.jar:?]* >>>> * at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.7.0.jar:?]* >>>> * at org.apache.storm.daemon.nimbus.main(Unknown Source) >>>> ~[storm-core-1.1.1.jar:1.1.1]* >>>> *2018-02-10 14:10:32.806 o.a.s.util main [ERROR] Halting process: >>>> ("Error on initialization")* >>>> *java.lang.RuntimeException: ("Error on initialization")* >>>> * at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341) >>>> ~[storm-core-1.1.1.jar:1.1.1]* >>>> * at clojure.lang.RestFn.invoke(RestFn.java:423) ~[clojure-1.7.0.jar:?]* >>>> * at >>>> org.apache.storm.daemon.nimbus$fn__11007$service_handler__11040.doInvoke(nimbus.clj:2448) >>>> ~[storm-core-1.1.1.jar:1.1.1]* >>>> * at clojure.lang.RestFn.invoke(RestFn.java:421) ~[clojure-1.7.0.jar:?]* >>>> * at >>>> org.apache.storm.daemon.nimbus$launch_server_BANG_.invoke(nimbus.clj:2536) >>>> ~[storm-core-1.1.1.jar:1.1.1]* >>>> * at org.apache.storm.daemon.nimbus$_launch.invoke(nimbus.clj:2569) >>>> ~[storm-core-1.1.1.jar:1.1.1]* >>>> * at org.apache.storm.daemon.nimbus$_main.invoke(nimbus.clj:2592) >>>> ~[storm-core-1.1.1.jar:1.1.1]* >>>> * at clojure.lang.AFn.applyToHelper(AFn.java:152) >>>> ~[clojure-1.7.0.jar:?]* >>>> * at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.7.0.jar:?]* >>>> * at org.apache.storm.daemon.nimbus.main(Unknown Source) >>>> ~[storm-core-1.1.1.jar:1.1.1]* >>>> >>>> I download a binary version of Storm 1.1.1 from >>>> http://www.apache.org/dyn/closer.lua/storm/apache-storm >>>> -1.1.1/apache-storm-1.1.1.tar.gz. >>>> ᐧ >>>> >>>> On Sat, Feb 10, 2018 at 2:02 PM, Wuyang Zhang < >>>> [email protected]> wrote: >>>> >>>>> Hi Jerry, >>>>> >>>>> I just retry to use the resource aware scheduler in a single node with >>>>> 4 slots without any further settings. The scheduler still kills the nimbus >>>>> process. Please find nimbus.log and storm.yaml attached. >>>>> >>>>> Without add the line >>>>> >>>>> storm.scheduler: >>>>> “org.apache.storm.scheduler.resource.ResourceAwareScheduler” >>>>> >>>>> in storm.yaml, everything works fine. >>>>> >>>>> I also tried >>>>> >>>>> conf.setTopologyStrategy(org.apache.storm.scheduler.resource.strategies.scheduling.DefaultResourceAwareStrategy.class); >>>>> >>>>> The cluster can run, but I cannot see any log related to >>>>> DefaultResourceAware. It seems like that it doesn't really run. >>>>> >>>>> Please take a look at the log and see if I miss anything to enable >>>>> them to work? >>>>> >>>>> Best, >>>>> Wuyang >>>>> ᐧ >>>>> >>>>> On Sat, Feb 10, 2018 at 12:23 PM, Zhang Wuyang < >>>>> [email protected]> wrote: >>>>> >>>>>> Hi Jerry, >>>>>> >>>>>> Thank you for the reply. >>>>>> >>>>>> Please find the storm.yaml below: >>>>>> >>>>>> ########### These MUST be filled in for a storm configuration >>>>>> storm.zookeeper.servers: >>>>>> - "10.10.21.1" >>>>>> - "10.10.21.2" >>>>>> - "10.10.21.3" >>>>>> - "10.10.21.4" >>>>>> - "10.10.21.5" >>>>>> - "10.10.21.6" >>>>>> - "10.10.21.7" >>>>>> >>>>>> nimbus.seeds: ["10.10.21.1"] >>>>>> >>>>>> storm.zookeeper.port: 2181 >>>>>> ###### These may optionally be filled in: >>>>>> >>>>>> worker.heap.memory.mb: 8192 >>>>>> >>>>>> supervisor.scheduler.meta: >>>>>> tag: 10.10.20.1 >>>>>> >>>>>> #storm.scheduler: storm.EmptyScheduler >>>>>> storm.scheduler: “org.apache.storm.scheduler.re >>>>>> source.ResourceAwareScheduler” >>>>>> supervisor.memory.capacity.mb: 2048.0 >>>>>> supervisor.cpu.capacity: 4000.0 >>>>>> supervisor.slots.ports: >>>>>> - 6700 >>>>>> - 6701 >>>>>> - 6702 >>>>>> - 6703 >>>>>> >>>>>> worker.childopts: "-Xmx%HEAP-MEM%m -Xms6000m -XX:+PrintGCDetails >>>>>> -Xloggc:artifacts/gc.log -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps >>>>>> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1M >>>>>> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=artifacts/heapdump" >>>>>> >>>>>> On Sat, Feb 10, 2018 at 12:16 PM, Jerry Peng < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Wuyang, >>>>>>> >>>>>>> Can you provide your whole storm.yaml ? >>>>>>> >>>>>>> Best, >>>>>>> >>>>>>> Jerry >>>>>>> >>>>>>> On Fri, Feb 9, 2018 at 9:44 PM, Wuyang Zhang < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I have the difficulty with the resource aware scheduler from the >>>>>>>> version 1.1.1. >>>>>>>> >>>>>>>> As suggested by the website, I add the following lines to >>>>>>>> storm.yaml: >>>>>>>> >>>>>>>> *storm.scheduler: “org.apache.storm.scheduler.re >>>>>>>> <http://org.apache.storm.scheduler.re/>source.ResourceAwareScheduler”* >>>>>>>> * supervisor.memory.capacity.mb: 2048.0* >>>>>>>> * supervisor.cpu.capacity: 400.0* >>>>>>>> >>>>>>>> Afterwards, I rerun the storm and the nimbus process has been >>>>>>>> killed. >>>>>>>> The last log I found from nimbus.log is: >>>>>>>> o.a.s.d.nimbus main [INFO] Using custom scheduler: “ >>>>>>>> org.apache.storm.scheduler.resource.ResourceAwareScheduler”. >>>>>>>> >>>>>>>> I suppose something goes wrong when it runs the scheduler. >>>>>>>> >>>>>>>> Do I miss anything to enable this scheduler? >>>>>>>> >>>>>>>> Best, >>>>>>>> Wuyang >>>>>>>> ᐧ >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> > >
