Hi Wuyang, This is a nice question, I didn’t find anything that computes it automatically, yet. By now you have to test your topology tuning the resource requirements until you don’t find pleasing values.
Cheers, Alessio > On 10 Feb 2018, at 21:42, Wuyang Zhang <[email protected]> wrote: > > Hi Jerry, > > I am not familiar with the pull operation on the Storm website... Can you do > it when you get a chance? > > Also, I wonder how can I accurately specify the resource requirement, > especially the cpu demand, for each task? > > Best, > Wuyang > ᐧ > > On Sat, Feb 10, 2018 at 3:21 PM, Jerry Peng <[email protected] > <mailto:[email protected]>> wrote: > Yup, an error in the documentation > > Feel free to submit a pull request to fix the documentation. > > Best, > > Jerry > > On Sat, Feb 10, 2018 at 1:42 PM, Wuyang Zhang <[email protected] > <mailto:[email protected]>> wrote: > Ok, I find it works by adding > storm.scheduler: org.apache.storm.scheduler.resource.ResourceAwareScheduler > > instead of adding > > storm.scheduler: “org.apache.storm.scheduler.re > <http://org.apache.storm.scheduler.re/>source.ResourceAwareScheduler” > > > suggested by the website.. > > ᐧ > > On Sat, Feb 10, 2018 at 2:22 PM, Wuyang Zhang <[email protected] > <mailto:[email protected]>> wrote: > When I redo the procedure, here is the more log > > 2018-02-10 14:10:32.791 o.a.s.d.nimbus main [INFO] Using custom scheduler: > “org.apache.storm.scheduler.re > <http://org.apache.storm.scheduler.re/>source.ResourceAwareScheduler” > 2018-02-10 14:10:32.793 o.a.s.d.nimbus main [ERROR] Error on initialization > of server service-handler > java.lang.ClassNotFoundException: “org.apache.storm.scheduler.re > <http://org.apache.storm.scheduler.re/>source.ResourceAwareScheduler” > at java.net.URLClassLoader.findClass(URLClassLoader.java:381) > ~[?:1.8.0_151] > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_151] > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) > ~[?:1.8.0_151] > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_151] > at java.lang.Class.forName0(Native Method) ~[?:1.8.0_151] > at java.lang.Class.forName(Class.java:264) ~[?:1.8.0_151] > at org.apache.storm.util$new_instance.invoke(util.clj:1027) > ~[storm-core-1.1.1.jar:1.1.1] > at org.apache.storm.daemon.nimbus$mk_scheduler.invoke(nimbus.cl > <http://nimbus.cl/>j:127) ~[storm-core-1.1.1.jar:1.1.1] > at org.apache.storm.daemon.nimbus$nimbus_data.invoke(nimbus.clj:215) > ~[storm-core-1.1.1.jar:1.1.1] > at > org.apache.storm.daemon.nimbus$fn__11007$exec_fn__1370__auto____11008.invoke(nimbus.clj:2451) > ~[storm-core-1.1.1.jar:1.1.1] > at clojure.lang.AFn.applyToHelper(AFn.java:156) ~[clojure-1.7.0.jar:?] > at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.7.0.jar:?] > at clojure.core$apply.invoke(core.clj:630) ~[clojure-1.7.0.jar:?] > at > org.apache.storm.daemon.nimbus$fn__11007$service_handler__11040.doInvoke(nimbus.clj:2448) > ~[storm-core-1.1.1.jar:1.1.1] > at clojure.lang.RestFn.invoke(RestFn.java:421) ~[clojure-1.7.0.jar:?] > at > org.apache.storm.daemon.nimbus$launch_server_BANG_.invoke(nimbus.clj:2536) > ~[storm-core-1.1.1.jar:1.1.1] > at org.apache.storm.daemon.nimbus$_launch.invoke(nimbus.clj:2569) > ~[storm-core-1.1.1.jar:1.1.1] > at org.apache.storm.daemon.nimbus$_main.invoke(nimbus.clj:2592) > ~[storm-core-1.1.1.jar:1.1.1] > at clojure.lang.AFn.applyToHelper(AFn.java:152) ~[clojure-1.7.0.jar:?] > at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.7.0.jar:?] > at org.apache.storm.daemon.nimbus.main(Unknown Source) > ~[storm-core-1.1.1.jar:1.1.1] > 2018-02-10 14:10:32.806 o.a.s.util main [ERROR] Halting process: ("Error on > initialization") > java.lang.RuntimeException: ("Error on initialization") > at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341) > ~[storm-core-1.1.1.jar:1.1.1] > at clojure.lang.RestFn.invoke(RestFn.java:423) ~[clojure-1.7.0.jar:?] > at > org.apache.storm.daemon.nimbus$fn__11007$service_handler__11040.doInvoke(nimbus.clj:2448) > ~[storm-core-1.1.1.jar:1.1.1] > at clojure.lang.RestFn.invoke(RestFn.java:421) ~[clojure-1.7.0.jar:?] > at > org.apache.storm.daemon.nimbus$launch_server_BANG_.invoke(nimbus.clj:2536) > ~[storm-core-1.1.1.jar:1.1.1] > at org.apache.storm.daemon.nimbus$_launch.invoke(nimbus.clj:2569) > ~[storm-core-1.1.1.jar:1.1.1] > at org.apache.storm.daemon.nimbus$_main.invoke(nimbus.clj:2592) > ~[storm-core-1.1.1.jar:1.1.1] > at clojure.lang.AFn.applyToHelper(AFn.java:152) ~[clojure-1.7.0.jar:?] > at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.7.0.jar:?] > at org.apache.storm.daemon.nimbus.main(Unknown Source) > ~[storm-core-1.1.1.jar:1.1.1] > > I download a binary version of Storm 1.1.1 from > http://www.apache.org/dyn/closer.lua/storm/apache-storm-1.1.1/apache-storm-1.1.1.tar.gz > > <http://www.apache.org/dyn/closer.lua/storm/apache-storm-1.1.1/apache-storm-1.1.1.tar.gz>. > ᐧ > > On Sat, Feb 10, 2018 at 2:02 PM, Wuyang Zhang <[email protected] > <mailto:[email protected]>> wrote: > Hi Jerry, > > I just retry to use the resource aware scheduler in a single node with 4 > slots without any further settings. The scheduler still kills the nimbus > process. Please find nimbus.log and storm.yaml attached. > > Without add the line > storm.scheduler: “org.apache.storm.scheduler.re > <http://org.apache.storm.scheduler.re/>source.ResourceAwareScheduler” > in storm.yaml, everything works fine. > > I also tried > conf.setTopologyStrategy(org.apache.storm.scheduler.resource.strategies.scheduling.DefaultResourceAwareStrategy.class); > The cluster can run, but I cannot see any log related to > DefaultResourceAware. It seems like that it doesn't really run. > > Please take a look at the log and see if I miss anything to enable them to > work? > > Best, > Wuyang > ᐧ > > On Sat, Feb 10, 2018 at 12:23 PM, Zhang Wuyang <[email protected] > <mailto:[email protected]>> wrote: > Hi Jerry, > > Thank you for the reply. > > Please find the storm.yaml below: > > ########### These MUST be filled in for a storm configuration > storm.zookeeper.servers: > - "10.10.21.1" > - "10.10.21.2" > - "10.10.21.3" > - "10.10.21.4" > - "10.10.21.5" > - "10.10.21.6" > - "10.10.21.7" > > nimbus.seeds: ["10.10.21.1"] > > storm.zookeeper.port: 2181 > ###### These may optionally be filled in: > > worker.heap.memory.mb: 8192 > > supervisor.scheduler.meta: > tag: 10.10.20.1 > > #storm.scheduler: storm.EmptyScheduler > storm.scheduler: “org.apache.storm.scheduler.re > <http://org.apache.storm.scheduler.re/>source.ResourceAwareScheduler” > supervisor.memory.capacity.mb: 2048.0 > supervisor.cpu.capacity: 4000.0 > supervisor.slots.ports: > - 6700 > - 6701 > - 6702 > - 6703 > > worker.childopts: "-Xmx%HEAP-MEM%m -Xms6000m -XX:+PrintGCDetails > -Xloggc:artifacts/gc.log -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps > -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1M > -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=artifacts/heapdump" > > On Sat, Feb 10, 2018 at 12:16 PM, Jerry Peng <[email protected] > <mailto:[email protected]>> wrote: > Wuyang, > > Can you provide your whole storm.yaml ? > > Best, > > Jerry > > On Fri, Feb 9, 2018 at 9:44 PM, Wuyang Zhang <[email protected] > <mailto:[email protected]>> wrote: > Hi, > > I have the difficulty with the resource aware scheduler from the version > 1.1.1. > > As suggested by the website, I add the following lines to storm.yaml: > > storm.scheduler: “org.apache.storm.scheduler.re > <http://org.apache.storm.scheduler.re/>source.ResourceAwareScheduler” > supervisor.memory.capacity.mb: 2048.0 > supervisor.cpu.capacity: 400.0 > > Afterwards, I rerun the storm and the nimbus process has been killed. > The last log I found from nimbus.log is: > o.a.s.d.nimbus main [INFO] Using custom scheduler: > “org.apache.storm.scheduler.re > <http://org.apache.storm.scheduler.re/>source.ResourceAwareScheduler”. > > I suppose something goes wrong when it runs the scheduler. > > Do I miss anything to enable this scheduler? > > Best, > Wuyang > ᐧ > > > > > > >
