Hi Wuyang,

This is a nice question, I didn’t find anything that computes it automatically, 
yet. By now you have to test your topology tuning the resource requirements 
until you don’t find pleasing values.

Cheers,

Alessio

> On 10 Feb 2018, at 21:42, Wuyang Zhang <wuy...@winlab.rutgers.edu> wrote:
> 
> Hi Jerry, 
> 
> I am not familiar with the pull operation on the Storm website... Can you do 
> it when you get a chance?
> 
> Also, I wonder how can I accurately specify the resource requirement, 
> especially the cpu demand, for each task?
> 
> Best, 
> Wuyang
> ᐧ
> 
> On Sat, Feb 10, 2018 at 3:21 PM, Jerry Peng <jerry.boyang.p...@gmail.com 
> <mailto:jerry.boyang.p...@gmail.com>> wrote:
> Yup, an error in the documentation
> 
> Feel free to submit a pull request to fix the documentation.
> 
> Best,
> 
> Jerry
> 
> On Sat, Feb 10, 2018 at 1:42 PM, Wuyang Zhang <wuy...@winlab.rutgers.edu 
> <mailto:wuy...@winlab.rutgers.edu>> wrote:
> Ok, I find it works by adding 
> storm.scheduler: org.apache.storm.scheduler.resource.ResourceAwareScheduler
> 
> instead of adding 
> 
> storm.scheduler: “org.apache.storm.scheduler.re 
> <http://org.apache.storm.scheduler.re/>source.ResourceAwareScheduler”
> 
> 
> suggested by the website..
> 
> ᐧ
> 
> On Sat, Feb 10, 2018 at 2:22 PM, Wuyang Zhang <wuy...@winlab.rutgers.edu 
> <mailto:wuy...@winlab.rutgers.edu>> wrote:
> When I redo the procedure, here is the more log
> 
> 2018-02-10 14:10:32.791 o.a.s.d.nimbus main [INFO] Using custom scheduler: 
> “org.apache.storm.scheduler.re 
> <http://org.apache.storm.scheduler.re/>source.ResourceAwareScheduler”
> 2018-02-10 14:10:32.793 o.a.s.d.nimbus main [ERROR] Error on initialization 
> of server service-handler
> java.lang.ClassNotFoundException: “org.apache.storm.scheduler.re 
> <http://org.apache.storm.scheduler.re/>source.ResourceAwareScheduler”
>       at java.net.URLClassLoader.findClass(URLClassLoader.java:381) 
> ~[?:1.8.0_151]
>       at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_151]
>       at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) 
> ~[?:1.8.0_151]
>       at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_151]
>       at java.lang.Class.forName0(Native Method) ~[?:1.8.0_151]
>       at java.lang.Class.forName(Class.java:264) ~[?:1.8.0_151]
>       at org.apache.storm.util$new_instance.invoke(util.clj:1027) 
> ~[storm-core-1.1.1.jar:1.1.1]
>       at org.apache.storm.daemon.nimbus$mk_scheduler.invoke(nimbus.cl 
> <http://nimbus.cl/>j:127) ~[storm-core-1.1.1.jar:1.1.1]
>       at org.apache.storm.daemon.nimbus$nimbus_data.invoke(nimbus.clj:215) 
> ~[storm-core-1.1.1.jar:1.1.1]
>       at 
> org.apache.storm.daemon.nimbus$fn__11007$exec_fn__1370__auto____11008.invoke(nimbus.clj:2451)
>  ~[storm-core-1.1.1.jar:1.1.1]
>       at clojure.lang.AFn.applyToHelper(AFn.java:156) ~[clojure-1.7.0.jar:?]
>       at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.7.0.jar:?]
>       at clojure.core$apply.invoke(core.clj:630) ~[clojure-1.7.0.jar:?]
>       at 
> org.apache.storm.daemon.nimbus$fn__11007$service_handler__11040.doInvoke(nimbus.clj:2448)
>  ~[storm-core-1.1.1.jar:1.1.1]
>       at clojure.lang.RestFn.invoke(RestFn.java:421) ~[clojure-1.7.0.jar:?]
>       at 
> org.apache.storm.daemon.nimbus$launch_server_BANG_.invoke(nimbus.clj:2536) 
> ~[storm-core-1.1.1.jar:1.1.1]
>       at org.apache.storm.daemon.nimbus$_launch.invoke(nimbus.clj:2569) 
> ~[storm-core-1.1.1.jar:1.1.1]
>       at org.apache.storm.daemon.nimbus$_main.invoke(nimbus.clj:2592) 
> ~[storm-core-1.1.1.jar:1.1.1]
>       at clojure.lang.AFn.applyToHelper(AFn.java:152) ~[clojure-1.7.0.jar:?]
>       at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.7.0.jar:?]
>       at org.apache.storm.daemon.nimbus.main(Unknown Source) 
> ~[storm-core-1.1.1.jar:1.1.1]
> 2018-02-10 14:10:32.806 o.a.s.util main [ERROR] Halting process: ("Error on 
> initialization")
> java.lang.RuntimeException: ("Error on initialization")
>       at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341) 
> ~[storm-core-1.1.1.jar:1.1.1]
>       at clojure.lang.RestFn.invoke(RestFn.java:423) ~[clojure-1.7.0.jar:?]
>       at 
> org.apache.storm.daemon.nimbus$fn__11007$service_handler__11040.doInvoke(nimbus.clj:2448)
>  ~[storm-core-1.1.1.jar:1.1.1]
>       at clojure.lang.RestFn.invoke(RestFn.java:421) ~[clojure-1.7.0.jar:?]
>       at 
> org.apache.storm.daemon.nimbus$launch_server_BANG_.invoke(nimbus.clj:2536) 
> ~[storm-core-1.1.1.jar:1.1.1]
>       at org.apache.storm.daemon.nimbus$_launch.invoke(nimbus.clj:2569) 
> ~[storm-core-1.1.1.jar:1.1.1]
>       at org.apache.storm.daemon.nimbus$_main.invoke(nimbus.clj:2592) 
> ~[storm-core-1.1.1.jar:1.1.1]
>       at clojure.lang.AFn.applyToHelper(AFn.java:152) ~[clojure-1.7.0.jar:?]
>       at clojure.lang.AFn.applyTo(AFn.java:144) ~[clojure-1.7.0.jar:?]
>       at org.apache.storm.daemon.nimbus.main(Unknown Source) 
> ~[storm-core-1.1.1.jar:1.1.1]
> 
> I download a binary version of  Storm 1.1.1 from 
> http://www.apache.org/dyn/closer.lua/storm/apache-storm-1.1.1/apache-storm-1.1.1.tar.gz
>  
> <http://www.apache.org/dyn/closer.lua/storm/apache-storm-1.1.1/apache-storm-1.1.1.tar.gz>.
> ᐧ
> 
> On Sat, Feb 10, 2018 at 2:02 PM, Wuyang Zhang <wuy...@winlab.rutgers.edu 
> <mailto:wuy...@winlab.rutgers.edu>> wrote:
> Hi Jerry, 
> 
> I just retry to use the resource aware scheduler in a single node with 4 
> slots without any further settings. The scheduler still kills the nimbus 
> process. Please find nimbus.log and storm.yaml attached. 
> 
> Without add the line 
> storm.scheduler: “org.apache.storm.scheduler.re 
> <http://org.apache.storm.scheduler.re/>source.ResourceAwareScheduler”
> in storm.yaml, everything works fine. 
> 
> I also tried 
> conf.setTopologyStrategy(org.apache.storm.scheduler.resource.strategies.scheduling.DefaultResourceAwareStrategy.class);
> The cluster can run, but I cannot see any log related to 
> DefaultResourceAware. It seems like that it doesn't really run. 
> 
> Please take a look at the log and see if I miss anything to enable them to 
> work?
> 
> Best, 
> Wuyang
> ᐧ
> 
> On Sat, Feb 10, 2018 at 12:23 PM, Zhang Wuyang <qingshanyou...@gmail.com 
> <mailto:qingshanyou...@gmail.com>> wrote:
> Hi Jerry, 
> 
> Thank you for the reply. 
> 
> Please find the storm.yaml below:
> 
> ########### These MUST be filled in for a storm configuration
>  storm.zookeeper.servers:
>      - "10.10.21.1"
>      - "10.10.21.2"
>      - "10.10.21.3"
>      - "10.10.21.4"
>      - "10.10.21.5"
>      - "10.10.21.6"
>      - "10.10.21.7"
> 
>  nimbus.seeds: ["10.10.21.1"]
> 
>  storm.zookeeper.port: 2181
> ###### These may optionally be filled in:
> 
>  worker.heap.memory.mb: 8192
> 
>  supervisor.scheduler.meta:
>     tag: 10.10.20.1
> 
> #storm.scheduler: storm.EmptyScheduler
>  storm.scheduler: “org.apache.storm.scheduler.re 
> <http://org.apache.storm.scheduler.re/>source.ResourceAwareScheduler”
>  supervisor.memory.capacity.mb: 2048.0
>  supervisor.cpu.capacity: 4000.0
>  supervisor.slots.ports:
>      - 6700
>      - 6701
>      - 6702
>      - 6703
> 
>  worker.childopts: "-Xmx%HEAP-MEM%m -Xms6000m -XX:+PrintGCDetails 
> -Xloggc:artifacts/gc.log -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps 
> -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1M 
> -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=artifacts/heapdump"
> 
> On Sat, Feb 10, 2018 at 12:16 PM, Jerry Peng <jerry.boyang.p...@gmail.com 
> <mailto:jerry.boyang.p...@gmail.com>> wrote:
> Wuyang,
> 
> Can you provide your whole storm.yaml ?
> 
> Best,
> 
> Jerry
> 
> On Fri, Feb 9, 2018 at 9:44 PM, Wuyang Zhang <wuy...@winlab.rutgers.edu 
> <mailto:wuy...@winlab.rutgers.edu>> wrote:
> Hi, 
> 
> I have the difficulty with the resource aware scheduler from the version 
> 1.1.1.
> 
> As suggested by the website, I add the following lines to storm.yaml:
> 
>  storm.scheduler: “org.apache.storm.scheduler.re 
> <http://org.apache.storm.scheduler.re/>source.ResourceAwareScheduler”
>  supervisor.memory.capacity.mb: 2048.0
>  supervisor.cpu.capacity: 400.0
> 
> Afterwards,  I rerun the storm and the nimbus process has been killed. 
> The last log I found from nimbus.log is:
> o.a.s.d.nimbus main [INFO] Using custom scheduler: 
> “org.apache.storm.scheduler.re 
> <http://org.apache.storm.scheduler.re/>source.ResourceAwareScheduler”.
> 
> I suppose something goes wrong when it runs the scheduler. 
> 
> Do I miss anything to enable this scheduler?
> 
> Best, 
> Wuyang
> ᐧ
> 
> 
> 
> 
> 
> 
> 

Reply via email to