Thank you!! . You answer is very helpful We've moved the topology to isolation and increased the number of resources (workers, executors) . The memory behaves much better now but .. the CPU getting crazy . 100% Our topology doesn't have any CPU consuming task . Does our high load get CPU to the limit? Should we add more resources or there is some configuration work to be done Thank you, Vladi
On Wed, Jul 9, 2014 at 4:00 PM, Nathan Leung <[email protected]> wrote: > As Jungtaek said, the issue is 0mq memory usage. 0mq library code is > accessed through jni so its memory usage is not governed by the jvm. By > default there is no high water mark so your memory usage can explode. If > you aren't using reliable message handling (emitting tuples with an id) and > setting topology max spout pending, that can help as well because it will > throttle tuples at the spout if there are too many in flight rather than > relying on back pressure (which as you've observed can perform quite poorly > with 0mq). > > If, after that, you still have OOM you should increase your heap size. > > Also the latest versions of storm use netty as the default transport which > has a java implementation (and I believe bounded queue size) so it will not > explode memory usage. > > -Nathan > On Jul 9, 2014 7:57 AM, "임정택" <[email protected]> wrote: > >> Hello. >> We have same issue from Storm+Trident / 0.9.0.1. >> In my experience, you can check your tuples these are sended to >> other workers. >> If you use 0.9.x, you can set ZeroMQ high watermark (value means message >> count). >> If you set this value to small but not 0 (unlimited), you can see RES >> doesn't go too high. Instead, tuple will be dropped. >> If you see dropping RES memory, it's time to tune configuration. >> We gave up and set worker to 1, and problem is gone. >> >> Hope this helps. >> HeartSaVioR (Jungtaek Lim) >> >> 2014년 7월 9일 수요일, Vladi Feigin<[email protected]>님이 작성한 메시지: >> >>> Hi, >>> >>> Our topology consumes almost 100% on the physical machines where it runs. >>> We have heavy load ( 5K events per sec) . >>> The Java Heap is configured with -Xmx1024m >>> But Linux top command shows very large figures (for the Storm process) : >>> VIRT=14G >>> RES=10G !! >>> Apparently that Java code (topology code) doesn't consumes it. So the >>> question is which Storm part does consume it? And why? >>> What should we check , reconfigure to avoid this? What we do wrong? >>> >>> At some point we get OOME in the bolts ... >>> The Storm version is 0.8.2 >>> >>> Thank you in advance, >>> Vladi >>> >>> >>> >>> >>> >>> >>> >> >> -- >> Name : 임 정택 >> Blog : http://www.heartsavior.net / http://dev.heartsavior.net >> Twitter : http://twitter.com/heartsavior >> LinkedIn : http://www.linkedin.com/in/heartsavior >> >>
