Our tests were done with 0.10 and 0.11. We did not test with 0.8 because we
upgraded to 0.10 a while ago.
I looked at the figures again. The large oscillations and big drop in
fibers mode that I gave were actually on the "heap used". The RSS figures
oscillate too but the drop is smaller. First RSS drop is from 1,500 MB to
1,000 MB and it does not seem to recover fully because the second drop is
from 1,600 MB to 1,270 MB. I did not run it long enough because it was
slowing down my machine but it looks like it was recovering heap ok but not
RSS.
With generators, there are small oscillations both on RSS and heap used and
no drift.
In callbacks mode RSS oscillates between 320 and 400 MB and "heap used"
between 230 and 300 MB. No drift.
I did not let the test run very long in all modes because my main question
was on whether there was a memory leak in our code or not and the
generators test convinced me that our code was fine but that fibers was
probably creating some fragmentation.
Note that I ran these tests before fixing the streamline issue that you
submitted. So the fibers anomaly was probably exacerbated by the fact that
too many fibers were created (but normally this should not cause a memory
drift because they the extra fibers should get gced - but maybe not fast
enough).
I just ran the test again (with a different dataset and different code).
Here are the results:
FIBERS (node 0.10.12):
{"rss":314937344,"heapTotal":283053568,"heapUsed":254641840,"pass":0,"time":29372},
{"rss":534265856,"heapTotal":499651840,"heapUsed":432355680,"pass":1,"time":29911},
{"rss":620900352,"heapTotal":581138688,"heapUsed":422644152,"pass":2,"time":29221},
{"rss":618565632,"heapTotal":578115072,"heapUsed":416696632,"pass":3,"time":28907},
{"rss":639131648,"heapTotal":596605696,"heapUsed":410939696,"pass":4,"time":29213},
{"rss":639836160,"heapTotal":596605696,"heapUsed":398062944,"pass":5,"time":29281},
{"rss":642822144,"heapTotal":598669568,"heapUsed":459811952,"pass":6,"time":28656},
{"rss":606179328,"heapTotal":562587904,"heapUsed":471224040,"pass":7,"time":29660},
{"rss":574820352,"heapTotal":532709888,"heapUsed":417162456,"pass":8,"time":29323},
{"rss":637779968,"heapTotal":592935128,"heapUsed":406728656,"pass":9,"time":29772},
{"rss":639041536,"heapTotal":593979096,"heapUsed":470931984,"pass":10,"time":29700},
{"rss":601161728,"heapTotal":556865496,"heapUsed":478325752,"pass":11,"time":29406},
{"rss":576159744,"heapTotal":532087000,"heapUsed":408100176,"pass":12,"time":29582},
{"rss":643297280,"heapTotal":598118872,"heapUsed":401238528,"pass":13,"time":29282},
{"rss":645103616,"heapTotal":599252256,"heapUsed":465132296,"pass":14,"time":29024},
{"rss":603955200,"heapTotal":558645832,"heapUsed":465093840,"pass":15,"time":29542},
{"rss":579842048,"heapTotal":534875208,"heapUsed":402461880,"pass":16,"time":29927},
{"rss":645910528,"heapTotal":599839048,"heapUsed":463863312,"pass":17,"time":30046},
{"rss":603533312,"heapTotal":559653704,"heapUsed":474944280,"pass":18,"time":30995},
{"rss":579911680,"heapTotal":534899272,"heapUsed":409166288,"pass":19,"time":30505},
{"rss":643850240,"heapTotal":597811272,"heapUsed":474930392,"pass":20,"time":29379},
{"rss":603017216,"heapTotal":557601864,"heapUsed":404188824,"pass":21,"time":29771},
{"rss":646098944,"heapTotal":599917680,"heapUsed":464544432,"pass":22,"time":29596},
GENERATORS (node 0.11.5):
{"rss":393945088,"heapTotal":360213504,"heapUsed":278081672,"pass":0,"time":48933},
{"rss":369987584,"heapTotal":335567360,"heapUsed":308752168,"pass":1,"time":46810},
{"rss":402051072,"heapTotal":366405120,"heapUsed":333544944,"pass":2,"time":48393},
{"rss":409866240,"heapTotal":373604608,"heapUsed":350195848,"pass":3,"time":46794},
{"rss":403386368,"heapTotal":366393088,"heapUsed":342079296,"pass":4,"time":48155},
{"rss":396079104,"heapTotal":357189888,"heapUsed":264438320,"pass":5,"time":47397},
{"rss":368582656,"heapTotal":330419712,"heapUsed":292616840,"pass":6,"time":47927},
{"rss":396505088,"heapTotal":359205632,"heapUsed":330376792,"pass":7,"time":47052},
{"rss":390201344,"heapTotal":352018176,"heapUsed":323006624,"pass":8,"time":47252},
{"rss":376860672,"heapTotal":338663168,"heapUsed":309587504,"pass":9,"time":46469},
{"rss":378093568,"heapTotal":339695104,"heapUsed":315238776,"pass":10,"time":47025},
{"rss":386592768,"heapTotal":348910336,"heapUsed":329006168,"pass":11,"time":46781},
{"rss":384413696,"heapTotal":344818688,"heapUsed":314528152,"pass":12,"time":48344},
{"rss":375791616,"heapTotal":336635392,"heapUsed":292589752,"pass":13,"time":46985},
{"rss":390516736,"heapTotal":352030208,"heapUsed":262808992,"pass":14,"time":47844},
Generators are slower but use less memory and mem usage seems more stable.
Bruno
On Thursday, September 19, 2013 9:52:40 PM UTC+2, spollack wrote:
>
> Hi Bruno, thanks for the suggestions. yes we are using streamline fibers
> mode.
>
> Several questions:
> 1) regarding the fiber mode memory oscillations that you saw: was this
> under node 0.10.x? and do you know if you had the same behavior under node
> 0.8.x? Also, the memory numbers you are quoting here are RSS usage, right?
> 2) do you know if streamline callbacks mode also avoids the memory
> oscillations?
>
> Thanks,
> Seth
>
>
--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines:
https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to [email protected]
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
---
You received this message because you are subscribed to the Google Groups
"nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/groups/opt_out.
