Are you frequently creating/slicing/toStringing any Buffers around a few MB? This might be calling AdjustAmountOfExternallyAllocatedMemory indirectly.
Have you tried running without CollectAllAvailableGarbage and monitoring your RAM usage after that? Joran Greef On 07 Jun 2013, at 12:07 PM, [email protected] wrote: No, we are not using that. Regards, Hitesh. On Thursday, June 6, 2013 2:33:43 PM UTC+5:30, Vyacheslav Egorov wrote: Do you report the size of your enormous Judy array to v8 via AdjustAmountOfExternallyAllocatedMemory? If you do try disabling that. Vyacheslav Egorov On Jun 6, 2013 10:22 AM, "Hitesh Gupta" <[email protected]> wrote: Hi, We are facing a similar problem. We have an XMPP server running over node.js on a machine with 3.8 GB RAM available. However, around 400mb heap usage, v8 starts seeing a false positive OOM, and starts triggering last resort gc. The detail problem description, and the gc trace can be found at https://groups.google.com/forum/?fromgroups#!topic/v8-users/pnhQsNxUhs4 . Please let us know if any cause or resolution has been identified for similar problems. Regards, Hitesh. On Monday, November 5, 2012 8:31:52 PM UTC+5:30, Joran Dirk Greef wrote: Thanks Vyacheslav. I thought it may be some kind of OOM situation, but was surprised that this would be the case, given all the memory available to the process. Running top command shows 32GB used memory but I assume this is all disk cache, since there are no other user programs shown in top apart from the node process itself which is shown to be around 2GB used memory. The node process accesses files where the total data set is over 32GB so it makes sense that Linux would grow the disk cache? Would something like overcommit_memory=1 help V8 here? It seems like V8 is seeing a false positive OOM. There really should be more than enough RAM. As to the kind of allocation, it seems to be caused by calling buffer.toString, which drops out to C++ to convert the buffer to a string which it passes back. So essentially any 1-2MB readFile('binary' or 'utf8' or 'ascii') seems to trigger it. Interestingly enough, reading the file as a pure buffer does not cause the allocation error and returns within a few ms. And then converting the buffer to a string manually in JS does not cause any further GC either. I will give your suggestion re: CollectAllAvailableGarbage a try and post the results here. What I was wanting to do was to set the GC limits very high, as you say, to try and prevent it from anything non-incremental, since the heap has millions of persistent objects. I was hoping there would be a way to configure this using flags, or make exposing gc cause V8 to refrain from doing anything non-incremental, except when gc() is called. Your help is much appreciated. On Monday, November 5, 2012 4:43:30 PM UTC+2, Vyacheslav Egorov wrote: Hello Joran, "last resort gc" means that there was an allocation failure that a normal GC could "resolve". Basically you are in a kinda OOM situation. I am kinda curious what kind of allocation it is. Probably it is some very big object. It can be that allocation attempt does not correctly fall into allocating from LO space. One thing though is that last resort GC can be much more lightweight for node.js application that it is currently. I doubt 7 GC in a row are very helpful. As a workaround you can go into Heap::CollectAllAvailableGarbage and replace everything inside with CollectGarbage(OLD_POINTER_SPACE, gc_reason); This should get rid of 7 repetitive GCs. I think for an application like yours it makes perfect sense to set internal GC limits very high and let incremental GC crunch things instead of falling back to non-incremental marking. But there are currently no way to configure GC like that. Vyacheslav Egorov On Mon, Nov 5, 2012 at 12:50 AM, Joran Dirk Greef <[email protected]> wrote: > Max-old-space-size is measured in MB not KB as you suggest. > > Further, max-new-space-size makes no difference to the GC trace given above, > whether it's passed as flag or not, big or small. > > On Monday, November 5, 2012 10:21:11 AM UTC+2, Yang Guo wrote: >> >> The short answer is: don't mess with GC settings if you don't know what >> you are doing. >> >> The long answer is: new space is the part of the heap where short-living >> objects are allocated. The GC scans new space on every collection and >> promotes long-living objects into the old space. You are setting the new >> space to ~19GB, which takes a while to scan. Furthermore, you are setting >> the old space to only 19MB, limiting the part of the heap where long-living >> objects are being moved to, hence the last resort GC. What you probably want >> is to specify a large old space size, but leave the new space size at >> default. >> >> Yang >> >> On Sunday, November 4, 2012 4:19:11 PM UTC+1, Joran Dirk Greef wrote: >>> >>> I am running Node v0.8.14 with --nouse_idle_notification --expose_gc >>> --max_old_space_size=19000 --max_new_space_size=19000000. >>> >>> I have a large object used as part of a BitCask style store, keeping a >>> few million entries. >>> >>> Calling gc() manually takes a 3 seconds which is fine as I call it every >>> 2 minutes. >>> >>> The machine has 32GB of RAM and all of this is available to the process, >>> there is nothing else running. >>> >>> The process sits at around 1.9GB of RAM. >>> >>> I have found an interesting test case where async reading a 1mb file in >>> Node takes longer and longer depending on how many entries are in the large >>> object discussed above: >>> >>> Node.fs.readFile('test', 'binary', End.timer()) >>> 347745 ms: Scavenge 1617.4 (1660.4) -> 1611.1 (1660.4) MB, 0 ms >>> [allocation failure]. >>> 350900 ms: Mark-sweep 1611.5 (1660.4) -> 1512.2 (1633.4) MB, 3153 ms >>> [last resort gc]. >>> 354072 ms: Mark-sweep 1512.2 (1633.4) -> 1512.0 (1592.4) MB, 3171 ms >>> [last resort gc]. >>> 357247 ms: Mark-sweep 1512.0 (1592.4) -> 1512.0 (1568.4) MB, 3175 ms >>> [last resort gc]. >>> 360426 ms: Mark-sweep 1512.0 (1568.4) -> 1512.0 (1567.4) MB, 3178 ms >>> [last resort gc]. >>> 363620 ms: Mark-sweep 1512.0 (1567.4) -> 1512.0 (1567.4) MB, 3193 ms >>> [last resort gc]. >>> 366802 ms: Mark-sweep 1512.0 (1567.4) -> 1511.6 (1567.4) MB, 3182 ms >>> [last resort gc]. >>> 369967 ms: Mark-sweep 1511.6 (1567.4) -> 1511.6 (1567.4) MB, 3164 ms >>> [last resort gc]. >>> 2012-11-04T14:59:30.700Z INFO 22230ms >>> >>> Reading the 1mb file before the large object is created is fast, the >>> bigger the object becomes the slower the file is to read. >>> >>> Why is last resort gc being called if gc is exposed and if the machine >>> has more than enough RAM? >>> >>> What was interesting was that this behabiour does not happen for V8 >>> 3.6.6.25 and earlier. >>> >>> The reason I can't use 3.6.6.25 however is that the heap is limited to >>> 1.9GB and I need more head room than that. >>> >>> Is there anyway I can disable the last resort GC? > > -- > v8-users mailing list > [email protected] > http://groups.google.com/group/v8-users -- -- v8-users mailing list [email protected] http://groups.google.com/group/v8-users --- You received this message because you are subscribed to the Google Groups "v8-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out. -- -- v8-users mailing list [email protected] http://groups.google.com/group/v8-users --- You received this message because you are subscribed to a topic in the Google Groups "v8-users" group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/v8-users/Xc5EriJtpd4/unsubscribe?hl=en. To unsubscribe from this group and all its topics, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out. -- -- v8-users mailing list [email protected] http://groups.google.com/group/v8-users --- You received this message because you are subscribed to the Google Groups "v8-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
