Do you report the size of your enormous Judy array to v8 via AdjustAmountOfExternallyAllocatedMemory? If you do try disabling that.
Vyacheslav Egorov On Jun 6, 2013 10:22 AM, "Hitesh Gupta" <[email protected]> wrote: > Hi, > We are facing a similar problem. We have an XMPP server running over > node.js on a machine with 3.8 GB RAM available. However, around 400mb heap > usage, v8 starts seeing a false positive OOM, and starts triggering last > resort gc. The detail problem description, and the gc trace can be found at > https://groups.google.com/forum/?fromgroups#!topic/v8-users/pnhQsNxUhs4 . > > Please let us know if any cause or resolution has been identified for > similar problems. > > Regards, > Hitesh. > > On Monday, November 5, 2012 8:31:52 PM UTC+5:30, Joran Dirk Greef wrote: >> >> Thanks Vyacheslav. >> >> I thought it may be some kind of OOM situation, but was surprised that >> this would be the case, given all the memory available to the process. >> Running top command shows 32GB used memory but I assume this is all disk >> cache, since there are no other user programs shown in top apart from the >> node process itself which is shown to be around 2GB used memory. The node >> process accesses files where the total data set is over 32GB so it makes >> sense that Linux would grow the disk cache? Would something like >> overcommit_memory=1 help V8 here? It seems like V8 is seeing a false >> positive OOM. There really should be more than enough RAM. >> >> As to the kind of allocation, it seems to be caused by calling >> buffer.toString, which drops out to C++ to convert the buffer to a string >> which it passes back. So essentially any 1-2MB readFile('binary' or 'utf8' >> or 'ascii') seems to trigger it. Interestingly enough, reading the file as >> a pure buffer does not cause the allocation error and returns within a few >> ms. And then converting the buffer to a string manually in JS does not >> cause any further GC either. >> >> I will give your suggestion re: CollectAllAvailableGarbage a try and post >> the results here. >> >> What I was wanting to do was to set the GC limits very high, as you say, >> to try and prevent it from anything non-incremental, since the heap has >> millions of persistent objects. I was hoping there would be a way to >> configure this using flags, or make exposing gc cause V8 to refrain from >> doing anything non-incremental, except when gc() is called. >> >> Your help is much appreciated. >> >> On Monday, November 5, 2012 4:43:30 PM UTC+2, Vyacheslav Egorov wrote: >>> >>> Hello Joran, >>> >>> "last resort gc" means that there was an allocation failure that a >>> normal GC could "resolve". Basically you are in a kinda OOM situation. >>> I am kinda curious what kind of allocation it is. Probably it is some >>> very big object. It can be that allocation attempt does not correctly >>> fall into allocating from LO space. >>> >>> One thing though is that last resort GC can be much more lightweight >>> for node.js application that it is currently. I doubt 7 GC in a row >>> are very helpful. As a workaround you can go into >>> Heap::**CollectAllAvailableGarbage and replace everything inside with >>> >>> CollectGarbage(OLD_POINTER_**SPACE, gc_reason); >>> >>> This should get rid of 7 repetitive GCs. I think for an application >>> like yours it makes perfect sense to set internal GC limits very high >>> and let incremental GC crunch things instead of falling back to >>> non-incremental marking. But there are currently no way to configure >>> GC like that. >>> Vyacheslav Egorov >>> >>> >>> On Mon, Nov 5, 2012 at 12:50 AM, Joran Dirk Greef <[email protected]> >>> wrote: >>> > Max-old-space-size is measured in MB not KB as you suggest. >>> > >>> > Further, max-new-space-size makes no difference to the GC trace given >>> above, >>> > whether it's passed as flag or not, big or small. >>> > >>> > On Monday, November 5, 2012 10:21:11 AM UTC+2, Yang Guo wrote: >>> >> >>> >> The short answer is: don't mess with GC settings if you don't know >>> what >>> >> you are doing. >>> >> >>> >> The long answer is: new space is the part of the heap where >>> short-living >>> >> objects are allocated. The GC scans new space on every collection and >>> >> promotes long-living objects into the old space. You are setting the >>> new >>> >> space to ~19GB, which takes a while to scan. Furthermore, you are >>> setting >>> >> the old space to only 19MB, limiting the part of the heap where >>> long-living >>> >> objects are being moved to, hence the last resort GC. What you >>> probably want >>> >> is to specify a large old space size, but leave the new space size at >>> >> default. >>> >> >>> >> Yang >>> >> >>> >> On Sunday, November 4, 2012 4:19:11 PM UTC+1, Joran Dirk Greef wrote: >>> >>> >>> >>> I am running Node v0.8.14 with --nouse_idle_notification --expose_gc >>> >>> --max_old_space_size=19000 --max_new_space_size=19000000. >>> >>> >>> >>> I have a large object used as part of a BitCask style store, keeping >>> a >>> >>> few million entries. >>> >>> >>> >>> Calling gc() manually takes a 3 seconds which is fine as I call it >>> every >>> >>> 2 minutes. >>> >>> >>> >>> The machine has 32GB of RAM and all of this is available to the >>> process, >>> >>> there is nothing else running. >>> >>> >>> >>> The process sits at around 1.9GB of RAM. >>> >>> >>> >>> I have found an interesting test case where async reading a 1mb file >>> in >>> >>> Node takes longer and longer depending on how many entries are in >>> the large >>> >>> object discussed above: >>> >>> >>> >>> Node.fs.readFile('test', 'binary', End.timer()) >>> >>> 347745 ms: Scavenge 1617.4 (1660.4) -> 1611.1 (1660.4) MB, 0 ms >>> >>> [allocation failure]. >>> >>> 350900 ms: Mark-sweep 1611.5 (1660.4) -> 1512.2 (1633.4) MB, 3153 >>> ms >>> >>> [last resort gc]. >>> >>> 354072 ms: Mark-sweep 1512.2 (1633.4) -> 1512.0 (1592.4) MB, 3171 >>> ms >>> >>> [last resort gc]. >>> >>> 357247 ms: Mark-sweep 1512.0 (1592.4) -> 1512.0 (1568.4) MB, 3175 >>> ms >>> >>> [last resort gc]. >>> >>> 360426 ms: Mark-sweep 1512.0 (1568.4) -> 1512.0 (1567.4) MB, 3178 >>> ms >>> >>> [last resort gc]. >>> >>> 363620 ms: Mark-sweep 1512.0 (1567.4) -> 1512.0 (1567.4) MB, 3193 >>> ms >>> >>> [last resort gc]. >>> >>> 366802 ms: Mark-sweep 1512.0 (1567.4) -> 1511.6 (1567.4) MB, 3182 >>> ms >>> >>> [last resort gc]. >>> >>> 369967 ms: Mark-sweep 1511.6 (1567.4) -> 1511.6 (1567.4) MB, 3164 >>> ms >>> >>> [last resort gc]. >>> >>> 2012-11-04T14:59:30.700Z INFO 22230ms >>> >>> >>> >>> Reading the 1mb file before the large object is created is fast, the >>> >>> bigger the object becomes the slower the file is to read. >>> >>> >>> >>> Why is last resort gc being called if gc is exposed and if the >>> machine >>> >>> has more than enough RAM? >>> >>> >>> >>> What was interesting was that this behabiour does not happen for V8 >>> >>> 3.6.6.25 and earlier. >>> >>> >>> >>> The reason I can't use 3.6.6.25 however is that the heap is limited >>> to >>> >>> 1.9GB and I need more head room than that. >>> >>> >>> >>> Is there anyway I can disable the last resort GC? >>> > >>> > -- >>> > v8-users mailing list >>> > [email protected] >>> > http://groups.google.com/**group/v8-users<http://groups.google.com/group/v8-users> >>> >> -- > -- > v8-users mailing list > [email protected] > http://groups.google.com/group/v8-users > --- > You received this message because you are subscribed to the Google Groups > "v8-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/groups/opt_out. > > > -- -- v8-users mailing list [email protected] http://groups.google.com/group/v8-users --- You received this message because you are subscribed to the Google Groups "v8-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
