Hi,
We are facing a similar problem. We have an XMPP server running over
node.js on a machine with 3.8 GB RAM available. However, around 400mb heap
usage, v8 starts seeing a false positive OOM, and starts triggering last
resort gc. The detail problem description, and the gc trace can be found
at https://groups.google.com/forum/?fromgroups#!topic/v8-users/pnhQsNxUhs4
.
Please let us know if any cause or resolution has been identified for
similar problems.
Regards,
Hitesh.
On Monday, November 5, 2012 8:31:52 PM UTC+5:30, Joran Dirk Greef wrote:
>
> Thanks Vyacheslav.
>
> I thought it may be some kind of OOM situation, but was surprised that
> this would be the case, given all the memory available to the process.
> Running top command shows 32GB used memory but I assume this is all disk
> cache, since there are no other user programs shown in top apart from the
> node process itself which is shown to be around 2GB used memory. The node
> process accesses files where the total data set is over 32GB so it makes
> sense that Linux would grow the disk cache? Would something like
> overcommit_memory=1 help V8 here? It seems like V8 is seeing a false
> positive OOM. There really should be more than enough RAM.
>
> As to the kind of allocation, it seems to be caused by calling
> buffer.toString, which drops out to C++ to convert the buffer to a string
> which it passes back. So essentially any 1-2MB readFile('binary' or 'utf8'
> or 'ascii') seems to trigger it. Interestingly enough, reading the file as
> a pure buffer does not cause the allocation error and returns within a few
> ms. And then converting the buffer to a string manually in JS does not
> cause any further GC either.
>
> I will give your suggestion re: CollectAllAvailableGarbage a try and post
> the results here.
>
> What I was wanting to do was to set the GC limits very high, as you say,
> to try and prevent it from anything non-incremental, since the heap has
> millions of persistent objects. I was hoping there would be a way to
> configure this using flags, or make exposing gc cause V8 to refrain from
> doing anything non-incremental, except when gc() is called.
>
> Your help is much appreciated.
>
> On Monday, November 5, 2012 4:43:30 PM UTC+2, Vyacheslav Egorov wrote:
>>
>> Hello Joran,
>>
>> "last resort gc" means that there was an allocation failure that a
>> normal GC could "resolve". Basically you are in a kinda OOM situation.
>> I am kinda curious what kind of allocation it is. Probably it is some
>> very big object. It can be that allocation attempt does not correctly
>> fall into allocating from LO space.
>>
>> One thing though is that last resort GC can be much more lightweight
>> for node.js application that it is currently. I doubt 7 GC in a row
>> are very helpful. As a workaround you can go into
>> Heap::CollectAllAvailableGarbage and replace everything inside with
>>
>> CollectGarbage(OLD_POINTER_SPACE, gc_reason);
>>
>> This should get rid of 7 repetitive GCs. I think for an application
>> like yours it makes perfect sense to set internal GC limits very high
>> and let incremental GC crunch things instead of falling back to
>> non-incremental marking. But there are currently no way to configure
>> GC like that.
>> Vyacheslav Egorov
>>
>>
>> On Mon, Nov 5, 2012 at 12:50 AM, Joran Dirk Greef <[email protected]>
>> wrote:
>> > Max-old-space-size is measured in MB not KB as you suggest.
>> >
>> > Further, max-new-space-size makes no difference to the GC trace given
>> above,
>> > whether it's passed as flag or not, big or small.
>> >
>> > On Monday, November 5, 2012 10:21:11 AM UTC+2, Yang Guo wrote:
>> >>
>> >> The short answer is: don't mess with GC settings if you don't know
>> what
>> >> you are doing.
>> >>
>> >> The long answer is: new space is the part of the heap where
>> short-living
>> >> objects are allocated. The GC scans new space on every collection and
>> >> promotes long-living objects into the old space. You are setting the
>> new
>> >> space to ~19GB, which takes a while to scan. Furthermore, you are
>> setting
>> >> the old space to only 19MB, limiting the part of the heap where
>> long-living
>> >> objects are being moved to, hence the last resort GC. What you
>> probably want
>> >> is to specify a large old space size, but leave the new space size at
>> >> default.
>> >>
>> >> Yang
>> >>
>> >> On Sunday, November 4, 2012 4:19:11 PM UTC+1, Joran Dirk Greef wrote:
>> >>>
>> >>> I am running Node v0.8.14 with --nouse_idle_notification --expose_gc
>> >>> --max_old_space_size=19000 --max_new_space_size=19000000.
>> >>>
>> >>> I have a large object used as part of a BitCask style store, keeping
>> a
>> >>> few million entries.
>> >>>
>> >>> Calling gc() manually takes a 3 seconds which is fine as I call it
>> every
>> >>> 2 minutes.
>> >>>
>> >>> The machine has 32GB of RAM and all of this is available to the
>> process,
>> >>> there is nothing else running.
>> >>>
>> >>> The process sits at around 1.9GB of RAM.
>> >>>
>> >>> I have found an interesting test case where async reading a 1mb file
>> in
>> >>> Node takes longer and longer depending on how many entries are in the
>> large
>> >>> object discussed above:
>> >>>
>> >>> Node.fs.readFile('test', 'binary', End.timer())
>> >>> 347745 ms: Scavenge 1617.4 (1660.4) -> 1611.1 (1660.4) MB, 0 ms
>> >>> [allocation failure].
>> >>> 350900 ms: Mark-sweep 1611.5 (1660.4) -> 1512.2 (1633.4) MB, 3153
>> ms
>> >>> [last resort gc].
>> >>> 354072 ms: Mark-sweep 1512.2 (1633.4) -> 1512.0 (1592.4) MB, 3171
>> ms
>> >>> [last resort gc].
>> >>> 357247 ms: Mark-sweep 1512.0 (1592.4) -> 1512.0 (1568.4) MB, 3175
>> ms
>> >>> [last resort gc].
>> >>> 360426 ms: Mark-sweep 1512.0 (1568.4) -> 1512.0 (1567.4) MB, 3178
>> ms
>> >>> [last resort gc].
>> >>> 363620 ms: Mark-sweep 1512.0 (1567.4) -> 1512.0 (1567.4) MB, 3193
>> ms
>> >>> [last resort gc].
>> >>> 366802 ms: Mark-sweep 1512.0 (1567.4) -> 1511.6 (1567.4) MB, 3182
>> ms
>> >>> [last resort gc].
>> >>> 369967 ms: Mark-sweep 1511.6 (1567.4) -> 1511.6 (1567.4) MB, 3164
>> ms
>> >>> [last resort gc].
>> >>> 2012-11-04T14:59:30.700Z INFO 22230ms
>> >>>
>> >>> Reading the 1mb file before the large object is created is fast, the
>> >>> bigger the object becomes the slower the file is to read.
>> >>>
>> >>> Why is last resort gc being called if gc is exposed and if the
>> machine
>> >>> has more than enough RAM?
>> >>>
>> >>> What was interesting was that this behabiour does not happen for V8
>> >>> 3.6.6.25 and earlier.
>> >>>
>> >>> The reason I can't use 3.6.6.25 however is that the heap is limited
>> to
>> >>> 1.9GB and I need more head room than that.
>> >>>
>> >>> Is there anyway I can disable the last resort GC?
>> >
>> > --
>> > v8-users mailing list
>> > [email protected]
>> > http://groups.google.com/group/v8-users
>>
>
--
--
v8-users mailing list
[email protected]
http://groups.google.com/group/v8-users
---
You received this message because you are subscribed to the Google Groups
"v8-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/groups/opt_out.
