On 10 February 2015 at 16:25, Andrew Thompson <thom...@freebsd.org> wrote:

> On 9 February 2015 at 23:28, Roger Pau Monné <roger....@citrix.com> wrote:
>> Hello,
>> El 09/02/15 a les 10.39, Andrew Thompson ha escrit:
>> > Hi,
>> >
>> >
>> > I have three VMs with Rackspace and one is behaving oddly with xenstore
>> > memory consumption. Here are the kernel versions and vmstat -m results.
>> >
> > As you can see the third VM is using 100MB in xenstore memory and it
>> seems
>> > to be climbing by 1-2MB per hour. Eventually all the processes go in to
>> > pfault state and it grinds to a halt.
>> That's certainly weird, are you doing something different on this VM as
>> compared to the others? Did you hot-add a nic, disk or ballooned memory?
>> Has the VM been saved/restored or migrated?
>> Tracking down this kind of xenstore leaks can be difficult without
>> having a way to reproduce them.
> A bit of trial and error with dtrace has narrowed this down. I can cause
> the leak by just opening /dev/xen/xenstore
> int main() {
>   open("/dev/xen/xenstore", O_RDWR, 0);
> }
> # vmstat -m | grep xenstore; ./open; vmstat -m | grep xenstore
>      xenstore  8739 104797K       -    56078  16,32,64,128,256,512
>      xenstore  8740 104809K       -    56079  16,32,64,128,256,512
> Using dtrace probes I can see that xs_dev_close is never called.

I think I have worked this out. Rackspace use an agent called nova-agent
which keeps and open handle on /dev/xen/xenstore. Since xenstore isnt using
the D_TRACKCLOSE flag it will not call d_close until the last reference is
dropped.  Since xenstore expects to malloc/free on open and close this
assumption breaks and will leak memory.

If i stop nova-agent I can see xs_dev_close being called and the memory
freed with testing with xenstore-read. The correct solution seems to be to
set D_TRACKCLOSE if I understand its purpose correctly.

freebsd-xen@freebsd.org mailing list
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Reply via email to