Re: VirtualBox mangles memory

[Bernhard Fröhlich]

Am 23.09.2015 4:10 vorm. schrieb "Mark Felder" :
>
>
>
> On Tue, Sep 22, 2015, at 15:57, Michael Butler wrote:
> > On 09/22/15 16:20, Adam Vande More wrote:
> > > On Tue, Sep 22, 2015 at 2:26 PM, Martin Birgmeier 
wrote:
> >
> > >> I am quite sure that the
> > >> cause is to be found there
> > >>
> > >
> > > I don't see any cause for surety in the info you have given us.  Have
you
> > > tried any basic steps like monitoring usage during these events or
running
> > > it under truss/valgrind etc to find out what is going on?
> > >
> >
> > Another data point ..
> >
> > I run it on -CURRENT and there have been a number of VM-related changes
> > of late but I don't have any problems with it.
> >
> > However, you *must* recompile (at least) the modules whenever the kernel
> > is rebuilt as they depend heavily on the underlying kernel structures,
> >
> >   imb
> >
>
> I agree. Just to be safe we really shouldn't provide packages for the
> virtualbox kmods. It's just asking for trouble.

No it's absolutely the right thing to provide them or as a package only
user on a release you would be screwed. As a STABLE or CURRENT user you
should never use any of them.
___
freebsd-emulation@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-emulation
To unsubscribe, send any mail to "freebsd-emulation-unsubscr...@freebsd.org"

Re: VirtualBox mangles memory

[Mark Felder]

On Wed, Sep 23, 2015, at 06:50, Bernhard Fröhlich wrote:
> Am 23.09.2015 4:10 vorm. schrieb "Mark Felder" :
> >
> >
> >
> > On Tue, Sep 22, 2015, at 15:57, Michael Butler wrote:
> > > On 09/22/15 16:20, Adam Vande More wrote:
> > > > On Tue, Sep 22, 2015 at 2:26 PM, Martin Birgmeier 
> wrote:
> > >
> > > >> I am quite sure that the
> > > >> cause is to be found there
> > > >>
> > > >
> > > > I don't see any cause for surety in the info you have given us.  Have
> you
> > > > tried any basic steps like monitoring usage during these events or
> running
> > > > it under truss/valgrind etc to find out what is going on?
> > > >
> > >
> > > Another data point ..
> > >
> > > I run it on -CURRENT and there have been a number of VM-related changes
> > > of late but I don't have any problems with it.
> > >
> > > However, you *must* recompile (at least) the modules whenever the kernel
> > > is rebuilt as they depend heavily on the underlying kernel structures,
> > >
> > >   imb
> > >
> >
> > I agree. Just to be safe we really shouldn't provide packages for the
> > virtualbox kmods. It's just asking for trouble.
> 
> No it's absolutely the right thing to provide them or as a package only
> user on a release you would be screwed. As a STABLE or CURRENT user you
> should never use any of them.


We don't build for every RELEASE. We always build with the oldest
supported version in that release train. Right now all packages for 10.2
users are being built on 10.1-RELEASE. This is bad for virtualbox users.
Beware of the dragon, etc etc.


-- 
  Mark Felder
  ports-secteam member
  f...@freebsd.org
___
freebsd-emulation@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-emulation
To unsubscribe, send any mail to "freebsd-emulation-unsubscr...@freebsd.org"

Re: VirtualBox mangles memory

[Martin Birgmeier]

I have always recompiled the vbox modules (in fact, all of VB) after the latest 
kernel installation, for both machines. This behavior started after I upgraded 
from FBSD 9.2 to 10.1 (and persisted with 10.2).

I have tried to refine the scenario where the problem appears, and the results 
are as follows.
- I run a 3 GB VB client on an 8 GB machine
- On this machine I start a dd with a blocksize of 128k to copy a 250G disk 
slice to standard output (the purpose being to make an image backup of the 
Windows partition).

The difference is how I start dd:
- Just running
dd if=/dev/ada0s2 of=/dev/null bs=128
  works.
- Running
dd if=/dev/ada0s2 bs=128k | cat > /dev/null
  works.
- *On another machine*, running
rsh -n  "dd if=/dev/ada0s2 bs=128k" > 
/dev/null
  produces "dd: stdout: Cannot allocate memory" quite soon (but after a 
non-deterministic number of blocks have been transferred).

If I do this using ssh, the behavior is slightly different but still erroneous:
ssh -n  "dd if=/dev/ada0s2 bs=128k" > 
/dev/null
  produces "Connection to  closed by remote 
host." without a message from dd.

I could not note any anomalies in memory usage as monitored by "top" and 
"systat -vm 1".

If, after the tests above, I shut down the VB client while keeping everything 
else the same, issuing the rsh command runs to completion correctly.

Note that in that other scenario where I was using "zfs send", "zfs send" was 
also invoked via "rsh -n".

So it seems that sending data via rsh while a VB client is running leads to the 
problem.

Could anyone try to reproduce this test case and tell me about the results?

Thank you in advance for your efforts.

-- Martin



On 09/22/15 22:57, Michael Butler wrote:
> On 09/22/15 16:20, Adam Vande More wrote:
>> On Tue, Sep 22, 2015 at 2:26 PM, Martin Birgmeier  wrote:
>>> I am quite sure that the
>>> cause is to be found there
>>>
>> I don't see any cause for surety in the info you have given us.  Have you
>> tried any basic steps like monitoring usage during these events or running
>> it under truss/valgrind etc to find out what is going on?
>>
> Another data point ..
>
> I run it on -CURRENT and there have been a number of VM-related changes
> of late but I don't have any problems with it.
>
> However, you *must* recompile (at least) the modules whenever the kernel
> is rebuilt as they depend heavily on the underlying kernel structures,
>
>   imb
>
>
>

___
freebsd-emulation@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-emulation
To unsubscribe, send any mail to "freebsd-emulation-unsubscr...@freebsd.org"

Re: VirtualBox mangles memory

[Kevin Oberman]

Just a quick reminder that, if you ever build a kernel, you probably want
to list all ports which generate kernel modules in a PORTS_MODULES line in
/etc/src.conf. E.g. "PORTS_MODULES=emulators/virtualbox-ose-kmod
multimedia/cuse4bsd-kmod". This will automatically rebuild these ports when
the kernel is rebuilt. You won't forget.

Caveat: This will fail for "make reinstallkernel" as the ports Mk files
lack a reinstall target, but it seems that relatively few people are aware
that this target even exists, even though it can be very useful.
--
Kevin Oberman, Part time kid herder and retired Network Engineer
E-mail: rkober...@gmail.com
PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683
___
freebsd-emulation@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-emulation
To unsubscribe, send any mail to "freebsd-emulation-unsubscr...@freebsd.org"

Re: VirtualBox mangles memory

[Martin Birgmeier]

In https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=195970, Larry
Rosenman suggested setting net.graph.maxdata=8192. I set it to 4096, and
the problem seems to have gone away (I am currently testing, and it has
not failed so far).

This is actually mentioned in
https://wiki.freebsd.org/VirtualBox/Tuning, but I would never have
guessed that the standard network stack is affected by setting this
tunable: I am not copying anything in/out of the VB client, but rather
run a completely independent regular IPv4 transfer alongside it. Maybe
the description there could be expanded accordingly.

Anyway, thanks to Larry for pointing me in the right direction.

-- Martin

___
freebsd-emulation@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-emulation
To unsubscribe, send any mail to "freebsd-emulation-unsubscr...@freebsd.org"

Re: VirtualBox mangles memory

[Bernhard Fröhlich]

To me this sounds a lot like vbox and zfs arc are fighting for resources.
What does the memory line in top look like when this problem happens?

A simple workaround would be to limit zfs arc in loader.conf if that is the
problem.

Am 22.09.2015 5:13 nachm. schrieb "Martin Birgmeier" :
>
> This did not happen when the machine was running FreeBSD 9.2 with
virtualbox-ose-4.3.20 installed. So I would not qualify this as "known
behavior".
>
> One server has 16G main memory, a VM gets 3G. The other has 8G main mem
and again 3G allocated to the VM.
>
> This is definitely a bug in how VirtualBox handles memory on FreeBSD 10.
I am pretty sure that something changed in FBSD 10 which was not properly
taken care of in VB.
>
> -- Martin
>
>
>
> On 09/21/15 22:10, Mark Felder wrote:
> >
> > On Sun, Sep 20, 2015, at 06:20, Martin Birgmeier wrote:
> >> On machines where VirtualBox is running, I get memory allocation
errors.
> >>
> >> 'dd' aborts with 'dd: stdout: Cannot allocate memory'
> >>
> >> 'zfs send' aborts with 'internal error: Cannot allocate memory'
> >>
> >> I have created https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=195970
> >> to track this.
> >>
> >> Could a knowledgeable person please have a look into this?
> >>
> >> FreeBSD 10.2, VirtualBox 4.3.30
> >>
> >> -- Martin
> >>
> > I'm not an expert, but I think this is known behavior. Virtualbox
> > requires a huge contiguous chunk of memory for the VM and that's
> > probably what you're running into.
> >
> > Can you give more information about the specs of the server itself and
> > the amount of resources allocated to VMs?
> >
>
> ___
> freebsd-emulation@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-emulation
> To unsubscribe, send any mail to "
freebsd-emulation-unsubscr...@freebsd.org"
___
freebsd-emulation@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-emulation
To unsubscribe, send any mail to "freebsd-emulation-unsubscr...@freebsd.org"

Re: VirtualBox mangles memory

[Martin Birgmeier]

I don't think it is related to zfs. The 8GB machine did not even have
zfs loaded or a zfs file system when I first reported the problem. (The
problem with 'zfs send' only happened recently on the 16GB machine.)

I'd like to know what has changed between FBSD 9 and 10 regarding memory
management and maybe memory tuning parameters. I am quite sure that the
cause is to be found there, and in VirtualBox not properly taking into
account these changes. Maybe it is just some page table or similar
temporarily running out of entries. The problem is easy to reproduce, so
if I get some good hints on what to tune I can check whether they help.

-- Martin

On 09/22/15 20:50, Bernhard Fröhlich wrote:
>
> To me this sounds a lot like vbox and zfs arc are fighting for
> resources. What does the memory line in top look like when this
> problem happens?
>
> A simple workaround would be to limit zfs arc in loader.conf if that
> is the problem.
>
> Am 22.09.2015 5:13 nachm. schrieb "Martin Birgmeier"  >:
> >
> > This did not happen when the machine was running FreeBSD 9.2 with
> virtualbox-ose-4.3.20 installed. So I would not qualify this as "known
> behavior".
> >
> > One server has 16G main memory, a VM gets 3G. The other has 8G main
> mem and again 3G allocated to the VM.
> >
> > This is definitely a bug in how VirtualBox handles memory on FreeBSD
> 10. I am pretty sure that something changed in FBSD 10 which was not
> properly taken care of in VB.
> >
> > -- Martin
> >
> >
> >
> > On 09/21/15 22:10, Mark Felder wrote:
> > >
> > > On Sun, Sep 20, 2015, at 06:20, Martin Birgmeier wrote:
> > >> On machines where VirtualBox is running, I get memory allocation
> errors.
> > >>
> > >> 'dd' aborts with 'dd: stdout: Cannot allocate memory'
> > >>
> > >> 'zfs send' aborts with 'internal error: Cannot allocate memory'
> > >>
> > >> I have created
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=195970
> > >> to track this.
> > >>
> > >> Could a knowledgeable person please have a look into this?
> > >>
> > >> FreeBSD 10.2, VirtualBox 4.3.30
> > >>
> > >> -- Martin
> > >>
> > > I'm not an expert, but I think this is known behavior. Virtualbox
> > > requires a huge contiguous chunk of memory for the VM and that's
> > > probably what you're running into.
> > >
> > > Can you give more information about the specs of the server itself and
> > > the amount of resources allocated to VMs?
> > >
> >
> > ___
> > freebsd-emulation@freebsd.org 
> mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-emulation
> > To unsubscribe, send any mail to
> "freebsd-emulation-unsubscr...@freebsd.org
> "
>

___
freebsd-emulation@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-emulation
To unsubscribe, send any mail to "freebsd-emulation-unsubscr...@freebsd.org"

Re: VirtualBox mangles memory

[Mark Felder]

On Tue, Sep 22, 2015, at 15:57, Michael Butler wrote:
> On 09/22/15 16:20, Adam Vande More wrote:
> > On Tue, Sep 22, 2015 at 2:26 PM, Martin Birgmeier  wrote:
> 
> >> I am quite sure that the
> >> cause is to be found there
> >>
> > 
> > I don't see any cause for surety in the info you have given us.  Have you
> > tried any basic steps like monitoring usage during these events or running
> > it under truss/valgrind etc to find out what is going on?
> > 
> 
> Another data point ..
> 
> I run it on -CURRENT and there have been a number of VM-related changes
> of late but I don't have any problems with it.
> 
> However, you *must* recompile (at least) the modules whenever the kernel
> is rebuilt as they depend heavily on the underlying kernel structures,
> 
>   imb
> 

I agree. Just to be safe we really shouldn't provide packages for the
virtualbox kmods. It's just asking for trouble.


-- 
  Mark Felder
  ports-secteam member
  f...@freebsd.org
___
freebsd-emulation@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-emulation
To unsubscribe, send any mail to "freebsd-emulation-unsubscr...@freebsd.org"

Re: VirtualBox mangles memory

[Michael Butler]

On 09/22/15 16:20, Adam Vande More wrote:
> On Tue, Sep 22, 2015 at 2:26 PM, Martin Birgmeier  wrote:

>> I am quite sure that the
>> cause is to be found there
>>
> 
> I don't see any cause for surety in the info you have given us.  Have you
> tried any basic steps like monitoring usage during these events or running
> it under truss/valgrind etc to find out what is going on?
> 

Another data point ..

I run it on -CURRENT and there have been a number of VM-related changes
of late but I don't have any problems with it.

However, you *must* recompile (at least) the modules whenever the kernel
is rebuilt as they depend heavily on the underlying kernel structures,

imb

___
freebsd-emulation@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-emulation
To unsubscribe, send any mail to "freebsd-emulation-unsubscr...@freebsd.org"

Re: VirtualBox mangles memory

[Martin Birgmeier]

This did not happen when the machine was running FreeBSD 9.2 with 
virtualbox-ose-4.3.20 installed. So I would not qualify this as "known 
behavior".

One server has 16G main memory, a VM gets 3G. The other has 8G main mem and 
again 3G allocated to the VM.

This is definitely a bug in how VirtualBox handles memory on FreeBSD 10. I am 
pretty sure that something changed in FBSD 10 which was not properly taken care 
of in VB.

-- Martin



On 09/21/15 22:10, Mark Felder wrote:
>
> On Sun, Sep 20, 2015, at 06:20, Martin Birgmeier wrote:
>> On machines where VirtualBox is running, I get memory allocation errors.
>>
>> 'dd' aborts with 'dd: stdout: Cannot allocate memory'
>>
>> 'zfs send' aborts with 'internal error: Cannot allocate memory'
>>
>> I have created https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=195970
>> to track this.
>>
>> Could a knowledgeable person please have a look into this?
>>
>> FreeBSD 10.2, VirtualBox 4.3.30
>>
>> -- Martin
>>
> I'm not an expert, but I think this is known behavior. Virtualbox
> requires a huge contiguous chunk of memory for the VM and that's
> probably what you're running into.
>
> Can you give more information about the specs of the server itself and
> the amount of resources allocated to VMs?
>

___
freebsd-emulation@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-emulation
To unsubscribe, send any mail to "freebsd-emulation-unsubscr...@freebsd.org"

Re: VirtualBox mangles memory

[Mark Felder]

On Sun, Sep 20, 2015, at 06:20, Martin Birgmeier wrote:
> On machines where VirtualBox is running, I get memory allocation errors.
> 
> 'dd' aborts with 'dd: stdout: Cannot allocate memory'
> 
> 'zfs send' aborts with 'internal error: Cannot allocate memory'
> 
> I have created https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=195970
> to track this.
> 
> Could a knowledgeable person please have a look into this?
> 
> FreeBSD 10.2, VirtualBox 4.3.30
> 
> -- Martin
> 

I'm not an expert, but I think this is known behavior. Virtualbox
requires a huge contiguous chunk of memory for the VM and that's
probably what you're running into.

Can you give more information about the specs of the server itself and
the amount of resources allocated to VMs?

-- 
  Mark Felder
  ports-secteam member
  f...@freebsd.org
___
freebsd-emulation@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-emulation
To unsubscribe, send any mail to "freebsd-emulation-unsubscr...@freebsd.org"

VirtualBox mangles memory

[Martin Birgmeier]

On machines where VirtualBox is running, I get memory allocation errors.

'dd' aborts with 'dd: stdout: Cannot allocate memory'

'zfs send' aborts with 'internal error: Cannot allocate memory'

I have created https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=195970
to track this.

Could a knowledgeable person please have a look into this?

FreeBSD 10.2, VirtualBox 4.3.30

-- Martin

___
freebsd-emulation@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-emulation
To unsubscribe, send any mail to "freebsd-emulation-unsubscr...@freebsd.org"