Re: Python on FreeBSD is slower than on Linux

[Alfred Perlstein]

Vladimir ,

Please run truss(1) against the python code and paste a subset here. Maybe it 
is doing many semaphore ops. 

Sent from my iPhone

> On Nov 13, 2015, at 12:49 AM, Mark Blackman  wrote:
> 
> Vladimir
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: Python on FreeBSD is slower than on Linux

[Alfred Perlstein]

I'm adding Freebsd-virtualization to this thread as both problems point 
to some possible issue with FreeBSD as a guest.  (although a bare metal 
comparison should likely be done as well).


-Alfred

On 11/12/15 11:26 AM, Vladimir Bogrecov wrote:

Hello,

I'm developing a little project on Python 3.5. The server's operating
system is FreeBSD 10.2. Today I decided to do a little test "just for fun"
and the result has confused me. I ran the following code

import random
import time


def test_sort(size):
 sequence = [i for i in range(0, size)]
 random.shuffle(sequence)
 start = time.time()
 ordered_sequence = sorted(sequence)
 print(time.time() - start)


if __name__ == '__main__':
 test_sort(100)

on FreeBSD 10.2 x64 and on Debian 8 x64. Both computers was the smallest
(5$ per month) virtual machines on the Digital Ocean (
https://www.digitalocean.com). The average result on the FreeBSD was 1.5
sec, on the Debian 1.0 sec. Both machines was created specially for test
and had not any customization. Could you help me to understand why python
is so slower on FreeBSD and may be there are some steps I can perform to
speed up the python to work not slower than on Debian.

I have found in Google the similar question:
https://lists.freebsd.org/pipermail/freebsd-python/2012-June/004306.html so
it has an interest not only for me.

P.S. I really like FreeBSD and I would be happy to solve this issue. If you
will have an interest to this issue I can provide SSH access for both
machines :)

Thank You!
___
freebsd-pyt...@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-python
To unsubscribe, send any mail to "freebsd-python-unsubscr...@freebsd.org"



___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Program for dynamically making taps/bridge topologies.

[Alfred Perlstein]

Hello,

Here at Norse we are using bhyve to run our appliance in a test environment.

We have a need for a test network to be made on demand, so this weekend 
I took some time and wrote a tool that allows you to specify a topology 
of bridges and taps that you can then use to create your virtual network.


In our case our test servers have a bridge0 that is static and bridges 
to our private test network.  However we also need a bridge to be 
dynamically that we will generate traffic over in isolation. We then 
need taps created and "assigned" or labeled properly for our QA suite to 
attach to our VMs properly.


We then also need the ability to query the topology and emit the correct 
taps to assign to each vm.


Finally we need the ability to tear down the virtual network once it's 
no longer needed.


This tool provides all three functions.

The tool is available here:
https://github.com/splbio/netmanager

Comments, pull requests and questions are welcome.

-Alfred
___
freebsd-virtualization@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
"freebsd-virtualization-unsubscr...@freebsd.org"

Re: backups of bhyve images

[Alfred Perlstein]

On 12/10/14 11:40 AM, Pete Wright wrote:


On 12/10/14 11:29, John-Mark Gurney wrote:

Pete Wright wrote this message on Mon, Dec 08, 2014 at 15:44 -0800:


On 12/08/14 15:30, Craig Rodrigues wrote:

   (3)  When you want to backup the VM, do a zfs snapshot take take a
snapshot of the ZFS zvol.

will this ensure that your zvol is consistent, or rather will the
filesystem overlaid on the zvol device be ensured it is consistent when
the hypervisor issues a snapshot command?

That's the beauty of FreeBSD... UFS provides this w/ soft updates, and
ZFS does this through COW...  In both cases, as far as I understand it,
it is safe to snapshot the FS...


hrm not sure I think that is a reliable methodology.  You'll be forced
to recover you VM's filesystem at best, and potentially have corrupt
data if blocks were still sitting in cache and had not returned a
filehande.  My guess is that this method will work with out issues
%80-%90 of the time - but you still expose yourself to possible data
loss or corruption.

I think you and others have mentioned the proper way to do this -
snapshot the guess filesystem from with-in the guest VM itself.

cheers,
-pete


Hence why in the original email I wrote:

*option 3 is the least safe imo as you can wind up with filesystem angry.*

in case 1 and 2 you'll have UFS snapshots that should be OK to restore from.
in case 4 you are also doing snapshot, but you switch to ZFS.



___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: backups of bhyve images

[Alfred Perlstein]

On Dec 8, 2014, at 8:33 AM, John wrote:

 Hello list,
 
 I have a few questions about creating backups to be stored offsite.
 
 If a guest is running, can I compress the image without it becoming 
 inconsistent? If not, can it be copied without it becoming inconsistent?  
 By inconsistent, I mean will I see weird effects and broken files if the 
 backup is restored? Previously I've shut the VM down to avoid this, 
 before archiving.
 
 I have each image on its own (external to the image) ZFS filesystem.  
 Internally the image is using ufs if freebsd, ext3fs if linux.  Would 
 using some ZFS method of duplication be better? In this case, would the 
 image become inconsistent?
 
 Basically, what I want to do is to run accurate backups without shutting 
 down and restarting the VM. Is this possible? If it isn't, I think the 
 only alternative is to make a script that shuts the vm down, copies it, 
 restarts the vm then runs its compression and backup-over-ssh routine.

[[ adding f...@freebsd.org in case I'm wrong ]]

If you are using UFS internally to the VMs then you'll need to send a snapshot 
that is consistent.

If you are just copying the files out from under a running vm you are going to 
get spaghettios for a filesystem if you try to recover as you need a true point 
in time snapshot.

I think a few better options would be:
1) Inside the VM create a UFS snapshot then dump that externally using tools.
2) Create the UFS snapshot, then make sure that the file/vzol is snapshotted 
using zfs.
3) Just snapshot the underlying zvol you've made the UFS image on and send that 
(you'll get a dirty FS on restore, but it *should* be recoverable with a simple 
fsck)  
4) Use zfs internally to the vm and send/receive the internal zfs.

option 3 is the least safe imo as you can wind up with filesystem angry.
in case 1 and 2 you'll have UFS snapshots that should be OK to restore from.
in case 4 you are also doing snapshot, but you switch to ZFS.

-Alfred

___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: bhyve tapN additions

[Alfred Perlstein]

On 10/22/14 2:54 PM, Peter Grehan wrote:

Hi Allan,


The tap device can also be created automatically by bhyve, make sure you
specify a different tap device for each instance of bhyve.


 That may require some devd scripting to get it to be added to a 
bridge group.


 We're looking at having the bridge interface have a cloneable 
tap-style interface so this step can be avoided i.e. point bhyve at a 
/dev/bridgeN device, and it will automatically become a part of that 
bridge.


Peter, that would be super helpful.  Thank you!

-Alfred

___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: Public Service Announcement: The name is bhyve

[Alfred Perlstein]

On 1/27/14 6:13 AM, Lars Engels wrote:

Am 2014-01-24 11:18, schrieb Michael Dexter:

Hello all,

It was BHyVe and Neel and Peter mercifully simplified it to bhyve.

It was never BHyve and how they came up with byhyve and byhve in
the same article I do not know:

http://www.serverwatch.com/server-news/open-source-freebsd-10-takes-on-virtualization.html 



Haters gonna hate?




Don't forget Beehyve:

http://www.admin-magazin.de/News/FreeBSD-10.0-fertiggestellt/%28language%29/ger-DE 



But don't confuse it with Beehive:

https://stbeehive.oracle.com/bcentral/


:-)



http://www.youtube.com/watch?v=-1GadTfGFvU

my eyes!

--
Alfred Perlstein

___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: bhyve in -current 4/14/13 can no longer support FreeBSD stable install

[Alfred Perlstein]

On 3/15/13 2:27 PM, Peter Grehan wrote:

Hi Alfred,


What will happen is during the extract process the install will hang. No
network IO happens and on the host I see bhyve's CPU hit 100% for each
core assigned.

For reference this is the bhyve command invoked via Neel's vmrun.sh.

0 3915 3908 0 103 0 2122208 103428 - R+ 2 1048:45.75 /usr/sbin/bhyve -c
2 -m 2048 -M 0 -AI -H -P -g 0 -s 0:0,hostbridge -s 1:0,virtio-net,tap0
-s 2:0,virtio-blk,../freebsd-stable.img -s
3:0,virtio-blk,../FreeBSD-9.1-STABLE-amd64-20130216-r246877-bootonly.iso
-S 31,uart,stdio freebsd-stable

I tried to gdb the bhyve process but that was bad news. Is there a set
of steps usually taken to help pinpoint what is going on when we this
state?


 Does the previous snapshot (r247640) work better ? The relevant 
change there was the recent virtio guest MFC.


 Some things to try:
  - boot bhyve uniprocessor
  - is the bhyve process performing any disk i/o, or is it just 
spinning ?
  the bhyvectl program can dump a lot of vm state - you can try 
something like


 sudo bhyvectl --get-all --cpu=0 --vm=your vm name
 sudo bhyvectl --get-all --cpu=1 --vm=your vm name

 If you do this repeatedly, it may be possible to see some patterns 
e.g. counters incrementing, %RIP values being the same etc.



Hmm, I keep seem to be getting this:
/usr/home/alfred # bhyvectl --get-all --cpu=0 --vm=freebsd-stable
lowmem0x/536870912
highmem0x0001/0
errno = 16
/usr/home/alfred # bhyvectl --get-all --cpu=1 --vm=freebsd-stable
lowmem0x/536870912
highmem0x0001/0
errno = 16



 Regarding gdb and bhyve, for debugging I usually start bhyve under 
gdb in the foreground and send output to another pty with the gdb set 
tty command (note the other pty should have a 'sleep 10' command 
run to avoid input being captured) - this way, a ctl-C in the gdb 
session will drop into gdb and allow breakpoints to be set etc.


I will try this next.  Thank you.



later,

Peter.
___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org




___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: question on bhyve + 9-stable

[Alfred Perlstein]

On 1/3/13 5:04 PM, Michael Dexter wrote:

On 1/2/13 2:53 PM, Alfred Perlstein wrote:

While the directions you and the rest of the bhyve developers put
together were clear, comprehensive and well written... they still are a
bit daunting for your average user...


There-in is the challenge. You can try bhyve with simply a special 
snapshot and yes, user's will soon have it by default. Creating guests 
however opens up a can of worms of opportunities to tailor them to 
your needs. jail(8) is the same in this regard. I am however talking 
to Kris about integrating bhyve into PC-BSD's Warden so that there can 
be at least one turn-key experience out there.




Yes, but please in no way take my comments as a reflection on the bhyve 
project.


The only issues I had were just due to that it has not been integrated 
into head yet.  I guess we may be stuck due to binutils for a while?  I 
am not sure.


-Alfred

___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org

Re: question on bhyve + 9-stable

[Alfred Perlstein]

Peter, this worked perfectly!!

Happy new year.

Let me know if you need any info or stuff done on this rig.  The point 
is for me to run FreeBSD 9-stable and FreeNAS to do development and testing.


-Alfred

On 12/30/12 9:04 AM, Peter Grehan wrote:

Hi Alfred,


I only have 1 really fast machine, so I've decided to give bhyve a whirl
and try to do -stable development inside of a bhyve instance.


 Great !


The problem I'm having is that:
1) the 9.1-RELEASE image does not seem to boot because it can not mount
the rootfs (iso). I think this may be due to missing virtualization
modules.


 Yep, that's right. Michael Dexter has pre-built virtio modules at

http://mirrors.nycbug.org/pub/BHyVe/r244024/

 .. and I guess the long-term solution for running older FreeBSD 
releases is to have bhyve implement block/network device models that 
will be recognized. AHCI looks like a good pick for a block model: any 
input on a network device would be appreciated (e1000 does seem to 
stick out).



2) building release iso from 9-stable, while it has the klds needed, the
image seems to causes the hypervisor to exit.


 That's the new TSC sync detection code, which does a CPUID with 
basically a random leaf value. bhyve should deal with that: a fix will 
be in when I'm back from vacation. But, in the meantime, there are two 
workarounds:


 - use single CPU only (-c 1), or

 - force the hard-coded hypervisor detection code in FreeBSD to 
trigger with a loader variable


set smbios.bios.vendor=Bochs

   ... and, since the TSC is actually OK, force the use of that as the 
timecounter


set kern.timecounter.invariant_tsc=1
set kern.timecounter.smp_tsc=1

later,

Peter.



___
freebsd-virtualization@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
To unsubscribe, send any mail to 
freebsd-virtualization-unsubscr...@freebsd.org