Terrible NFS performance under 9.2-RELEASE?
While setting up a test for other purposes, I noticed some really horrible NFS performance issues. To explore this, I set up a test environment with two FreeBSD 9.2-RELEASE-p3 virtual machines running under KVM. The NFS server is configured to serve a 2 gig mfs on /mnt. The performance of the virtual network is outstanding: Server: $ iperf -c 172.20.20.169 Client connecting to 172.20.20.169, TCP port 5001 TCP window size: 1.00 MByte (default) [ 3] local 172.20.20.162 port 59717 connected with 172.20.20.169 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 16.1 GBytes 13.8 Gbits/sec $ iperf -s Server listening on TCP port 5001 TCP window size: 1.00 MByte (default) [ 4] local 172.20.20.162 port 5001 connected with 172.20.20.169 port 45655 [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.0 sec 15.8 GBytes 13.6 Gbits/sec Client: $ iperf -s Server listening on TCP port 5001 TCP window size: 1.00 MByte (default) [ 4] local 172.20.20.169 port 5001 connected with 172.20.20.162 port 59717 [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.0 sec 16.1 GBytes 13.8 Gbits/sec ^C$ iperf -c 172.20.20.162 Client connecting to 172.20.20.162, TCP port 5001 TCP window size: 1.00 MByte (default) [ 3] local 172.20.20.169 port 45655 connected with 172.20.20.162 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 15.8 GBytes 13.6 Gbits/sec The performance of the mfs filesystem on the server is also good. Server: $ sudo mdconfig -a -t swap -s 2g md0 $ sudo newfs -U -b 4k -f 4k /dev/md0 /dev/md0: 2048.0MB (4194304 sectors) block size 4096, fragment size 4096 using 43 cylinder groups of 48.12MB, 12320 blks, 6160 inodes. with soft updates super-block backups (for fsck_ffs -b #) at: 144, 98704, 197264, 295824, 394384, 492944, 591504, 690064, 788624, 887184, 985744, 1084304, 1182864, 1281424, 1379984, 1478544, 1577104, 1675664, 1774224, 1872784, 1971344, 2069904, 2168464, 2267024, 2365584, 2464144, 2562704, 2661264, 2759824, 2858384, 2956944, 3055504, 3154064, 3252624, 3351184, 3449744, 3548304, 3646864, 3745424, 3843984, 3942544, 4041104, 4139664 $ sudo mount /dev/md0 /mnt $ cd /mnt $ sudo iozone -e -I -s 512m -r 4k -i 0 -i 1 -i 2 Iozone: Performance Test of File I/O Version $Revision: 3.420 $ [...] random random KB reclen write rewritereadrereadread write 524288 4 560145 1114593 933699 831902 56347 158904 iozone test complete. But introduce NFS into the mix and everything falls apart. Client: $ sudo mount -o tcp,nfsv3 f12.phxi:/mnt /mnt $ cd /mnt $ sudo iozone -e -I -s 512m -r 4k -i 0 -i 1 -i 2 Iozone: Performance Test of File I/O Version $Revision: 3.420 $ [...] random random KB reclen write rewritereadrereadread write 524288 4 672462923 103295 1272407 172475 196 And the above took 48 minutes to run, compared to 14 seconds for the local version. So it's 200x slower over NFS. The random write test is over 800x slower. Of course NFS is slower, that's expected, but it definitely wasn't this exaggerated in previous releases. To emphasize that iozone reflects real workloads here, I tried doing an svn co of the 9-STABLE source tree over NFS but after two hours it was still in llvm so I gave up. While all this not-much-of-anything NFS traffic is going on, both systems are essentially idle. The process on the client sits in newnfs wait state with nearly no CPU. The server is completely idle except for the occasional 0.10% in an nfsd thread, which otherwise spend their lives in rpcsvc wait state. Server iostat: $ iostat -x -w 10 md0 extended device statistics device r/s w/skr/skw/s qlen svc_t %b [...] md00.0 36.0 0.0 0.00 1.2 0 md00.0 38.8 0.0 0.00 1.5 0 md00.0 73.6 0.0 0.00 1.0 0 md00.0 53.3 0.0 0.00 2.5 0 md00.0 33.7 0.0 0.00 1.1 0 md00.0 45.5 0.0 0.00 1.8 0 Server nfsstat: $ nfsstat -s -w 10 GtAttr Lookup Rdlink Read Write Rename Access Rddir [...] 0 0 0471816 0 0 0 0 0 0480751 0
Re: panic: AP #1 (PHY# 1) failed!
Hi Sergey, Now I can say it happens not every time but one time on two or three good boots. Are you destroying the VM after the bhyve process exits and before each load ? i.e. bhyvectl --destroy --vm=vmname If not: that's required. If so, then could be a bug and yes we may need more info. later, Peter. ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: emulators/petitecloud 0.2.1 available
On Fri, Jan 17, 2014 at 09:36:27PM -0500, Aryeh Friedman wrote: On Fri, Jan 17, 2014 at 7:25 PM, Aryeh Friedman aryeh.fried...@gmail.comwrote: On Fri, Jan 17, 2014 at 7:11 PM, Mark Martinec mark.martinec+free...@ijs.si wrote: http://www.PetiteCloud.org -- http://www.petitecloud.org/downloads/port.tar.gz # make === src-petitecloud-aegis-0.2.1 depends on file: /usr/local/sbin/pkg - found = src-petitecloud-aegis-0.2.1.tar.gz doesn't seem to exist in /usr/ports/distfiles/. = Attempting to fetch ftp://ftp.petitecloud.org/petitecloud/0.2.1/aegis/src- petitecloud-aegis-0.2.1.tar.gz fetch: ftp://ftp.petitecloud.org/petitecloud/0.2.1/aegis/src- petitecloud-aegis-0.2.1.tar.gz: File unavailable (e.g., file not found, no access) There is no directory 'aegis' there, but there is an 'aryeh'. Also there is no src-petitecloud-aegis-0.2.1.tar.gz in it. URL should probably be: ftp://ftp.petitecloud.org/petitecloud/0.2.1/aryeh/src- petitecloud-aryeh-0.2.1.tar.gz Will look into this... for now just use aegis... you will need to modify the Makefile... since the Makefile itself is autogenerated we will post new port files later tonight with the fix in them btw if you're interested in helping with development you will also need devel/aegis in addition to the port's dependencies Fixed and posted new port files to petitecloud.org... side question should I update the PR for the port or create a new one? Please update the existing one. pgpXGUKPYzGvg.pgp Description: PGP signature
kldload vmm partial lockup
I’ve been trying to get bhyve to work on a system with the following specs: FreeBSD 10.0 64 bit ASUS P6T Deluxe Intel i7 920 According to what I’ve researched my cpu should have the VT-D extensions. However many users of this motherboard have complained about ASUS not properly supporting VT-D with Xen and something about a problem with tables getting corrupt due to a problem with the bios. On the third link below which is the Xen wiki it even states that this motherboard needs a bios update which is not available to the public. https://communities.intel.com/thread/28389 http://vip.asus.com/forum/view.aspx?id=20090402224408018SLanguage=en-usboard_id=1 http://wiki.xen.org/wiki/VTd_HowTo I am curious though if this would be what would be causing my problem with bhyve? I suppose I am willing to just by a new motherboard if so. I can run kldload vmm and I see a bunch of text fly by and then something about uhci interrupt problem. As soon as that happens the messages repeats my network drops and no keyboard input. However the message repeats and it doesn’t appear that the system has fully locked up otherwise. However on another system that works I notice that kldload VMM does not show this information it just loads the module. Is there a way I can gather more logs somehow so that I can determine what text is appearing before the uhci errors? Perhaps some kind of dump procedure? Or is there a way I can confirm that freebsd can properly see and utilize the VT-D extensions. VirtualBox runs fine with the VT extensions enabled. I also made sure VirtualBox wasn’t installed or loaded before I tried bhyve. I’ve tried to research for a week or two before posting here. Joe Maloney ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: kldload vmm partial lockup
On Sun, Jan 19, 2014 at 5:21 PM, Joe Maloney jpm...@gmail.com wrote: I’ve been trying to get bhyve to work on a system with the following specs: FreeBSD 10.0 64 bit ASUS P6T Deluxe Intel i7 920 According to what I’ve researched my cpu should have the VT-D extensions. However many users of this motherboard have complained about ASUS not properly supporting VT-D with Xen and something about a problem with tables getting corrupt due to a problem with the bios. On the third link below which is the Xen wiki it even states that this motherboard needs a bios update which is not available to the public. https://communities.intel.com/thread/28389 http://vip.asus.com/forum/view.aspx?id=20090402224408018SLanguage=en-usboard_id=1 http://wiki.xen.org/wiki/VTd_HowTo I am curious though if this would be what would be causing my problem with bhyve? I suppose I am willing to just by a new motherboard if so. I can run kldload vmm and I see a bunch of text fly by and then something about uhci interrupt problem. As soon as that happens the messages repeats my network drops and no keyboard input. However the message repeats and it doesn’t appear that the system has fully locked up otherwise. However on another system that works I notice that kldload VMM does not show this information it just loads the module. Is there a way I can gather more logs somehow so that I can determine what text is appearing before the uhci errors? Perhaps some kind of dump procedure? Or is there a way I can confirm that freebsd can properly see and utilize the VT-D extensions. VirtualBox runs fine with the VT extensions enabled. I also made sure VirtualBox wasn’t installed or loaded before I tried bhyve. I’ve tried to research for a week or two before posting here. Joe Maloney According to this, your processor supports VT-x, but not VT-d ; http://ark.intel.com/products/37147/Intel-Core-i7-920-Processor-8M-Cache-2_66-ghz-4_80-gts-Intel-qpi -Tom ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: kldload vmm partial lockup
On Sun, Jan 19, 2014 at 5:34 PM, Thomas Hoffmann trh...@gmail.com wrote: On Sun, Jan 19, 2014 at 5:21 PM, Joe Maloney jpm...@gmail.com wrote: I’ve been trying to get bhyve to work on a system with the following specs: FreeBSD 10.0 64 bit ASUS P6T Deluxe Intel i7 920 According to what I’ve researched my cpu should have the VT-D extensions. However many users of this motherboard have complained about ASUS not properly supporting VT-D with Xen and something about a problem with tables getting corrupt due to a problem with the bios. On the third link below which is the Xen wiki it even states that this motherboard needs a bios update which is not available to the public. https://communities.intel.com/thread/28389 http://vip.asus.com/forum/view.aspx?id=20090402224408018SLanguage=en-usboard_id=1 http://wiki.xen.org/wiki/VTd_HowTo I am curious though if this would be what would be causing my problem with bhyve? I suppose I am willing to just by a new motherboard if so. I can run kldload vmm and I see a bunch of text fly by and then something about uhci interrupt problem. As soon as that happens the messages repeats my network drops and no keyboard input. However the message repeats and it doesn’t appear that the system has fully locked up otherwise. However on another system that works I notice that kldload VMM does not show this information it just loads the module. Is there a way I can gather more logs somehow so that I can determine what text is appearing before the uhci errors? Perhaps some kind of dump procedure? Or is there a way I can confirm that freebsd can properly see and utilize the VT-D extensions. VirtualBox runs fine with the VT extensions enabled. I also made sure VirtualBox wasn’t installed or loaded before I tried bhyve. I’ve tried to research for a week or two before posting here. Joe Maloney According to this, your processor supports VT-x, but not VT-d ; http://ark.intel.com/products/37147/Intel-Core-i7-920-Processor-8M-Cache-2_66-ghz-4_80-gts-Intel-qpi -Tom But this i7-920 processor does: http://ark.intel.com/products/43126/Intel-Core-i7-920XM-Processor-Extreme-Edition-8M-Cache-2_00-GHz Not sure which you have. -Tom ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Fwd: kldload vmm partial lockup
-- Forwarded message -- From: Aryeh Friedman aryeh.fried...@gmail.com Date: Sun, Jan 19, 2014 at 6:26 PM Subject: Re: kldload vmm partial lockup To: Thomas Hoffmann trh...@gmail.com You should see the proposed handbook entry on virtualization ( http://www.petitecloud.org/handbook.jsp) for more info... note to the list Dee will be doing some additional proofing of this soon and we will submit again On Sun, Jan 19, 2014 at 5:55 PM, Thomas Hoffmann trh...@gmail.com wrote: On Sun, Jan 19, 2014 at 5:34 PM, Thomas Hoffmann trh...@gmail.com wrote: On Sun, Jan 19, 2014 at 5:21 PM, Joe Maloney jpm...@gmail.com wrote: I’ve been trying to get bhyve to work on a system with the following specs: FreeBSD 10.0 64 bit ASUS P6T Deluxe Intel i7 920 According to what I’ve researched my cpu should have the VT-D extensions. However many users of this motherboard have complained about ASUS not properly supporting VT-D with Xen and something about a problem with tables getting corrupt due to a problem with the bios. On the third link below which is the Xen wiki it even states that this motherboard needs a bios update which is not available to the public. https://communities.intel.com/thread/28389 http://vip.asus.com/forum/view.aspx?id=20090402224408018SLanguage=en-usboard_id=1 http://wiki.xen.org/wiki/VTd_HowTo I am curious though if this would be what would be causing my problem with bhyve? I suppose I am willing to just by a new motherboard if so. I can run kldload vmm and I see a bunch of text fly by and then something about uhci interrupt problem. As soon as that happens the messages repeats my network drops and no keyboard input. However the message repeats and it doesn’t appear that the system has fully locked up otherwise. However on another system that works I notice that kldload VMM does not show this information it just loads the module. Is there a way I can gather more logs somehow so that I can determine what text is appearing before the uhci errors? Perhaps some kind of dump procedure? Or is there a way I can confirm that freebsd can properly see and utilize the VT-D extensions. VirtualBox runs fine with the VT extensions enabled. I also made sure VirtualBox wasn’t installed or loaded before I tried bhyve. I’ve tried to research for a week or two before posting here. Joe Maloney According to this, your processor supports VT-x, but not VT-d ; http://ark.intel.com/products/37147/Intel-Core-i7-920-Processor-8M-Cache-2_66-ghz-4_80-gts-Intel-qpi -Tom But this i7-920 processor does: http://ark.intel.com/products/43126/Intel-Core-i7-920XM-Processor-Extreme-Edition-8M-Cache-2_00-GHz Not sure which you have. -Tom ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org -- Aryeh M. Friedman, Lead Developer, http://www.PetiteCloud.org -- Aryeh M. Friedman, Lead Developer, http://www.PetiteCloud.org ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: emulators/petitecloud 0.2.1 available
On Sun, Jan 19, 2014 at 5:15 PM, Aryeh Friedman aryeh.fried...@gmail.comwrote: Fixed and posted new port files to petitecloud.org... side question should I update the PR for the port or create a new one? Please update the existing one. Already done via GNATS If any committers are willing to take it that is ;-) -- Aryeh M. Friedman, Lead Developer, http://www.PetiteCloud.org -- Aryeh M. Friedman, Lead Developer, http://www.PetiteCloud.org ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
Re: kldload vmm partial lockup
Hi Joe, On Sun, Jan 19, 2014 at 2:21 PM, Joe Maloney jpm...@gmail.com wrote: I’ve been trying to get bhyve to work on a system with the following specs: FreeBSD 10.0 64 bit ASUS P6T Deluxe Intel i7 920 According to what I’ve researched my cpu should have the VT-D extensions. However many users of this motherboard have complained about ASUS not properly supporting VT-D with Xen and something about a problem with tables getting corrupt due to a problem with the bios. On the third link below which is the Xen wiki it even states that this motherboard needs a bios update which is not available to the public. https://communities.intel.com/thread/28389 http://vip.asus.com/forum/view.aspx?id=20090402224408018SLanguage=en-usboard_id=1 http://wiki.xen.org/wiki/VTd_HowTo I am curious though if this would be what would be causing my problem with bhyve? I suppose I am willing to just by a new motherboard if so. I can run kldload vmm and I see a bunch of text fly by and then something about uhci interrupt problem. As soon as that happens the messages repeats my network drops and no keyboard input. However the message repeats and it doesn’t appear that the system has fully locked up otherwise. However on another system that works I notice that kldload VMM does not show this information it just loads the module. Is there a way I can gather more logs somehow so that I can determine what text is appearing before the uhci errors? Perhaps some kind of dump procedure? Or is there a way I can confirm that freebsd can properly see and utilize the VT-D extensions. VirtualBox runs fine with the VT extensions enabled. I also made sure VirtualBox wasn’t installed or loaded before I tried bhyve. I’ve tried to research for a week or two before posting here. I have a patch to not initialize the iommu unless there are passthru devices explicitly configured for bhyve to use. It is available here: http://people.freebsd.org/~neel/patches/bhyve_iommu_init.patch Could you give it a spin and see if it helps with the ASUS system? best Neel Joe Maloney ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org ___ freebsd-virtualization@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to freebsd-virtualization-unsubscr...@freebsd.org
