Re: CFT - Xen infrastructure and block I/O improvements
On 17 Sep 2010, at 16:36, Justin T. Gibbs wrote: On 9/17/2010 1:27 AM, Rui Paulo wrote: On 17 Sep 2010, at 07:55, Rui Paulo wrote: On 17 Sep 2010, at 02:44, Justin T. Gibbs wrote: At Spectra Logic, we are using FreeBSD amd64 under Xen to serve storage to other Xen domains. Over the past 9 months, we've made several changes to FreeBSD's Xen support. These include: ... Justin, this is quite a big diff (16k lines). I wonder if you can create separate diffs (xenstore, blkback, xenbus, etc.) for easier review and commenting. '... and comment'. The bulk of the patch is due to code reorganization. Unfortunately SVN doesn't reflect the copies properly in a diff, but at least the diff will apply properly even in a non-SVN backed source tree. This original patch should be the best for testers. The following SVN operations were made to the source tree to clean up it's organization and to import functionality from the vendor (1 file): # Does not apply to FreeBSD's NewBus method for dealing with XenBus devices svn delete sys/xen/xenbus/init.txt # Linux version of backend XenBus service routines. Never ported to FreeBSD. # OBE: See xenbusb.c, xenbusb_if.m, xenbusb_front.c xenbusb_back.c svn delete sys/xen/xenbus/xenbus_probe_backend.c # Split XenStore into its own tree. XenBus is a software layer built on top # of XenStore. The old arrangement and the naming of some structures and # functions blurred these lines making it difficult to discern what services # are provided by which layer and what times these services are available # (e.g. during system startup and shutdown). mkdir sys/xen/xenstore svn add sys/xen/xenstore svn move sys/xen/xenbus/xenbus_dev.c sys/xen/xenstore/xenstore_dev.c svn copy sys/xen/xenbus/xenbusvar.h sys/xen/xenstore/xenstorevar.h svn move sys/xen/xenbus/xenbus_xs.c sys/xen/xenstore/xenstore.c # Split up XenBus code into methods available for use by client # drivers (xenbus.c) and code used by the XenBus bus code to # enumerate, attach, detach, and service bus drivers. svn move sys/xen/xenbus/xenbus_client.c sys/xen/xenbus/xenbus.c svn move sys/xen/xenbus/xenbus_probe.c sys/xen/xenbus/xenbusb.c svn copy sys/xen/xenbus/xenbusb.c sys/xen/xenbus/xenbusb.h # Merged with xenstore.c svn delete sys/xen/xenbus/xenbus_comms.c svn delete sys/xen/xenbus/xenbus_comms.h # Merged with new XenBus control driver mkdir sys/dev/xen/control svn add sys/dev/xen/control svn move sys/xen/reboot.c sys/dev/xen/control/control.c # New file from Xen vendor with macros and structures used by # a block back driver to service requires from a VM running a # different ABI (e.g. amd64 back with i386 front). svn add sys/xen/blkif.h These alone account for 6k lines of svn diff. A diff against a tree with these operations already made may make more sense to a reviewer. You can download this diff from here: http://people.FreeBSD.org/~gibbs/FreeBSD-head-xen_post-svn-ops_2010-09-17_diffs.txt It isn't much shorter since the additional context has amplified the changes. The bulk is largely caused by refactoring, and comments. It will probably be easier to just review the files in sys/xen/xenstore and sys/xen/xenbus in their entirety, but the comments bellow (boiled down from our SCM system), when coupled with the diffs, should give you enough information to understand the intentions behind the changes. Oh, IC. It's still pretty large and I don't have the necessary time, but it appears that this is good to go (am afraid the past people who have worked on Xen are now gone, no?). -- Rui Paulo ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: CFT - Xen infrastructure and block I/O improvements
On 17 Sep 2010, at 02:44, Justin T. Gibbs wrote: At Spectra Logic, we are using FreeBSD amd64 under Xen to serve storage to other Xen domains. Over the past 9 months, we've made several changes to FreeBSD's Xen support. These include: o Support for backend devices (e.g. blkback) o Extensions to the Xen para-virtualized block API to allow for larger and more outstanding I/Os. o A completely rewritten block back driver with support for fronting I/O to both raw devices and files. o General cleanup and documentation of the XenBus and XenStore support code. o Robustness and performance updates for the block front driver. o Fixes to the netfront driver. Some of these changes have already been pushed back into FreeBSD, but the bulk of them need additional testing, especially under i386 PV, before they can be committed. If you work in the Xen area, I'd appreciate your review and/or testing of these changes. http://people.freebsd.org/~gibbs/FreeBSD-head-xen-diffs_2010_09_16.txt Justin, this is quite a big diff (16k lines). I wonder if you can create separate diffs (xenstore, blkback, xenbus, etc.) for easier review and commenting. Regards, -- Rui Paulo ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: CFT - Xen infrastructure and block I/O improvements
On 17 Sep 2010, at 07:55, Rui Paulo wrote: On 17 Sep 2010, at 02:44, Justin T. Gibbs wrote: At Spectra Logic, we are using FreeBSD amd64 under Xen to serve storage to other Xen domains. Over the past 9 months, we've made several changes to FreeBSD's Xen support. These include: o Support for backend devices (e.g. blkback) o Extensions to the Xen para-virtualized block API to allow for larger and more outstanding I/Os. o A completely rewritten block back driver with support for fronting I/O to both raw devices and files. o General cleanup and documentation of the XenBus and XenStore support code. o Robustness and performance updates for the block front driver. o Fixes to the netfront driver. Some of these changes have already been pushed back into FreeBSD, but the bulk of them need additional testing, especially under i386 PV, before they can be committed. If you work in the Xen area, I'd appreciate your review and/or testing of these changes. http://people.freebsd.org/~gibbs/FreeBSD-head-xen-diffs_2010_09_16.txt Justin, this is quite a big diff (16k lines). I wonder if you can create separate diffs (xenstore, blkback, xenbus, etc.) for easier review and commenting. '... and comment'. Regards, -- Rui Paulo ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
Re: CFT - Xen infrastructure and block I/O improvements
On 9/17/2010 1:27 AM, Rui Paulo wrote: On 17 Sep 2010, at 07:55, Rui Paulo wrote: On 17 Sep 2010, at 02:44, Justin T. Gibbs wrote: At Spectra Logic, we are using FreeBSD amd64 under Xen to serve storage to other Xen domains. Over the past 9 months, we've made several changes to FreeBSD's Xen support. These include: ... Justin, this is quite a big diff (16k lines). I wonder if you can create separate diffs (xenstore, blkback, xenbus, etc.) for easier review and commenting. '... and comment'. The bulk of the patch is due to code reorganization. Unfortunately SVN doesn't reflect the copies properly in a diff, but at least the diff will apply properly even in a non-SVN backed source tree. This original patch should be the best for testers. The following SVN operations were made to the source tree to clean up it's organization and to import functionality from the vendor (1 file): # Does not apply to FreeBSD's NewBus method for dealing with XenBus devices svn delete sys/xen/xenbus/init.txt # Linux version of backend XenBus service routines. Never ported to FreeBSD. # OBE: See xenbusb.c, xenbusb_if.m, xenbusb_front.c xenbusb_back.c svn delete sys/xen/xenbus/xenbus_probe_backend.c # Split XenStore into its own tree. XenBus is a software layer built on top # of XenStore. The old arrangement and the naming of some structures and # functions blurred these lines making it difficult to discern what services # are provided by which layer and what times these services are available # (e.g. during system startup and shutdown). mkdir sys/xen/xenstore svn add sys/xen/xenstore svn move sys/xen/xenbus/xenbus_dev.c sys/xen/xenstore/xenstore_dev.c svn copy sys/xen/xenbus/xenbusvar.h sys/xen/xenstore/xenstorevar.h svn move sys/xen/xenbus/xenbus_xs.c sys/xen/xenstore/xenstore.c # Split up XenBus code into methods available for use by client # drivers (xenbus.c) and code used by the XenBus bus code to # enumerate, attach, detach, and service bus drivers. svn move sys/xen/xenbus/xenbus_client.c sys/xen/xenbus/xenbus.c svn move sys/xen/xenbus/xenbus_probe.c sys/xen/xenbus/xenbusb.c svn copy sys/xen/xenbus/xenbusb.c sys/xen/xenbus/xenbusb.h # Merged with xenstore.c svn delete sys/xen/xenbus/xenbus_comms.c svn delete sys/xen/xenbus/xenbus_comms.h # Merged with new XenBus control driver mkdir sys/dev/xen/control svn add sys/dev/xen/control svn move sys/xen/reboot.c sys/dev/xen/control/control.c # New file from Xen vendor with macros and structures used by # a block back driver to service requires from a VM running a # different ABI (e.g. amd64 back with i386 front). svn add sys/xen/blkif.h These alone account for 6k lines of svn diff. A diff against a tree with these operations already made may make more sense to a reviewer. You can download this diff from here: http://people.FreeBSD.org/~gibbs/FreeBSD-head-xen_post-svn-ops_2010-09-17_diffs.txt It isn't much shorter since the additional context has amplified the changes. The bulk is largely caused by refactoring, and comments. It will probably be easier to just review the files in sys/xen/xenstore and sys/xen/xenbus in their entirety, but the comments bellow (boiled down from our SCM system), when coupled with the diffs, should give you enough information to understand the intentions behind the changes. -- Justin sys/conf/files: Adjust kernel build spec for new XenBus/XenStore layout and added Xen functionality. sys/dev/xen/balloon/balloon.c: sys/dev/xen/netfront/netfront.c: sys/dev/xen/blkfront/blkfront.c: sys/xen/xenbus/... sys/xen/xenstore/... o Rename XenStore APIs and structures from xenbus_* to xs_*. o Adjust to use of M_XENBUS and M_XENSTORE malloc types for allocation of objects returned by these APIs. o Adjust for changes in the bus interface for Xen drivers. sys/dev/xen/blkback/blkback.c: Rewrite the Block Back driver to attach properly via newbus, operate correctly in both PV and HVM mode regardless of domain (e.g. can be in a DOM other than 0), and to deal with the latest metadata available in XenStore for block devices. Allow users to specify a file as a backend to blkback, in addition to character devices. Use the namei lookup to figure out whether it's a device node or a file, and use the appropriate interface. One current issue with the file interface is that we're effectively limited to having a single command at a time outstanding. To get around this, we may need to try using the vnode pager more directly, or perhaps coming up with a direct interface into ZFS. (i.e. something similar to zvols, but without the performance issues.) This will impact reads more than writes, because writes are cached but with random reads you have to go all the way down to the disk, so you suffer the full latency of the stack. sys/dev/xen/blkback/blkback.c: sys/xen/interface/io/blkif.h: sys/xen/blkif.h: sys/dev/xen/blkfront/blkfront.c:
CFT - Xen infrastructure and block I/O improvements
At Spectra Logic, we are using FreeBSD amd64 under Xen to serve storage to other Xen domains. Over the past 9 months, we've made several changes to FreeBSD's Xen support. These include: o Support for backend devices (e.g. blkback) o Extensions to the Xen para-virtualized block API to allow for larger and more outstanding I/Os. o A completely rewritten block back driver with support for fronting I/O to both raw devices and files. o General cleanup and documentation of the XenBus and XenStore support code. o Robustness and performance updates for the block front driver. o Fixes to the netfront driver. Some of these changes have already been pushed back into FreeBSD, but the bulk of them need additional testing, especially under i386 PV, before they can be committed. If you work in the Xen area, I'd appreciate your review and/or testing of these changes. http://people.freebsd.org/~gibbs/FreeBSD-head-xen-diffs_2010_09_16.txt Thanks, Justin ___ freebsd-current@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to freebsd-current-unsubscr...@freebsd.org
