[Openstack] glance performance gains via sendfile()
Hi Reynolds, I've been looking into your interesting idea around sendfile()[1] usage, here are a few initial thoughts: - There's potentially even more speed-up to be harnessed in serving out images from the filesystem store via sendfile(), than from using it client-side on the initial upload (based on the assumption that images would typically uploaded once but downloaded many times, also that the download time is more crucial for perceived responsiveness as an instance being spun up by nova may be held up until the image is retreived from glance, unless already cached). - I'd suspect that some of the potential gain on the client side is currently thrown away by the syntax of the glance add CLI, specifically the use of shell redirection to pass the image content: glance add name=MyImage /path/to/image/file I'm open to correction, but this seems to needlessly copy the image content via user-space, even if the glance client avoids a second copy internally via the sendfile() usage. So I'd propose to also add a new cmd line arg to allow the file be directly specified, e.g. glance add name=MyImage path=/path/to/image/file This would have different semantics to the location field, e.g location=file:///path/to/image/file (which would imply that the content is not uploaded to the remote store). - The structure of typical pysendfile usage gets in the way of glance's image iterator pattern. On the client side this is more an an incovenience, requiring some restructuring of the code. However on the service-side, it seems we're a bit a hamstrung by the WSGI/webob APIs. For example the webob.response.body_file is filelike but doesn't expose a fileno attribute as there's no real underlying FD available intially, so the following kind of approach isn't workable: sendfile(response.body_file.fileno(), filestore_path.fileno(), ...) Seems a better approach would be to allow glance to be optionally deployed on a WSGI container that directly supports the [2] wsgi.file_wrapper extension (e.g. mod_wsgi on Apache, or uWSGI) or even allow one of the non-standard headers like X-Sendfile or X-Accel-Redirect to be set where supported. In case, I'll crack on with the basic client-side usage to begin with, so that we can quantify the performance gain. Cheers, Eoghan [1] http://code.google.com/p/pysendfile/ [2] http://www.python.org/dev/peps/pep-0333/#optional-platform-specific-file-handling ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] glance performance gains via sendfile()
If one wants to experiment with the performance effects of sendfile(), the netperf benchmark http://www.netperf.org/ has a TCP_SENDFILE test which complements the TCP_STREAM test. It can also report CPU utilization and service demand to allow a comparison of efficiency. netperf -H destination -t TCP_SENDFILE -F file -c -C -l 30 will run a 30-second TCP_SENDFILE tests using file as the data source (one is created if no -F option is specified) sending to destination (assumes that netserver has been launched on destination. The corresponding TCP_STREAM test would be the obvious substitution. One area of investigation would be the effect of send size on things. That can be accomplished with a test-specific (following a -- on the command line) -m option: netperf ...as above... -- -m 64K would cause netperf to send 65536 bytes in each send call. The manual for the current top-of-trunk version of netperf is at: http://www.netperf.org/svn/netperf2/trunk/doc/netperf.html and the top-of-trunk bits can be pulled via subversion pointing at http://www.netperf.org/svn/netperf2/trunk happy benchmarking, rick jones For example, between a pair of Ubuntu 11.04 systems with Mellanox 10GbE, and a pair of X5650 processors each (so 24 CPUs): ~$ ./netperf -p 12866 -H ndestination -c -C -l 30 -- -P 12867 -m 64K MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 12867 AF_INET to destination () port 12867 AF_INET : demo Recv SendSend Utilization Service Demand Socket Socket Message Elapsed Send Recv SendRecv Size SizeSize Time Throughput localremote local remote bytes bytes bytessecs.10^6bits/s % S % S us/KB us/KB 87380 16384 6553630.00 9271.36 2.52 2.72 0.535 0.576 ~$ ./netperf -t TCP_SENDFILE -p 12866 -H destination -c -C -l 30 -- -P 12867 -m 64K TCP SENDFILE TEST from 0.0.0.0 (0.0.0.0) port 12867 AF_INET to destination () port 12867 AF_INET : demo Recv SendSend Utilization Service Demand Socket Socket Message Elapsed Send Recv SendRecv Size SizeSize Time Throughput localremote local remote bytes bytes bytessecs.10^6bits/s % S % S us/KB us/KB 87380 16384 6553630.00 9332.46 0.82 2.71 0.173 0.572 It would be good to repeat each a couple times, but in this case at least, we see a considerable drop in sending side CPU utilization and service demand, the latter being a direct measure of efficiency. (the socket sizes are simply what they were at the onset of the connection, not by the end. for that, use omni output selectors - http://www.netperf.org/svn/netperf2/trunk/doc/netperf.html#Omni-Output-Selection - the test-specific -P option is to explicitly select port numbers for the data connection to deal with firewalls in my test environment - similarly for the global -p option selecting the port number on which netserver at destination is waiting) With a smaller send size the results may be a bit different: ~$ ./netperf -p 12866 -H destination -c -C -l 30 -- -P 12867 MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 12867 AF_INET to destination () port 12867 AF_INET : demo Recv SendSend Utilization Service Demand Socket Socket Message Elapsed Send Recv SendRecv Size SizeSize Time Throughput localremote local remote bytes bytes bytessecs.10^6bits/s % S % S us/KB us/KB 87380 16384 1638430.00 9332.43 2.64 2.74 0.556 0.578 ~$ ./netperf -t TCP_SENDFILE -p 12866 -H destination -c -C -l 30 -- -P 12867 TCP SENDFILE TEST from 0.0.0.0 (0.0.0.0) port 12867 AF_INET to destination () port 12867 AF_INET : demo Recv SendSend Utilization Service Demand Socket Socket Message Elapsed Send Recv SendRecv Size SizeSize Time Throughput localremote local remote bytes bytes bytessecs.10^6bits/s % S % S us/KB us/KB 87380 16384 1638430.00 9351.32 1.26 2.73 0.264 0.574 Mileage will vary depending on link-type, CPU's present, etc etc etc... ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
