On 11/01/2013 02:21 PM, Clayton Coleman wrote: > > > ----- Original Message ----- >>>>> I was blindly assuming we want to pull in eventlet support, with the >>>>> implicit understanding that we will be doing some form of timeslicing and >>>>> async io bound waiting in the API... but would like to hear others weigh >>>>> in before I add the monkey_patch and stub code around script startup. >>>> >>>> I'm not so sure that bringing in eventlet should be done by default. It >>>> adds complexity and if most/all of the API calls will be doing some call >>>> to a native C library like libmysql that blocks, I'm not sure there is >>>> going to be much benefit to using eventlet versus multiplexing the >>>> servers using full OS processes -- either manually like some of the >>>> projects do with the workers=N configuration and forking, or using more >>>> traditional multiplexing solutions like running many mod_wsgi or uwsgi >>>> workers inside Apache or nginx. >>>> >>> >>> What about callouts to heat/keystone APIs? >> >> Sure, it's possible to do that with eventlet. It's also possible to do >> that with a queueing system. For example, the API server could send an >> RPC message to a queue and a separate process could work on executing >> the individual tasks associated with a particular application build-pack. > > I guess I was asking because event patterns are generally more efficient for > memory than multiprocess, assuming that the underlying codebase isn't > fighting the event system at every step. Are your concerns with eventlet > based on that mismatch (bugs, problems with eventlet across the various > projects and libraries that OpenStack uses) or more that you believe we > should start, at the very beginning, with the pattern of building everything > as a distributed ordered task flow since we know at least some of our > interactions are asynchronous? There are at least a few network IO > operations that will be synchronous to our flows - while they are not likely > to be a large percentage of the API time, they may definitely block threads > for a period of time. > >> >> The design of Zuul [1], especially with the relatively recent addition >> of Gearman and the nodepool orchestrator [2] that the openstack-infra >> folks wrote would, IMO, be a worthy place to look for inspiration, since >> Solum essentially is handling a similar problem domain -- distributed >> processing of ordered tasks. >> >> Best, >> -jay > > Familiar with Gearman, will look through nodepool/Zuul.
BTW - I was just brainstorming the other day that we might want to have a few of our things- nodepool might be one of them - start to take advantage of taskflow as well. In general though, I think we can all agree that zuul can handle a pretty high load. :) _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
