Here's the updated architecture based on the conversations we've had. https://docs.google.com/document/d/1PtFKcizW3bmP3QBv771tSGMO4VaCiLkWlr-oDzn- FqU/edit
Anyone outside Canonical can view it this time. Follow up with comments here please. On Friday 04 May 2012 15:50:15 Julian Edwards wrote: > Hi > > Gavin also has a proposal to come, but this is the basis of what I discussed > with Robert earlier. > > The main aim is hyperscale. That is, we want to support hundreds of > thousands of nodes. Cobbler simply doesn't allow us to do that. In > addition we are currently tied into driving it synchronously so that we can > be sure it synced its own database, which causes bugs like: > https://bugs.launchpad.net/maas/+bug/989355 > > The other main issue surrounds DHCP/TFTP and PXE booting. When we write out > config files for these services we need to make sure there are no conflicts > with multiple processes/threads doing the same thing. > > Finally, we need to separate appserver requests from provisioning tasks > because the latter can be long running (as per the above bug). > > To this end we discussed: > > * Use Celery as a task queue (can use Rabbit or Django's DB as a broker) > → Django has a Celery plugin which makes this very easy > * Have one queue per pserv process > * Each pserv is responsible for servicing the tasks on that queue > * It scales by having multiple queues each with another pserv, or just > multiple pservs pulling from one queue. > * The pserv can read the database by using Django's ORM, and write to it by > sending API requests back to Django's appserver threads (again, scalable). > > We also talked about batching requests on the Rabbit task queues because we > might want to accumulate several requests that change the configs for DHCP > etc. For now though we can process serially and optimise later. We'd need > to think about how to implement locking though if there are multiple > writers to the same files. > > J -- Mailing list: https://launchpad.net/~maas-devel Post to : [email protected] Unsubscribe : https://launchpad.net/~maas-devel More help : https://help.launchpad.net/ListHelp
