Thanks, Jay. This covers my feelings pretty much as well. I am concerned as well that it is a 180 degree turn 3 weeks before feature freeze. I like the abstraction, but I would like us to keep the support for redis. I think SQL is critical for the enterprise and private clouds, but at Rackspace's scale, especially with regards to globalization, I think we are going to need some kind of keystore.
My feeling is that we put this in Austin +1 and add support for other datastores. That will also give us time to write up a blueprint and have an in depth discussion about it at the summit in November. I have added this to the agenda for the next release meeting on Sept 14. Rick On 09/10/2010 11:56 AM, Jay Pipes wrote: > Hi Vish, > > Such a large patch has taken me quite some time to digest. There is a > larger discussion on large patches without any specifications, but > I'll leave that for a later time! :) > > I am torn on this one, mostly because I spent a bunch of time > attempting to do the datastore refactoring myself (as did Justin Santa > Barbara), and thus I know the dragons that live in this layer of the > code :) > > One of the things that both Justin and I had tried was to keep an > abstraction layer that would allow both NoSQL as well as SQL data > stores to be used. Unfortunately, it seems that this patch removes > the ability to use ReDIS, among other NoSQL stores. I think this is a > mistake, and although I like much of the code in this patch, I was > hoping that SQLAlchemy could be hidden behind an abstraction layer > that would play nicely with the non-relational data stores. > > As this patch stands, we take a 180 degree turn away from NoSQL data > stores and back into the relatively comfortable norms of the SQL > databases. While there's nothing particularly wrong with SQL > databases (as you know, I'm a fan of many of them ;) ), I think that > keeping non-relational data store capabilities is pretty critical. > > After an email discussion with SQLAlchemy's Michael Bayer about > SQLAlchemy's future with NoSQL data stores. Although there is an > issue in the SQLAlchemy trac system about this (see here: > http://www.sqlalchemy.org/trac/ticket/1518) the likelihood of this > module seeing the light of day is unlikely in the next year or two. > > So...what to do? There are at least four options I can see: > > 1) Go forward with this patch and add NoSQL stores back at some later > time by ourselves > 2) Go forward with this patch and wait until SQLAlchemy properly > supports key value stores > 3) Delay this patch until after the Austin release and have a larger > discussion about it here and at the next summit > 4) Go back to the drawing board and try again with a less ambitious > set of patches that incrementally changes the way the data stores > work. > > I'm personally on the fence. I'd prefer to at least delay the patch > until after Austin, but I understand there are now at least 4 branches > that depend on this one, which makes things, well, a bit difficult. > > -jay > > On Tue, Aug 31, 2010 at 8:46 PM, Vishvananda Ishaya > <[email protected]> wrote: >> I've proposed a merge of the orm refactor branch that a large part of the >> nasa/anso team has been working on. I'm hoping everyone can pick it apart >> and we end up with a really clean system that everyone likes. I've copied >> the description of the change and issues below. If the mailing list debates >> get too complicated, we should just organize a time to discuss it in IRC. >> >> Proposing merge to get feedback on orm refactoring. I am very interested in >> feedback to all of these changes. >> >> This is a huge set of changes, that touches almost all of the files. I'm >> sure I have broken quite a bit, but better to take the plunge now than to >> postpone this until later. The idea is to allow for pluggable backends >> throughout the code. >> >> Brief Overview >> For compute/volume/network, there are multiple classes >> service - responsible for rpc >> this currently uses the existing cast and call in rpc.py and a little bit >> of magic >> to call public methods on the manager class. >> each service also reports its state into the database every 10 seconds >> manager - responsible for managing respective object classes >> all the business logic for the classes go here >> db (db_driver) - responsible for abstracting database access >> driver (domain_driver) - responsible for executing actual shell commands and >> implementation >> >> Compute hasn't been fully cleaned up, but to get an idea of how it works, >> take a look >> at volume and network >> >> Known issues/Things to be done: >> >> * nova-api accesses db objects directly >> It seems cleaner to have only the managers dealing with their respective >> objects. This would >> mean code for 'run_instances' would move into the manager class and it >> would do the initial >> setup and cast out to the remote service >> >> * db code uses flat methods to define its interface >> In my mind this is a little prettier as an abstract base class, but driver >> loading code >> can load a module or a class. It works, so I'm not sure it needs to be >> changed but feel >> free to debate it. >> >> * Service classes have no code in them >> Not sure if this is a problem for people, but the magic of calling the >> manager's methods is >> done in the base class. We could remove the magic from the base class and >> explicitly >> wrap methods that we want to make available via rpc if this seems nasty. >> >> * AuthManager Projects/Users/Roles are not integrated into this system. >> In order for everything to live happily in the backend, we need some type >> of adaptor for LDAP >> >> * Context is not passed properly across rabbit >> Context should probably be changed to a simple dictionary so that it can >> be >> passed properly through the queue >> >> * No authorization checks on access to objects >> We need to decide on which layer auth checks should happen. >> >> * Some of the methods in ComputeManager need to be moved into other >> layers/managers >> * Compute driver layer should be abstracted more cleanly >> * Flat networking is untested and may need to be reworked >> * Some of the api commands are not working yet >> * Nova Swift Authentication needs to be refactored(Todd is working on this) >> >> _______________________________________________ >> Mailing list: https://launchpad.net/~nova >> Post to : [email protected] >> Unsubscribe : https://launchpad.net/~nova >> More help : https://help.launchpad.net/ListHelp >> >> > > _______________________________________________ > Mailing list: https://launchpad.net/~nova > Post to : [email protected] > Unsubscribe : https://launchpad.net/~nova > More help : https://help.launchpad.net/ListHelp
signature.asc
Description: OpenPGP digital signature
_______________________________________________ Mailing list: https://launchpad.net/~nova Post to : [email protected] Unsubscribe : https://launchpad.net/~nova More help : https://help.launchpad.net/ListHelp
