On 09/04/2014 03:08 AM, Flavio Percoco wrote: > Greetings, > > Last Tuesday the TC held the first graduation review for Zaqar. During > the meeting some concerns arose. I've listed those concerns below with > some comments hoping that it will help starting a discussion before the > next meeting. In addition, I've added some comments about the project > stability at the bottom and an etherpad link pointing to a list of use > cases for Zaqar. > > # Concerns > > - Concern on operational burden of requiring NoSQL deploy expertise to > the mix of openstack operational skills > > For those of you not familiar with Zaqar, it currently supports 2 nosql > drivers - MongoDB and Redis - and those are the only 2 drivers it > supports for now. This will require operators willing to use Zaqar to > maintain a new (?) NoSQL technology in their system. Before expressing > our thoughts on this matter, let me say that: > > 1. By removing the SQLAlchemy driver, we basically removed the chance > for operators to use an already deployed "OpenStack-technology" > 2. Zaqar won't be backed by any AMQP based messaging technology for > now. Here's[0] a summary of the research the team (mostly done by > Victoria) did during Juno > 3. We (OpenStack) used to require Redis for the zmq matchmaker > 4. We (OpenStack) also use memcached for caching and as the oslo > caching lib becomes available - or a wrapper on top of dogpile.cache - > Redis may be used in place of memcached in more and more deployments. > 5. Ceilometer's recommended storage driver is still MongoDB, although > Ceilometer has now support for sqlalchemy. (Please correct me if I'm wrong). > > That being said, it's obvious we already, to some extent, promote some > NoSQL technologies. However, for the sake of the discussion, lets assume > we don't. > > I truly believe, with my OpenStack (not Zaqar's) hat on, that we can't > keep avoiding these technologies. NoSQL technologies have been around > for years and we should be prepared - including OpenStack operators - to > support these technologies. Not every tool is good for all tasks - one > of the reasons we removed the sqlalchemy driver in the first place - > therefore it's impossible to keep an homogeneous environment for all > services. > > With this, I'm not suggesting to ignore the risks and the extra burden > this adds but, instead of attempting to avoid it completely by not > evolving the stack of services we provide, we should probably work on > defining a reasonable subset of NoSQL services we are OK with > supporting. This will help making the burden smaller and it'll give > operators the option to choose. > > [0] http://blog.flaper87.com/post/marconi-amqp-see-you-later/
I've been one of the consistent voices concerned about a hard requirement on adding NoSQL into the mix. So I'll explain that thinking a bit more. I feel like when the TC makes an integration decision previously this has been about evaluating the project applying for integration, and if they met some specific criteria they were told about some time in the past. I think that's the wrong approach. It's a locally optimized approach that fails to ask the more interesting question. Is OpenStack better as a whole if this is a mandatory component of OpenStack? Better being defined as technically better (more features, less janky code work arounds, less unexpected behavior from the stack). Better from the sense of easier or harder to run an actual cloud by our Operators (taking into account what kinds of moving parts they are now expected to manage). Better from the sense of a better user experience in interacting with OpenStack as whole. Better from a sense that the OpenStack release will experience less bugs, less unexpected cross project interactions, an a greater overall feel of consistency so that the OpenStack API feels like one thing. https://dague.net/2014/08/26/openstack-as-layers/ One of the interesting qualities of Layers 1 & 2 is they all follow an AMQP + RDBMS pattern (excepting swift). You can have a very effective IaaS out of that stack. They are the things that you can provide pretty solid integration testing on (and if you look at where everything stood before the new TC mandates on testing / upgrade that was basically what was getting integration tested). (Also note, I'll accept Barbican is probably in the wrong layer, and should be a Layer 2 service.) While large shops can afford to have a dedicated team to figure out how to make mongo or redis HA, provide monitoring, have a DR plan for when a huricane requires them to flip datacenters, that basically means OpenStack heads further down the path of "only for the big folks". I don't want OpenStack to be only for the big folks, I want OpenStack to be for all sized folks. I really do want to have all the local small colleges around here have OpenStack clouds, because it's something that people believe they can do and manage. I know the people that work in this places, they all come out to the LUG I run. We've talked about this. OpenStack is basically seen as too complex for them to use as it stands, and that pains me a ton. So I think Zaqar is good software, and really useful part of our ecosystem, but this added step function burden of a 3rd class of support software that has to be maintained... seems like it takes us further away from OpenStack at a small scale. If we were thinking about Zaqar as a thing that we could replace olso.messaging with, that becomes interesting in a different way, because we could instead of having 3 classes of support software, remain at 2, just take a sideways shift on one of them. But that's not actually the path we are on. So, honestly, I'll probably remain -1 on the final integration vote, not because Zaqar is bad, but because I'm feeling more firmly that for OpenStack to not leave the small deployers behind we need to redefine the tightly integrated piece of OpenStack to basically the Layer 1 & 2 parts of my diagram, and consider the rest of the layers exciting parts of our ecosystem that more advanced users may choose to deploy to meet their needs. Smaller tent, big ecosystem, easier on ramp. I realize that largely means Zaqar would be caught up in a definition discussion outside of it's control, and that's kind of unfortunate, as Flavio and team have been doing a bang up job of late. But we need to stop considering "integration" as the end game of all interesting software in the OpenStack ecosystem, and I think it's better to have that conversation sooner rather than later. Anyway, my $0.02. -Sean -- Sean Dague http://dague.net _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
