Re: [openstack-dev] [nova] Distributed Database

Excerpts from Mark Doffman's message of 2016-05-03 17:05:54 -0700: > This thread has been a depressing read. > First, I apologize if any of my actions have caused you any undue stress. > I understand that the content is supposed to be distributed databases > but for me it has become an

Re: [openstack-dev] [nova] Distributed Database

On 04/05/16 19:00, "Edward Leafe" wrote: >On May 3, 2016, at 7:05 PM, Mark Doffman wrote: > >> This thread has been a depressing read. >> >> I understand that the content is supposed to be distributed databases but >> for me it has become an

Re: [openstack-dev] [nova] Distributed Database

On 5/4/2016 12:00 PM, Edward Leafe wrote: On May 3, 2016, at 7:05 PM, Mark Doffman wrote: This thread has been a depressing read. I understand that the content is supposed to be distributed databases but for me it has become an inquisition of cellsV2. Thanks

Re: [openstack-dev] [nova] Distributed Database

On May 3, 2016, at 7:05 PM, Mark Doffman wrote: > This thread has been a depressing read. > > I understand that the content is supposed to be distributed databases but for > me it has become an inquisition of cellsV2. Thanks for bringing this up, and I feel a lot

Re: [openstack-dev] [nova] Distributed Database

On Tue, May 3, 2016, at 08:05 PM, Mark Doffman wrote: > This thread has been a depressing read. > > I understand that the content is supposed to be distributed databases > but for me it has become an inquisition of cellsV2. > > Our question has clearly become "Should we continue efforts on >

Re: [openstack-dev] [nova] Distributed Database

This thread has been a depressing read. I understand that the content is supposed to be distributed databases but for me it has become an inquisition of cellsV2. Our question has clearly become "Should we continue efforts on cellsV2?", which I will address head-on. We shouldn't be afraid

Re: [openstack-dev] [nova] Distributed Database

Excerpts from Andrew Laski's message of 2016-05-03 14:46:08 -0700: > > On Mon, May 2, 2016, at 01:13 PM, Edward Leafe wrote: > > On May 2, 2016, at 10:51 AM, Mike Bayer wrote: > > > > >> Concretely, we think that there are three possible approaches: > > >> 1) We can use

Re: [openstack-dev] [nova] Distributed Database

On Mon, May 2, 2016, at 01:13 PM, Edward Leafe wrote: > On May 2, 2016, at 10:51 AM, Mike Bayer wrote: > > >> Concretely, we think that there are three possible approaches: > >> 1) We can use the SQLAlchemy API as the common denominator between a > >> relational and

Re: [openstack-dev] [nova] Distributed Database

Excerpts from Mike Bayer's message of 2016-05-03 09:04:00 -0700: > > On 05/02/2016 01:48 PM, Clint Byrum wrote: > >> > > > > FWIW, I agree with you. If you're going to use SQLAlchemy, use it to > > take advantage of the relational model. > > > > However, how is what you describe a win? Whether

Re: [openstack-dev] [nova] Distributed Database

Excerpts from Edward Leafe's message of 2016-05-03 08:20:36 -0700: > On May 3, 2016, at 6:45 AM, Miles Gould wrote: > > >> This DB could be an RDBMS or Cassandra, depending on the deployer's > >> preferences > > AFAICT this would mean introducing and maintaining a layer that

Re: [openstack-dev] [nova] Distributed Database

On 05/02/2016 01:48 PM, Clint Byrum wrote: FWIW, I agree with you. If you're going to use SQLAlchemy, use it to take advantage of the relational model. However, how is what you describe a win? Whether you use SELECT .. FOR UPDATE, or a stored procedure, the lock is not distributed, and

Re: [openstack-dev] [nova] Distributed Database

On May 3, 2016, at 6:45 AM, Miles Gould wrote: >> This DB could be an RDBMS or Cassandra, depending on the deployer's >> preferences > AFAICT this would mean introducing and maintaining a layer that abstracts > over RDBMSes and Cassandra. That's a big abstraction, over two

Re: [openstack-dev] [nova] Distributed Database

On 02/05/16 18:43, Jay Pipes wrote: This DB could be an RDBMS or Cassandra, depending on the deployer's preferences AFAICT this would mean introducing and maintaining a layer that abstracts over RDBMSes and Cassandra. That's a big abstraction, over two quite different systems, and it would be

Re: [openstack-dev] [nova] Distributed Database

Excerpts from Jay Pipes's message of 2016-05-02 10:43:21 -0700: > On 05/02/2016 11:51 AM, Mike Bayer wrote: > > On 05/02/2016 07:38 AM, Matthieu Simonin wrote: > >> As far as we understand the idea of an ORM is to hide the relational > >> database with an Object oriented API. > > > > I actually

Re: [openstack-dev] [nova] Distributed Database

Excerpts from Mike Bayer's message of 2016-05-02 08:51:58 -0700: > > Well IMO that's actually often a problem. My goal across Openstack > projects in general is to allow them to make use of SQL more effectively > than they do right now; for example, in Neutron I am helping them to > move a

Re: [openstack-dev] [nova] Distributed Database

On 05/02/2016 11:51 AM, Mike Bayer wrote: On 05/02/2016 07:38 AM, Matthieu Simonin wrote: As far as we understand the idea of an ORM is to hide the relational database with an Object oriented API. I actually disagree with that completely. The reason ORMs are so maligned is because of this

Re: [openstack-dev] [nova] Distributed Database

On May 2, 2016, at 10:51 AM, Mike Bayer wrote: >> Concretely, we think that there are three possible approaches: >> 1) We can use the SQLAlchemy API as the common denominator between a >> relational and non-relational implementation of the db.api component. These >> two

Re: [openstack-dev] [nova] Distributed Database

On 05/02/2016 07:38 AM, Matthieu Simonin wrote: As far as we understand the idea of an ORM is to hide the relational database with an Object oriented API. I actually disagree with that completely. The reason ORMs are so maligned is because of this misconception; developer attempts to

Re: [openstack-dev] [nova] Distributed Database

- Mail original - > De: "Mike Bayer" <mba...@redhat.com> > À: "OpenStack Development Mailing List (not for usage questions)" > <openstack-dev@lists.openstack.org> > Envoyé: Jeudi 28 Avril 2016 18:57:59 > Objet: Re: [openstack-dev] [nova] D

Re: [openstack-dev] [nova] Distributed Database

Excerpts from Matt Riedemann's message of 2016-04-29 05:38:17 -0700: > > So, we're all here in person this week (with 1 day left). The Nova team > has a meetup session all day (Salon A in the Hilton). Clint/Ed, can you > guys show up to that and bring these issues up in person so we can >

Re: [openstack-dev] [nova] Distributed Database

On Fri, 29 Apr 2016, Matt Riedemann wrote: So, we're all here in person this week (with 1 day left). The Nova team has a meetup session all day (Salon A in the Hilton). Clint/Ed, can you guys show up to that and bring these issues up in person so we can actually talk through this? Preferably

Re: [openstack-dev] [nova] Distributed Database

On 4/28/2016 8:54 PM, Edward Leafe wrote: On Apr 28, 2016, at 5:35 PM, Clint Byrum wrote: - Vitess [2] is a proven technology that serves _every_ request to Youtube, and provides a familiar SQL interface with sharding built in. Shard by project ID and you can just use

Re: [openstack-dev] [nova] Distributed Database

Excerpts from Mike Bayer's message of 2016-04-28 22:16:54 -0500: > > On 04/28/2016 08:25 PM, Edward Leafe wrote: > > > Your own tests showed that a single RDBMS instance doesn’t even break a > > sweat > > under your test loads. I don’t see why we need to shard it in the first > > place,

Re: [openstack-dev] [nova] Distributed Database

On 04/28/2016 08:25 PM, Edward Leafe wrote: Your own tests showed that a single RDBMS instance doesn’t even break a sweat under your test loads. I don’t see why we need to shard it in the first place, especially if in doing so we add another layer of complexity and another dependency in order

Re: [openstack-dev] [nova] Distributed Database

On Apr 28, 2016, at 5:35 PM, Clint Byrum wrote: > - Vitess [2] is a proven technology that serves _every_ request to > Youtube, and provides a familiar SQL interface with sharding built > in. Shard by project ID and you can just use regular index semantics. > Or if that's

Re: [openstack-dev] [nova] Distributed Database

On Apr 28, 2016, at 1:09 PM, Jay Pipes wrote: > nova list as an admin (or any user frankly) should be a proxy call to Project > Searchlight and elasticsearch. > > elasticsearch is a great interface for this kind of operation. We should use > it. Oh, that’s great! A

Re: [openstack-dev] [nova] Distributed Database

Excerpts from Jay Pipes's message of 2016-04-28 13:09:29 -0500: > On 04/28/2016 08:44 AM, Edward Leafe wrote: > > On Apr 24, 2016, at 3:28 PM, Robert Collins > > wrote: > >> For instance, the things I think are essential for a distributed > >> database based datastore:

Re: [openstack-dev] [nova] Distributed Database

On 04/28/2016 08:44 AM, Edward Leafe wrote: On Apr 24, 2016, at 3:28 PM, Robert Collins wrote: For instance, the things I think are essential for a distributed database based datastore: - good single-machine developer story. Must not need a physical cluster to hack

Re: [openstack-dev] [nova] Distributed Database

On 04/28/2016 08:44 AM, Edward Leafe wrote: On Apr 24, 2016, at 3:28 PM, Robert Collins wrote: For instance, the things I think are essential for a distributed database based datastore: - good single-machine developer story. Must not need a physical cluster to

Re: [openstack-dev] [nova] Distributed Database

On Apr 24, 2016, at 3:28 PM, Robert Collins wrote: > For instance, the things I think are essential for a distributed > database based datastore: > - good single-machine developer story. Must not need a physical > cluster to hack on OpenStack > - deal gracefully with

Re: [openstack-dev] [nova] Distributed Database

On 04/25/2016 03:06 AM, Chris Dent wrote: On Mon, 25 Apr 2016, Clint Byrum wrote: RPC being a different animal, we're, frankly, abusing RabbitMQ in silly ways. There are a _massive_ pile of simpler things just waiting to be tried: Yes, very much yes. I think much of the rest of the

Re: [openstack-dev] [nova] Distributed Database

On Mon, 25 Apr 2016, Clint Byrum wrote: RPC being a different animal, we're, frankly, abusing RabbitMQ in silly ways. There are a _massive_ pile of simpler things just waiting to be tried: Yes, very much yes. I think much of the rest of the discussions about improving the architecture of

Re: [openstack-dev] [nova] Distributed Database

Excerpts from Andrew Laski's message of 2016-04-22 14:32:59 -0700: > > On Fri, Apr 22, 2016, at 04:27 PM, Ed Leafe wrote: > > OK, so I know that Friday afternoons are usually the worst times to > > write a blog post and start an email discussion, and that the Friday > > immediately before a

Re: [openstack-dev] [nova] Distributed Database

On Apr 24, 2016, at 3:28 PM, Robert Collins wrote: > Heh, glad you said it! :) -- Ed Leafe __ OpenStack Development Mailing List (not for usage questions) Unsubscribe:

Re: [openstack-dev] [nova] Distributed Database

On 24 April 2016 at 12:15, Edward Leafe wrote: > On Apr 24, 2016, at 6:41 AM, Michael Bayer wrote: > >> I'm only seeking to counter what appears to be the premise of your blog >> post, which is that distributed and SQL are mutually exclusive. When you >>

Re: [openstack-dev] [nova] Distributed Database

On Apr 24, 2016, at 6:41 AM, Michael Bayer wrote: > I'm only seeking to counter what appears to be the premise of your blog post, > which is that distributed and SQL are mutually exclusive. When you say, > "why don't we just query the database?" You can make distributed

Re: [openstack-dev] [nova] Distributed Database

On Sunday, April 24, 2016, Ed Leafe wrote: > On Apr 23, 2016, at 11:33 PM, Mike Bayer > > wrote: > > > > Facebook and LinkedIn have built distributed database systems based on > MySQL at profoundly massive scales. Openstack's problem I'm going to

Re: [openstack-dev] [nova] Distributed Database

On Apr 23, 2016, at 11:33 PM, Mike Bayer wrote: > > Facebook and LinkedIn have built distributed database systems based on MySQL > at profoundly massive scales. Openstack's problem I'm going to guess isn't as > hard as that. That's always the problem with citing

Re: [openstack-dev] [nova] Distributed Database

On Apr 23, 2016, at 10:08 PM, Jay Pipes wrote: > > think it's amusing that Redis was Nova's original "database". :) > > I "fondly" remember having to work with the original Redis store interface > for complex data. Oh the fun that was had. I think my first commits to

Re: [openstack-dev] [nova] Distributed Database

On 04/22/2016 04:27 PM, Ed Leafe wrote: OK, so I know that Friday afternoons are usually the worst times to write a blog post and start an email discussion, and that the Friday immediately before a Summit is the absolute worst, but I did it anyway.

Re: [openstack-dev] [nova] Distributed Database

- Mail original - > De: "Edward Leafe" <e...@leafe.com> > À: "OpenStack Development Mailing List (not for usage questions)" > <openstack-dev@lists.openstack.org> > Envoyé: Samedi 23 Avril 2016 19:12:03 > Objet: Re: [openstack-dev] [nova] D

Re: [openstack-dev] [nova] Distributed Database

On 04/22/2016 04:40 PM, Davanum Srinivas wrote: Ed, fyi, i just got a ping about this effort: https://github.com/BeyondTheClouds/BeyondTheClouds.github.io/raw/master/DOCS/PAPERS/2015/nova-description/BTC-DISCOVERY-Overview.pdf https://github.com/BeyondTheClouds/rome

Re: [openstack-dev] [nova] Distributed Database

On Apr 23, 2016, at 10:10 AM, Thierry Carrez wrote: >> I think replacing nova's persistent storage layer with a distributed >> database would have a great effect - but I do not think it would have >> anything to do with the database itself. It would come from the act that

Re: [openstack-dev] [nova] Distributed Database

Monty Taylor wrote: On 04/22/2016 03:27 PM, Ed Leafe wrote: OK, so I know that Friday afternoons are usually the worst times to write a blog post and start an email discussion, and that the Friday immediately before a Summit is the absolute worst, but I did it anyway.

Re: [openstack-dev] [nova] Distributed Database

On 04/22/2016 05:29 PM, Dan Smith wrote: This would be a very interesting direction to explore. Focus on the pain points of the message queue and then look at addressing the beast that is the database layer separately. I am going to toss support in behind a lot of what has been said in this

Re: [openstack-dev] [nova] Distributed Database

> This would be a very interesting direction to explore. Focus on the pain > points of the message queue and then look at addressing the beast that > is the database layer separately. I am going to toss support in behind a > lot of what has been said in this thread already. But I really wanted to

Re: [openstack-dev] [nova] Distributed Database

On Fri, Apr 22, 2016 at 2:57 PM, Dan Smith wrote: > > However I do want to point out that cells v2 is not just about dealing > > with scale in the database. The message queue is another consideration, > > and as far as I know there is not an analog to the "distributed > >

Re: [openstack-dev] [nova] Distributed Database

> However I do want to point out that cells v2 is not just about dealing > with scale in the database. The message queue is another consideration, > and as far as I know there is not an analog to the "distributed > database" option available for the persistence layer. Yeah, it's actually *mostly*

Re: [openstack-dev] [nova] Distributed Database

On 04/22/2016 04:32 PM, Andrew Laski wrote: On Fri, Apr 22, 2016, at 04:27 PM, Ed Leafe wrote: OK, so I know that Friday afternoons are usually the worst times to write a blog post and start an email discussion, and that the Friday immediately before a Summit is the absolute worst, but I did

Re: [openstack-dev] [nova] Distributed Database

On Fri, Apr 22, 2016, at 04:27 PM, Ed Leafe wrote: > OK, so I know that Friday afternoons are usually the worst times to > write a blog post and start an email discussion, and that the Friday > immediately before a Summit is the absolute worst, but I did it anyway. > >

Re: [openstack-dev] [nova] Distributed Database

On 04/22/2016 03:27 PM, Ed Leafe wrote: OK, so I know that Friday afternoons are usually the worst times to write a blog post and start an email discussion, and that the Friday immediately before a Summit is the absolute worst, but I did it anyway.

Re: [openstack-dev] [nova] Distributed Database

Ed, fyi, i just got a ping about this effort: https://github.com/BeyondTheClouds/BeyondTheClouds.github.io/raw/master/DOCS/PAPERS/2015/nova-description/BTC-DISCOVERY-Overview.pdf https://github.com/BeyondTheClouds/rome https://beyondtheclouds.github.io/ They have a WG session:

[openstack-dev] [nova] Distributed Database

OK, so I know that Friday afternoons are usually the worst times to write a blog post and start an email discussion, and that the Friday immediately before a Summit is the absolute worst, but I did it anyway. http://blog.leafe.com/index.php/2016/04/22/distributed_data_nova/ Summary: we are