Re: [openstack-dev] [oslo.db]A proposal for DB read/write separation

2014-09-04 Thread Jay Pipes
On 09/02/2014 07:15 AM, Duncan Thomas wrote: On 11 August 2014 19:26, Jay Pipes jaypi...@gmail.com wrote: The above does not really make sense for MySQL Galera/PXC clusters *if only Galera nodes are used in the cluster*. Since Galera is synchronously replicated, there's no real point in

Re: [openstack-dev] [oslo.db]A proposal for DB read/write separation

2014-09-02 Thread Duncan Thomas
On 11 August 2014 19:26, Jay Pipes jaypi...@gmail.com wrote: The above does not really make sense for MySQL Galera/PXC clusters *if only Galera nodes are used in the cluster*. Since Galera is synchronously replicated, there's no real point in segregating writers from readers, IMO. Better to

Re: [openstack-dev] [oslo.db]A proposal for DB read/write separation

2014-08-13 Thread Mike Wilson
: [openstack-dev] [oslo.db]A proposal for DB read/write separation On Aug 10, 2014, at 11:17 AM, Li Ma skywalker.n...@gmail.com wrote: How about Galera multi-master cluster? As Mike Bayer said, it is virtually synchronous by default. It is still possible that outdated rows are queried that make

Re: [openstack-dev] [oslo.db]A proposal for DB read/write separation

2014-08-11 Thread Jay Pipes
Hi Li, comments inline. On 08/08/2014 12:03 AM, Li Ma wrote: Getting a massive amount of information from data storage to be displayed is where most of the activity happens in OpenStack. The two activities of reading data and writing (creating, updating and deleting) data are fundamentally

Re: [openstack-dev] [oslo.db]A proposal for DB read/write separation

2014-08-10 Thread Amrith Kumar
[mailto:geekinu...@gmail.com] Sent: Friday, August 08, 2014 7:35 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [oslo.db]A proposal for DB read/write separation Li Ma, This is interesting, In general I am in favor of expanding the scope of any read

Re: [openstack-dev] [oslo.db]A proposal for DB read/write separation

2014-08-10 Thread Mike Bayer
=marks From: Mike Wilson [mailto:geekinu...@gmail.com] Sent: Friday, August 08, 2014 7:35 PM To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [oslo.db]A proposal for DB read/write separation Li Ma, This is interesting, In general

Re: [openstack-dev] [oslo.db]A proposal for DB read/write separation

2014-08-10 Thread Li Ma
Thanks for all the detailed analysis, Mike W, Mike B, and Roman. For a production-ready database system, replication is a must I think. So, the questions are which replication mode is suitable for OpenStack and which way is suitable for OpenStack to improve performance and scalability of DB

Re: [openstack-dev] [oslo.db]A proposal for DB read/write separation

2014-08-10 Thread Mike Bayer
On Aug 10, 2014, at 11:17 AM, Li Ma skywalker.n...@gmail.com wrote: How about Galera multi-master cluster? As Mike Bayer said, it is virtually synchronous by default. It is still possible that outdated rows are queried that make results not stable. not sure if I said that :). I know

Re: [openstack-dev] [oslo.db]A proposal for DB read/write separation

2014-08-10 Thread Li Ma
mba...@redhat.com To: OpenStack Development Mailing List (not for usage questions) openstack-dev@lists.openstack.org Sent: 星期日, 2014年 8 月 10日 下午 11:57:47 Subject: Re: [openstack-dev] [oslo.db]A proposal for DB read/write separation On Aug 10, 2014, at 11:17 AM, Li Ma skywalker.n...@gmail.com wrote

Re: [openstack-dev] [oslo.db]A proposal for DB read/write separation

2014-08-08 Thread Roman Podoliaka
Hi Li, How are you going to make this separation transparent? I mean, generally, in a function code, you can't know in advance if the transaction will be read-only or it will contain an INSERT/UPDATE/DELETE statement. On the other hand, as a developer, you could analyze the DB queries that can be

Re: [openstack-dev] [oslo.db]A proposal for DB read/write separation

2014-08-08 Thread Mike Bayer
On Aug 8, 2014, at 12:03 AM, Li Ma skywalker.n...@gmail.com wrote: So, I'd like to propose a transparent read/write separation method for oslo.db that every project may happily takes advantage of it without any code modification. A single transaction begins, which is to emit a series of

Re: [openstack-dev] [oslo.db]A proposal for DB read/write separation

2014-08-08 Thread Mike Wilson
Li Ma, This is interesting, In general I am in favor of expanding the scope of any read/write separation capabilities that we have. I'm not clear what exactly you are proposing, hopefully you can answer some of my questions inline. The thing I had thought of immediately was detection of whether

[openstack-dev] [oslo.db]A proposal for DB read/write separation

2014-08-07 Thread Li Ma
Getting a massive amount of information from data storage to be displayed is where most of the activity happens in OpenStack. The two activities of reading data and writing (creating, updating and deleting) data are fundamentally different. The optimization for these two opposite database