[openstack-dev] [keystone] [tripleo] multi-region Keystone via stretched Galera cluster

2018-06-04 Thread Michael Bayer
Hey list - as mentioned in the May 11 Keystone meeting, I am working on one of the canonical approaches to producing a multi-region Keystone service, which is by having overcloud-specific keystone services interact with a Galera database that is running masters across multiple overclouds. The

Re: [openstack-dev] [oslo.db] innodb OPTIMIZE TABLE ?

2018-04-09 Thread Michael Bayer
On Mon, Apr 9, 2018 at 5:53 AM, Gorka Eguileor <gegui...@redhat.com> wrote: > On 06/04, Michael Bayer wrote: >> On Wed, Apr 4, 2018 at 5:00 AM, Gorka Eguileor <gegui...@redhat.com> wrote: >> > On 03/04, Jay Pipes wrote: >> >> On 04/03/2018 11:07 AM, Michae

Re: [openstack-dev] [oslo.db] innodb OPTIMIZE TABLE ?

2018-04-06 Thread Michael Bayer
On Wed, Apr 4, 2018 at 5:00 AM, Gorka Eguileor <gegui...@redhat.com> wrote: > On 03/04, Jay Pipes wrote: >> On 04/03/2018 11:07 AM, Michael Bayer wrote: >> > The MySQL / MariaDB variants we use nowadays default to >> > innodb_file_per_table=ON and we also

Re: [openstack-dev] [oslo.db] innodb OPTIMIZE TABLE ?

2018-04-03 Thread Michael Bayer
On Tue, Apr 3, 2018 at 11:41 AM, Jay Pipes <jaypi...@gmail.com> wrote: > On 04/03/2018 11:07 AM, Michael Bayer wrote: >> > > Yes. > >> b. oslo.db script to run generically, yes or no? > > > No. Just have Triple-O install galera_innoptimizer and run it in a cr

[openstack-dev] [oslo.db] innodb OPTIMIZE TABLE ?

2018-04-03 Thread Michael Bayer
The MySQL / MariaDB variants we use nowadays default to innodb_file_per_table=ON and we also set this flag to ON in installer tools like TripleO. The reason we like file per table is so that we don't grow an enormous ibdata file that can't be shrunk without rebuilding the database. Instead,

Re: [openstack-dev] [oslo.db] upcoming warnings in MySQL 5.6, 5.7 for BLOB columns

2018-03-14 Thread Michael Bayer
Forgot the links: [1] https://bugs.mysql.com/bug.php?id=79317 [2] https://github.com/PyMySQL/PyMySQL/issues/644 On Wed, Mar 14, 2018 at 11:53 AM, Michael Bayer <mba...@redhat.com> wrote: > hey all - > > Just looking to see if we think this will impact openstack. MySQL 5.6 > an

[openstack-dev] [oslo.db] upcoming warnings in MySQL 5.6, 5.7 for BLOB columns

2018-03-14 Thread Michael Bayer
hey all - Just looking to see if we think this will impact openstack. MySQL 5.6 and 5.7, but not yet MariaDB, now emits an erroneous warning when you try to send a binary value to the database, because it sees the client connection is supposed to use the utf8 or utf8mb4 charsets, assumes all

Re: [openstack-dev] [oslo.db] [all] please DO NOT IMPORT from oslo_db.tests.* ! projects doing this need to revert ASAP

2018-02-19 Thread Michael Bayer
Summarizing all the reviews: Doug's proposed check in oslo_db.tests: https://review.openstack.org/#/c/545859/ Mark oslo_db internal fixtures private: https://review.openstack.org/545862 Cinder: https://review.openstack.org/545860 Neutron: https://review.openstack.org/545868 Ironic:

Re: [openstack-dev] [oslo.db] [all] please DO NOT IMPORT from oslo_db.tests.* ! projects doing this need to revert ASAP

2018-02-19 Thread Michael Bayer
exclude` option which should solve the issue as > well. > > > > 2018-02-19 17:41 GMT+02:00 Michael Bayer <mba...@redhat.com>: >> >> On Mon, Feb 19, 2018 at 10:39 AM, Andrey Kurilin <andr.kuri...@gmail.com> >> wrote: >> > Can someone explain me th

Re: [openstack-dev] [oslo.db] [all] please DO NOT IMPORT from oslo_db.tests.* ! projects doing this need to revert ASAP

2018-02-19 Thread Michael Bayer
wow that's heavy-handed. should that be in an oslo utility package of some kind ? On Mon, Feb 19, 2018 at 10:52 AM, Doug Hellmann wrote: > Excerpts from Doug Hellmann's message of 2018-02-19 10:15:34 -0500: >> Excerpts from Michael Bayer's message of 2018-02-19 10:00:59

Re: [openstack-dev] [oslo.db] [all] please DO NOT IMPORT from oslo_db.tests.* ! projects doing this need to revert ASAP

2018-02-19 Thread Michael Bayer
te against packaged dependencies, which fails because we don't package oslo_db.tests. > > > 2018-02-19 17:00 GMT+02:00 Michael Bayer <mba...@redhat.com>: >> >> Hi list - >> >> Apparently Cinder was misled by my deprecations within the >> oslo

Re: [openstack-dev] [oslo.db] [all] please DO NOT IMPORT from oslo_db.tests.* ! projects doing this need to revert ASAP

2018-02-19 Thread Michael Bayer
On Mon, Feb 19, 2018 at 10:15 AM, Doug Hellmann wrote: > Excerpts from Michael Bayer's message of 2018-02-19 10:00:59 -0500: >> Hi list - >> >> Apparently Cinder was misled by my deprecations within the >> oslo_db.sqlalchemy.test_base package of DbFixture and DbTestCase,

[openstack-dev] [oslo.db] [all] please DO NOT IMPORT from oslo_db.tests.* ! projects doing this need to revert ASAP

2018-02-19 Thread Michael Bayer
Hi list - Apparently Cinder was misled by my deprecations within the oslo_db.sqlalchemy.test_base package of DbFixture and DbTestCase, and in https://review.openstack.org/#/c/522290/ the assumption was made that these should be imported from oslo_db.tests.sqlalchemy.This is an immense mistake

Re: [openstack-dev] [neutron][l2gw] Preventing DB out-of-sync

2017-12-08 Thread Michael Bayer
On Wed, Dec 6, 2017 at 3:46 AM, Peng Liu wrote: > Hi, > > During working on this patch[0], I encounter some DB out-of-sync problem. I > think maybe the design can be improved. Here is my thought, all comments are > welcome. see also https://review.openstack.org/#/c/490834/

Re: [openstack-dev] [neutron][oslo.db] db retry madness

2017-09-08 Thread Michael Bayer
. It's copying things passed into the API > request so they are usually dictionaries with fewer than 10 keys and values > with small strings or lists of only a few entries, making the copy > operations cheap. > > If we did want to get rid of the copy operation, it would probably be >

[openstack-dev] [neutron][oslo.db] db retry madness

2017-09-07 Thread Michael Bayer
I'm trying to get a handle on neutron's retry decorators.I'm assisting a developer in the networking_odl project who wishes to make use of neutron's retry_if_session_inactive decorator. The developer has run into several problems, including that networking_odl methods here seem to accept a

Re: [openstack-dev] [oslo][oslo.db][keystone] A POC of Keystone over CockroachDB

2017-09-06 Thread Michael Bayer
On Mon, Sep 4, 2017 at 12:06 PM, Ronan-Alexandre Cherrueau wrote: > Hi folks, > > Recently in the Inria's Discovery initiative[1], we got in touch with > CockroachLabs guys with an idea: make Keystone supports CockorachDB. So > we give it a try and you can find

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

2017-07-27 Thread Michael Bayer
proposed: https://review.openstack.org/#/c/487902/ On Thu, Jul 27, 2017 at 9:46 AM, Michael Bayer <mba...@redhat.com> wrote: > On Wed, Jul 26, 2017 at 8:06 PM, Jay Pipes <jaypi...@gmail.com> wrote: >> Isn't that exactly what I'm proposing below? :) > >

Re: [openstack-dev] [oslo][oslo.db] nominating Jay Pipes for oslo-db-core

2017-07-27 Thread Michael Bayer
Yeah if Jay has the time that's a big +1 from me! On Thu, Jul 27, 2017 at 10:04 AM, Doug Hellmann wrote: > I have noticed that Jay has been very deeply involved in several > recent design discussions about oslo.db, and he obviously has a > great deal of experience in the

Re: [openstack-dev] [all] [oslo.db] [relational database users] heads up for a MariaDB issue that will affect most projects

2017-07-27 Thread Michael Bayer
aybe* mariadb's CHECK constraint here already has a predictable name. I'd have to get a 10.2 instance running and see what it comes up with. > > 2017-07-24 23:21 GMT+08:00 Michael Bayer <mba...@redhat.com>: >> >> hey good news, the owner of the issue upstream found that th

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

2017-07-27 Thread Michael Bayer
On Wed, Jul 26, 2017 at 8:06 PM, Jay Pipes wrote: > Isn't that exactly what I'm proposing below? :) yes, I'm agreeing with you! __ OpenStack Development Mailing List (not for usage questions)

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

2017-07-26 Thread Michael Bayer
On Jul 26, 2017 7:45 PM, "Jay Pipes" <jaypi...@gmail.com> wrote: On 07/26/2017 07:06 PM, Octave J. Orgeron wrote: > Hi Michael, > > On 7/26/2017 4:28 PM, Michael Bayer wrote: > >> >> it at all. >> thinking out loud >> >> o

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

2017-07-26 Thread Michael Bayer
On Jul 26, 2017 6:28 PM, "Michael Bayer" <mba...@redhat.com> wrote: On Wed, Jul 26, 2017 at 6:19 PM, Michael Bayer <mba...@redhat.com> wrote: > On Wed, Jul 26, 2017 at 5:30 PM, Michael Bayer <mba...@redhat.com> wrote: >> >> There is a bigger problem

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

2017-07-26 Thread Michael Bayer
On Wed, Jul 26, 2017 at 6:19 PM, Michael Bayer <mba...@redhat.com> wrote: > On Wed, Jul 26, 2017 at 5:30 PM, Michael Bayer <mba...@redhat.com> wrote: >> >> There is a bigger problem with this entire series of changes, whether >> or not the "ndb" keywo

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

2017-07-26 Thread Michael Bayer
On Wed, Jul 26, 2017 at 5:30 PM, Michael Bayer <mba...@redhat.com> wrote: > > There is a bigger problem with this entire series of changes, whether > or not the "ndb" keyword is present. Which is that projects need to > add new columns, tables, and make d

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

2017-07-26 Thread Michael Bayer
hecks need to be put in place in oslo.db that would sum the total row size of any given table and raise an error if the limit is surpassed. On Wed, Jul 26, 2017 at 5:40 PM, Octave J. Orgeron <octave.orge...@oracle.com> wrote: > Hi Michael, > > Comments below.. > >

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

2017-07-26 Thread Michael Bayer
flags, which is a more lengthy process than the rest of > this. > > I don't see the point in trying to make this stuff anymore complicated. > > Octave > > > On 7/25/2017 12:20 PM, Michael Bayer wrote: >> >> On Mon, Jul 24, 2017 at 5:41 PM, Michael Bayer

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

2017-07-26 Thread Michael Bayer
covered. Octave On 7/25/2017 12:20 PM, Michael Bayer wrote: > On Mon, Jul 24, 2017 at 5:41 PM, Michael Bayer <mba...@redhat.com> wrote: > >> oslo_db.sqlalchemy.String(255, ndb_type=TINYTEXT) -> VARCHAR(255) for most >>> dbs, TINYTEXT for ndb >>> oslo_

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

2017-07-25 Thread Michael Bayer
On Mon, Jul 24, 2017 at 5:41 PM, Michael Bayer <mba...@redhat.com> wrote: >> oslo_db.sqlalchemy.String(255, ndb_type=TINYTEXT) -> VARCHAR(255) for most >> dbs, TINYTEXT for ndb >> oslo_db.sqlalchemy.String(4096, ndb_type=TEXT) -> VARCHAR(4096)

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

2017-07-24 Thread Michael Bayer
On Mon, Jul 24, 2017 at 5:10 PM, Octave J. Orgeron wrote: > I don't think it makes sense to make these global. We don't need to change > all occurrences of String(255) to TinyText for example. We make that > determination through understanding the table structure and

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

2017-07-24 Thread Michael Bayer
On Mon, Jul 24, 2017 at 3:37 PM, Octave J. Orgeron wrote: > For these, here is a brief synopsis: > > AutoStringTinyText, will convert a column to the TinyText type. This is used > for cases where a 255 varchar string needs to be converted to a text blob to > make the

Re: [openstack-dev] [all] [oslo.db] [relational database users] heads up for a MariaDB issue that will affect most projects

2017-07-24 Thread Michael Bayer
to duck it. On Mon, Jul 24, 2017 at 11:16 AM, Michael Bayer <mba...@redhat.com> wrote: > On Mon, Jul 24, 2017 at 10:37 AM, Doug Hellmann <d...@doughellmann.com> wrote: >> Excerpts from Michael Bayer's message of 2017-07-23 16:39:20 -0400: >>> Hey list - >>> &g

Re: [openstack-dev] [all] [oslo.db] [relational database users] heads up for a MariaDB issue that will affect most projects

2017-07-24 Thread Michael Bayer
On Mon, Jul 24, 2017 at 10:37 AM, Doug Hellmann wrote: > Excerpts from Michael Bayer's message of 2017-07-23 16:39:20 -0400: >> Hey list - >> >> It appears that MariaDB as of version 10.2 has made an enhancement >> that overall is great and fairly historic in the MySQL

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

2017-07-24 Thread Michael Bayer
On Mon, Jul 24, 2017 at 10:01 AM, Jay Pipes wrote: > I would much prefer to *add* a brand new schema migration that handles > conversion of the entire InnoDB schema at a certain point to an > NDB-compatible one *after* that point. That way, we isolate the NDB changes > to one

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

2017-07-23 Thread Michael Bayer
On Sun, Jul 23, 2017 at 6:10 PM, Jay Pipes <jaypi...@gmail.com> wrote: > Glad you brought this up, Mike. I was going to start a thread about this. > Comments inline. > > On 07/23/2017 05:02 PM, Michael Bayer wrote: > Well, besides that point (which I agree with), that is

[openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

2017-07-23 Thread Michael Bayer
I've been working with Octave Oregon in assisting with new rules and datatypes that would allow projects to support the NDB storage engine with MySQL. To that end, we've made changes to oslo.db in [1] to support this, and there are now a bunch of proposals such as [2] [3] to implement new

[openstack-dev] [all] [oslo.db] [relational database users] heads up for a MariaDB issue that will affect most projects

2017-07-23 Thread Michael Bayer
Hey list - It appears that MariaDB as of version 10.2 has made an enhancement that overall is great and fairly historic in the MySQL community, they've made CHECK constraints finally work. For all of MySQL's existence, you could emit a CREATE TABLE statement that included CHECK constraint, but

Re: [openstack-dev] [oslo.config] how to deprecate a name but still have it as conf.

2017-07-18 Thread Michael Bayer
On Tue, Jul 18, 2017 at 1:02 PM, Doug Hellmann wrote: > Option renaming was originally meant as an operatior-facing feature > to handle renames for values coming from the config file, but not > as they are used in code. mtreinish added >

[openstack-dev] [oslo.config] how to deprecate a name but still have it as conf.

2017-07-18 Thread Michael Bayer
In oslo.db, I'd like to rename the option "idle_timeout" to "connection_recycle_time". Following the pattern of using DeprecatedOpt, we get this: cfg.IntOpt('connection_recycle_time', default=3600, deprecated_opts=[cfg.DeprecatedOpt('idle_timeout',

Re: [openstack-dev] [keystone][nova][neutron][all] Rolling upgrades: database triggers and oslo.versionedobjects

2016-09-01 Thread Michael Bayer
On Thursday, September 1, 2016, Jeremy Stanley wrote: > > I don't read that at all as suggesting "the problem is solved, go > away" but rather "help us make it better for everyone, don't just > take one project off in a new direction and leave the others > behind." I can

Re: [openstack-dev] [nova] Distributed Database

2016-04-24 Thread Michael Bayer
On Sunday, April 24, 2016, Ed Leafe wrote: > On Apr 23, 2016, at 11:33 PM, Mike Bayer > > wrote: > > > > Facebook and LinkedIn have built distributed database systems based on > MySQL at profoundly massive scales. Openstack's problem I'm going to

Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-22 Thread Michael Bayer
On Feb 22, 2015, at 10:20 AM, Yuriy Taraday yorik@gmail.com wrote: On Sun Feb 22 2015 at 6:27:16 AM Michael Bayer mba...@redhat.com wrote: On Feb 21, 2015, at 9:49 PM, Joshua Harlow harlo...@outlook.com wrote: Some comments/questions inline... Mike Bayer wrote

Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-21 Thread Michael Bayer
On Feb 21, 2015, at 9:49 PM, Joshua Harlow harlo...@outlook.com wrote: Some comments/questions inline... Mike Bayer wrote: Yuriy Taradayyorik@gmail.com wrote: On Fri Feb 20 2015 at 9:14:30 PM Joshua Harlowharlo...@outlook.com wrote: This feels like something we could do in