Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

Hi Jay, Comments below.. I apologize for being a bit verbose, but I know some folks on the list are not familiar with NDB, so I do go into some details below. Thanks, Octave On 7/27/2017 11:49 AM, Jay Pipes wrote: I guess we're really getting into the weeds here. On 07/27/2017 12:51 PM,

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

I guess we're really getting into the weeds here. On 07/27/2017 12:51 PM, Octave J. Orgeron wrote: Hi Jay, Comments below.. On 7/26/2017 5:43 PM, Jay Pipes wrote: On 07/26/2017 07:06 PM, Octave J. Orgeron wrote: Hi Michael, On 7/26/2017 4:28 PM, Michael Bayer wrote: it at all. thinking

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

Hi Jay, Comments below.. On 7/26/2017 5:43 PM, Jay Pipes wrote: On 07/26/2017 07:06 PM, Octave J. Orgeron wrote: Hi Michael, On 7/26/2017 4:28 PM, Michael Bayer wrote: it at all. thinking out loud oslo_db.sqlalchemy.types.String(255, mysql_small_rowsize=64)

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

Hi Jay, There are number of other projects that are not using oslo.db for their migrations so that all of the flags and options are passed along. Good examples are things like Barbican, Murano, Glance, etc. Thanks, Octave On 7/26/2017 5:26 PM, Jay Pipes wrote: On 07/26/2017 07:01 PM,

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

proposed: https://review.openstack.org/#/c/487902/ On Thu, Jul 27, 2017 at 9:46 AM, Michael Bayer wrote: > On Wed, Jul 26, 2017 at 8:06 PM, Jay Pipes wrote: >> Isn't that exactly what I'm proposing below? :) > > yes, I'm agreeing with you!

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

On Wed, Jul 26, 2017 at 8:06 PM, Jay Pipes wrote: > Isn't that exactly what I'm proposing below? :) yes, I'm agreeing with you! __ OpenStack Development Mailing List (not for usage questions)

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

On 07/26/2017 07:58 PM, Michael Bayer wrote: On Jul 26, 2017 7:45 PM, "Jay Pipes" > wrote: On 07/26/2017 07:06 PM, Octave J. Orgeron wrote: Hi Michael, On 7/26/2017 4:28 PM, Michael Bayer wrote: it at all.

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

On Jul 26, 2017 7:45 PM, "Jay Pipes" wrote: On 07/26/2017 07:06 PM, Octave J. Orgeron wrote: > Hi Michael, > > On 7/26/2017 4:28 PM, Michael Bayer wrote: > >> >> it at all. >> thinking out loud >> >> oslo_db.sqlalchemy.types.String(255, mysql_small_rowsize=64) >>

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

On 07/26/2017 07:06 PM, Octave J. Orgeron wrote: Hi Michael, On 7/26/2017 4:28 PM, Michael Bayer wrote: it at all. thinking out loud oslo_db.sqlalchemy.types.String(255, mysql_small_rowsize=64) oslo_db.sqlalchemy.types.String(255, mysql_small_rowsize=sa.TINYTEXT)

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

On 07/26/2017 07:09 PM, Octave J. Orgeron wrote: Hi Michael, That is something we are working towards having our own CI where we could catch these things. I do think that having a function in the utils module of oslo.db to test the length of a table row would be handy though for projects to

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

[snip] > > I do think a real migration that simply reduces the sizes of selected > columns is the best approach in this case, and that the types like > AutoStringXYZ should go away completely. > > To that end I've proposed reverting the one ndb patchset that has > merged which is the one in

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

On 07/26/2017 07:01 PM, Octave J. Orgeron wrote: Either way though, we'll have to ... still have to deal with any migrations that don't make proper use of oslo.db. Could you elaborate on the above? What about the Nova migrations aren't making proper use of oslo.db? Could you provide an example

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

Hi Michael, That is something we are working towards having our own CI where we could catch these things. I do think that having a function in the utils module of oslo.db to test the length of a table row would be handy though for projects to leverage as part of their unit tests. Octave On

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

Hi Michael, On 7/26/2017 4:28 PM, Michael Bayer wrote: it at all. thinking out loud oslo_db.sqlalchemy.types.String(255, mysql_small_rowsize=64) oslo_db.sqlalchemy.types.String(255, mysql_small_rowsize=sa.TINYTEXT) oslo_db.sqlalchemy.types.String(255, mysql_small_rowsize=sa.TEXT) so if you

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

Hi Michael, Yeah, the data types are the same database wise, it's just the total row size that different between InnoDB and NDB when it comes to table structure. So it's more of a decision point of: A. Change the column size or type across the database types. B. Have a mechanism to

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

On Jul 26, 2017 6:28 PM, "Michael Bayer" wrote: On Wed, Jul 26, 2017 at 6:19 PM, Michael Bayer wrote: > On Wed, Jul 26, 2017 at 5:30 PM, Michael Bayer wrote: >> >> There is a bigger problem with this entire series of changes, whether >>

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

On Wed, Jul 26, 2017 at 6:19 PM, Michael Bayer wrote: > On Wed, Jul 26, 2017 at 5:30 PM, Michael Bayer wrote: >> >> There is a bigger problem with this entire series of changes, whether >> or not the "ndb" keyword is present. Which is that projects need to

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

On Wed, Jul 26, 2017 at 5:30 PM, Michael Bayer wrote: > > There is a bigger problem with this entire series of changes, whether > or not the "ndb" keyword is present. Which is that projects need to > add new columns, tables, and make datatype changes all the time, and > they

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

I realize now that we are in fact going for a total "row size", when I was under the impression that ndb had a simple limit of 64 characters for a VARCHAR. As I was going on the completely wrong assumptions, I'd like to rethink the approach of datatypes. I do think a real migration that simply

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

Hi Michael, Comments below.. On 7/26/2017 1:08 PM, Michael Bayer wrote: On Jul 25, 2017 3:38 PM, "Octave J. Orgeron" > wrote: Hi Michael, I understand that you want to abstract this completely away inside of

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

On Tue, Jul 25, 2017 at 3:27 PM, Octave J. Orgeron wrote: > 5. Evaluated against the other columns in the table, which one makes the > most sense to adjust? well, the above point is one I've been trying to get a straight answer on for a long time. "evaluated against

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

Still working through a full response but wanted to point something important out about something you say below On 07/25/2017 03:27 PM, Octave J. Orgeron wrote: If we really want to remove the complexity here, why don't we just change the sizes and types on these handful of table columns

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

On Jul 25, 2017 3:38 PM, "Octave J. Orgeron" wrote: Hi Michael, I understand that you want to abstract this completely away inside of oslo.db. However, the reality is that making column changes based purely on the size and type of that column, without understanding

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

Hi Michael, I understand that you want to abstract this completely away inside of oslo.db. However, the reality is that making column changes based purely on the size and type of that column, without understanding what that column is being used for is extremely dangerous. You could end up

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

On Mon, Jul 24, 2017 at 5:41 PM, Michael Bayer wrote: >> oslo_db.sqlalchemy.String(255, ndb_type=TINYTEXT) -> VARCHAR(255) for most >> dbs, TINYTEXT for ndb >> oslo_db.sqlalchemy.String(4096, ndb_type=TEXT) -> VARCHAR(4096) for most >> dbs, TEXT for ndb >>

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

Hi Graham, Comments below.. On 7/25/2017 5:04 AM, Graham Hayes wrote: On 24/07/17 20:37, Octave J. Orgeron wrote: Rather than having all the projects make use of oslo_db.sqlalchemy.ndb.AutoStringTinyText / AutoStringSize, we add new generic types to oslo.db :

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

On 24/07/17 20:37, Octave J. Orgeron wrote: >> >> Rather than having all the projects make use of >> oslo_db.sqlalchemy.ndb.AutoStringTinyText / AutoStringSize, we add new >> generic types to oslo.db : >> >> oslo_db.sqlalchemy.types.SmallString >> oslo_db.sqlalchemy.types.String >> >> (or

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

On Mon, Jul 24, 2017 at 5:10 PM, Octave J. Orgeron wrote: > I don't think it makes sense to make these global. We don't need to change > all occurrences of String(255) to TinyText for example. We make that > determination through understanding the table structure and

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

Hi Michael, Comments below.. On 7/24/2017 2:49 PM, Michael Bayer wrote: On Mon, Jul 24, 2017 at 3:37 PM, Octave J. Orgeron wrote: For these, here is a brief synopsis: AutoStringTinyText, will convert a column to the TinyText type. This is used for cases where a

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

On Mon, Jul 24, 2017 at 3:37 PM, Octave J. Orgeron wrote: > For these, here is a brief synopsis: > > AutoStringTinyText, will convert a column to the TinyText type. This is used > for cases where a 255 varchar string needs to be converted to a text blob to > make the

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

Hi Michael, Comments below.. On 7/24/2017 9:13 AM, Michael Bayer wrote: On Mon, Jul 24, 2017 at 10:01 AM, Jay Pipes wrote: I would much prefer to *add* a brand new schema migration that handles conversion of the entire InnoDB schema at a certain point to an

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

Hi Jay, Comments below.. On 7/24/2017 8:01 AM, Jay Pipes wrote: +Dan Smith Good morning Mike :) Comments inline... On 07/23/2017 08:05 PM, Michael Bayer wrote: On Sun, Jul 23, 2017 at 6:10 PM, Jay Pipes wrote: Glad you brought this up, Mike. I was going to start a

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

Hi Jay, Comments below.. Thanks, Octave On 7/23/2017 4:10 PM, Jay Pipes wrote: Glad you brought this up, Mike. I was going to start a thread about this. Comments inline. On 07/23/2017 05:02 PM, Michael Bayer wrote: I've been working with Octave Oregon in assisting with new rules and

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

Hi Mike, Thanks for putting this together. Comments below.. Thanks, Octave On 7/23/2017 3:02 PM, Michael Bayer wrote: I've been working with Octave Oregon in assisting with new rules and datatypes that would allow projects to support the NDB storage engine with MySQL. To that end, we've made

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

On Mon, Jul 24, 2017 at 10:01 AM, Jay Pipes wrote: > I would much prefer to *add* a brand new schema migration that handles > conversion of the entire InnoDB schema at a certain point to an > NDB-compatible one *after* that point. That way, we isolate the NDB changes > to one

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

> So, I see your point here, but my concern here is that if we *modify* an > existing schema migration that has already been tested to properly apply > a schema change for MySQL/InnoDB and PostgreSQL with code that is > specific to NDB, we introduce the potential for bugs where users report > that

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

+Dan Smith Good morning Mike :) Comments inline... On 07/23/2017 08:05 PM, Michael Bayer wrote: On Sun, Jul 23, 2017 at 6:10 PM, Jay Pipes wrote: Glad you brought this up, Mike. I was going to start a thread about this. Comments inline. On 07/23/2017 05:02 PM, Michael

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

On 07/24/2017 08:05 AM, Michael Bayer wrote: On Sun, Jul 23, 2017 at 6:10 PM, Jay Pipes wrote: Glad you brought this up, Mike. I was going to start a thread about this. Comments inline. On 07/23/2017 05:02 PM, Michael Bayer wrote: Well, besides that point (which I agree

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

On Sun, Jul 23, 2017 at 6:10 PM, Jay Pipes wrote: > Glad you brought this up, Mike. I was going to start a thread about this. > Comments inline. > > On 07/23/2017 05:02 PM, Michael Bayer wrote: > Well, besides that point (which I agree with), that is attempting to change > an

Re: [openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

Glad you brought this up, Mike. I was going to start a thread about this. Comments inline. On 07/23/2017 05:02 PM, Michael Bayer wrote: I've been working with Octave Oregon in assisting with new rules and datatypes that would allow projects to support the NDB storage engine with MySQL. To

[openstack-dev] [oslo.db] [ndb] ndb namespace throughout openstack projects

I've been working with Octave Oregon in assisting with new rules and datatypes that would allow projects to support the NDB storage engine with MySQL. To that end, we've made changes to oslo.db in [1] to support this, and there are now a bunch of proposals such as [2] [3] to implement new