On Wed, Jul 26, 2017 at 6:19 PM, Michael Bayer <[email protected]> wrote: > On Wed, Jul 26, 2017 at 5:30 PM, Michael Bayer <[email protected]> wrote: >> >> There is a bigger problem with this entire series of changes, whether >> or not the "ndb" keyword is present. Which is that projects need to >> add new columns, tables, and make datatype changes all the time, and >> they will not have any idea about the requirements for ndb or even >> that it exists, nor will anyone have access to this platform for >> development nor should they be expected to worry about it. If they >> not only have to fill in dozens of special "ndb" or generic-but-needed >> by ndb flags, and then if they even have to worry about the sum of all >> the sizes in a row, that means the ndb implementation will be >> continuously broken across many projects in every release unless ndb >> developers are checking every database change in every project at all >> times. Is that level of effort part of the plan? > > OK, I apologize, you answered that here: > > https://review.openstack.org/#/c/427970/26 > > >> Now considering that my company is heavily invested in using MySQL Cluster >> (NDB) and that we use the kola framework, we have to keep an eye on each of >> the services to make sure that it works. This is why you see lots of other >> patches that I'm working on to fix services like Cinder, Neutron, Nova, etc. >> So has time goes by, we will continue to make patches to enable these >> services to work with NDB. > > If we were to approach this as real migrations that change the lengths > of datatypes, that means the tables must be created at InnoDB first > and migrated to NDB within the migrations. Because NDB will disallow > the initial creation, right? > > if the proposal is to modify the actual sizes in the original > migration files, that's not something that can be done, unfortunately, > it would be hugely risky because those migrations represent a snapshot > of the actual schema. > > If we *do* need to keep doing something like the "AutoStringXYZ" > approach I really want to change those names and not have any "ndb." > in it at all.
thinking out loud oslo_db.sqlalchemy.types.String(255, mysql_small_rowsize=64) oslo_db.sqlalchemy.types.String(255, mysql_small_rowsize=sa.TINYTEXT) oslo_db.sqlalchemy.types.String(255, mysql_small_rowsize=sa.TEXT) so if you don't have mysql_small_rowsize, nothing happens. > > But all the options here seem kind of icky. > > > > > > >> >> >> >> >> >> >> >>> >>> I don't see a way of automating that and making it maintainable without a >>> lot more overhead in code and people. If we really want to remove the >>> complexity here, why don't we just change the sizes and types on these >>> handful of table columns so that they fit within both InnoDB and NDB? That >>> way we don't need these functions and the tables are exactly the same? That >>> would only leave us with the createtable, savepoint/rollback, etc. stuff to >>> address which is already taken care of in the ndb module in oslo.db? Then we >>> just fix the foreign key stuff as I've been doing, since it has zero impact >>> on InnoDB deployments and if anything ensures things are consistent. That >>> would then leave us to really focus on fixing migrations to use oslo.db and >>> pass the correct flags, which is a more lengthy process than the rest of >>> this. >>> >>> I don't see the point in trying to make this stuff anymore complicated. >>> >>> Octave >>> >>> >>> On 7/25/2017 12:20 PM, Michael Bayer wrote: >>>> >>>> On Mon, Jul 24, 2017 at 5:41 PM, Michael Bayer <[email protected]> wrote: >>>>>> >>>>>> oslo_db.sqlalchemy.String(255, ndb_type=TINYTEXT) -> VARCHAR(255) for >>>>>> most >>>>>> dbs, TINYTEXT for ndb >>>>>> oslo_db.sqlalchemy.String(4096, ndb_type=TEXT) -> VARCHAR(4096) for most >>>>>> dbs, TEXT for ndb >>>>>> oslo_db.sqlalchemy.String(255, ndb_size=64) -> VARCHAR(255) on most dbs, >>>>>> VARCHAR(64) on ndb >>>>>> >>>>>> This way, we can override the String with TINYTEXT or TEXT or change the >>>>>> size for ndb. >>>>>>> >>>>>>> oslo_db.sqlalchemy.String(255) -> VARCHAR(255) on most dbs, >>>>>>> TINYTEXT() on ndb >>>>>>> oslo_db.sqlalchemy.String(255, ndb_size=64) -> VARCHAR(255) on >>>>>>> most dbs, VARCHAR(64) on ndb >>>>>>> oslo_db.sqlalchemy.String(50) -> VARCHAR(50) on all dbs >>>>>>> oslo_db.sqlalchemy.String(64) -> VARCHAR(64) on all dbs >>>>>>> oslo_db.sqlalchemy.String(80) -> VARCHAR(64) on most dbs, >>>>>>> TINYTEXT() >>>>>>> on ndb >>>>>>> oslo_db.sqlalchemy.String(80, ndb_size=55) -> VARCHAR(64) on most >>>>>>> dbs, VARCHAR(55) on ndb >>>>>>> >>>>>>> don't worry about implementation, can the above declaration -> >>>>>>> datatype mapping work ? >>>>>>> >>>>>>> >>>>>> In my patch for Neutron, you'll see a lot of the AutoStringText() calls >>>>>> to >>>>>> replace exceptionally long String columns (4096, 8192, and larger). >>>>> >>>>> MySQL supports large VARCHAR now, OK. yeah this could be >>>>> String(8192, ndb_type=TEXT) as well. >>>> >>>> OK, no, sorry each time I think of this I keep seeing the verbosity of >>>> imports etc. in the code, because if we had: >>>> >>>> String(80, ndb_type=TEXT) >>>> >>>> then we have to import both String and TEXT, and then what if there's >>>> ndb.TEXT, the code is still making an ndb-specific decision, etc. >>>> >>>> I still see that this can be mostly automated from a simple ruleset >>>> based on the size: >>>> >>>> length <= 64 : VARCHAR(length) on all backends >>>> length > 64, length <= 255: VARCHAR(length) for most backends, >>>> TINYTEXT for ndb >>>> length > 4096: VARCHAR(length) for most backends, TEXT for ndb >>>> >>>> the one case that seems outside of this is: >>>> >>>> String(255) where they have an index or key on the VARCHAR, and in >>>> fact they only need < 64 characters to be indexed. In that case you >>>> don't want to use TINYTEXT, right? So one exception: >>>> >>>> oslo_db.sqlalchemy.types.String(255, indexable=True) >>>> >>>> e.g. a declarative hint to the oslo_db backend to not use a LOB type. >>>> >>>> then we just need oslo_db.sqlalchemy.types.String, and virtually >>>> nothing except the import has to change, and a few keywords. >>>> >>>> What we're trying to do in oslo_db is as much as possible state the >>>> intent of a structure or datatype declaratively, and leave as much of >>>> the implementation up to oslo_db itself. >>>> >>>> __________________________________________________________________________ >>>> OpenStack Development Mailing List (not for usage questions) >>>> Unsubscribe: [email protected]?subject:unsubscribe >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>> >>> >>> >>> __________________________________________________________________________ >>> OpenStack Development Mailing List (not for usage questions) >>> Unsubscribe: [email protected]?subject:unsubscribe >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: [email protected]?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
