Hi Morgan,
Comments below..
Thanks,
Octave
On 2/6/2017 1:04 PM, Morgan Fainberg wrote:
On Thu, Feb 2, 2017 at 2:28 PM, Octave J. Orgeron
<[email protected] <mailto:[email protected]>> wrote:
That refers to the total length of the row. InnoDB has a limit of
65k and NDB is limited to 14k.
A simple example would be the volumes table in Cinder where the
row length goes beyond 14k. So in the IF logic block, I change
columns types that are vastly oversized such as status and
attach_status, which by default are 255 chars. So to determine a
more appropriate size, I look through the Cinder code to find
where the possible options/states are for those columns. Then I
cut it down to a more reasonable size. I'm very careful when I cut
the size of a string column to ensure that all of the possible
values can be contained.
In cases where a column is extremely large for capturing the
outputs of a command, I will change the type to Text or TinyText
depending on the length required. A good example of this is in the
agents table of Neutron where there is a column for configurations
that has a string length of 4096 characters, which I change to
Text. Text blobs are stored differently and do not count against
the row length.
So
https://github.com/openstack/keystone/blob/master/keystone/common/sql/core.py#L117
would not be an issue with the 14k limit, simply limits for things
such as VARCHAR would be affected (in other words, we wouldn't need to
change keystone's implementation since we already use sql.text here)?
Correct. Having done these patches for Kilo and Mitaka, I can say that
Keystone has been the easiest to patch up. I haven't had to make any
column changes at all. All I've had to do is change the mysql_engine
setting for each table to use the value from mysql_storage_engine. So it
hasn't had any impact on the table schema or structure.
I've also observed differences between Kilo, Mitaka, and tip where
even for InnoDB some of these tables are getting wider than can be
supported. So in the case of Cinder, some of the columns have been
shifted to separate tables to fit within 65k. I've seen the same
thing in Neutron. So I fully expect that some of the services that
have table bloat will have to cut the lengths or break the tables
up over time anyways. As that happens, it reduces the amount of
work for me, which is a good thing.
The most complicated database schemas to patch up are cinder,
glance, neutron, and nova due to the size and complexity of their
tables. Those also have a lot of churn between releases where the
schema changes more often. Other services like keystone, heat, and
ironic are considerably easier to work with and have well laid out
tables that don't change much.
FTR: Keystone also supports "no-downtime-upgrades" (just pending some
functional tests before we apply for the tag) and we will be looking
to move towards Alembic, so make sure that the code supplied can
easily be swapped out between SQL-A-Migrate and Alembic (IIRC most
projects want to move to alembic, but it is varying levels of
difficulty to do so and therefore different priorities).
There are some things that I do like about Alembic and the way that it
heals the database. But there will probably be some tricky conversions
going from SQL Alchemy.
I look forward to solid NDB support; having using NDB in the past to
support another project, I always thought it could be an interesting
choice to back OpenStack (++ to what Monty said eariler).
Thanks! I think the benefits will outweigh the investment for everyone.
Thanks,
Octave
On 2/2/2017 1:25 PM, Mike Bayer wrote:
On 02/02/2017 02:52 PM, Mike Bayer wrote:
But more critically I noticed you referred to altering the names of
columns to suit NDB. How will this be accomplished? Changing
a column
name in an openstack application is no longer trivial, because
online
upgrades must be supported for applications like Nova and
Neutron. A
column name can't just change to a new name, both columns have
to exist
and logic must be added to keep these columns synchronized.
correction, the phrase was "Row character length limits 65k ->
14k" - does this refer to the total size of a row? I guess rows
that store JSON or tables like keystone tokens are what you had
in mind here, can you give specifics ?
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
[email protected]?subject:unsubscribe
<mailto:[email protected]?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
--
Oracle <http://www.oracle.com/>
Octave J. Orgeron | Sr. Principal Architect and Software Engineer
Oracle Linux OpenStack
500 Eldorado Blvd. | Broomfield, CO 80021
Certified Oracle Enterprise Architect: Systems Infrastructure
<http://www.oracle.com/us/solutions/enterprise-architecture/index.html>
Green Oracle <http://www.oracle.com/commitment> Oracle is
committed to developing practices and products that help protect
the environment
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
[email protected]?subject:unsubscribe
<http://[email protected]?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [email protected]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
--
Oracle <http://www.oracle.com/>
Octave J. Orgeron | Sr. Principal Architect and Software Engineer
Oracle Linux OpenStack
<tel:+17206161550> 500 Eldorado Blvd. | Broomfield, CO 80021
Certified Oracle Enterprise Architect: Systems Infrastructure
<http://www.oracle.com/us/solutions/enterprise-architecture/index.html>
Green Oracle <http://www.oracle.com/commitment> Oracle is committed to
developing practices and products that help protect the environment
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [email protected]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
