On 05/17/2017 02:38 PM, Sean Dague wrote:
Some of the concerns/feedback has been "please describe things that are
harder by this being an abstraction", so examples are provided.
so let's go through this list:
- OpenStack services taking a more active role in managing the DBMS
mmmm, "managing" is vague to me, are we referring to the database
service itself, e.g. starting / stopping / configuring? installers
like tripleo do this now, pacemaker is standard in HA for control of
services, I think I need some background here as to what the more active
role would look like.
- The ability to have zero down time upgrade for services such as
Keystone.
So "zero down time upgrades" seems to have broken into:
* "expand / contract with the code carefully dancing around the
existence of two schema concepts simultaneously", e.g. nova, neutron.
AFAIK there is no particular issue supporting multiple backends on this
because we use alembic or sqlalchemy-migrate to abstract away basic
ALTER TABLE types of feature.
* "expand / contract using server side triggers to reconcile the two
schema concepts", e.g. keystone. This is more difficult because there
is currently no "trigger" abstraction layer. Triggers represent more
of an imperative programming model vs. typical SQL, which is why I've
not taken on trying to build a one-size-fits-all abstraction for this in
upstream Alembic or SQLAlchemy. However, it is feasible to build a
"one-size-that-fits-openstack-online-upgrades" abstraction. I was
trying to gauge interest in helping to create this back in the
"triggers" thread, in my note at
http://lists.openstack.org/pipermail/openstack-dev/2016-August/102345.html,
which also referred to some very raw initial code examples. However, it
received strong pushback from a wide range of openstack veterans, which
led me to believe this was not a thing that was happening. Apparently
Keystone has gone ahead and used triggers anyway, however I was not
pulled into that process. But if triggers are to be "blessed" by at
least some projects, I can likely work on this problem for MySQL /
Postgresql agnosticism. If keystone is using triggers right now for
online upgrades, I would ask, are they currently working on Postgresql
as well with PG-specific triggers, or does Postgresql degrade into a
"non-online" migration scenario if you're running Keystone?
- Consistent UTF8 4 & 5 byte support in our APIs
"5 byte support" appears to refer to utf-8's ability to be...well a
total of 6 bytes. But in practice, unicode itself only needs 4 bytes
and that is as far as any database supports right now since they target
unicode (see https://en.wikipedia.org/wiki/UTF-8#Description). That's
all any database we're talking about supports at most. So...lets assume
this means four bytes.
From the perspective of database-agnosticism with regards to database
and driver support for non-ascii characters, this problem has been
solved by SQLAlchemy well before Python 3 existed when many DBAPIs would
literally crash if they received a u'' string, and the rest of them
would churn out garbage; SQLAlchemy implemented a full encode/decode
layer on top of the Python DBAPI to fix this. The situation is vastly
improved now that all DBAPIs support unicode natively.
However, on the MySQL side there is this complexity that their utf-8
support is a 3-byte only storage model, and you have to use utf8mb4 if
you want the four byte model. I'm not sure right now what projects are
specifically hitting issues related to this.
Postgresql doesn't have such a limitation. If your Postgresql server
or specific database is set up for utf-8 (which should be the case),
then you get full utf-8 character set support.
So I don't see the problem of "consistent utf8 support" having much to
do with whether or not we support Posgtresql - you of course need your
"CREATE DATABASE" to include the utf8 charset like we do on MySQL, but
that's it.
- The requirement that Postgresql libraries are compiled for new users
trying to just run unit tests (no equiv is true for mysql because of
the pure python driver).
I would suggest that new developers for whom the presence of things like
postgresql client libraries is a challenge (but somehow they are running
a MySQL server for their pure python driver to talk to?) don't actually
have to worry about running the tests against Postgresql, this is how
the "opportunistic" testing model in oslo.db has always worked; it only
runs for the backends that you have set up.
Also, openstack got all the way through Kilo approximately using the
native python-MySQL driver which required a compiled client library as
well as the MySQL dependencies be installed. The psycopg2 driver has a
ton of whl's up on pypi (https://pypi.python.org/pypi/psycopg2) and all
linux distros supply it as a package in any case, so an actual "compile"
should not be needed. Also, this is Openstack....it's basic existence
is a kind of this vastly enormous glue between thousands of native
(ultimately C-compiled) libraries and packages, and it runs only
on...linux. So this is just a weird point to bring up. Seems a
little red herring to me.
- Consistency around case sensitivity collation defaults that lead to
strange bugs around searching/updating names in resources.
Finally, a real issue-ish thing that requires a resolution. So the
good news here is that while MySQL is defaulting to case-insensitive
collations (which I assume we like) but Postgresql has almost no support
for case-insensitive collations (unless you use this odd CITEXT
datatype), it is possible to make a case-sensitive collation style
become case-insensitive at SQL query time much more easily than it would
be to go the other way.
SQLAlchemy already has some case-insensitive operators, most notably
"ilike()", which is a case-insensitive "LIKE" that is backend agnostic.
If these search queries are just using LIKE then they only need use
ilike() from SQLAlchemy instead of like().
If we are talking about the full range of operators like ==, !=, <, >
etc., and/or if we are also concerned that developers may use like()
when they really need to use ilike(), the issue can be addressed at the
typing level as well. Using SQLAlchemy, a String datatype that
guarantees case-insensitive comparisons is straightforward to construct.
This would be a Python side replacement for the String type, and
possibly Text, Unicode, etc. as needed. It does **not** imply any
database schema migrations. The hypothetical CaseInsensitiveString
would override all the comparison operators to ensure that on the
Postgresql (or other case-sensitive) backend, both sides of the
expression are embedded within the SQL LOWER() function, so that these
comparisons act case insensitively. The trick then is to get the
downstream projects to use this type (which would be in oslo.db) in
their models, which is the usual herding cats job. But this is a pretty
solvable problem.
so to sum up:
1. "more active management role" - not totally sure what that refers to
beyond what is already present
2. "zero downtime upgrades" - to the extent that projects use server
side constructs like triggers, effort is needed to produce
implementations for both backends. I can help with this effort but the
downstream projects need to care and be open to reviewing / merging
gerrits and things. Not sure what the state of trigger-based online
upgrades is for Postgresql right now, or if there is a degradation mode
present in Keystone w/ Postgresql.
3. utf-8 - no problem
4. native clients - red herring ?
5. case-insensitive - both query-level and model-level solutions exist
that work within our existing frameworks and patterns
A statement around support was also put in there, because support only
meant QA jobs, or only developers for some folks. I think it's important
to ensure we paint the whole picture with how people get support in an
Open Source project.
There seems to be general agreement that we need to be more honest with
users, and that we've effectively been lying to them.
I feel like the current sticking points come down to whether:
* it's important that the operator community largely is already in one
camp or not
* future items listed that are harder are important enough to justify a
strict trade off here
* it's ok to have the proposal have a firm lean in tone, even though
it's set of concrete actions are pretty reversible and don't commit to
future removal of postgresql
Also, as I stated on IRC, if some set of individuals came through and
solved all the future problems on the list for us as a community, my
care on how many DBs supported would drastically decrease. Because its
the fact that it's costing us solving real problems that we want to
solve (by making them too complex for anyone to take on), is my key
concern. For folks asking the question about what they could do to make
pg a first class citizen, that's a pretty good starting point.
-Sean
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
