Re: [openstack-dev] [all][oslo][db][docs] RFC: drop support for libpq < 9.1
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On 07/10/14 18:05, Mike Bayer wrote: > > On Oct 7, 2014, at 8:29 AM, Ihar Hrachyshka > wrote: > >> >> That said, I wonder how we're going to manage cases when those >> *global* settings for the whole server should be really limited >> to specific databases. Isn't it better to enforce utf8 on service >> side, since we already know that we always run against utf8 >> database? > > I think whenever we do a ?CREATE DATABASE?, we should make sure the > desired encodings are set up at that level. I?ve seen lots of > migration scripts that are listing through tables and setting each > table individually to utf-8, and that?s less than ideal, though I > suppose that?s more of a retroactive fix. > >> >> Please let me clarify... Do you say that setting client encoding >> on oslo.db side is actually ok? I haven't suggested to enforce >> utf8 per column/table, though I guess we're already there. > > The way we are setting client encoding now should be fine, if you > could clarify what about that isn?t working for PG 8.4 that would > be helpful.IMHO especially on Postgresql we should be able to > get away with not having it. MySQLdb is not as nicely implemented > as far as encoding (including the use_unicode speed issues) so it > may be more necessary there. PostgreSQL part is more for consistency with MySQL, but also to avoid enforcing users to set utf8 globally for their SQL server. I think the less we demand on configuration level the better (assuming we decided on using utf8 globally for all our databases). > > But yes what I really *dont* want is the encoding set up on every > column definition, e.g. ?VARCHAR(20) CHARACTER SET ?utf-8??, that?s > been suggested before and that would be terrible. I don?t think > Postgresql has quite the same thing available (only collation per > column). > >> >> Forgoing, again, means some users running with utf8 client >> encoding, others without it. I strive towards consistency here, >> that's why I try to find a way to set the setting for all clients >> we support (and the question is *which* of those clients we >> really support). >> >> The thing that is not supported in psql < 9.1 is a connection >> option added to libpq as of: >> http://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=02e14562a806a96f38120c96421d39dfa7394192 > >> > OK but that is just the connection parameter, when you pass > client_encoding to SQLAlchemy?s psycopg2 dialect, the encoding is > set using psycopg2?s set_client_encoding() method: > http://initd.org/psycopg/docs/connection.html#connection.set_client_encoding. > This ultimately emits ?SET client_encoding TO ?utf8?? on the > connection: > > conn_set_client_encoding -> > https://github.com/psycopg/psycopg2/blob/master/psycopg/connection_int.c#L1188 > > pq_set_guc_locked -> > https://github.com/psycopg/psycopg2/blob/master/psycopg/pqpath.c#L709 > > ?SET client_encoding TO ? is supported in all Postgresql > versions, here?s 8.0: > http://www.postgresql.org/docs/8.0/static/multibyte.html > > So there?s no issue with Postgresql 8.2 here as far as client > encoding, the libpq feature isn?t used for that. Also, it > defaults to the encoding that is set for the database (e.g. server > side), so if we make sure CREATE DATABASE is emitted with the > encoding, then we wouldn?t even need to set it (and perhaps we > shouldn?t if the database is set to a different encoding). As was revealed during IRC discussion with Mike, libpq 'client_encoding' and SQLAlchemy 'client_encoding' are different beasts, and we have the latter into our possession when using libpq < 9.1, so it's better to use that one to avoid raising a minimum version bar. > > > > > > ___ OpenStack-dev > mailing list OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.22 (Darwin) iQEcBAEBCgAGBQJUNn5OAAoJEC5aWaUY1u57Y98H/2PGpZg4IeUKPqT2IGRtCOp3 Ln52cJ3ew/unL0n1JdBW/CAYuN2/c8jsFeWsFGG4P7zMrLh+AFUtmq/4xQShyLkM YimaCzziN3beojVjPOKCYhMNNnS83pS7FIDXdEn/cdadFliiDX7m5rjMG40l9WOD JFR5WfWPFrvura4VUDMjStmgFANxuxLboDy95xmdVomSL9m4Ms+/OZOPkcrV39Iu H3scbIgZVEd8SZ1lpEQYMO4QXgSSfgEV+3efQbZADLYO8ig82a4h+nVN1079RAos DfgxRdu1kK5NaPUFxbtbxhhTYbIuKRHf183tv6qJv6uPHgo06sLe5ie+is2q4LA= =kSU1 -END PGP SIGNATURE- ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][oslo][db][docs] RFC: drop support for libpq < 9.1
On Oct 7, 2014, at 8:29 AM, Ihar Hrachyshka wrote: > > That said, I wonder how we're going to manage cases when those > *global* settings for the whole server should be really limited to > specific databases. Isn't it better to enforce utf8 on service side, > since we already know that we always run against utf8 database? I think whenever we do a “CREATE DATABASE”, we should make sure the desired encodings are set up at that level. I’ve seen lots of migration scripts that are listing through tables and setting each table individually to utf-8, and that’s less than ideal, though I suppose that’s more of a retroactive fix. > > Please let me clarify... Do you say that setting client encoding on > oslo.db side is actually ok? I haven't suggested to enforce utf8 per > column/table, though I guess we're already there. The way we are setting client encoding now should be fine, if you could clarify what about that isn’t working for PG 8.4 that would be helpful.IMHO especially on Postgresql we should be able to get away with not having it. MySQLdb is not as nicely implemented as far as encoding (including the use_unicode speed issues) so it may be more necessary there. But yes what I really *dont* want is the encoding set up on every column definition, e.g. “VARCHAR(20) CHARACTER SET ‘utf-8’”, that’s been suggested before and that would be terrible. I don’t think Postgresql has quite the same thing available (only collation per column). > > Forgoing, again, means some users running with utf8 client encoding, > others without it. > I strive towards consistency here, that's why I try > to find a way to set the setting for all clients we support (and the > question is *which* of those clients we really support). > > The thing that is not supported in psql < 9.1 is a connection option > added to libpq as of: > http://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=02e14562a806a96f38120c96421d39dfa7394192 OK but that is just the connection parameter, when you pass client_encoding to SQLAlchemy’s psycopg2 dialect, the encoding is set using psycopg2’s set_client_encoding() method: http://initd.org/psycopg/docs/connection.html#connection.set_client_encoding. This ultimately emits “SET client_encoding TO ‘utf8’” on the connection: conn_set_client_encoding -> https://github.com/psycopg/psycopg2/blob/master/psycopg/connection_int.c#L1188 pq_set_guc_locked -> https://github.com/psycopg/psycopg2/blob/master/psycopg/pqpath.c#L709 “SET client_encoding TO ” is supported in all Postgresql versions, here’s 8.0: http://www.postgresql.org/docs/8.0/static/multibyte.html So there’s no issue with Postgresql 8.2 here as far as client encoding, the libpq feature isn’t used for that. Also, it defaults to the encoding that is set for the database (e.g. server side), so if we make sure CREATE DATABASE is emitted with the encoding, then we wouldn’t even need to set it (and perhaps we shouldn’t if the database is set to a different encoding). ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][oslo][db][docs] RFC: drop support for libpq < 9.1
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On 06/10/14 19:10, Mike Bayer wrote: > > On Oct 6, 2014, at 9:56 AM, Ihar Hrachyshka > wrote: > >>> >>> But we can do better. We should also enforce utf8 on client >>> side, so that there is no way to run with a different encoding, >>> and so that we may get rid of additional options in sql >>> connection strings. I've sent a patch for oslo.db [4] to do >>> just that. > > i would recommend that we definitely do *not* set explicit client > encodings on all columns, and go with the most minimal approach for > whatever the target database is.For example, with Postgresql > 8.4, utf-8 is not an issue so long as client_encoding is set within > postgresql.conf.I’ve had this kind of discussion many times in > the past with folks who are trying to “protect” some subset of > users that don’t have this setting in their conf file, either > because they installed from an OSX package or some other weird > reason, and generally I’m not buying it.There’s no need to > build tremendous verbosity and fine-grained specificity throughout > a system in order to appease very narrow edge cases like this. > It’s not just about potential performance problems, it’s also just > a schema and code management issue, in that it is complexity that > IMHO is just not needed. - From server side, indeed, proper configuration described in documentation may be enough. Though atm we don't document PostgreSQL settings anywhere. [As for MySQL, both server and client encodings are indeed advised to be set to utf8.] That said, I wonder how we're going to manage cases when those *global* settings for the whole server should be really limited to specific databases. Isn't it better to enforce utf8 on service side, since we already know that we always run against utf8 database? > > For this reason I’m pretty ambivalent overall about any kind of > utf-8 hardcoding within the application connection logic. IMHO > this should be handled at the configurational layer as much as > possible. Though as long as it’s just an application time setting, > and not something hardwired throughout the schema (implying we now > have to *rely* upon UTF-8 encoding explicitly for all columns > everywhere…and then what happens if we are on some target database > that doesn’t quite do things the same way, e.g. DB2, oracle, > others?), it’s not *too* big of a deal for me if it solves some > problems right now. Please let me clarify... Do you say that setting client encoding on oslo.db side is actually ok? I haven't suggested to enforce utf8 per column/table, though I guess we're already there. > > short answer, there should be no need to drop PG8.X support for > this reason. If the client encoding setting is something we want > hardcoded in the app layer and it fails for those versions (which > I’m not familiar with? what is the thing that is not actually > supported prior to libpq 9.1 ? psycopg2’s set_client_encoding, > really? this doesn’t sound familiar two me), we can detect that > and forego the setting - sqlalchemy dialect has server_version_info > for this reason. Forgoing, again, means some users running with utf8 client encoding, others without it. I strive towards consistency here, that's why I try to find a way to set the setting for all clients we support (and the question is *which* of those clients we really support). The thing that is not supported in psql < 9.1 is a connection option added to libpq as of: http://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=02e14562a806a96f38120c96421d39dfa7394192 As I said before, we have other ways to set client encoding for psql clients that use libpq < 9.1, and the thing I try to determine is whether we consider supporting those versions of libpq. /Ihar -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.22 (Darwin) iQEcBAEBCgAGBQJUM9yfAAoJEC5aWaUY1u571T0IALBgfRSL0PY1q9PC/vpiee2B d++dVM+CU/SRhXh7zlY703K/sdRVvsPGlUNkzPjT0TGa81LbXPKtwVqPIQeUpI5i a+X9HKHksaeidgLyuOPbSic+hFSoQESoYBswLfb0+vhT9JEn2XwzEyJo83c4MDH3 grgi5Crk+xx2uAa3mnEKCkeCOiq6t6V+7s7DKM8qdAEZOvKWlY3JK5W+AF6Smjlj XJkywLXkDy+t+jMMIo2XqlR7xIVoy4wSTZRgbqrHpBqxqhL3/wzk31JnQgant8A+ 2HF8p+Goy0CxoLOYb0LCE4JiHh6aoHXCcUOXW9BmcVBxfRg7el3OcKWJb9g2Q+M= =78+Z -END PGP SIGNATURE- ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][oslo][db][docs] RFC: drop support for libpq < 9.1
On Mon, Oct 06, 2014, Ihar Hrachyshka wrote: > Maybe it is indeed wasteful, I don't have numbers; though the fact is > we don't allow any migrations for databases with any non utf8 tables > as of [1]. The code was copied in multiple projects (Nova, Glance > among other things). Also, the same check is present in oslo.db that > is used by most projects now. > > Also we have migration rules in multiple projects that end up with > converting all tables to utf8. F.e. it's done in Nova [2]. > > So we already run against utf8 tables. Though we don't ever tell users > to create their databases with utf8 as default charset, opening a can > of worms. We also don't ever tell users to at least set use_unicode=0 > when running against MySQLdb (which is the default and the only > supported MySQL driver as of Juno) to avoid significant performance > drop [3] (same is true for oursql driver, but that one is at least not > recommended to users thru official docs). This is a situation where Openstack is frustratingly inconsistent. While we don't provide any guidance about the default charset, Nova now creates all tables with the utf8 charset and provided a migration[1] to fix deployments done before this change. The same cannot be said for Glance. It inherited the utf8 check, but never provided a migration to fix deployments done before this change. It still creates tables with no default charset leading to a situation where you can deploy Glance with default values but then end up not being able to run any future migrations. Glance did have a flag to disable that check, but it was recently removed[2] with no automated migrations to resolve earlier deployments (like many of ours). This frustratingly got approved and merged after my -1 and with no explanation why they were doing this to operators. It felt like I was getting the gerrit equivalent of a middle finger. All of that said, I'm 100% for making all of the projects more consistent and use utf8 (where it makes sense). [1]: https://review.openstack.org/#/c/3946/ [2]: https://review.openstack.org/#/c/120002/ JE ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][oslo][db][docs] RFC: drop support for libpq < 9.1
On Oct 6, 2014, at 9:56 AM, Ihar Hrachyshka wrote: > > > > But we can do better. We should also enforce utf8 on client side, > > so that there is no way to run with a different encoding, and so > > that we may get rid of additional options in sql connection > > strings. I've sent a patch for oslo.db [4] to do just that. i would recommend that we definitely do *not* set explicit client encodings on all columns, and go with the most minimal approach for whatever the target database is.For example, with Postgresql 8.4, utf-8 is not an issue so long as client_encoding is set within postgresql.conf.I’ve had this kind of discussion many times in the past with folks who are trying to “protect” some subset of users that don’t have this setting in their conf file, either because they installed from an OSX package or some other weird reason, and generally I’m not buying it.There’s no need to build tremendous verbosity and fine-grained specificity throughout a system in order to appease very narrow edge cases like this. It’s not just about potential performance problems, it’s also just a schema and code management issue, in that it is complexity that IMHO is just not needed. For this reason I’m pretty ambivalent overall about any kind of utf-8 hardcoding within the application connection logic. IMHO this should be handled at the configurational layer as much as possible. Though as long as it’s just an application time setting, and not something hardwired throughout the schema (implying we now have to *rely* upon UTF-8 encoding explicitly for all columns everywhere…and then what happens if we are on some target database that doesn’t quite do things the same way, e.g. DB2, oracle, others?), it’s not *too* big of a deal for me if it solves some problems right now. short answer, there should be no need to drop PG8.X support for this reason. If the client encoding setting is something we want hardcoded in the app layer and it fails for those versions (which I’m not familiar with? what is the thing that is not actually supported prior to libpq 9.1 ? psycopg2’s set_client_encoding, really? this doesn’t sound familiar two me), we can detect that and forego the setting - sqlalchemy dialect has server_version_info for this reason. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][oslo][db][docs] RFC: drop support for libpq < 9.1
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 On 06/10/14 15:11, Jay Pipes wrote: > On 10/06/2014 08:30 AM, Ihar Hrachyshka wrote: Hi all, > > tl;dr we should enforce utf8 on server and client sides of db > connection, and this requires changes to docs and oslo.db. The > latter may require raising effective libpq version dep to 9.1+. > > Recently I was working on making sure we always run using utf8 on > both database server and client side, and that we do it with all > the proper connection options set (f.e. mysqldb requires > use_unicode=0 to avoid performance drop - a thing that is not set > anywhere in gate as of now, and not documented). > > For server side, it's a matter of how we create databases, and in > devstack, it's usually utf8 (though, for unclear reason, latin1 is > still enforced for nova for initial creation due to Folsom (sic!) > bug; I've reverted to utf8 with [1]). > > For client side, it's not that clear. Though devstack enforces utf8 > in SQL connection strings, we don't ask to do so in official docs > [2]. There is a bug [3] in LP that seems to be relevant. > > I think the proper way is to enforce utf8 everywhere. The first > step would be to make sure we suggest adding charset=utf8 to SQL > connection strings, and configure SQL servers to use utf8. That is > something to be covered by docs team. > > But we can do better. We should also enforce utf8 on client side, > so that there is no way to run with a different encoding, and so > that we may get rid of additional options in sql connection > strings. I've sent a patch for oslo.db [4] to do just that. > >> So, the subject of this email is a bit misleading :) You are >> suggesting a change to the UTF-8 charset enforcement here, and >> below you are suggesting removing libpq<9.1 for related reasons. > >> I suppose setting of utf8 on the client side, and supporting utf8 >> on the server side, but I do *not* recommend using a blanket >> policy of charset=utf8 on every text column on the database side. >> It really is wasteful for certain columns like uuids that are >> used in joins all over the schema... Maybe it is indeed wasteful, I don't have numbers; though the fact is we don't allow any migrations for databases with any non utf8 tables as of [1]. The code was copied in multiple projects (Nova, Glance among other things). Also, the same check is present in oslo.db that is used by most projects now. Also we have migration rules in multiple projects that end up with converting all tables to utf8. F.e. it's done in Nova [2]. So we already run against utf8 tables. Though we don't ever tell users to create their databases with utf8 as default charset, opening a can of worms. We also don't ever tell users to at least set use_unicode=0 when running against MySQLdb (which is the default and the only supported MySQL driver as of Juno) to avoid significant performance drop [3] (same is true for oursql driver, but that one is at least not recommended to users thru official docs). [1]: https://review.openstack.org/#/c/64764/ [2]: https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/migrate_repo/versions/216_havana.py#L1539 [3]: http://docs.sqlalchemy.org/en/rel_0_9/dialects/mysql.html#module-sqlalchemy.dialects.mysql.mysqldb > >> Best, -jay > > Though as you can see, the patch fails on py26 postgresql unit > tests. This is because I set charset_encoding for PostgreSQL via > libpq connection option that is available since PostgreSQL 9.1 > only. The failure is because we run py26 test suite on CentOS 6.5 > that still uses PostgreSQL 8.4.*, meaning the option is not > available for us. > > Now, there are several other ways available for us to enforce > encoding without using the option. First, we may set encoding thru > PGCLIENTENCODING environment variable. Second, we may execute "SET > CLIENT_ENCODING TO utf8" after connecting to psql server. > > Now, the question is whether it's worth reimplementing the feature > using some of the available hacks that will work with PSQL < 9.1. > We're about to drop Python 2.6 support in Kilo, and that probably > means that CentOS 6.5 will be dropped from gate too. Do we have > any other distribution of interest that uses both python 2.7 and > psql < 9.1, and is going to run Kilo pieces? > > On Red Hat side, we're not interested in running Juno on EL6 that > uses old psql. As for EL7, it has a psql version that supports the > connection option. On Ubuntu side, the only LTS release that is > still supported and uses psql < 9.1 is Lucid, but it's based on > py26, so it should not be an issue. Any other consumers that could > be affected by effectively raising minimum version bar for psql to > 9.1 in Kilo? > > Though we've (almost) decided to drop py26 support in Kilo, we > didn't have similar discussions for other components that we use. > So I'd like to hear what's our take on that. Do we really drop > *py26* support, or maybe what we really do is we drop *CentOS 6.5* >
Re: [openstack-dev] [all][oslo][db][docs] RFC: drop support for libpq < 9.1
On 10/06/2014 08:30 AM, Ihar Hrachyshka wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Hi all, tl;dr we should enforce utf8 on server and client sides of db connection, and this requires changes to docs and oslo.db. The latter may require raising effective libpq version dep to 9.1+. Recently I was working on making sure we always run using utf8 on both database server and client side, and that we do it with all the proper connection options set (f.e. mysqldb requires use_unicode=0 to avoid performance drop - a thing that is not set anywhere in gate as of now, and not documented). For server side, it's a matter of how we create databases, and in devstack, it's usually utf8 (though, for unclear reason, latin1 is still enforced for nova for initial creation due to Folsom (sic!) bug; I've reverted to utf8 with [1]). For client side, it's not that clear. Though devstack enforces utf8 in SQL connection strings, we don't ask to do so in official docs [2]. There is a bug [3] in LP that seems to be relevant. I think the proper way is to enforce utf8 everywhere. The first step would be to make sure we suggest adding charset=utf8 to SQL connection strings, and configure SQL servers to use utf8. That is something to be covered by docs team. But we can do better. We should also enforce utf8 on client side, so that there is no way to run with a different encoding, and so that we may get rid of additional options in sql connection strings. I've sent a patch for oslo.db [4] to do just that. So, the subject of this email is a bit misleading :) You are suggesting a change to the UTF-8 charset enforcement here, and below you are suggesting removing libpq<9.1 for related reasons. I suppose setting of utf8 on the client side, and supporting utf8 on the server side, but I do *not* recommend using a blanket policy of charset=utf8 on every text column on the database side. It really is wasteful for certain columns like uuids that are used in joins all over the schema... Best, -jay Though as you can see, the patch fails on py26 postgresql unit tests. This is because I set charset_encoding for PostgreSQL via libpq connection option that is available since PostgreSQL 9.1 only. The failure is because we run py26 test suite on CentOS 6.5 that still uses PostgreSQL 8.4.*, meaning the option is not available for us. Now, there are several other ways available for us to enforce encoding without using the option. First, we may set encoding thru PGCLIENTENCODING environment variable. Second, we may execute "SET CLIENT_ENCODING TO utf8" after connecting to psql server. Now, the question is whether it's worth reimplementing the feature using some of the available hacks that will work with PSQL < 9.1. We're about to drop Python 2.6 support in Kilo, and that probably means that CentOS 6.5 will be dropped from gate too. Do we have any other distribution of interest that uses both python 2.7 and psql < 9.1, and is going to run Kilo pieces? On Red Hat side, we're not interested in running Juno on EL6 that uses old psql. As for EL7, it has a psql version that supports the connection option. On Ubuntu side, the only LTS release that is still supported and uses psql < 9.1 is Lucid, but it's based on py26, so it should not be an issue. Any other consumers that could be affected by effectively raising minimum version bar for psql to 9.1 in Kilo? Though we've (almost) decided to drop py26 support in Kilo, we didn't have similar discussions for other components that we use. So I'd like to hear what's our take on that. Do we really drop *py26* support, or maybe what we really do is we drop *CentOS 6.5* support?.. On a more general note, do we track those kind of external dependencies anywhere? There are lots of them: neutron depends on specific versions of openvswitch; oslo.db depends on specific versions of underlying backend libraries, etc. Comments? [1]: https://review.openstack.org/126267 [2]: http://docs.openstack.org/icehouse/install-guide/install/yum/content/nova-controller.html [3]: https://bugs.launchpad.net/glance/+bug/1279000 [4]: https://review.openstack.org/111236 /Ihar -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.22 (Darwin) iQEcBAEBCgAGBQJUMotwAAoJEC5aWaUY1u57IQQIAN2yHfxbj0DYDMHE40OGbEwU 8fis/KOAxSyUEhHbbECS8ODGjHNK9SeuddUplzL5sKVbYIMZgt3wKXGT/rT5ex0C iMmc7/mb6JGGTUYvR1sF2EXxioJ3kWJSt5oQqBhVT14y2g8rwqN6tQaTusfDVZWN yZ7KCmQDc+3HMyY3EmYCp6V1ndubLbx8Oh56/jtdSGdNtc7V4IwYIF3F4WQaKBcB 54tZkAO0wvlbbRrXqEiyf99/dVLDiMquM3LNKEbYg4zjDUEX74fGJ3mscQtWdDLv 0TcqOvQ2pwLvpksf30kvBBHGcNCF9x8Ov8H4vg5wRTD4bkpjEDwmXzQXTk7xfHM= =aYRM -END PGP SIGNATURE- ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [all][oslo][db][docs] RFC: drop support for libpq < 9.1
-BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Hi all, tl;dr we should enforce utf8 on server and client sides of db connection, and this requires changes to docs and oslo.db. The latter may require raising effective libpq version dep to 9.1+. Recently I was working on making sure we always run using utf8 on both database server and client side, and that we do it with all the proper connection options set (f.e. mysqldb requires use_unicode=0 to avoid performance drop - a thing that is not set anywhere in gate as of now, and not documented). For server side, it's a matter of how we create databases, and in devstack, it's usually utf8 (though, for unclear reason, latin1 is still enforced for nova for initial creation due to Folsom (sic!) bug; I've reverted to utf8 with [1]). For client side, it's not that clear. Though devstack enforces utf8 in SQL connection strings, we don't ask to do so in official docs [2]. There is a bug [3] in LP that seems to be relevant. I think the proper way is to enforce utf8 everywhere. The first step would be to make sure we suggest adding charset=utf8 to SQL connection strings, and configure SQL servers to use utf8. That is something to be covered by docs team. But we can do better. We should also enforce utf8 on client side, so that there is no way to run with a different encoding, and so that we may get rid of additional options in sql connection strings. I've sent a patch for oslo.db [4] to do just that. Though as you can see, the patch fails on py26 postgresql unit tests. This is because I set charset_encoding for PostgreSQL via libpq connection option that is available since PostgreSQL 9.1 only. The failure is because we run py26 test suite on CentOS 6.5 that still uses PostgreSQL 8.4.*, meaning the option is not available for us. Now, there are several other ways available for us to enforce encoding without using the option. First, we may set encoding thru PGCLIENTENCODING environment variable. Second, we may execute "SET CLIENT_ENCODING TO utf8" after connecting to psql server. Now, the question is whether it's worth reimplementing the feature using some of the available hacks that will work with PSQL < 9.1. We're about to drop Python 2.6 support in Kilo, and that probably means that CentOS 6.5 will be dropped from gate too. Do we have any other distribution of interest that uses both python 2.7 and psql < 9.1, and is going to run Kilo pieces? On Red Hat side, we're not interested in running Juno on EL6 that uses old psql. As for EL7, it has a psql version that supports the connection option. On Ubuntu side, the only LTS release that is still supported and uses psql < 9.1 is Lucid, but it's based on py26, so it should not be an issue. Any other consumers that could be affected by effectively raising minimum version bar for psql to 9.1 in Kilo? Though we've (almost) decided to drop py26 support in Kilo, we didn't have similar discussions for other components that we use. So I'd like to hear what's our take on that. Do we really drop *py26* support, or maybe what we really do is we drop *CentOS 6.5* support?.. On a more general note, do we track those kind of external dependencies anywhere? There are lots of them: neutron depends on specific versions of openvswitch; oslo.db depends on specific versions of underlying backend libraries, etc. Comments? [1]: https://review.openstack.org/126267 [2]: http://docs.openstack.org/icehouse/install-guide/install/yum/content/nova-controller.html [3]: https://bugs.launchpad.net/glance/+bug/1279000 [4]: https://review.openstack.org/111236 /Ihar -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.22 (Darwin) iQEcBAEBCgAGBQJUMotwAAoJEC5aWaUY1u57IQQIAN2yHfxbj0DYDMHE40OGbEwU 8fis/KOAxSyUEhHbbECS8ODGjHNK9SeuddUplzL5sKVbYIMZgt3wKXGT/rT5ex0C iMmc7/mb6JGGTUYvR1sF2EXxioJ3kWJSt5oQqBhVT14y2g8rwqN6tQaTusfDVZWN yZ7KCmQDc+3HMyY3EmYCp6V1ndubLbx8Oh56/jtdSGdNtc7V4IwYIF3F4WQaKBcB 54tZkAO0wvlbbRrXqEiyf99/dVLDiMquM3LNKEbYg4zjDUEX74fGJ3mscQtWdDLv 0TcqOvQ2pwLvpksf30kvBBHGcNCF9x8Ov8H4vg5wRTD4bkpjEDwmXzQXTk7xfHM= =aYRM -END PGP SIGNATURE- ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
