Re: [openstack-dev] [keystone] sqlite doesn't support migrations
Along these lines, OpenStack is now maintaining sqlachemy-migrate, and we have our first patch up for better SQLite support, taken from nova, https://review.openstack.org/#/c/37656/ Do we want to go the direction of explicitly not supporting sqllite and not running migrations with it, like keystone. If so we may not want the patch above to get merged. On Wed, Jul 17, 2013 at 10:40 AM, Adam Young ayo...@redhat.com wrote: On 07/16/2013 10:58 AM, Thomas Goirand wrote: On 07/16/2013 03:55 PM, Michael Still wrote: On Tue, Jul 16, 2013 at 4:17 PM, Thomas Goirand z...@debian.org wrote: Could you explain a bit more what could be done to fix it in an easy way, even if it's not efficient? I understand that ALTER doesn't work well. Though would we have the possibility to just create a new temporary table with the correct fields, and copy the existing content in it, then rename the temp table so that it replaces the original one? There are a bunch of nova migrations that already work that way... Checkout the *sqlite* files in nova/db/sqlalchemy/migrate_**repo/versions/ Why can't we do that with Keystone then? Is it too much work? It doesn't seem hard to do (just probably a bit annoying and boring ...). Does it represent too much work for the case of Keystone? Thomas __**_ OpenStack-dev mailing list OpenStack-dev@lists.openstack.**org OpenStack-dev@lists.openstack.org http://lists.openstack.org/**cgi-bin/mailman/listinfo/**openstack-devhttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev In general, yes, the SQlite migrations have eaten up far more time than the benefit they provide. Sqlite migrations don't buy us much. A live deployment should not be on sqlite. It is OK to use it for testing, though. As such, it should be possible to generate a sqlite scheme off the model the way that the unit tests do. You just will not be able to migrate it forward: later changes will require regenerating the database. Agreed, so lets change the default database away from SQLite everywhere and we can just override the default when needed for testing. For any non-trivial deployent, we should encourage the use of Mysql or Postgresql. Migrations for these will be full supported. From a unit test perspective, we are going to disable the sqlite migration tests. Users that need to perform testing of migrations should instead test them against both mysql and postgresql. the IBMers out there will also be responsible for keeping on top of the DB2 variations. The [My/postgre]Sql migration tests will be run as part of the gate. The current crop of unit tests for the SQL backend use the module based approach to generate a sqlite database. This approach is fine. Sqlite, when run in memory, provides a very fast sanity check of our logic. MySQL and PostgreSQL versions would take far, far too long to execute for everyday development. We will, however, make it possible to execute those body of unit tests against both postgresql and mysql as part of the gate. Wer will also execute against the LDAP backend. IN order to be able to perform that, however, we need to fix bunch of the tests so that they run at 100% first. __**_ OpenStack-dev mailing list OpenStack-dev@lists.openstack.**org OpenStack-dev@lists.openstack.org http://lists.openstack.org/**cgi-bin/mailman/listinfo/**openstack-devhttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [keystone] sqlite doesn't support migrations
On 07/16/2013 12:07 PM, Michael Still wrote: On Tue, Jul 16, 2013 at 1:44 PM, Thomas Goirand z...@debian.org wrote: In Debian, by policy, any package should be able to be installed using DEBIAN_FRONTEND=noninteractive apt-get install. What I do in my postinst is calling db_sync, because that isn't something our users should even care about, since it can be automated. The final result is that, for many package like Keystone and Glance, simply doing apt-get install is enough to make it work, without needing any configuration file edition. I want to be able to keep that nice feature. Is there any policy about how long a package can take to install? No. db_sync might make many minutes for large installations. For example, folsom - grizzly takes about 7 minutes in my tests. Michael Gosh!!! I haven't experienced that! Or is it that my upgrade path was broken somehow? :) I shall test more. Thomas ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [keystone] sqlite doesn't support migrations
On 07/16/2013 01:50 PM, Dolph Mathews wrote: Make it work is an entirely different goal than make a production-ready deployment. If your goal in using sqlite is just to make it work then I'm not sure that I would expect such an install to survive to the next release, anyway... rendering migration support as a nice-to-have. I can't imagine that any end users would be happy with a sqlite-based deployment for anything other than experimentation and testing. In that case, IMO we should strongly state that fact, and don't let our users believe that SQLite is supported (my view is that if upgrades aren't, then it's as if SQLite wasn't supported at all). If the support for SQLite (db upgrades) has to go, I will understand and adapt. I haven't and probably wont find the time for doing the actual work to support SQLite upgrades, and therefore, it probably is easier for me to state, though, I believe it is my duty to raise my concerns, and tell that I do not support this decision. I'm glad you spoke up Thanks! :) What direction do you think this should take? Your thoughts? I'd still like to pursue dropping support for sqlite migrations, albeit not as aggressively as I would have preferred. With a stakeholder, I think it's requisite to continue support through Havana. Perhaps at the fall summit we can evaluate our position on both alembic and sqlite migrations. Could you explain a bit more what could be done to fix it in an easy way, even if it's not efficient? I understand that ALTER doesn't work well. Though would we have the possibility to just create a new temporary table with the correct fields, and copy the existing content in it, then rename the temp table so that it replaces the original one? Cheers, Thomas ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [keystone] sqlite doesn't support migrations
On Tue, Jul 16, 2013 at 4:05 PM, Thomas Goirand z...@debian.org wrote: On 07/16/2013 12:07 PM, Michael Still wrote: On Tue, Jul 16, 2013 at 1:44 PM, Thomas Goirand z...@debian.org wrote: In Debian, by policy, any package should be able to be installed using DEBIAN_FRONTEND=noninteractive apt-get install. What I do in my postinst is calling db_sync, because that isn't something our users should even care about, since it can be automated. The final result is that, for many package like Keystone and Glance, simply doing apt-get install is enough to make it work, without needing any configuration file edition. I want to be able to keep that nice feature. Is there any policy about how long a package can take to install? No. Hmmm. Ok. I do think its something for you to track though. db_sync might make many minutes for large installations. For example, folsom - grizzly takes about 7 minutes in my tests. Gosh!!! I haven't experienced that! Or is it that my upgrade path was broken somehow? :) I shall test more. That's with a DB that records 30 million instances. If you're using a trivial test database then you're not replicating the experience for a real deployment. Michael -- Rackspace Australia ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [keystone] sqlite doesn't support migrations
On Tue, Jul 16, 2013 at 4:17 PM, Thomas Goirand z...@debian.org wrote: Could you explain a bit more what could be done to fix it in an easy way, even if it's not efficient? I understand that ALTER doesn't work well. Though would we have the possibility to just create a new temporary table with the correct fields, and copy the existing content in it, then rename the temp table so that it replaces the original one? There are a bunch of nova migrations that already work that way... Checkout the *sqlite* files in nova/db/sqlalchemy/migrate_repo/versions/ Cheers, Michael -- Rackspace Australia ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [keystone] sqlite doesn't support migrations
On Tue, Jul 16, 2013 at 12:55 AM, Michael Still mi...@stillhq.com wrote: On Tue, Jul 16, 2013 at 4:17 PM, Thomas Goirand z...@debian.org wrote: Could you explain a bit more what could be done to fix it in an easy way, even if it's not efficient? I understand that ALTER doesn't work well. Though would we have the possibility to just create a new temporary table with the correct fields, and copy the existing content in it, then rename the temp table so that it replaces the original one? There are a bunch of nova migrations that already work that way... Checkout the *sqlite* files in nova/db/sqlalchemy/migrate_repo/versions/ Cheers, Michael -- Rackspace Australia ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev If we really want to support the concept of SQLite migrations, the way nova does it seems to be the most sane. I'm not 100% convinced that SQLite migrations are worth supporting, but then again, I am not the target to use them (besides in a simple development capacity, and I still validate against MySQL/Postgres mostly). If there is a demand for SQLite, I'd say Michael has hit it on the head and the way Nova handles this is a fairly clean mechanism and far more supportable over the short and medium term(s) compared to working around migrate issues with SQLite and the limited Alter support. In one of the discussions in IRC I had offered to help with the effort of moving away from SQLite migration testing; if the nova-way is the way we want to go, I'll be happy to help contribute to that. Cheers, Morgan Fainberg ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [keystone] sqlite doesn't support migrations
On 07/16/2013 03:55 PM, Michael Still wrote: On Tue, Jul 16, 2013 at 4:17 PM, Thomas Goirand z...@debian.org wrote: Could you explain a bit more what could be done to fix it in an easy way, even if it's not efficient? I understand that ALTER doesn't work well. Though would we have the possibility to just create a new temporary table with the correct fields, and copy the existing content in it, then rename the temp table so that it replaces the original one? There are a bunch of nova migrations that already work that way... Checkout the *sqlite* files in nova/db/sqlalchemy/migrate_repo/versions/ Why can't we do that with Keystone then? Is it too much work? It doesn't seem hard to do (just probably a bit annoying and boring ...). Does it represent too much work for the case of Keystone? Thomas ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [keystone] sqlite doesn't support migrations
On 15/07/13 09:26, Thomas Goirand wrote: Dolph, If you do that, then you will be breaking Debian packages, as they expect Sqlite as the default, for example when using DEBIAN_FRONTEND=noninteractive apt-get install keystone (if you choose MySQL, then you need to enter admin credentials to setup the db). I will receive tons of piupart failures reports if we can't upgrade with SQLite. I would really be disappointed if this happens, and get into situations where I have RC bugs which I can't realistically close by myself. So really, if it is possible, continue to support it, at least from one release to the next. Why not just change the default for Debian? Sqlite isn't particularly useful for actual deployments anyway. Cheers, -- Stephen Gran Senior Systems Integrator - guardian.co.uk Please consider the environment before printing this email. -- Visit guardian.co.uk - website of the year www.guardian.co.ukwww.observer.co.uk www.guardiannews.com www.guardian.co.uk/australia On your mobile, visit m.guardian.co.uk or download the Guardian iPhone app www.guardian.co.uk/iphone and iPad edition www.guardian.co.uk/iPad Save up to 32% by subscribing to the Guardian and Observer - choose the papers you want and get full digital access. Visit subscribe.guardian.co.uk - This e-mail and all attachments are confidential and may also be privileged. If you are not the named recipient, please notify the sender and delete the e-mail and all attachments immediately. Do not disclose the contents to another person. You may not use the information for any purpose, or store, or copy, it in any way. Guardian News Media Limited is not liable for any computer viruses or other material transmitted with or as part of this e-mail. You should employ virus checking software. Guardian News Media Limited A member of Guardian Media Group plc Registered Office PO Box 68164 Kings Place 90 York Way London N1P 2AP Registered in England Number 908396 ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [keystone] sqlite doesn't support migrations
On 07/15/2013 04:32 PM, Stephen Gran wrote: On 15/07/13 09:26, Thomas Goirand wrote: Dolph, If you do that, then you will be breaking Debian packages, as they expect Sqlite as the default, for example when using DEBIAN_FRONTEND=noninteractive apt-get install keystone (if you choose MySQL, then you need to enter admin credentials to setup the db). I will receive tons of piupart failures reports if we can't upgrade with SQLite. I would really be disappointed if this happens, and get into situations where I have RC bugs which I can't realistically close by myself. So really, if it is possible, continue to support it, at least from one release to the next. Why not just change the default for Debian? Sqlite isn't particularly useful for actual deployments anyway. Because that is the only backend that will work without providing credentials on the keyboard, so it is the only one that will work in a non-interactive session of apt-get (which is used for all automated tests in Debian, including piuparts). Thomas ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [keystone] sqlite doesn't support migrations
On 2013/07/13, Stephen Gran a wrote : On 15/07/13 10:46, Thomas Goirand wrote: On 07/15/2013 04:32 PM, Stephen Gran wrote: On 15/07/13 09:26, Thomas Goirand wrote: Dolph, If you do that, then you will be breaking Debian packages, as they expect Sqlite as the default, for example when using DEBIAN_FRONTEND=noninteractive apt-get install keystone (if you choose MySQL, then you need to enter admin credentials to setup the db). I will receive tons of piupart failures reports if we can't upgrade with SQLite. I would really be disappointed if this happens, and get into situations where I have RC bugs which I can't realistically close by myself. So really, if it is possible, continue to support it, at least from one release to the next. Why not just change the default for Debian? Sqlite isn't particularly useful for actual deployments anyway. Because that is the only backend that will work without providing credentials on the keyboard, so it is the only one that will work in a non-interactive session of apt-get (which is used for all automated tests in Debian, including piuparts). It strikes me that making the least useful option for users the default in order to pass a test suite is suboptimal. I'm sure this conversation would be better continued off list if you're interested. Cheers, -- Stephen Gran Senior Systems Integrator - guardian.co.uk I would have to agree here. If the case for using a suboptimal solution here is to pass tests another approach should be taken. If this is a legitimate issue, maybe we should look at what Neutron is doing, as they are using alembric for migrations already. Cheers, Morgan Fainberg -- Sent from my iPhone (please excuse the brevity and any typos) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [keystone] sqlite doesn't support migrations
On 07/15/2013 05:46 AM, Thomas Goirand wrote: On 07/15/2013 04:32 PM, Stephen Gran wrote: On 15/07/13 09:26, Thomas Goirand wrote: Dolph, If you do that, then you will be breaking Debian packages, as they expect Sqlite as the default, for example when using DEBIAN_FRONTEND=noninteractive apt-get install keystone (if you choose MySQL, then you need to enter admin credentials to setup the db). I will receive tons of piupart failures reports if we can't upgrade with SQLite. I would really be disappointed if this happens, and get into situations where I have RC bugs which I can't realistically close by myself. So really, if it is possible, continue to support it, at least from one release to the next. Why not just change the default for Debian? Sqlite isn't particularly useful for actual deployments anyway. Because that is the only backend that will work without providing credentials on the keyboard, so it is the only one that will work in a non-interactive session of apt-get (which is used for all automated tests in Debian, including piuparts). That is a really, really, really bad reason. Thomas ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [keystone] sqlite doesn't support migrations
On Sun, Jul 14, 2013 at 12:29 PM, Boris Pavlovic bo...@pavlovic.me wrote: Hi Dolph, Yes, I agree that there are some problems with sqlite and migrations. But I am not agree with approach of fully removing sqlite. It is pretty useful for testing. To be clear, I don't have any interest in dropping support for running keystone against sqlite as backend for development / testing. It's certainly useful for running functional tests in-memory. However, such databases don't need to be created by db_sync or using any migrations whatsoever. It's the migrations themselves where I want to drop support for sqlite. No one is migrating production data in a sqlite database. So the right approach that we are trying to provide in whole OpenStack is next: 1) Use alembic as migration tool (don't support sqlite in migrations) In my original email, I failed to cite where I got the idea (because I couldn't find a link, perhaps someone else knows of it?), but the same concern was expressed by some alembic-related documentation or blog post that I had recently read. I don't expect to see keystone making much progress towards switching to alembic until icehouse or even later. In the mean time, I'd like to do what we can to limit our pain while working with sqlaclhemy-migrate (which mostly involves writing entirely different code paths to handle sqlite's lack of support for proper schema evolution). 2) Test migrations in two ways: a) run all migrations with real data against supported backends (mysql, psql) b) test that models and migrations are synced in all backends (mysql, psql) +1 3) Unit tests should be run against DB that was created from models (not from migrations) +1 4) Unit tests should support all backends (sqlite also) +1 If you are interested in this sphere I could try to describe current state in more words. I think we're already in complete agreement! I just wasn't clear in my original email (apologies!). Best regards, Boris Pavlovic On Sat, Jul 13, 2013 at 12:19 AM, Monty Taylor mord...@inaugust.comwrote: On 07/11/2013 01:12 PM, Dolph Mathews wrote: Just as a general statement, outside the scope of openstack, I don't think sqlite is intended to support schema evolution. From the sqlite docs [1]: SQLite supports a limited subset of ALTER TABLE. [...] It is not possible to rename a column, remove a column, or add or remove constraints from a table. We've been through hell trying to support migrations on sqlite, because we test against sqlite, and because we test our migrations... on sqlite. So, we've already shot ourselves in the foot. We're clearly moving towards gating against mysql + postgresql, so in the mean time, let's limit the amount of effort we put into further support sqlite migrations until we can safely rip it out altogether. [1]: http://www.sqlite.org/lang_altertable.html I agree. The reason to use sqlite in unitests and stuff is because it's easy and doesn't require users and system things and everything. If we're spending extra effort to maintain the simple thing, then it's probably not a simple thing. As an aside, (ignore the fact that I'm a former Drizzle core dev) it might be worthwhile taking 30 minutes one day and exploring a drizzle database test fixture. One of the things we did in drizzle was make it not need any bootstrapping and to work sanely with no config files ... so launching a drizzle on a spare port, running database tests against it and then deleting it should actually be super simple - and at the worst no harder than doing what glance does in their functional tests. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- -Dolph ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [keystone] sqlite doesn't support migrations
On Fri, Jul 12, 2013 at 3:19 PM, Monty Taylor mord...@inaugust.com wrote: On 07/11/2013 01:12 PM, Dolph Mathews wrote: Just as a general statement, outside the scope of openstack, I don't think sqlite is intended to support schema evolution. From the sqlite docs [1]: SQLite supports a limited subset of ALTER TABLE. [...] It is not possible to rename a column, remove a column, or add or remove constraints from a table. We've been through hell trying to support migrations on sqlite, because we test against sqlite, and because we test our migrations... on sqlite. So, we've already shot ourselves in the foot. We're clearly moving towards gating against mysql + postgresql, so in the mean time, let's limit the amount of effort we put into further support sqlite migrations until we can safely rip it out altogether. [1]: http://www.sqlite.org/lang_altertable.html I agree. The reason to use sqlite in unitests and stuff is because it's easy and doesn't require users and system things and everything. If we're spending extra effort to maintain the simple thing, then it's probably not a simple thing. I agree that it's easy for unit functional testing. It's a simple solution and works fairly well, although I know we'd catch more issues if we ran our functional tests against a database that supported static typing, real booleans, etc. As an aside, (ignore the fact that I'm a former Drizzle core dev) it might be worthwhile taking 30 minutes one day and exploring a drizzle database test fixture. One of the things we did in drizzle was make it not need any bootstrapping and to work sanely with no config files ... so launching a drizzle on a spare port, running database tests against it and then deleting it should actually be super simple - and at the worst no harder than doing what glance does in their functional tests. That sounds like an viable improvement over sqlite in general... unfortunately, the drizzle site appears to be unmaintained? (at least at the moment) The documentation link [1] from here [2] returns a 404, and these docs [3] return a 403. Launchpad bug activity [4] doesn't seem particularly active either :-/ [1] http://docs.drizzle.org/ [2] http://www.drizzle.org/content/documentation [3] https://drizzle.readthedocs.org/en/latest/ [4] https://bugs.launchpad.net/drizzle/+bugs?orderby=-date_last_updatedsearch=Searchfield.status%3Alist=FIXCOMMITTEDfield.status%3Alist=FIXRELEASED ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- -Dolph ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [keystone] sqlite doesn't support migrations
On 07/15/2013 11:07 PM, Adam Young wrote: On 07/15/2013 05:46 AM, Thomas Goirand wrote: On 07/15/2013 04:32 PM, Stephen Gran wrote: On 15/07/13 09:26, Thomas Goirand wrote: Dolph, If you do that, then you will be breaking Debian packages, as they expect Sqlite as the default, for example when using DEBIAN_FRONTEND=noninteractive apt-get install keystone (if you choose MySQL, then you need to enter admin credentials to setup the db). I will receive tons of piupart failures reports if we can't upgrade with SQLite. I would really be disappointed if this happens, and get into situations where I have RC bugs which I can't realistically close by myself. So really, if it is possible, continue to support it, at least from one release to the next. Why not just change the default for Debian? Sqlite isn't particularly useful for actual deployments anyway. Because that is the only backend that will work without providing credentials on the keyboard, so it is the only one that will work in a non-interactive session of apt-get (which is used for all automated tests in Debian, including piuparts). That is a really, really, really bad reason. Ok, then, I think I didn't express myself correctly, so I will try again. In Debian, by policy, any package should be able to be installed using DEBIAN_FRONTEND=noninteractive apt-get install. What I do in my postinst is calling db_sync, because that isn't something our users should even care about, since it can be automated. The final result is that, for many package like Keystone and Glance, simply doing apt-get install is enough to make it work, without needing any configuration file edition. I want to be able to keep that nice feature. If we remove support for upgrading from one version to the next, then either I should remove the support for this, or make a special case for when sqlite is in use, and not setup any database in that case. Or the other option is to completely remove sqlite support (if we remove the possibility to upgrade, then I believe it should be done), and only do db_sync whenever the database is setup and working. That would also mean to not start the daemon either, in such a case. This removes really a lot of automated package testings, and I don't think it is a bad reason (don't we have a strong culture of automated testing inside the project?). If the support for SQLite (db upgrades) has to go, I will understand and adapt. I haven't and probably wont find the time for doing the actual work to support SQLite upgrades, and therefore, it probably is easier for me to state, though, I believe it is my duty to raise my concerns, and tell that I do not support this decision. What direction do you think this should take? Your thoughts? Cheers, Thomas Goirand (zigo) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [keystone] sqlite doesn't support migrations
On Mon, Jul 15, 2013 at 10:44 PM, Thomas Goirand z...@debian.org wrote: On 07/15/2013 11:07 PM, Adam Young wrote: On 07/15/2013 05:46 AM, Thomas Goirand wrote: On 07/15/2013 04:32 PM, Stephen Gran wrote: On 15/07/13 09:26, Thomas Goirand wrote: Dolph, If you do that, then you will be breaking Debian packages, as they expect Sqlite as the default, for example when using DEBIAN_FRONTEND=noninteractive apt-get install keystone (if you choose MySQL, then you need to enter admin credentials to setup the db). I will receive tons of piupart failures reports if we can't upgrade with SQLite. I would really be disappointed if this happens, and get into situations where I have RC bugs which I can't realistically close by myself. So really, if it is possible, continue to support it, at least from one release to the next. Why not just change the default for Debian? Sqlite isn't particularly useful for actual deployments anyway. Because that is the only backend that will work without providing credentials on the keyboard, so it is the only one that will work in a non-interactive session of apt-get (which is used for all automated tests in Debian, including piuparts). That is a really, really, really bad reason. Ok, then, I think I didn't express myself correctly, so I will try again. In Debian, by policy, any package should be able to be installed using DEBIAN_FRONTEND=noninteractive apt-get install. What I do in my postinst is calling db_sync, because that isn't something our users should even care about, since it can be automated. The final result is that, for many package like Keystone and Glance, simply doing apt-get install is enough to make it work, without needing any configuration file edition. I want to be able to keep that nice feature. Make it work is an entirely different goal than make a production-ready deployment. If your goal in using sqlite is just to make it work then I'm not sure that I would expect such an install to survive to the next release, anyway... rendering migration support as a nice-to-have. I can't imagine that any end users would be happy with a sqlite-based deployment for anything other than experimentation and testing. If we remove support for upgrading from one version to the next, then either I should remove the support for this, or make a special case for when sqlite is in use, and not setup any database in that case. As a parallel, we special case sqlite:///:memory: for testing purposes by automatically building the schema from models, without running migrations. Or the other option is to completely remove sqlite support (if we remove the possibility to upgrade, then I believe it should be done), and only do db_sync whenever the database is setup and working. That would also mean to not start the daemon either, in such a case. This removes really a lot of automated package testings, and I don't think it is a bad reason (don't we have a strong culture of automated testing inside the project?). If the support for SQLite (db upgrades) has to go, I will understand and adapt. I haven't and probably wont find the time for doing the actual work to support SQLite upgrades, and therefore, it probably is easier for me to state, though, I believe it is my duty to raise my concerns, and tell that I do not support this decision. I'm glad you spoke up, as I wasn't aware anyone was dependent on support for sqlite migrations outside of our own tests. Thanks! What direction do you think this should take? Your thoughts? I'd still like to pursue dropping support for sqlite migrations, albeit not as aggressively as I would have preferred. With a stakeholder, I think it's requisite to continue support through Havana. Perhaps at the fall summit we can evaluate our position on both alembic and sqlite migrations. Cheers, Thomas Goirand (zigo) ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- -Dolph ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [keystone] sqlite doesn't support migrations
On 07/11/2013 01:12 PM, Dolph Mathews wrote: Just as a general statement, outside the scope of openstack, I don't think sqlite is intended to support schema evolution. From the sqlite docs [1]: SQLite supports a limited subset of ALTER TABLE. [...] It is not possible to rename a column, remove a column, or add or remove constraints from a table. We've been through hell trying to support migrations on sqlite, because we test against sqlite, and because we test our migrations... on sqlite. So, we've already shot ourselves in the foot. We're clearly moving towards gating against mysql + postgresql, so in the mean time, let's limit the amount of effort we put into further support sqlite migrations until we can safely rip it out altogether. [1]: http://www.sqlite.org/lang_altertable.html I agree. The reason to use sqlite in unitests and stuff is because it's easy and doesn't require users and system things and everything. If we're spending extra effort to maintain the simple thing, then it's probably not a simple thing. As an aside, (ignore the fact that I'm a former Drizzle core dev) it might be worthwhile taking 30 minutes one day and exploring a drizzle database test fixture. One of the things we did in drizzle was make it not need any bootstrapping and to work sanely with no config files ... so launching a drizzle on a spare port, running database tests against it and then deleting it should actually be super simple - and at the worst no harder than doing what glance does in their functional tests. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
