Re: [openstack-dev] [Fuel] Testing DB migrations

2015-03-09 Thread Igor Kalnitsky
Hi, guys,

Indeed, it's a hot topic since it looks like there's no silver bullet
at all. As OpenStack community, we should move toward oslo.db
approach, but it may require hard effort from our side.

Meantime, as a part of bp/consume-external-ubuntu [1] I've prepared a
base class for testing migrations [2]. It's rough and could be
improved, but it works and I ask all contributors to test their
migrations from now on. You could use this testcase as an example [3].

[1]: https://blueprints.launchpad.net/fuel/+spec/consume-external-ubuntu
[2]: 
https://github.com/stackforge/fuel-web/blob/b1cb2f73c147c394fd6a7d91667f61859e6bc20a/nailgun/nailgun/test/base.py#L1125-L1146
[3]: 
https://github.com/stackforge/fuel-web/blob/b1cb2f73c147c394fd6a7d91667f61859e6bc20a/nailgun/nailgun/test/unit/test_migration_fuel_6_1.py#L23-L191

Thanks,
Igor

On Fri, Mar 6, 2015 at 5:15 PM, Roman Podoliaka rpodoly...@mirantis.com wrote:
 Hi all,

 You could take a look at how this is done in OpenStack projects [1][2]

 Most important parts:
 1) use the same RDBMS you use in production
 2) test migration scripts on data, not on empty schema
 3) test corner cases (adding a NOT NULL column without a server side
 default value, etc)
 4) do a separate migration scripts run with large data sets to make
 sure you don't introduce slow migrations [3]

 Thanks,
 Roman

 [1] 
 https://github.com/openstack/nova/blob/fb642be12ef4cd5ff9029d4dc71c7f5d5e50ce29/nova/tests/unit/db/test_migrations.py#L66-L833
 [2] 
 https://github.com/openstack/oslo.db/blob/0058c6510bfc6c41c830c38f3a30b5347a703478/oslo_db/sqlalchemy/test_migrations.py#L40-L273
 [3] 
 http://josh.people.rcbops.com/2013/12/third-party-testing-with-turbo-hipster/

 On Fri, Mar 6, 2015 at 4:50 PM, Nikolay Markov nmar...@mirantis.com wrote:
 We already run unit tests only using real Postgresql. But this still doesn't
 answer the question how we should test migrations.

 On Fri, Mar 6, 2015 at 5:24 PM, Boris Bobrov bbob...@mirantis.com wrote:

 On Friday 06 March 2015 16:57:19 Nikolay Markov wrote:
  Hi everybody,
 
  From time to time some bugs appear regarding failed database migrations
  during upgrade and we have High-priority bug for 6.1 (
  https://bugs.launchpad.net/fuel/+bug/1391553) on testing this migration
  process. I want to start a thread for discussing how we're going to do
  it.
 
  I don't see any obvious solution, but we can at least start adding tests
  together with any changes in migrations, which will use a number of
  various
  fake environments upgrading and downgrading DB.
 
  Any thoughts?

 In Kyestone adding unit tests and running them in in-memory sqlite was
 proven
 ineffective.The only solution we've come to is to run all db-related tests
 against real rdbmses.

 --
 Best regards,
 Boris Bobrov

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




 --
 Best regards,
 Nick Markov

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Fuel] Testing DB migrations

2015-03-06 Thread Boris Bobrov
On Friday 06 March 2015 16:57:19 Nikolay Markov wrote:
 Hi everybody,
 
 From time to time some bugs appear regarding failed database migrations
 during upgrade and we have High-priority bug for 6.1 (
 https://bugs.launchpad.net/fuel/+bug/1391553) on testing this migration
 process. I want to start a thread for discussing how we're going to do it.
 
 I don't see any obvious solution, but we can at least start adding tests
 together with any changes in migrations, which will use a number of various
 fake environments upgrading and downgrading DB.
 
 Any thoughts?

In Kyestone adding unit tests and running them in in-memory sqlite was proven 
ineffective.The only solution we've come to is to run all db-related tests 
against real rdbmses.

-- 
Best regards,
Boris Bobrov

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Fuel] Testing DB migrations

2015-03-06 Thread Nikolay Markov
We already run unit tests only using real Postgresql. But
this still doesn't answer the question how we should test migrations.

On Fri, Mar 6, 2015 at 5:24 PM, Boris Bobrov bbob...@mirantis.com wrote:

 On Friday 06 March 2015 16:57:19 Nikolay Markov wrote:
  Hi everybody,
 
  From time to time some bugs appear regarding failed database migrations
  during upgrade and we have High-priority bug for 6.1 (
  https://bugs.launchpad.net/fuel/+bug/1391553) on testing this migration
  process. I want to start a thread for discussing how we're going to do
 it.
 
  I don't see any obvious solution, but we can at least start adding tests
  together with any changes in migrations, which will use a number of
 various
  fake environments upgrading and downgrading DB.
 
  Any thoughts?

 In Kyestone adding unit tests and running them in in-memory sqlite was
 proven
 ineffective.The only solution we've come to is to run all db-related tests
 against real rdbmses.

 --
 Best regards,
 Boris Bobrov

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




-- 
Best regards,
Nick Markov
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [Fuel] Testing DB migrations

2015-03-06 Thread Roman Podoliaka
Hi all,

You could take a look at how this is done in OpenStack projects [1][2]

Most important parts:
1) use the same RDBMS you use in production
2) test migration scripts on data, not on empty schema
3) test corner cases (adding a NOT NULL column without a server side
default value, etc)
4) do a separate migration scripts run with large data sets to make
sure you don't introduce slow migrations [3]

Thanks,
Roman

[1] 
https://github.com/openstack/nova/blob/fb642be12ef4cd5ff9029d4dc71c7f5d5e50ce29/nova/tests/unit/db/test_migrations.py#L66-L833
[2] 
https://github.com/openstack/oslo.db/blob/0058c6510bfc6c41c830c38f3a30b5347a703478/oslo_db/sqlalchemy/test_migrations.py#L40-L273
[3] 
http://josh.people.rcbops.com/2013/12/third-party-testing-with-turbo-hipster/

On Fri, Mar 6, 2015 at 4:50 PM, Nikolay Markov nmar...@mirantis.com wrote:
 We already run unit tests only using real Postgresql. But this still doesn't
 answer the question how we should test migrations.

 On Fri, Mar 6, 2015 at 5:24 PM, Boris Bobrov bbob...@mirantis.com wrote:

 On Friday 06 March 2015 16:57:19 Nikolay Markov wrote:
  Hi everybody,
 
  From time to time some bugs appear regarding failed database migrations
  during upgrade and we have High-priority bug for 6.1 (
  https://bugs.launchpad.net/fuel/+bug/1391553) on testing this migration
  process. I want to start a thread for discussing how we're going to do
  it.
 
  I don't see any obvious solution, but we can at least start adding tests
  together with any changes in migrations, which will use a number of
  various
  fake environments upgrading and downgrading DB.
 
  Any thoughts?

 In Kyestone adding unit tests and running them in in-memory sqlite was
 proven
 ineffective.The only solution we've come to is to run all db-related tests
 against real rdbmses.

 --
 Best regards,
 Boris Bobrov

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




 --
 Best regards,
 Nick Markov

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev