Re: [openstack-dev] [release][oslo] oslotest release 1.7.0 (liberty)

2015-06-10 Thread Victor Sergeyev
Hi All,

this oslotest release break oslo.messaging gate - unittest fails
with ImportError, on import mox from six.moves - [0].
It caused by commit [1], which removed adding mox to six moves.

There is a fix for oslo.messaging - [2] - please, help to merge it.

[0] - http://paste.openstack.org/show/281091/
[1] -
https://github.com/openstack/oslotest/commit/9e0c8ad2c251274128499a7fcfb591c488d27d2b
[2] - https://review.openstack.org/190136

Thanks,
Victor

On Tue, Jun 9, 2015 at 6:14 PM, d...@doughellmann.com wrote:

 We are content to announce the release of:

 oslotest 1.7.0: Oslo test framework

 This release is part of the liberty release series.

 With source available at:

 http://git.openstack.org/cgit/openstack/oslotest

 For more details, please see the git log history below and:

 http://launchpad.net/oslotest/+milestone/1.7.0

 Please report issues through launchpad:

 http://bugs.launchpad.net/oslotest

 Changes in oslotest 1.6.0..1.7.0
 

 5bd9b31 Updated from global requirements
 ff9b38e Fix argument handling in oslo_run_cross_tests
 960e444 Add class to deal with clouds.yaml support
 1c0863f Remove unneeded runtime pbr dep
 5f2e635 Updated from global requirements
 f8a5a3c Advertise support for Python3.4 / Remove support for Python 3.3
 a063f25 Do not sync run_cross_tests.sh
 0782ab7 Remove unused discover dependency
 9e0c8ad Remove six.moves call

 Diffstat (except docs and test files)
 -

 openstack-common.conf  |  7 --
 oslotest/__init__.py   |  1 -
 oslotest/functional.py | 59
 ++
 oslotest/moxstubout.py |  2 +-
 requirements.txt   |  3 +--
 setup.cfg  |  6 +
 tox.ini|  3 +--
 8 files changed, 83 insertions(+), 30 deletions(-)


 Requirements updates
 

 diff --git a/requirements.txt b/requirements.txt
 index 674130c..1486ede 100644
 --- a/requirements.txt
 +++ b/requirements.txt
 @@ -5,2 +4,0 @@
 -pbr=0.6,!=0.7,1.0
 -discover
 @@ -14,0 +13 @@ mox3=0.7.0
 +os-client-config=1.2.0



 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [oslo] Adding Mehdi Abaakouk (sileht) to oslo-core

2015-05-07 Thread Victor Sergeyev
+1 from me

On Thu, May 7, 2015 at 5:52 PM, Jay Pipes jaypi...@gmail.com wrote:

 Note an oslo core, but big +1 from me.

 On 05/07/2015 10:36 AM, Davanum Srinivas wrote:

 Dear Oslo folks,

 I'd like to propose adding Mehdi Abaakouk to oslo-core. He is already
 leading the oslo.messaging team and helping with Tooz, and futurist
 efforts.

 I am hoping to get Mehdi more involved across the board in Oslo.

 Thanks,
 Dims


 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible

2015-04-22 Thread Victor Sergeyev
Hi, All,

My 2c are:

- yes, oslo.db supports python 3 (unittests passes, at least :) )
- MySQL-python still default MySQL DB driver in OpenStack, but at the
moment the only DB driver for MySQL in python3 environment is PyMySQL, so I
think, it's ok to use it with python 3.



On Wed, Apr 22, 2015 at 1:15 PM, Victor Stinner vstin...@redhat.com wrote:

 Hi,

 It's moving fast. I'm currently working on porting remaining libraries to
 prepare my spec for nova.

  oslo.db -- looks like it is almost there

 I don't know the status of oslo.db support of Python 3. I guess that it
 already works on Python 3, it's just a matter of running tests with MySQL
 (MySQL-Python blocks again here).

  oslo.messaging -- same

 Two changes of my Python 3 changes were already merged last week. Three
 remaining Python 3 changes are almost there, they are mostly blocked by the
 freeze on requirements, but changes are already approved. The blocker is
 the bump of eventlet to 0.17.3:
 https://review.openstack.org/#/c/172132/

  paste -- almost there

 I fixed that last night (!) with the release of Paste 2.0. It's the first
 Paste release since 2010. Paste 2.0 has an experimental support of Python
 3. It should be enough for nova, according to my quick tests in my local
 nova repository where I fixed most obvious Python 3 issues. Ian Bicking
 allowed me to publish new versions of Paste.

  sqlalchemy-migrate -- almost there

 It already supports Python 3, so it doesn't block.

  suds -- with the suds fork shouldn't be too hard

 You should just switch to the fork. I contacted the author of suds-jurko:
 he plans to rename its component from suds-jurko to suds. So his fork
 would be simple drop-in which would not require any change in OpenStack
 except of requirements.

  websockify -- unknown

 It announces Python 3.3 and 3.4 support and has a py34 env in tox.ini. I
 didn't check yet if it supports Python 3, but since the project is active
 (last commit 12 hours ago), it's shouldn't be hard to fix them quickly.

  libvirt-python -- unknown

 Oh, I missed this dependency. It's my next target :-)

  mysql-python -- alternitive looks viable.

 Yes, it's possible to replace it with PyMySQL. Does anyone know the status
 of this switch?

  Based on there being two unknowns, and a lot of dependencies that are
 just almost there, and a few we may want to migrate off of, I was assuming
 addressing those issues would make it hard for us to make nova python3
 compatible for Liberty.

 My plan for Liberty is not to support fully Python 3 in nova, but only
 start the port (well, continue to be exact). The idea is to fix syntax
 errors and obvious issues like dict.iteritems() = six.iteritems(dict),
 then more complex changes. Maybe it's possible to finish the port in a
 cycle, but it really depends on the time taken to merge patches.

 I started to port nova in my local repository. Some unit tests already
 pass.

 nova already uses six in many places, so it's not like we really start
 from scratch, the port is already an ingoing process.

 Victor

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [oslo] eventlet 0.17.3 is now fully Python 3 compatible

2015-04-09 Thread Victor Sergeyev
Thanks for your work on this! :)

On Thu, Apr 9, 2015 at 7:25 PM, Victor Stinner vstin...@redhat.com wrote:

 Hi,

 During the last OpenStack Summit at Paris, we discussed how we can port
 OpenStack to Python 3, because eventlet was not compatible with Python 3.
 There are multiple approaches: port eventlet to Python 3, replace eventlet
 with asyncio, replace eventlet with threads, etc. We decided to not take a
 decision and instead investigate all options.

 I fixed 4 issues with monkey-patching in Python 3 (importlib, os.open(),
 threading.RLock, threading.Thread). Good news: the just released eventlet
 0.17.3 includes these fixes and it is now fully compatible with Python 3!
 For example, the Oslo Messaging test suite now pass with this eventlet
 version! Currently, eventlet is disabled in Oslo Messaging on Python 3
 (eventlet tests are skipped).

 I just sent a patch for requirements and Oslo Messaging to bump to
 eventlet 0.17.3, but it will have to wait until everyone has master as
 Liberty.

https://review.openstack.org/#/c/172132/
https://review.openstack.org/#/c/172135/

 It becomes possible to port more projects depending on eventlet to Python
 3!

 Liberty cycle will be a good opportunity to port more OpenStack components
 to Python 3. Most OpenStack clients and Common Libraries are *already*
 Python 3 compatible, see the wiki page:

https://wiki.openstack.org/wiki/Python3

 --

 To replace eventlet, I wrote a spec to replace it with asyncio:

https://review.openstack.org/#/c/153298/

 Joshua Harlow wrote a spec to replace eventlet with threads:

https://review.openstack.org/#/c/156711/

 But then he wrote a single spec Replace eventlet + monkey-patching with
 ?? which covers threads and asyncio:

https://review.openstack.org/#/c/164035/

 Victor

 __
 OpenStack Development Mailing List (not for usage questions)
 Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] olso.db 1.5.0 release

2015-02-26 Thread Victor Sergeyev
Hi folks!

The Oslo team is pleased to announce the release of: oslo.db - OpenStack
common DB library

Changes from the previous release:

$ git log --oneline --no-merges 1.4.1..1.5.0
7bfdb6a Make DBAPI class work with mocks correctly
96cabf4 Updated from global requirements
a3a1bdd Imported Translations from Transifex
ab20754 Fix PyMySQL reference error detection
99e2ab6 Updated from global requirements
6ccea34 Organize provisioning to use testresources
eeb7ea2 Add retry decorator allowing to retry DB operations on request
d78e3aa Imported Translations from Transifex
dcd137a Implement backend-specific drop_all_objects for provisioning.
3fb5098 Refactor database migration manager to use given engine
afcc3df Fix 0 version handling in migration_cli manager
f81653b Updated from global requirements
efdefa9 Fix PatchStacktraceTest for pypy
c0a4373 Update Oslo imports to remove namespace package
1b7c295 Retry query if db deadlock error is received
046e576 Ensure DBConnectionError is raised on failed revalidate


Please report issues through launchpad:
 http://bugs.launchpad.net/oslo.db
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] oslo.db 1.4.1 release

2015-01-13 Thread Victor Sergeyev
Hello folks!

The Oslo team is pleased to announce the release of oslo.db - 1.4.1

This is a bugfix release to address a problem found in the 1.4.0 release -
 oslo.db API change break neutron functional tests

Change in openstack/oslo.db  1.4.0..1.4.1

commit b1fc55c7ce6004311379f4002fdceddcc8da9784
Author: Mike Bayer mike...@zzzcomputing.com
Date:   Mon Jan 12 17:26:23 2015 -0500

Restore the check_foreign_keys() method.

This method was prematurely removed from oslo.db without
a deprecation phase, when Alembic added support for
foreign key autodetection.   oslo.db continues to use
alembic for this purpose, however the
ModelsMigrationsSync.check_foreign_keys() method is restored
directly for those projects which were calling into it
explicitly.

diffstat
 oslo_db/sqlalchemy/test_migrations.py   |  72 +
 oslo_db/tests/sqlalchemy/test_migrations.py | 187 +


Please report issues through launchpad:
 http://bugs.launchpad.net/oslo.db
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] oslo.db 1.2.0 released

2014-12-03 Thread Victor Sergeyev
Hello Folks

The Oslo team is pleased to announce the release of oslo.db 1.2.0.

This release includes several bug fixes as well as many other changes:

$ git log --abbrev-commit --pretty=oneline --no-merges 1.1.0..1.2.0
f740e3b Imported Translations from Transifex
ca1ad56 Make test_models pass on py3
549ba15 Repair include_object to accommodate new objects
10e8d15 Add table name to foreign keys diff
ddd11df Updated from global requirements
2269848 Handle Galera deadlock on SELECT FOR UPDATE
4b2058b Add exception filter for _sqlite_dupe_key_error
b6d363d Add info on how to run unit tests
7f755bf Ensure is_backend_avail() doesn't leave open connections
c54d3a9 Updated from global requirements
2099177 Add pbr to installation requirements
135701b Fix python3.x scoping issues with removed 'de' variable

Thanks Doug Hellmann, Joshua Harlow, Mike Bayer, Oleksii Chuprykov, Roman
Podoliaka for contributing to this release.

For more details, please see the git log history below and
https://launchpad.net/oslo.db/+milestone/1.2.0
Please report issues through launchpad: https://launchpad.net/oslo.db


  diffstat (except docs and test files):

 CONTRIBUTING.rst   | 39 +++
 .../locale/en_GB/LC_MESSAGES/oslo.db-log-info.po   |  8 ++--
 oslo.db/locale/fr/LC_MESSAGES/oslo.db-log-error.po | 56
++
 oslo.db/locale/fr/LC_MESSAGES/oslo.db-log-info.po  | 33 +
 .../locale/fr/LC_MESSAGES/oslo.db-log-warning.po   | 40 
 oslo/db/sqlalchemy/exc_filters.py  | 21 ++--
 oslo/db/sqlalchemy/session.py  | 10 ++--
 oslo/db/sqlalchemy/test_migrations.py  |  2 +-
 oslo/db/sqlalchemy/utils.py|  3 +-
 requirements.txt   |  1 +
 test-requirements-py2.txt  |  2 +-
 test-requirements-py3.txt  |  2 +-
 tests/sqlalchemy/test_exc_filters.py   | 13 +
 tests/sqlalchemy/test_migrations.py|  7 ++-
 tests/sqlalchemy/test_models.py| 31 +---
 15 files changed, 233 insertions(+), 35 deletions(-)

  Requirements updates:

 diff --git a/requirements.txt b/requirements.txt
 index cc50660..f8a0d8c 100644
 --- a/requirements.txt
 +++ b/requirements.txt
 @@ -4,0 +5 @@
 +pbr=0.6,!=0.7,1.0
 diff --git a/test-requirements-py2.txt b/test-requirements-py2.txt
 index 13cea90..ac5c18a 100644
 --- a/test-requirements-py2.txt
 +++ b/test-requirements-py2.txt
 @@ -19 +19 @@ testscenarios=0.4
 -testtools=0.9.36
 +testtools=0.9.36,!=1.2.0
 diff --git a/test-requirements-py3.txt b/test-requirements-py3.txt
 index 4f195da..58b9a3d 100644
 --- a/test-requirements-py3.txt
 +++ b/test-requirements-py3.txt
 @@ -18 +18 @@ testscenarios=0.4
 -testtools=0.9.36
 +testtools=0.9.36,!=1.2.0
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] oslo.db 1.1.0 released

2014-11-18 Thread Victor Sergeyev
Matt,

As for race in Nova - it caused by deprecated is_backend_avail() function,
which calls _ensure_backenv_available() method, which creates a SQLAlchemy
engine and opens a test connection, but doesn't call engine.dispose().
Depending on Python interpreter version, this connection may remain open
for some time.

So there are such ways to fix Nova:
- wait for oslo.db 1.1.1 which will include fix for this method - see patch
[1]
- remove is_backend_avail() helper usage in Nova - patch [2] refactor Nova
opportunistic DB tests and remove that method.

[1] https://review.openstack.org/#/c/135293/
[2] https://review.openstack.org/#/c/103920/


On Tue, Nov 18, 2014 at 5:22 AM, Matt Riedemann mrie...@linux.vnet.ibm.com
wrote:



 On 11/17/2014 9:36 AM, Victor Sergeyev wrote:

 Hello All!

 Oslo team is pleased to announce the new release of Oslo database
 handling library - oslo.db 1.1.0

 List of changes:
 $ git log --oneline --no-merges  1.0.2..master
 1b0c2b1 Imported Translations from Transifex
 9aa02f4 Updated from global requirements
 766ff5e Activate pep8 check that _ is imported
 f99e1b5 Assert exceptions based on API, not string messages
 490f644 Updated from global requirements
 8bb12c0 Updated from global requirements
 4e19870 Reorganize DbTestCase to use provisioning completely
 2a6dbcd Set utf8 encoding for mysql and postgresql
 1b41056 ModelsMigrationsSync: Add check for foreign keys
 8fb696e Updated from global requirements
 ba4a881 Remove extraneous vim editor configuration comments
 33011a5 Remove utils.drop_unique_constraint()
 64f6062 Improve error reporting for backend import failures
 01a54cc Ensure create_engine() retries the initial connection test
 26ec2fc Imported Translations from Transifex
 9129545 Use fixture from oslo.config instead of oslo-incubator
 2285310 Move begin ping listener to a connect listener
 7f9f4f1 Create a nested helper function that will work on py3.x
 b42d8f1 Imported Translations from Transifex
 4fa3350 Start adding a environment for py34/py33
 b09ee9a Explicitly depend on six in requirements file
 7a3e091 Unwrap DialectFunctionDispatcher from itself.
 0928d73 Updated from global requirements
 696f3c1 Use six.wraps instead of functools.wraps
 8fac4c7 Update help string to use database
 fc8eb62 Use __qualname__ if we can
 6a664b9 Add description for test_models_sync function
 8bc1fb7 Use the six provided iterator mix-in
 436dfdc ModelsMigrationsSync:add correct server_default check for Enum
 2075074 Add history/changelog to docs
 c9e5fdf Add run_cross_tests.sh script

 Thanks Andreas Jaeger, Ann Kamyshnikova, Christian Berendt, Davanum
 Srinivas, Doug Hellmann, Ihar Hrachyshka, James Carey, Joshua Harlow,
 Mike Bayer, Oleksii Chuprykov, Roman Podoliaka for contributing to this
 release.

 Please report issues to the bug tracker: https://bugs.launchpad.net/
 oslo.db


 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


 And...the nova postgresql opportunistic DB tests are failing quite
 frequently due to some race introduced by the new library version [1].

 [1] https://bugs.launchpad.net/oslo.db/+bug/1393633

 --

 Thanks,

 Matt Riedemann



 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] oslo.db 1.1.0 released

2014-11-18 Thread Victor Sergeyev
Well, we can re-define is_backend_avail() in Nova code and use it instead
of oslo.db's function. it looks a bit ugly but it will be fast

 Also can I request that when deprecating methods in oslo libraries we
 use a standard deprecation mechanism so that warnings are emitted when
 this method is used. I didn't find anything in our unit tests logs. It

ok, will suggest

 It is odd that it exposed after the release (and not before), any idea
 which oslo.db change impacted this?

This bug depends fro python version so we haven't catch it locally :(



On Tue, Nov 18, 2014 at 6:47 PM, Sean Dague s...@dague.net wrote:

 Is there a more minimal version of
 https://review.openstack.org/#/c/103920/ that *just* fixes this issue.

 So we can evaluate the refactor on it's own, but get the bug fixed more
 immediately.

 Also can I request that when deprecating methods in oslo libraries we
 use a standard deprecation mechanism so that warnings are emitted when
 this method is used. I didn't find anything in our unit tests logs. It
 would be helpful to keep us ahead of this in the future.

 It is odd that it exposed after the release (and not before), any idea
 which oslo.db change impacted this?

 -Sean

 On 11/18/2014 11:17 AM, Victor Sergeyev wrote:
  Matt,
 
  As for race in Nova - it caused by deprecated is_backend_avail()
  function, which calls _ensure_backenv_available() method, which creates
  a SQLAlchemy engine and opens a test connection, but doesn't call
  engine.dispose(). Depending on Python interpreter version, this
  connection may remain open for some time.
 
  So there are such ways to fix Nova:
  - wait for oslo.db 1.1.1 which will include fix for this method - see
  patch [1]
  - remove is_backend_avail() helper usage in Nova - patch [2] refactor
  Nova opportunistic DB tests and remove that method.
 
  [1] https://review.openstack.org/#/c/135293/
  [2] https://review.openstack.org/#/c/103920/
 
 
  On Tue, Nov 18, 2014 at 5:22 AM, Matt Riedemann
  mrie...@linux.vnet.ibm.com mailto:mrie...@linux.vnet.ibm.com wrote:
 
 
 
  On 11/17/2014 9:36 AM, Victor Sergeyev wrote:
 
  Hello All!
 
  Oslo team is pleased to announce the new release of Oslo database
  handling library - oslo.db 1.1.0
 
  List of changes:
  $ git log --oneline --no-merges  1.0.2..master
  1b0c2b1 Imported Translations from Transifex
  9aa02f4 Updated from global requirements
  766ff5e Activate pep8 check that _ is imported
  f99e1b5 Assert exceptions based on API, not string messages
  490f644 Updated from global requirements
  8bb12c0 Updated from global requirements
  4e19870 Reorganize DbTestCase to use provisioning completely
  2a6dbcd Set utf8 encoding for mysql and postgresql
  1b41056 ModelsMigrationsSync: Add check for foreign keys
  8fb696e Updated from global requirements
  ba4a881 Remove extraneous vim editor configuration comments
  33011a5 Remove utils.drop_unique_constraint()
  64f6062 Improve error reporting for backend import failures
  01a54cc Ensure create_engine() retries the initial connection
 test
  26ec2fc Imported Translations from Transifex
  9129545 Use fixture from oslo.config instead of oslo-incubator
  2285310 Move begin ping listener to a connect listener
  7f9f4f1 Create a nested helper function that will work on py3.x
  b42d8f1 Imported Translations from Transifex
  4fa3350 Start adding a environment for py34/py33
  b09ee9a Explicitly depend on six in requirements file
  7a3e091 Unwrap DialectFunctionDispatcher from itself.
  0928d73 Updated from global requirements
  696f3c1 Use six.wraps instead of functools.wraps
  8fac4c7 Update help string to use database
  fc8eb62 Use __qualname__ if we can
  6a664b9 Add description for test_models_sync function
  8bc1fb7 Use the six provided iterator mix-in
  436dfdc ModelsMigrationsSync:add correct server_default check
  for Enum
  2075074 Add history/changelog to docs
  c9e5fdf Add run_cross_tests.sh script
 
  Thanks Andreas Jaeger, Ann Kamyshnikova, Christian Berendt,
 Davanum
  Srinivas, Doug Hellmann, Ihar Hrachyshka, James Carey, Joshua
  Harlow,
  Mike Bayer, Oleksii Chuprykov, Roman Podoliaka for contributing
  to this
  release.
 
  Please report issues to the bug tracker:
  https://bugs.launchpad.net/__oslo.db
  https://bugs.launchpad.net/oslo.db
 
 
  _
  OpenStack-dev mailing list
  OpenStack-dev@lists.openstack.__org
  mailto:OpenStack-dev@lists.openstack.org
 
 http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
  
 http://lists.openstack.org/cgi-bin/mailman/listinfo

[openstack-dev] oslo.db 1.1.0 released

2014-11-17 Thread Victor Sergeyev
Hello All!

Oslo team is pleased to announce the new release of Oslo database handling
library - oslo.db 1.1.0

List of changes:
$ git log --oneline --no-merges  1.0.2..master
1b0c2b1 Imported Translations from Transifex
9aa02f4 Updated from global requirements
766ff5e Activate pep8 check that _ is imported
f99e1b5 Assert exceptions based on API, not string messages
490f644 Updated from global requirements
8bb12c0 Updated from global requirements
4e19870 Reorganize DbTestCase to use provisioning completely
2a6dbcd Set utf8 encoding for mysql and postgresql
1b41056 ModelsMigrationsSync: Add check for foreign keys
8fb696e Updated from global requirements
ba4a881 Remove extraneous vim editor configuration comments
33011a5 Remove utils.drop_unique_constraint()
64f6062 Improve error reporting for backend import failures
01a54cc Ensure create_engine() retries the initial connection test
26ec2fc Imported Translations from Transifex
9129545 Use fixture from oslo.config instead of oslo-incubator
2285310 Move begin ping listener to a connect listener
7f9f4f1 Create a nested helper function that will work on py3.x
b42d8f1 Imported Translations from Transifex
4fa3350 Start adding a environment for py34/py33
b09ee9a Explicitly depend on six in requirements file
7a3e091 Unwrap DialectFunctionDispatcher from itself.
0928d73 Updated from global requirements
696f3c1 Use six.wraps instead of functools.wraps
8fac4c7 Update help string to use database
fc8eb62 Use __qualname__ if we can
6a664b9 Add description for test_models_sync function
8bc1fb7 Use the six provided iterator mix-in
436dfdc ModelsMigrationsSync:add correct server_default check for Enum
2075074 Add history/changelog to docs
c9e5fdf Add run_cross_tests.sh script

Thanks Andreas Jaeger, Ann Kamyshnikova, Christian Berendt, Davanum
Srinivas, Doug Hellmann, Ihar Hrachyshka, James Carey, Joshua Harlow, Mike
Bayer, Oleksii Chuprykov, Roman Podoliaka for contributing to this release.

Please report issues to the bug tracker: https://bugs.launchpad.net/oslo.db
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [oslo][all] Alembic migrations for SQLite

2014-11-12 Thread Victor Sergeyev
Great job, Mike, thanks!

Doug, as for migration from sqlalchemy-migrate to Alembic - at the moment
it's hard to realize this because we suppose to keep backward compatibility
with available migrations. I'd like to re-review existing approach with
Mike and Roman, create road-map for this migration, and try to migrate one
of the projects as Proof-Of-Concept.

On Tue, Nov 11, 2014 at 5:58 PM, Doug Hellmann d...@doughellmann.com
wrote:


 On Nov 10, 2014, at 10:55 AM, Mike Bayer mba...@redhat.com wrote:

  Bonjour openstackers -
 
  While you were all sipping champagne on the Champs-Élysées, I took some
 time to tackle one of the two most critically wanted features in Alembic,
 which is that of being able to migrate tables on a SQLite database with
 some degree of sanity.   My immediate focus on Alembic was spurred on
 partially because some changes to the test suite pushed it into 0.7.0, and
 then we got a very large number of bug fixes in, so the urgency to get
 0.7.0 is relatively high; but what good is a pseudo-major point release
 without some big new features ?
 
  The feature works similarly to that of SQLAlchemy-Migrate, but I’m
 hoping in a way that is more controllable and open-ended.   I would
 encourage all those interested to take a look at the basic mode of
 operation over at
 http://alembic.readthedocs.org/en/latest/tutorial.html#running-batch-migrations-for-sqlite-and-other-databases.
  Highlights include that several table operations can take place within one
 “move and copy” operation at once, and the system can also be applied to
 other databases if one so desired (not a common use case but some have
 expressed interest in this being possible…so it is!).   The format of a
 SQLite-compatible migration script will change slightly, though for the
 better as per-table operations are grouped together, and the scripts of
 course in the default case continue to work exactly as before on all other
 target databases.

 This sounds great, Mike, and I’m sure it will make it much easier for us
 to use sqlite databases for unit/functional tests.

 
  I know that a handful of projects have moved or started with Alembic,
 and I’d like to continue pushing Alembic to be the single solution across
 all projects.  There’s some work in oslo.db to define a common
 environmental format as well (see https://review.openstack.org/#/c/112842/).
 I would encourage projects to continue to evaluate moving their migrations
 over to Alembic at some point in the future, which should also include
 sending me emails/ircs/bug reports about what additional features/fixes are
 needed.

 I want to stress this. We ran into issues with sqlalchemy-migrate during
 Juno, and we’re having an increasingly difficult time finding reviewers
 even though OpenStack (not Oslo) has adopted the library. The Oslo team is
 working on improving Alembic support with the intent to deprecate support
 for sqlalchemy-migrate in the “near future” (not Kilo). Please work with
 Mike, Viktor, Roman, and the rest of the oslo.db team to identify any holes
 that would delay your ability to start using alembic for new migrations in
 the next release cycle.

 
  The next major feature for Alembic, which I will tentatively use this
 week to see if I can get it online, is the multiple heads/branch resolution
 feature (
 https://bitbucket.org/zzzeek/alembic/issue/167/multiple-heads-branch-resolution-support)
 which a *lot* of people are really asking for.   This feature would allow
 independent migration series to coexist simultaneously, as well “merge
 point” migrations that join disparate branches back into a single stream.
  The risk level for this feature is significantly higher than that of the
 SQLite migration feature, as while the SQLite migration feature lives
 entirely within a new API that is otherwise unused, the multiple branch
 feature makes some fundamental changes to how versioning is performed.   So
 while I’d like to get this in 0.7.0 as well, if it gets into the weeds I
 may have to push 0.7.0 without it as there’s really a crapload of other
 fixes to be pushed.
 
  Thanks for reading!
 
  - mike
 
 
 
 
  ___
  OpenStack-dev mailing list
  OpenStack-dev@lists.openstack.org
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] oslo.db 0.5.0 released

2014-09-16 Thread Victor Sergeyev
Hello All!

Oslo team is pleased to announce the new Oslo database handling library
release - oslo.db 0.5.0

List of changes:

$ git log --oneline --no-merges 0.4.0..0.5.0
c785bee Updated from global requirements
ac05c2a Imported Translations from Transifex
57f499e Add a check for SQLite transactional state
bdcfb4a Let oslotest manage the six.move setting for mox
4ed63fa Fix DBReferenceError on MySQL and SQLite
49ae6c0 Renaming in WalkVersionsMixin
cae1202 Clean up documentation
fb577ec Use single quotes for db schema sanity check
ab47516 warn against sorting requirements
be6f9a0 ModelsMigrationsSync:Override compare_server_default
e187a4c Updated from global requirements
cbf995e Imported Translations from Transifex
f2f0960 Add doc8 to tox environment docs
92d9a1c Use oslo.i18n
78fd290 Repair pysqlite transaction support
41dada1 Extract logging setup into a separate function
7c57798 Updated from global requirements
977869d Remove reliance on create_engine() from TestsExceptionFilter
626d762 Consolidate sqlite and mysql event listeners
5ab43ee Use dialect dispatch for engine initiailization.
4683475 Add get_non_innodb_tables() to utils
686005e Added check to see whether oslotest is installed

Please report issues to the bug tracker: https://bugs.launchpad.net/oslo.db
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [oslo][db] Nominating Mike Bayer for the oslo.db core reviewers team

2014-08-20 Thread Victor Sergeyev
Sure, here is my +1

Folks, it looks like, that there is no no objection to this proposal.

Welcome to the team, Mike!

On Tue, Aug 19, 2014 at 12:02 PM, Flavio Percoco fla...@redhat.com wrote:

 On 08/15/2014 10:26 PM, Doug Hellmann wrote:
 
  On Aug 15, 2014, at 10:00 AM, Ben Nemec openst...@nemebean.com wrote:
 
  On 08/15/2014 08:20 AM, Russell Bryant wrote:
  On 08/15/2014 09:13 AM, Jay Pipes wrote:
  On 08/15/2014 04:21 AM, Roman Podoliaka wrote:
  Hi Oslo team,
 
  I propose that we add Mike Bayer (zzzeek) to the oslo.db core
  reviewers team.
 
  Mike is an author of SQLAlchemy, Alembic, Mako Templates and some
  other stuff we use in OpenStack. Mike has been working on OpenStack
  for a few months contributing a lot of good patches and code reviews
  to oslo.db [1]. He has also been revising the db patterns in our
  projects and prepared a plan how to solve some of the problems we
 have
  [2].
 
  I think, Mike would be a good addition to the team.
 
  Uhm, yeah... +10 :)
 
  ^2 :-)
 
 
  What took us so long to do this? :-)
 
  +1 obviously.
 
  I did think it would be a good idea to wait a *little* while and make
 sure we weren’t going to scare him off. ;-)
 
  Seriously, Mike has been doing a great job collaborating with the
 existing team and helping us make oslo.db sane.
 
  +1


 Big +1


 --
 @flaper87
 Flavio Percoco

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] oslo.db 0.4.0 released

2014-08-20 Thread Victor Sergeyev
Hello Folks!

Oslo team is pleased to announce the new Oslo database handling library
release - oslo.db 0.4.0
Thanks all for contributions to this release.

Feel free to report issues using the launchpad tracker:
https://bugs.launchpad.net/oslo and mark them with ``db`` tag.

See the full list of changes:

$ git log --oneline --no-merges 0.3.0..0.4.0
ee176a8 Implement a dialect-level function dispatch system
6065b21 Move to oslo.utils
deeda38 Restore correct source file encodings
4dde38b Handle DB2 SmallInteger type for
change_deleted_column_type_to_boolean
4c18fca Imported Translations from Transifex
69f16bf Fixes comments to pass E265 check.
e1dbd31 Fixes indentations to pass E128 check.
423c17e Uses keyword params for i18n string to pass H703
3cb5927 Adds empty line to multilines docs to pass H405
0996c5d Updates one line docstring with dot to pass H402
a3ca010 Changes import orders to pass H305 check
584a883 Fixed DeprecationWarning in exc_filters
fc2fc90 Imported Translations from Transifex
3b17365 oslo.db.exceptions module documentation
c919585 Updated from global requirements
4685631 Extension of DBDuplicateEntry exception
7cb512c oslo.db.options module documentation
c0d9f36 oslo.db.api module documentation
93d95d4 Imported Translations from Transifex
e83e4ca Use SQLAlchemy cursor execute events for tracing
d845a16 Remove sqla_07 from tox.ini
9722ab6 Updated from global requirements
3bf8941 Specify raise_on_warnings=False for mysqlconnector
1814bf8 Make MySQL regexes generic across MySQL drivers
62729fb Allow tox tests with complex OS_TEST_DBAPI_CONNECTION URLs
a9e3af2 Raise DBReferenceError on foreign key violation
b69899e Add host argument to get_connect_string()
9a6aa50 Imported Translations from Transifex
f817555 Don't drop pre-existing database before tests
4499da7 Port _is_db_connection_error check to exception filters
9d5ab2a Integrate the ping listener into the filter system.
cbae81e Add disconnect modification support to exception handling
0a6c8a8 Implement new exception interception and filtering layer
69a4a03 Implement the SQLAlchemy ``handle_error()`` event.
f96deb8 Remove moxstubout.py from oslo.db
7d78e3e Added check for DB2 deadlock error
2df7e88 Bump hacking to version 0.9.2
c34c32e Opportunistic migration tests
108e2bd Move all db exception to exception.py
35afdf1 Enable skipped tests from test_models.py
e68a53b Use explicit loops instead of list comprehensions
44e96a8 Imported Translations from Transifex
817fd44 Allow usage of several iterators on ModelBase
baf30bf Add DBDuplicateEntry detection for mysqlconnector driver
4796d06 Check for mysql_sql_mode is not None in create_engine()
01b916c remove definitions of Python Source Code Encoding

Thanks,
Victor
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [neutron][all] switch from mysqldb to another eventlet aware mysql client

2014-08-18 Thread Victor Sergeyev
Hello Doug, All.

 This release is currently blocked on landing some changes in projects
using the library so they don’t break when the new version starts using
different exception classes. We’re tracking that work in
https://etherpad.openstack.org/p/sqla_exceptions_caught

 It looks like we’re down to 2 patches, one for cinder (
https://review.openstack.org/#/c/111760/) and one for glance (
https://review.openstack.org/#/c/109655).

At the moment these patches are merged, so the exception issue was fixed in
all core OS projects.

But unfortunately, there is another blocker for the oslo.db release - Heat
uses BaseMigrationTestCase class, which was removed from oslo.db in patch
https://review.openstack.org/#/c/93424/ , so the new oslo.db release will
break unittests in Heat. Here is the patch, which should fix this issue -
https://review.openstack.org/#/c/109658/

I really hope, that this patch is the last release blocker :)
Roman, folks - please fix me, if I miss something



On Fri, Aug 15, 2014 at 11:07 PM, Doug Hellmann d...@doughellmann.com
wrote:


 On Aug 15, 2014, at 9:29 AM, Ihar Hrachyshka ihrac...@redhat.com wrote:

  Signed PGP part
  Some updates on the matter:
 
  - oslo-spec was approved with narrowed scope which is now 'enabled
  mysqlconnector as an alternative in gate' instead of 'switch the
  default db driver to mysqlconnector'. We'll revisit the switch part
  the next cycle once we have the new driver running in gate and real
  benchmarking is heavy-lifted.
 
  - there are several patches that are needed to make devstack and
  tempest passing deployment and testing. Those are collected under the
  hood of: https://review.openstack.org/#/c/114207/ Not much of them.
 
  - we'll need a new oslo.db release to bump versions (this is needed to
  set raise_on_warnings=False for the new driver, which was incorrectly
  set to True in sqlalchemy till very recently). This is expected to be
  released this month (as per Roman Podoliaka).

 This release is currently blocked on landing some changes in projects
 using the library so they don’t break when the new version starts using
 different exception classes. We’re tracking that work in
 https://etherpad.openstack.org/p/sqla_exceptions_caught

 It looks like we’re down to 2 patches, one for cinder (
 https://review.openstack.org/#/c/111760/) and one for glance (
 https://review.openstack.org/#/c/109655). Roman, can you verify that
 those are the only two projects that need changes for the exception issue?

 
  - once the corresponding patch for sqlalchemy-migrate is merged, we'll
  also need a new version released for this.
 
  - on PyPI side, no news for now. The last time I've heard from Geert
  (the maintainer of MySQL Connector for Python), he was working on
  this. I suspect there are some legal considerations running inside
  Oracle. I'll update once I know more about that.

 If we don’t have the new package on PyPI, how do we plan to include it in
 the gate? Are there options to allow an exception, or to make the mirroring
 software download it anyway?

 Doug

 
  - once all the relevant patches land in affected projects and
  devstack, I'm going to introduce a separate gate job to run against
  mysqlconnector.
 
  Cheers,
  /Ihar
 
  On 22/07/14 15:03, Ihar Hrachyshka wrote:
   FYI: I've moved the spec to oslo space since the switch is not
   really limited to neutron, and most of coding is to be done in
   oslo.db (though not much anyway).
  
   New spec: https://review.openstack.org/#/c/108355/
  
   On 09/07/14 13:17, Ihar Hrachyshka wrote:
   Hi all,
  
   Multiple projects are suffering from db lock timeouts due to
   deadlocks deep in mysqldb library that we use to interact with
   mysql servers. In essence, the problem is due to missing
   eventlet support in mysqldb module, meaning when a db lock is
   encountered, the library does not yield to the next green thread,
   allowing other threads to eventually unlock the grabbed lock, and
   instead it just blocks the main thread, that eventually raises
   timeout exception (OperationalError).
  
   The failed operation is not retried, leaving failing request not
served. In Nova, there is a special retry mechanism for
   deadlocks, though I think it's more a hack than a proper fix.
  
   Neutron is one of the projects that suffer from those timeout
   errors a lot. Partly it's due to lack of discipline in how we do
   nested calls in l3_db and ml2_plugin code, but that's not
   something to change in foreseeable future, so we need to find
   another solution that is applicable for Juno. Ideally, the
   solution should be applicable for Icehouse too to allow
   distributors to resolve existing deadlocks without waiting for
   Juno.
  
   We've had several discussions and attempts to introduce a
   solution to the problem. Thanks to oslo.db guys, we now have more
   or less clear view on the cause of the failures and how to easily
   fix them. The solution is to switch mysqldb to 

Re: [openstack-dev] Where should a test for eventlet and oslo.db interaction go?

2014-07-11 Thread Victor Sergeyev
Hello all.

After discussion in IRC, I agree, that we should take care about
interaction with eventlet at least because right now eventlet is the
OpenStack production configuration. So we must be sure, that we'll get no
issues, when we work with eventlet.

So I agree, that it makes a sense - to add a test for eventlet and
sqlalchemy interaction to oslo.db. You convinced me :)


On Thu, Jul 10, 2014 at 7:05 PM, Mike Bayer mba...@redhat.com wrote:


 On 7/10/14, 7:47 AM, Sean Dague wrote:

 Honestly, that seems weird to me.

 oslo.db is built as a common layer for OpenStack services.

 eventlet is used by most OpenStack services.

 There are lots of known issues with eventlet vs. our db access patterns.

 Knowing that the db layer works in the common OpenStack pattern seems
 really important. And something that seems to make sense very close to
 the db code itself.


 Yeah I am +1 on this, the use of eventlet is very prominent throughout
 openstack components, and oslo.db is intended as a glue layer between
 SQLAlchemy and those apps.   The patterns that are used with eventlet
 should be tested at the oslo.db level, as oslo.db is responsible for
 configuration of the driver and additionally IMO should be taking on a much
 greater role in establishing transactional patterns which also have an
 impact on these issues.

 SQLAlchemy itself never spawns any threads.  However, we certainly have a
 crap-ton of tests that test the connection pool and other
 concurrency-sensitive areas in the context of many threads being run.  It's
 a critical use case so we test against it.

 oslo.db should at every turn be attempting to remove redundancy from
 downstream projects.   If ten projects all use eventlet, they shouldn't all
 have to replicate the same test over and over that should just be upwards
 of them.







   -Sean

 On 07/10/2014 07:42 AM, Victor Sergeyev wrote:

  Hello Angus!


 IMO, the simple answer on your question is - tests for eventlet and
 oslo.db interaction should be in the same place, where eventlet and
 oslo.db interact. :)


 A little digression - we suppose, that oslo.db should neither know, nor
 take care whether target projects use eventlet/gevent/OS
 threads/multiple processes/callbacks/etc for handling concurrency -
 oslo.db just can't (and should not) make such decisions for users. For
 the very same reason SQLAlchemy doesn't do that.


 Thanks,

 Victor


 On Thu, Jul 10, 2014 at 10:55 AM, Angus Lees 
 gusl...@gmail.commailto:gusl...@gmail.com gusl...@gmail.com wrote:

 We have an issue with neutron (and presumably elsewhere), where
 mysqldb and eventlet may deadlock, until the mysqldb deadlock timer
 fires.
 I believe it's responsible for ~all of these failures:
 
 http://logstash.openstack.org/#eyJzZWFyY2giOiJcIkxvY2sgd2FpdCB0aW1lb3V0IGV4Y2VlZGVkOyB0cnkgcmVzdGFydGluZyB0cmFuc2FjdGlvblwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDA0OTcwMzgwMjc0fQ==

 Now, the fix is one thing and is underway (the current favourite
 option is just switching to a different mysql client library) - my
 question here is instead about this test:

 https://review.openstack.org/#/c/104436/

 This test (as written) is against oslo.db and drives eventlet +
 sqlalchemy to confirm that the current sqlalchemy driver does _not_
 have the above deadlock observed with mysqldb.  I think it (or some
 version of it) is an important test, but the oslo.db guys don't want
 it in their testsuite since they've purged every explicit mention of
 eventlet.  I'm sympathetic to this pov.

 I think we should have something like this test *somewhere*, at
 least as long as we're using eventlet frequently.

 I'm a bit new to openstack, so I'm lost in a maze of testing
 options.  Could some kind member of the TC point to where this test
 *should* go?

 --
  - Gus

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 mailto:OpenStack-dev@lists.openstack.org 
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




 ___
 OpenStack-dev mailing 
 listOpenStack-dev@lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



 ___
 OpenStack-dev mailing 
 listOpenStack-dev@lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Where should a test for eventlet and oslo.db interaction go?

2014-07-10 Thread Victor Sergeyev
Hello Angus!

IMO, the simple answer on your question is - tests for eventlet and oslo.db
interaction should be in the same place, where eventlet and oslo.db
interact. :)

A little digression - we suppose, that oslo.db should neither know, nor
take care whether target projects use eventlet/gevent/OS threads/multiple
processes/callbacks/etc for handling concurrency - oslo.db just can't (and
should not) make such decisions for users. For the very same reason
SQLAlchemy doesn't do that.

Thanks,
Victor


On Thu, Jul 10, 2014 at 10:55 AM, Angus Lees gusl...@gmail.com wrote:

 We have an issue with neutron (and presumably elsewhere), where mysqldb
 and eventlet may deadlock, until the mysqldb deadlock timer fires.
 I believe it's responsible for ~all of these failures:

 http://logstash.openstack.org/#eyJzZWFyY2giOiJcIkxvY2sgd2FpdCB0aW1lb3V0IGV4Y2VlZGVkOyB0cnkgcmVzdGFydGluZyB0cmFuc2FjdGlvblwiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiI2MDQ4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxNDA0OTcwMzgwMjc0fQ==

 Now, the fix is one thing and is underway (the current favourite option is
 just switching to a different mysql client library) - my question here is
 instead about this test:

 https://review.openstack.org/#/c/104436/

 This test (as written) is against oslo.db and drives eventlet + sqlalchemy
 to confirm that the current sqlalchemy driver does _not_ have the above
 deadlock observed with mysqldb.  I think it (or some version of it) is an
 important test, but the oslo.db guys don't want it in their testsuite since
 they've purged every explicit mention of eventlet.  I'm sympathetic to this
 pov.

 I think we should have something like this test *somewhere*, at least as
 long as we're using eventlet frequently.

 I'm a bit new to openstack, so I'm lost in a maze of testing options.
  Could some kind member of the TC point to where this test *should* go?

 --
  - Gus

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] oslo.db 0.3.0 released

2014-07-09 Thread Victor Sergeyev
Hello Folks!

The Oslo team is pleased to announce the release of oslo.db 0.3.0

oslo.db is the database handling library.

This release includes the following changes:

$ git log --oneline --no-merges 0.2.0..0.3.0

404de36 Add a base test case for DB schema comparison
a1fd49f Test for distinct SQLAlchemy major releases
385b823 Updated from global requirements
167ffb6 Add __contains__ to ModelBase to fully behave like a dict
e734e2b Fix test to not assume eventlet isn't present
d7da57c Avoid usage of mutables as default args
819671a Updated from global requirements
3a6a5b0 Add _wrap_db_error support for postgresql


Please report issues using the launchpad tracker:
https://bugs.launchpad.net/oslo and mark them with ``db`` tag.

Thanks,
Victor
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [Oslo][DB] oslo.db released

2014-06-19 Thread Victor Sergeyev
Hello, Folks!

Please, be informed, that the oslo.db library has been released and is
available on PyPi.
See [1] for the source code, [2] for the documentation.

There is an example of how to switch an OpenStack project to oslo.db - see
patch to Ironic [3]. oslo.db team is going to eventually make all consuming
projects use the library instead of common db code (which is now obsolete).

[1] http://git.openstack.org/cgit/openstack/oslo.db/
[2] http://docs.openstack.org/developer/oslo.db/
[3] https://review.openstack.org/#/c/42159/

Thanks,
Victor
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Fwd: [Eventletdev] Eventlet 0.15 pre-release testers needed

2014-06-16 Thread Victor Sergeyev
Hello Folks!

AFAIK, new eventlet version have limited PY3 support. So, unfortunately, it
is not ready for work with python 3 in production. See issue [1] on github.
Anyway, at least we will be in able to install eventlet in PY3 environment
and (I hope) to test some openstack features (or mock eventlet calls).
Everybody, who want to help with eventlet PY3 porting (I think, python3
migration team should be interesting in it) are welcomed to contribute.
Please, find eventlet repository on github [2] :)

I've copied this email to the eventlet maintainer - Sergey Shepelev. Maybe
he will add some more details.

[1] https://github.com/eventlet/eventlet/issues/6
[2] https://github.com/eventlet/eventlet

Thanks,
Victor



On Fri, Jun 13, 2014 at 9:59 PM, Chuck Thier cth...@gmail.com wrote:

 Just a FYI for those interested in the next eventlet version.  It also
 looks like they have a python 3 branch ready to start testing with.

 --
 Chuck

 -- Forwarded message --
 From: Sergey Shepelev temo...@gmail.com
 Date: Fri, Jun 13, 2014 at 1:18 PM
 Subject: [Eventletdev] Eventlet 0.15 pre-release testers needed
 To: eventletdev eventlet...@lists.secondlife.com, Noah Glusenkamp 
 n...@empowerengine.com, Victor Sergeyev viktor.serge...@gmail.com,
 ja...@stasiak.at


 Hello, everyone.

 TL;DR: please test these versions in Python2 and Python3:
 pip install URL should work
 (master)

 https://github.com/eventlet/eventlet/archive/6c4823c80575899e98afcb12f84dcf4d54e277cd.zip
 (py3-greenio branch, on top of master)

 https://github.com/eventlet/eventlet/archive/9e666c78086a1eb0c05027ec6892143dfa5c32bd.zip

 I am going to make Eventlet 0.15 release in coming week or two and your
 feedback would be greatly appreciated because it's the first release since
 we started work on Python3 compatibility. So please try to run your project
 in Python3 too, if you can.


 ___
 Click here to unsubscribe or manage your list subscription:
 https://lists.secondlife.com/cgi-bin/mailman/listinfo/eventletdev



 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova] plan for moving to using oslo.db

2014-05-12 Thread Victor Sergeyev
Hello Matt.

Thanks a lot for your working on this!
In my opinion, these steps are correct. Please see a few minor notes below.

1 - Yes, you right, oslo.db is not in global-requirements now. Blueprint
``Split openstack.common.db code into a separate oslo.db library `` [1]  is
not completed at the moment. Please see ``work items`` section in [1] for
the actual bp state

2,3 - Looks good to me :)

4 - If you want, you test it locally even without oslo.db in
global-requirements. You can replace nova.openstack.common.db with oslo.db
on  and run unittests. This will help us to find and fix potential issues
with porting to oslo.db (if any).
If you will have any issues or questions, please feel free to ping me or
Roman Podoliaka via irc or e-mail.

5 - ???

6 - Profit!

[1] https://blueprints.launchpad.net/oslo/+spec/oslo-db-lib

Thanks,
Victor


On Mon, May 5, 2014 at 5:47 PM, Matt Riedemann
mrie...@linux.vnet.ibm.comwrote:

 Just wanted to get some thoughts down while they are in my head this
 morning.

 Oslo DB is now a library [1].  I'm trying to figure out what the steps are
 to getting Nova to using that so we can rip out the sync'ed common db code.

 1. Looks like it's not in global-requirements yet [2], so that's probably
 a first step.

 2. We'll want to cut a sqlalchemy-migrate release once this patch is
 merged [3]. This moves a decent chunk of unique constraint patch code out
 of oslo and into sqlalchemy-migrate where it belongs so we can run unit
 tests with sqlite to drop unique constraints.

 3. Rip this [4] out of oslo.db once migrate is updated and released.

 4. Replace nova.openstack.common.db with oslo.db.

 5. ???

 6. Profit!

 Did I miss anything?

 [1] http://git.openstack.org/cgit/openstack/oslo.db/
 [2] http://git.openstack.org/cgit/openstack/requirements/tree/
 global-requirements.txt
 [3] https://review.openstack.org/#/c/87773/
 [4] https://review.openstack.org/#/c/31016/

 --

 Thanks,

 Matt Riedemann


 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [oslo][db] oslo.db repository review request

2014-04-18 Thread Victor Sergeyev
Hello all,

During Icehouse release cycle our team has been working on splitting of
openstack common db code into a separate library blueprint [1]. At the
moment the issues, mentioned in this bp and [2] are solved and we are
moving forward to graduation of oslo.db. You can find the new oslo.db code
at [3]

So, before moving forward, I want to ask Oslo team to review oslo.db
repository [3] and especially the commit, that allows the unit tests to
pass [4].

Thanks,
Victor

[1] https://blueprints.launchpad.net/oslo/+spec/oslo-db-lib
[2] https://wiki.openstack.org/wiki/Oslo/GraduationStatus#oslo.db
[3] https://github.com/malor/oslo.db
[4]
https://github.com/malor/oslo.db/commit/276f7570d7af4a7a62d0e1ffb4edf904cfbf0600
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [oslo-incubator] removal of slave_connection from db.sqlalchemy.session

2014-03-05 Thread Victor Sergeyev
Hello All.

We suppose to have common database code oslo.db library. So we decided to
let end applications to cope with engines, not oslo.db. For example, see
work with slave engine in Nova [1]. These is also patch to oslo with more
details - [2]

Also, Darren, please inform us about your usecase a bit.

[1]
https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L95
[2] https://review.openstack.org/#/c/68684/


On Wed, Mar 5, 2014 at 6:35 PM, Doug Hellmann
doug.hellm...@dreamhost.comwrote:




 On Wed, Mar 5, 2014 at 10:43 AM, Alexei Kornienko 
 alexei.kornie...@gmail.com wrote:

  Hello Darren,

 This option is removed since oslo.db will no longer manage engine objects
 on it's own. Since it will not store engines it cannot handle query
 dispatching.

 Every project that wan't to use slave_connection will have to implement
 this logic (creation of the slave engine and query dispatching) on it's own.


 If we are going to have multiple projects using that feature, we will have
 to restore it to oslo.db. Just because the primary API won't manage global
 objects doesn't mean we can't have a secondary API that does.

 Doug




 Regards,


 On 03/05/2014 05:18 PM, Darren Birkett wrote:

 Hi,

  I'm wondering why in this commit:


 https://github.com/openstack/oslo-incubator/commit/630d3959b9d001ca18bd2ed1cf757f2eb44a336f

  ...the slave_connection option was removed.  It seems like a useful
 option to have, even if a lot of projects weren't yet using it.

  Darren


 ___
 OpenStack-dev mailing 
 listOpenStack-dev@lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [nova][oslo] Changes to oslo-incubator sync workflow

2014-02-20 Thread Victor Sergeyev
Hello All

I and Roman Podoliaka are familiar with the changes made to common db code,
so we are ready to help with syncing it to OS projects.
But we want to ask you for more activity in reviewing of these patches.

Thanks, Victor


On Thu, Feb 20, 2014 at 4:27 AM, Doug Hellmann
doug.hellm...@dreamhost.comwrote:




 On Wed, Feb 19, 2014 at 8:13 PM, Joe Gordon joe.gord...@gmail.com wrote:

 Hi All,

 As many of you know most oslo-incubator code is wildly out of sync.
 Assuming we consider it a good idea to sync up oslo-incubator code
 before cutting Icehouse, then we have a problem.

 Today oslo-incubator code is synced in ad-hoc manor, resulting in
 duplicated efforts and wildly out of date code. Part of the challenges
 today are backwards incompatible changes and new oslo bugs. I expect
 that once we get a single project to have an up to date oslo-incubator
 copy it will make syncing a second project significantly easier. So
 because I (hopefully) have some karma built up in nova, I would like
 to volunteer nova to be the guinea pig.


 Thank you for volunteering to spear-head this, Joe.


 To fix this I would like to propose starting an oslo-incubator/nova
 sync team. They would be responsible for getting nova's oslo code up
 to date.  I expect this work to involve:
 * Reviewing lots of oslo sync patches
 * Tracking the current sync patches
 * Syncing over the low hanging fruit, modules that work without changing
 nova.
 * Reporting bugs to oslo team
 * Working with oslo team to figure out how to deal with backwards
 incompatible changes
   * Update nova code or make oslo module backwards compatible
 * Track all this
 * Create a roadmap for other projects to follow (re: documentation)

 I am looking for volunteers to help with this effort, any takers?


 I will help, especially with reviews and tracking.

 We are going to want someone from the team working on the db modules to
 participate as well, since we know that's one area where the API has
 diverged some (although we did take backwards compatibility into account).
 Victor, can you help find us a volunteer?

 Doug





 best,
 Joe Gordon

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [oslo][nova] Issues syncing latest db changes

2014-02-06 Thread Victor Sergeyev
Hello Joe.

Thanks for pointing this issue. We will investigate this situation and fix
it.

In the future in such cases you can just create a bug on launchpad.
Also feel free to ping me (and another db maintainers) in IRC.

Thanks,
Victor


On Wed, Feb 5, 2014 at 9:12 PM, Joe Gordon joe.gord...@gmail.com wrote:

 Hi Boris, Roman, Victor (oslo-incubator db maintainers),

 Last night I stumbled across bug https://launchpad.net/bugs/1272500 in
 nova, which says the issue has been fixed in the latest oslo-incubator
 code. So I ran:

 ./update.sh --base nova --dest-dir ../nova --modules db.sqlalchemy

 https://review.openstack.org/#/c/71191/

 And that appeared to fix the specific issues I was seeing from Bug
 1272500, but it introduced some new failures.


 I would like to get nova unit test working with sqlite 3.8.2-1 if
 possible. How can this situation be resolved?


 best,
 Joe

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [oslo] team meeting Friday 31 Jan 1400 UTC

2014-01-29 Thread Victor Sergeyev
Hello All.

Also I have a proposition to discuss current graduation status of oslo.db
code.
This code going to move into a separate library (it will be soon, I hope),
so it's would be nice to look at it's state/issues/and-so-on due to speed
up graduation process and avoid any confusion in the future.
See blueprint [1] for more details

[1] https://blueprints.launchpad.net/oslo/+spec/oslo-db-lib

Thanks,
Victor.


On Tue, Jan 28, 2014 at 6:50 PM, Doug Hellmann
doug.hellm...@dreamhost.comwrote:




 On Tue, Jan 28, 2014 at 3:55 AM, Flavio Percoco fla...@redhat.com wrote:

 On 27/01/14 14:57 -0500, Doug Hellmann wrote:

 The Oslo team has a few items we need to discuss, so I'm calling a
 meeting for
 this Friday, 31 Jan. Our normal slot is 1400 UTC Friday in
 #openstack-meeting.
 The agenda [1] includes 2 items (so far):

 1. log translations (see the other thread started today)
 2. parallelizing our tests


 We should also discuss the process to pull out packages from oslo. I
 mean, check if anything has changed in terms of stability since the
 summit and what our plans are for moving forward with this.


 I added an item to discuss managing graduating code that is still in the
 incubator (the rpc discussion this week brought it to the front of my
 mind). Is that the same thing you mean?

 Doug




 Cheers,
 flaper


  If you have anything else you would like to discuss, please add it to the
 agenda.

 See you Friday!
 Doug


 [1] https://wiki.openstack.org/wiki/Meetings/Oslo#Agenda_for_
 Next_Meeting


  ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



 --
 @flaper87
 Flavio Percoco

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] [oslo] dependency analysis for graduating more oslo libraries

2014-01-15 Thread Victor Sergeyev
Hello All.

As for lockutils - a few days ago I wondered why we used custom oslo module
instead of lockfile library [1]. AFAIK, it must be due to this bug [2]
(please fix me, if I wrong).

This library is available on github [3] and there is a pull-request fixing
the bug [4], but unfortunately it hasn’t been merged yet. And seems that
this project is not maintained anymore - author told, that he “haven't done
anything with it in a few years” [5].

So my question is - can we start maintaining this library (put it on
stackforge) and use it instead of oslo.lockutils? Or maybe we could include
parts of oslo.lockutils into lockfile? Or vice versa, incorporate lockfile
into oslo.lockutils?

[1] https://pypi.python.org/pypi/lockfile
[2] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=632857
[3] https://github.com/smontanaro/pylockfile
[4] https://github.com/smontanaro/pylockfile/pull/3
[5] https://github.com/smontanaro/pylockfile/pull/3#issuecomment-32085554


On Wed, Jan 15, 2014 at 1:00 AM, Michael Still mi...@stillhq.com wrote:

 On Wed, Jan 15, 2014 at 9:27 AM, Ben Nemec openst...@nemebean.com wrote:

  It would be nice to get lockutils graduated to solve some of the issues
  mentioned in the oslo.db section, but I believe we do have an outstanding
  question regarding its behavior without lock_path being set.  I think
 Clint
  was on board with Sean's proposed solution after quite a bit of
 discussion
  (
 http://lists.openstack.org/pipermail/openstack-dev/2013-December/021620.html
 ),
  so it's possible we could just restore that patch and call it done, but
 it
  should probably be addressed somehow before graduation.

 I committed a while ago (at the last summit IIRC) to working on
 getting lockutils released as a library, but I haven't managed to get
 that done yet. If it is blocking other people I can prioritise that
 work to being higher on my todo list.

 Part of the problem here is that its my first oslo graduation, so I
 need to figure out what to do...

 Michael

 --
 Rackspace Australia

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


Re: [openstack-dev] Revisiting current column number limit in code

2013-10-30 Thread Victor Sergeyev
By the way, pep8 says, that it is okay to increase the nominal line length
from 80 to 100 characters. See
http://www.python.org/dev/peps/pep-0008/#maximum-line-length

I'm ok with 80 characters, but just for information - is there any plans
to to increase the line length in OpenStack?

Victor


On Wed, Oct 30, 2013 at 8:52 AM, Aaron Rosen aro...@nicira.com wrote:

 Here's what mine usually looks like :)

 http://www.python.org/dev/peps/pep-0008/#maximum-line-length


 On Tue, Oct 29, 2013 at 11:19 PM, Ravi Kumar Pasumarthy 
 ravi...@thoughtworks.com wrote:

 Hello,

 Looks like the current column limit for the code base is 80. Because of
 this large space on the right hand side is not used. Please find the
 attached pictures of couple of developer screens working on openstack
 codebase.

 Are there any thoughts to increase the current column limit to 120
 atleast.

 How does this help ? I can see more lines of code, it is easier on the
 eyes. Some external monitors supports rotation, but for laptops we cannot
 rotate.

 So is it possible to revisit the column number limit ?

 Thanks,
 Ravi Kumar Pasumarthy
 Lead Consultant

 ThoughtWorks

 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



 ___
 OpenStack-dev mailing list
 OpenStack-dev@lists.openstack.org
 http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] Move pep8 requirements to a separate file

2013-10-24 Thread Victor Sergeyev
Hello all,

I noticed, that when I run tests using tox I get some redundant modules
installed to tox virtual environments. At the moment, pep8 specific
requirements (such as pep8, flake8 and so on) are stated in
test-requirements.txt file, which is meant to contain testing specific
modules (nose, testtools, etc). So when we run tox, it installs those
unnecessary pep8 libraries into py26, py27 environments. The same is true
for pep8 environment - tox installs all libraries from
test-requirements.txt there, but only pep8 specific modules are actually
needed.

I think, it would be nice to move pep8 specific requirements from
test-requirements.txt to a separate file (pep8-requirements.txt, perhaps)
to avoid this situation. It would save network traffic and reduce the time
it takes to create virtual environments.

Thoughts? Does it sound reasonable?

Victor.
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [Glance] Replacing Glance DB code to Oslo DB code.

2013-08-16 Thread Victor Sergeyev
Hello All.

Glance cores (Mark Washenberger, Flavio Percoco, Iccha Sethi) have some
questions about Oslo DB code, and why is it so important to use it instead
of custom implementation and so on. As there were a lot of questions it was
really hard to answer on all this questions in IRC. So we decided that
mailing list is better place for such things.

List of main questions:

1. What includes oslo DB code?
2. Why is it safe to replace custom implementation by Oslo DB code?
3. Why oslo DB code is better than custom implementation?
4. Why oslo DB code won’t slow up project development progress?
5. What we are going actually to do in Glance?
6. What is the current status?

Answers:

1. What includes oslo DB code?

Currently Oslo code improves different aspects around DB:
-- Work with SQLAlchemy models, engine and session
-- Lot of tools for work with SQLAlchemy
-- Work with unique keys
-- Base test case for work with database
-- Test migrations against different backends
-- Sync DB Models with actual schemas in DB (add test that they are
equivalent)


2. Why is it safe to replace custom implementation by Oslo DB code?

Oslo module, as base openstack module, takes care about code quality.
Usually, common code more readable (most of flake8 checks enabled in Oslo)
and have better test coverage.  Also it was tested in different use-cases
(in production also) in an other projects so bugs in Oslo code were already
fixed. So we can be sure, that we use high-quality code.


3. Why oslo DB code is better than custom implementation?

There are some arguments pro Oslo database code

-- common code collects useful features from different projects
Different utils, for work with database, common test class, module for
database migration, and  other features are already in Oslo db code. Patch
on automatic retry db.api query if db connection lost on review at the
moment. If we use Oslo db code we should not care, how to port these (and
others - in the future) features to Glance - it will came to all projects
automaticly when it will came to Oslo.

-- unified project work with database
As it was already said,  It can help developers work with database in a
same way in different projects. It’s useful if developer work with db in a
few projects - he use same base things and got no surprises from them.

-- it’s will reduce time for running tests.
Maybe it’s minor feature, but it’s also can be important. We can removed
some tests for base `DB` classes (such as session, engines, etc)  and
replaced for work with DB to mock calls.


4. Why oslo DB code won’t slow up project development progress?

Oslo code for work with database already in such projects as Nova, Neutron,
Celiometer and Ironic. AFAIK, these projects development speed doesn’t
decelerated (please fix me, If I’m wrong). Work with database level already
improved and tested in Oslo project, so we can concentrate on work with
project features. All features, that already came to oslo code will be
available in Glance, but if you want to add some specific feature to
project *just now* you will be able to do it in project code.


5. What we are going actually to do in Glance?

-- Improve test coverage of DB API layer
We are going to increase test coverage of glance/db/sqlalchemy/api module
and fix bugs, if found.

-- Run DB API tests on all backends
-- Use Oslo migrations base test case for test migrations against different
backends
There are lot of different things in SQl backends. For example work with
casting.
In current SQLite we are able to store everything in column (with any
type). Mysql will try to convert value to required type, and postgresql
will raise IntegrityError.
If we will improve this feature, we will be sure, that all Glance DB
migrations will run correctly on all backends.

-- Use Oslo code for SA models, engine and session
-- Use Oslo SA utils
Using common code for work with database was already discussed and approved
for all projects. So we are going to implement common code for work with
database instead of Glance implementation.

-- Fix work with session and transactions
Our work items in Glance:
- don't pass session instances to public DB methods
- use explicit transactions only when necessary
- fix incorrect usage of sessions throughout the DB-related code

-- Optimize methods
When we will have tests for all functions in glance/db/sqlalchemy/api
module it’s will be safe to refactor api methods. It will make these
functions more clean, readable and faster.

The main ideas are:
- identify and remove unused methods
- consolidate duplicate methods when possible
- ensure SQLAlchemy objects are not leaking out of the API
- ensure related methods are grouped together and named consistently

-- Add missing unique constraints
We should provide missed unique constraints, based on database queries from
glance.db.sqlalchemy.api module. It’s will reduce data duplication and
became one more step to Glance database normalization.

-- Sync models definitions with DB 

Re: [openstack-dev] [Nova][Oslo-incubator] Automatic retry db.api query if database connection lost

2013-07-29 Thread Victor Sergeyev
Hello.

Any suggestions, please?

On Mon, Jul 22, 2013 at 11:39 AM, Victor Sergeyev vserge...@mirantis.comwrote:

 Hi All.

 There is a blueprint (
 https://blueprints.launchpad.net/nova/+spec/db-reconnect) by Devananda
 van der Veen, which goal is to implement reconnection to a database and
 retrying of the last operation if a db connection fails. I’m working on the
 implementation of this BP in oslo-incubator (
 https://review.openstack.org/#/c/33831/).

 Function _raise_if_db_connection_lost() was added to _wrap_db_error()
 decorator defined in openstack/common/db/sqlalchemy/session.py. This
 function catches sqlalchemy.exc.OperationalError and finds database error
 code in this exception. If this error code is on `database has gone away`
 error codes list, this function raises DBConnectionError exception.

 Decorator for db.api methods was added to openstack/common/db/api.py.
 We can apply this decorator to methods in db.sqlalchemy.api (not to
 individual queries).
 It catches DBConnectionError exception and retries the last query in a
 loop until it succeeds, or until the timeout is reached. The timeout value
 is configurable with min, max, and increment options.
 We suppose that all db.api methods are executed inside a single
 transaction, so retrying the whole method, when a connection is lost,
 should be safe.

 I would really like to receive some comments about the following
 suggestions:

 1. I can’t imagine a situation when we lose connection to an SQLite DB.
 Also, as far as I know, SQLite is not used in production at the moment, so
 we don't handle this case.

 2. Please, leave some comments about  `database has gone away` error codes
 list in MySQL and PostgreSQL.

 3. Johannes Erdfelt suggested that “retrying the whole method, even if
 it's in a transaction, is only safe the entire method is idempotent. A
 method could execute successfully in the database, but the connection could
 be dropped before the final status is sent to the client.”
  I agree, that this situation can cause data corruption in a database (e.
 g., if we try to insert something to a database), but I’m not sure, how
 RDBMS handle this. Also, I haven't succeeded in creation of a functional
 test case, that would allow to reproduce the described situation easily.


 Thanks, Victor

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


[openstack-dev] [Nova][Oslo-incubator] Automatic retry db.api query if database connection lost

2013-07-22 Thread Victor Sergeyev
Hi All.

There is a blueprint (
https://blueprints.launchpad.net/nova/+spec/db-reconnect) by Devananda van
der Veen, which goal is to implement reconnection to a database and
retrying of the last operation if a db connection fails. I’m working on the
implementation of this BP in oslo-incubator (
https://review.openstack.org/#/c/33831/).

Function _raise_if_db_connection_lost() was added to _wrap_db_error()
decorator defined in openstack/common/db/sqlalchemy/session.py. This
function catches sqlalchemy.exc.OperationalError and finds database error
code in this exception. If this error code is on `database has gone away`
error codes list, this function raises DBConnectionError exception.

Decorator for db.api methods was added to openstack/common/db/api.py.
We can apply this decorator to methods in db.sqlalchemy.api (not to
individual queries).
It catches DBConnectionError exception and retries the last query in a loop
until it succeeds, or until the timeout is reached. The timeout value is
configurable with min, max, and increment options.
We suppose that all db.api methods are executed inside a single
transaction, so retrying the whole method, when a connection is lost,
should be safe.

I would really like to receive some comments about the following
suggestions:

1. I can’t imagine a situation when we lose connection to an SQLite DB.
Also, as far as I know, SQLite is not used in production at the moment, so
we don't handle this case.

2. Please, leave some comments about  `database has gone away` error codes
list in MySQL and PostgreSQL.

3. Johannes Erdfelt suggested that “retrying the whole method, even if it's
in a transaction, is only safe the entire method is idempotent. A method
could execute successfully in the database, but the connection could be
dropped before the final status is sent to the client.”
I agree, that this situation can cause data corruption in a database (e.
g., if we try to insert something to a database), but I’m not sure, how
RDBMS handle this. Also, I haven't succeeded in creation of a functional
test case, that would allow to reproduce the described situation easily.


Thanks, Victor
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev