Re: [openstack-dev] [neutron][db] online schema upgrades

2015-06-16 Thread Mike Bayer
On 6/16/15 11:41 AM, Ihar Hrachyshka wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 - - instead of migrating data with alembic rules, migrate it in runtime. There should be a abstraction layer that will make sure that data is migrated into new schema fields and objects, while preservin

Re: [openstack-dev] [neutron][db] online schema upgrades

2015-06-17 Thread Mike Bayer
On 6/17/15 12:40 PM, Ihar Hrachyshka wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 06/17/2015 11:27 AM, Anna Kamyshnikova wrote: Ihar, thanks for bringing this up! This is very interesting and I think it worth trying. I'm +1 on that and want to participate in this work. Awesome

Re: [openstack-dev] [neutron][db] online schema upgrades

2015-06-18 Thread Mike Bayer
my initial proposal for scripted expand/contract migrations is up: https://review.openstack.org/#/c/192937/ On 6/18/15 5:54 AM, Anna Kamyshnikova wrote: On Wed, Jun 17, 2015 at 8:23 PM, Mike Bayer <mailto:mba...@redhat.com>> wrote: On 6/17/15 12:40 PM, Ihar Hrachys

Re: [openstack-dev] [all] DevStack switching from MySQL-python to PyMySQL

2015-06-18 Thread Mike Bayer
On 6/18/15 1:28 PM, Matt Riedemann wrote: It's not only neutron, I saw some pymysql failures in nova the other day for 'too many connections' or some such related error. "too many connections" is an error raised by MySQL when more connections are attempting to connect than the max_con

Re: [openstack-dev] [all] DevStack switching from MySQL-python to PyMySQL

2015-06-18 Thread Mike Bayer
On 6/18/15 3:53 PM, Sean Dague wrote: On 06/18/2015 03:47 PM, Mike Bayer wrote: On 6/18/15 1:28 PM, Matt Riedemann wrote: It's not only neutron, I saw some pymysql failures in nova the other day for 'too many connections' or some such related error. "too many conn

Re: [openstack-dev] Python 2.6 failed jobs for SQLA-Migrate

2015-07-01 Thread Mike Bayer
On 6/30/15 9:51 PM, Thomas Goirand wrote: Hi, Mike nicely tried to help me to get sqla-migrate to work with sqlalchemy 1.0.6 which is now in Debian. But there's some failures in Python 2.6: https://review.openstack.org/#/c/197144/ Do we still care about them? Can we get them removed from -mi

Re: [openstack-dev] Python 2.6 failed jobs for SQLA-Migrate

2015-07-01 Thread Mike Bayer
On 7/1/15 10:03 AM, Mike Bayer wrote: On 6/30/15 9:51 PM, Thomas Goirand wrote: Hi, Mike nicely tried to help me to get sqla-migrate to work with sqlalchemy 1.0.6 which is now in Debian. But there's some failures in Python 2.6: https://review.openstack.org/#/c/197144/ Do we still

Re: [openstack-dev] [all] Please do *not* use git (and specifically "git log") when generating the docs

2016-02-18 Thread Mike Bayer
On 02/18/2016 04:39 PM, Dolph Mathews wrote: On Thu, Feb 18, 2016 at 11:17 AM, Thomas Goirand mailto:z...@debian.org>> wrote: Hi, I've seen Reno doing it, then some more. It's time that I raise the issue globally in this list before the epidemic spreads to the whole of OpenSt

Re: [openstack-dev] [nova] nova-compute blocking main thread under heavy disk IO

2016-02-22 Thread Mike Bayer
On 02/22/2016 11:30 AM, Chris Friesen wrote: On 02/22/2016 11:17 AM, Jay Pipes wrote: On 02/22/2016 10:43 AM, Chris Friesen wrote: Hi all, We've recently run into some interesting behaviour that I thought I should bring up to see if we want to do anything about it. Basically the problem see

Re: [openstack-dev] [oslo] oslo.db reset session?

2016-02-23 Thread Mike Bayer
On 02/23/2016 09:22 AM, Sean Dague wrote: With enginefascade working coming into projects, there seems to be some new bits around oslo.db global sessions. The effect of this on tests is a little problematic. Because it builds global state which couples between tests. I've got a review to use m

Re: [openstack-dev] [oslo] documentation on using the oslo.db opportunistic test feature

2016-02-23 Thread Mike Bayer
On 02/22/2016 08:02 PM, Sean Dague wrote: Before migrating into oslo.db the opportunistic testing for database backends was pretty simple. Create an openstack_citest@openstack_citest pw:openstack_citest and you could get tests running on mysql. This no longer seems to be the case. this is sti

Re: [openstack-dev] [oslo] documentation on using the oslo.db opportunistic test feature

2016-02-23 Thread Mike Bayer
On 02/22/2016 08:08 PM, Davanum Srinivas wrote: Sean, You need to set the env variable like so. See testenv:mysql-python for example OS_TEST_DBAPI_ADMIN_CONNECTION=mysql://openstack_citest:openstack_citest@localhost you should not need to set this if you're using the default URL. The defau

Re: [openstack-dev] [oslo] documentation on using the oslo.db opportunistic test feature

2016-02-23 Thread Mike Bayer
On 02/22/2016 08:18 PM, Sean Dague wrote: On 02/22/2016 08:08 PM, Davanum Srinivas wrote: Sean, You need to set the env variable like so. See testenv:mysql-python for example OS_TEST_DBAPI_ADMIN_CONNECTION=mysql://openstack_citest:openstack_citest@localhost Thanks, Dims [1] http://codesear

Re: [openstack-dev] [oslo] oslo.db reset session?

2016-02-23 Thread Mike Bayer
;t yet impact all of those existing MySQLOpportunisticTest classes it has. On Tue, Feb 23, 2016 at 6:09 PM, Mike Bayer wrote: On 02/23/2016 09:22 AM, Sean Dague wrote: With enginefascade working coming into projects, there seems to be some new bits around oslo.db global sessions. The effect of this on tes

Re: [openstack-dev] [oslo] documentation on using the oslo.db opportunistic test feature

2016-02-23 Thread Mike Bayer
On 02/23/2016 12:20 PM, Sean Dague wrote: On 02/23/2016 11:29 AM, Mike Bayer wrote: On 02/22/2016 08:18 PM, Sean Dague wrote: On 02/22/2016 08:08 PM, Davanum Srinivas wrote: Sean, You need to set the env variable like so. See testenv:mysql-python for example

Re: [openstack-dev] [tricircle] Using in-memory database for unit tests

2016-03-23 Thread Mike Bayer
On 03/23/2016 01:33 AM, Vega Cai wrote: On 22 March 2016 at 12:09, Shinobu Kinjo mailto:shinobu...@gmail.com>> wrote: Thank you for your comment (inline for my message). On Tue, Mar 22, 2016 at 11:53 AM, Vega Cai mailto:luckyveg...@gmail.com>> wrote: > Let me try to explain some

Re: [openstack-dev] Deleting a cluster in Sahara SQL/PyMYSQL Error

2016-03-24 Thread Mike Bayer
Id recommend filing a bug in Launchpad against Sahara for that. Can you reproduce it ? On 03/23/2016 07:10 PM, Jerico Revote wrote: Hello, When trying to delete a cluster in sahara, I'm getting the following error: code 500 and message 'Internal Server Error' 2016-03-23 17:25:21.651 18827

Re: [openstack-dev] Deleting a cluster in Sahara SQL/PyMYSQL Error

2016-03-25 Thread Mike Bayer
on "validating", then tried deleting it but get stuck again on "deleting", and seeing that SQL error. Will submit a bug @ Launchpad. On 25 March 2016 at 01:56, Mike Bayer mailto:mba...@redhat.com>> wrote: Id recommend filing a bug in Launchpad a

Re: [openstack-dev] [oslo] temporarily disabling python 3.x testing for oslo.messaging and oslo.rootwrap

2015-01-27 Thread Mike Bayer
Doug Hellmann wrote: > The infra team has been working hard to update our Python 3 testing for all > projects to run on 3.4 instead of 3.3. Two of the last projects to be able to > shift are oslo.messaging and oslo.rootwrap. The test suites for both projects > trigger a segfault bug in the 3

[openstack-dev] [oslo.db] PyMySQL review

2015-01-28 Thread Mike Bayer
Hey list - As many are aware, we’ve been looking for the one MySQL driver to rule them all. As has been the case for some weeks now, that driver is PyMySQL, meeting all the critical requirements we have of: 1. pure python, so eventlet patchable, 2. Python 3 support 3. Is released on Pypi (whi

Re: [openstack-dev] [oslo.db] PyMySQL review

2015-01-28 Thread Mike Bayer
Mike Bayer wrote: > Hey list - > > As many are aware, we’ve been looking for the one MySQL driver to rule them > all. As has been the case for some weeks now, that driver is PyMySQL, > meeting all the critical requirements we have of: 1. pure python, so > eventlet patc

Re: [openstack-dev] [oslo.db] PyMySQL review

2015-01-28 Thread Mike Bayer
Johannes Erdfelt wrote: > On Wed, Jan 28, 2015, Mike Bayer wrote: >> I can envision turning this driver into a total monster, adding >> C-speedups where needed but without getting in the way of async >> patching, adding new APIs for explicit async, and everything else

Re: [openstack-dev] [oslo.db] PyMySQL review

2015-01-29 Thread Mike Bayer
Roman Podoliaka wrote: > > On native threads vs green threads: I very much like the Keystone > approach, which allows to run the service using either eventlet or > Apache. It would be great, if we could do that for other services as > well. but why do we need two approaches to be at all expli

Re: [openstack-dev] [all][tc] SQL Schema Downgrades and Related Issues

2015-01-29 Thread Mike Bayer
Morgan Fainberg wrote: > > Are downward migrations really a good idea for us to support? Is this > downward migration path a sane expectation? In the real world, would any one > really trust the data after migrating downwards? It’s a good idea for a migration script to include a rudimentary

Re: [openstack-dev] [oslo.db] PyMySQL review

2015-01-29 Thread Mike Bayer
Angus Lees wrote: > > If we absolutely can't switch to another mysql driver, another option that > was suggested recently (and passes the above test) is using > eventlet.monkey_patch(MySQLdb=True). I haven't done the investigation to > find out why that isn't the default, or what the downs

Re: [openstack-dev] [Murano] SQLite support - drop or not?

2015-01-30 Thread Mike Bayer
Andrew Pashkin wrote: > Working on this issue I encountered another problem. > > Most indices in the project has no names and because of that, > developer must reverse-engineer them in every migration. > Read about that also here [1]. > > SQLAlchemy and Alembic provide feature for generation

Re: [openstack-dev] [oslo.db][nova] Deprecating use_slave in Nova

2015-01-30 Thread Mike Bayer
Matthew Booth wrote: > At some point in the near future, hopefully early in L, we're intending > to update Nova to use the new database transaction management in > oslo.db's enginefacade. > > Spec: > http://git.openstack.org/cgit/openstack/oslo-specs/plain/specs/kilo/make-enginefacade-a-facade

Re: [openstack-dev] [Neutron] Multiple template libraries being used in tree

2015-02-02 Thread Mike Bayer
Sean Dague wrote: > On 02/02/2015 04:20 PM, Mark McClain wrote: >> You’re right that the Mako dependency is really a side effect from Alembic. >> We used jinja for tempting radvd because it is used by the projects within >> the OpenStack ecosystem and also used in VPNaaS. > > Jinja is far m

Re: [openstack-dev] [oslo.db][nova] Use of asynchronous slaves in Nova (was: Deprecating use_slave in Nova)

2015-02-02 Thread Mike Bayer
Matthew Booth wrote: > > Based on my current (and still sketchy) understanding, I think we can > define 3 classes of database node: > > 1. Read/write > 2. Synchronous read-only > 3. Asynchronous read-only > > and 3 code annotations: > > * Writer (must use class 1) > * Reader (prefer class 2

Re: [openstack-dev] [Neutron] Multiple template libraries being used in tree

2015-02-02 Thread Mike Bayer
Sean Dague wrote: > On 02/02/2015 06:06 PM, Mike Bayer wrote: >> Sean Dague wrote: >> >>> On 02/02/2015 04:20 PM, Mark McClain wrote: >>>> You’re right that the Mako dependency is really a side effect from >>>> Alembic. We used ji

Re: [openstack-dev] [Murano] SQLite support - drop or not?

2015-02-03 Thread Mike Bayer
Andrew Pashkin wrote: > Mike Bayer wrote: >> The patch seems to hardcode the conventions for MySQL and Postgresql. >> The first thought I had was that in order to remove the dependence >> on them here, you’d need to instead simply turn off the >> “naming_convent

Re: [openstack-dev] [Murano] SQLite support - drop or not?

2015-02-03 Thread Mike Bayer
Andrew Pashkin wrote: > Mike Bayer wrote: >> there’s always a naming convention in place; all databases other than >> SQLite produce them on the fly if you don’t specify one. The purpose >> of the Alembic/SQLAlchemy naming_convention feature is so that you >> h

Re: [openstack-dev] [all][oslo.db][nova] TL; DR Things everybody should know about Galera

2015-02-04 Thread Mike Bayer
Matthew Booth wrote: > A: start transaction; > B: start transaction; > A: insert into foo values(1); > B: insert into foo values(1); <-- 'regular' DB would block here, and > report an error on A's commit > A: commit; <-- success > B: commit; <-- KABOOM > > Conf

Re: [openstack-dev] [all][oslo.db][nova] TL; DR Things everybody should know about Galera

2015-02-04 Thread Mike Bayer
Matthew Booth wrote: > This means that even for 'synchronous' slaves, if a client makes an RPC > call which writes a row to write master A, then another RPC call which > expects to read that row from synchronous slave node B, there's no > default guarantee that it'll be there. Can I get some

Re: [openstack-dev] [all][oslo.db][nova] TL; DR Things everybody should know about Galera

2015-02-04 Thread Mike Bayer
Matthew Booth wrote: > A: start transaction; > A: insert into foo values(1) > A: commit; > B: select * from foo; <-- May not contain the value we inserted above[3] I’ve confirmed in my own testing that this is accurate. the wsrep_causal_reads flag does resolve this, and it is settable on a per

Re: [openstack-dev] [all][oslo.db][nova] TL; DR Things everybody should know about Galera

2015-02-04 Thread Mike Bayer
Jay Pipes wrote: > No, this is not correct. There is nothing different about Galera here versus > any asynchronously replicated database. A single thread, issuing statements > in two entirely *separate sessions*, load-balanced across an entire set of > database cluster nodes, may indeed see

Re: [openstack-dev] [all][oslo.db][nova] TL; DR Things everybody should know about Galera

2015-02-05 Thread Mike Bayer
Attila Fazekas wrote: > I have a question related to deadlock handling as well. > > Why the DBDeadlock exception is not caught generally for all api/rpc request ? > > The mysql recommendation regarding to Deadlocks [1]: > "Normally, you must write your applications so that they are always >

Re: [openstack-dev] [all] Replace eventlet with asyncio

2015-02-15 Thread Mike Bayer
ase of Trollius 1.0 > * July 14, 2014: Patch `Add a 'greenio' oslo.messaging executor (spec) > <https://review.openstack.org/#/c/104792/>`_ merged into > openstack/oslo-specs. > * July 7, 2014: Patch `Fix AMQPListener for polling with timeout > <https://review.openstack.or

[openstack-dev] [cinder] [oslo] MySQL connection shared between green threads concurrently

2015-02-16 Thread Mike Bayer
hi all - I’ve been researching this cinder issue https://bugs.launchpad.net/cinder/+bug/1417018 and I’ve found something concerning. Basically there’s a race condition here which is occurring because a single MySQL connection is shared between two green threads. This occurs while Cinder is d

Re: [openstack-dev] [cinder] [oslo] MySQL connection shared between green threads concurrently

2015-02-17 Thread Mike Bayer
Doug Hellmann wrote: >> >> So I’m not really sure what’s going on here. Cinder seems to have some >> openstack greenlet code of its own in >> cinder/openstack/common/threadgroup.py, I don’t know the purpose of this >> code. SQLAlchemy’s connection pool has been tested a lot with eventlet >>

Re: [openstack-dev] [cinder] [oslo] MySQL connection shared between green threads concurrently

2015-02-17 Thread Mike Bayer
Mike Bayer wrote: > > I haven’t confirmed this yet today but based on some greenlet research as > well as things I observed with PDB yesterday, my suspicion is that Cinder’s > startup code runs in a traditional thread, at the same time the service is > allowing connections

Re: [openstack-dev] [cinder] [oslo] MySQL connection shared between green threads concurrently

2015-02-17 Thread Mike Bayer
Doug Hellmann wrote: >> My question is then how is it that such an architecture would be >> possible, >> that Cinder’s service starts up without greenlets yet allows >> greenlet-based >> requests to come in before this critical task is complete? Shouldn’t the >> various oslo systems be providin

Re: [openstack-dev] [cinder] [oslo] MySQL connection shared between green threads concurrently

2015-02-17 Thread Mike Bayer
Mike Bayer wrote: > > > Doug Hellmann wrote: > >>> My question is then how is it that such an architecture would be >>> possible, >>> that Cinder’s service starts up without greenlets yet allows >>> greenlet-based >>> requests to com

Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-19 Thread Mike Bayer
Doug Hellmann wrote: >> 5) Allow this sort of connection sharing to continue for a deprecation >> period with apppropriate logging, then make it a hard failure. >> >> This would provide services time to find and fix any sharing problems >> they might have, but would delay the timeframe for a f

Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-19 Thread Mike Bayer
Joshua Harlow wrote: > Doug Hellmann wrote: >> On Thu, Feb 19, 2015, at 01:09 PM, Ben Nemec wrote: >>> Hi, >>> >>> Mike Bayer recently tracked down an issue with database errors in Cinder >>> to a single database connection being shared over multi

Re: [openstack-dev] [Keystone] Deprecation of Eventlet deployment in Kilo (Removal for "M"-release)

2015-02-19 Thread Mike Bayer
Morgan Fainberg wrote: > The Keystone development team is planning to deprecate deployment of Keystone > under Eventlet during the Kilo cycle. Support for deploying under eventlet > will be dropped as of the “M”-release of OpenStack. > > The reasoning behind this move is multifaceted but the

Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-20 Thread Mike Bayer
Doug Hellmann wrote: > > And I don't think we want the database library doing anything with this > case at all. Recovery code is tricky, and often prevents valid use cases > (perhaps the parent *meant* for the child to reuse the open connection > and isn't going to continue using it so there w

Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-20 Thread Mike Bayer
Doug Hellmann wrote: > > > On Fri, Feb 20, 2015, at 11:28 AM, Mike Bayer wrote: >> Doug Hellmann wrote: >> >>> And I don't think we want the database library doing anything with this >>> case at all. Recovery code is tricky, and often prevents va

Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-21 Thread Mike Bayer
Yuriy Taraday wrote: > On Fri Feb 20 2015 at 9:14:30 PM Joshua Harlow wrote: > > This feels like something we could do in the service manager base class, > > maybe by adding a "post fork" hook or something. > > +1 to that. > > I think it'd be nice to have the service __init__() maybe be some

Re: [openstack-dev] olso.db 1.5.0 release

2015-03-02 Thread Mike Bayer
Matt Riedemann wrote: > > > On 2/26/2015 3:22 AM, Victor Sergeyev wrote: >> Hi folks! >> >> The Oslo team is pleased to announce the release of: oslo.db - OpenStack >> common DB library >> >> Changes from the previous release: >> >> $ git log --oneline --no-merges 1.4.1..1.5.0 >> 7bfdb6a M

Re: [openstack-dev] [nova] blueprint about multiple workers supported in nova-scheduler

2015-03-04 Thread Mike Bayer
Attila Fazekas wrote: > Hi, > > I wonder what is the planned future of the scheduling. > > The scheduler does a lot of high field number query, > which is CPU expensive when you are using sqlalchemy-orm. > Does anyone tried to switch those operations to sqlalchemy-core ? An upcoming feature

Re: [openstack-dev] [all] SQLAlchemy performance suite and upcoming features (was: [nova] blueprint about multiple workers)

2015-03-04 Thread Mike Bayer
Mike Bayer wrote: > > > Attila Fazekas wrote: > >> Hi, >> >> I wonder what is the planned future of the scheduling. >> >> The scheduler does a lot of high field number query, >> which is CPU expensive when you are using sqlalchemy-orm. &g

Re: [openstack-dev] [all] SQLAlchemy performance suite and upcoming features (was: [nova] blueprint about multiple workers)

2015-03-05 Thread Mike Bayer
worth to convert the results to dict, > when you access the data multiple times. > > dict is also the typical input type for the json serializers. > > The plain dict is good enough if you do not want to mange > which part is changed, especially when you are not planning to

Re: [openstack-dev] [horizon] Do No Evil

2015-03-08 Thread Mike Bayer
Ian Wells wrote: > With apologies for derailing the question, but would you care to tell us what > evil you're planning on doing? I find it's always best to be informed about > these things. All of us, every day, do lots of things that someone is going to think is evil. From eating meat, to

Re: [openstack-dev] [Keystone]ON DELETE RESTRICT VS ON DELETE CASCADE

2015-03-08 Thread Mike Bayer
Morgan Fainberg wrote: > In general I'd say that cascade is the right approach. There are some very > limited cases where restrict should be used. Overall, I'd like to see less > reliance on FK constraints anywhere. can you elaborate on your reasoning that FK constraints should be used less

Re: [openstack-dev] [Keystone]ON DELETE RESTRICT VS ON DELETE CASCADE

2015-03-09 Thread Mike Bayer
; OpenStack Development Mailing List (not for usage questions) > Subject: Re: [openstack-dev] [Keystone]ON DELETE RESTRICT VS ON DELETE CASCADE > > > > On March 8, 2015 at 11:24:37 AM, David Stanek (dsta...@dstanek.com) wrote: > > > On Sun, Mar 8, 2015 at 1:37 PM

Re: [openstack-dev] [Keystone]ON DELETE RESTRICT VS ON DELETE CASCADE

2015-03-09 Thread Mike Bayer
Clint Byrum wrote: > Excerpts from David Stanek's message of 2015-03-08 11:18:05 -0700: >> On Sun, Mar 8, 2015 at 1:37 PM, Mike Bayer wrote: >> >>> can you elaborate on your reasoning that FK constraints should be used less >>> overall? or do you ju

Re: [openstack-dev] [Keystone]ON DELETE RESTRICT VS ON DELETE CASCADE

2015-03-09 Thread Mike Bayer
Morgan Fainberg wrote: > > > On Monday, March 9, 2015, Mike Bayer wrote: > > > Wei D wrote: > > > +1, > > > > > > > > I am fan of checking the constraints in the controller level instead of > > relying on FK constraints itself,

Re: [openstack-dev] [Keystone]ON DELETE RESTRICT VS ON DELETE CASCADE

2015-03-09 Thread Mike Bayer
Clint Byrum wrote: > > So I think I didn't speak clearly enough here. The benchmarks are of > course needed, but there's a tipping point when write activity gets to > a certain level where it's cheaper to let it get a little skewed and > correct asynchronously. This is not unique to SQL, this

Re: [openstack-dev] [Keystone]ON DELETE RESTRICT VS ON DELETE CASCADE

2015-03-10 Thread Mike Bayer
Clint Byrum wrote: > > Please try to refrain from using false equivalence. ACID stands for > Atomicity, Consistency, Isolation, Durability. Nowhere in there does it > stand for "referential integrity”. This point is admittedly controversial as I’ve had this debate before, but it is common th

Re: [openstack-dev] [Keystone]ON DELETE RESTRICT VS ON DELETE CASCADE

2015-03-10 Thread Mike Bayer
Adam Young wrote: > On 03/09/2015 01:26 PM, Mike Bayer wrote: >> Im about -1000 on disabling foreign key constraints. > So was I. We didn't do it out of performance. > > Since I am responsible for tipping over this particular cow, let me explain. > > No, is too

Re: [openstack-dev] [Keystone]ON DELETE RESTRICT VS ON DELETE CASCADE

2015-03-10 Thread Mike Bayer
Mike Bayer wrote: > >> I'm not entirely sure what you've said above actually prevents coders >> from relying on the constraints. Being careful about deleting all of the >> child rows before a parent is good practice. I have seen code like this &g

Re: [openstack-dev] [cinder] cinder is broken until someone fixes the forking code

2015-03-11 Thread Mike Bayer
er than fail hard. But Cinder should really at least prevent against this whole condition from occurring anyway. > -Josh > > Mike Bayer wrote: >> Hello Cinder - >> >> I’d like to note that for issue >> https://bugs.launchpad.net/oslo.db/+bug/1417018, no solut

[openstack-dev] [cinder] cinder is broken until someone fixes the forking code

2015-03-12 Thread Mike Bayer
Hello Cinder - I’d like to note that for issue https://bugs.launchpad.net/oslo.db/+bug/1417018, no solution that actually solves the problem for Cinder is scheduled to be committed anywhere. The patch I proposed for oslo.db is on hold, and the patch proposed for oslo.incubator in the service code

Re: [openstack-dev] [cinder] cinder is broken until someone fixes the forking code

2015-03-12 Thread Mike Bayer
Mike Perez wrote: > On 11:49 Wed 11 Mar , Walter A. Boring IV wrote: >> We have this patch in review currently. I think this one should >> 'fix' it no? >> >> Please review. >> >> https://review.openstack.org/#/c/163551/ > > Loo

[openstack-dev] [nova] if by "archived" you mean, "wipes out your tables completely", then sure, it works fine

2015-03-12 Thread Mike Bayer
Hello Nova - Not sure if I’m just staring at this for too long, or if archive_deleted_rows_for_table() is just not something we ever use. Because it looks like it’s really, really broken very disastrously, and I’m wondering if I’m just missing something in front of me. Let’s look at what it do

Re: [openstack-dev] [nova] if by "archived" you mean, "wipes out your tables completely", then sure, it works fine

2015-03-13 Thread Mike Bayer
t/nova/+spec/db-purge-engine > [3] > > ----- Original Message - >> From: "Mike Bayer" >> To: "OpenStack Development Mailing List (not for usage questions)" >> >> Sent: Friday, March 13, 2015 12:29:55 AM >> Subject: [openstack-dev] [nova] if b

Re: [openstack-dev] [Keystone]ON DELETE RESTRICT VS ON DELETE CASCADE

2015-03-13 Thread Mike Bayer
Adam Young wrote: > On 03/10/2015 10:23 AM, Mike Bayer wrote: >> if *that’s* >> what you mean, that’s known as a “polymorphic foreign key”, and >> it is not actually a foreign key at all, it is a terrible antipattern >> started by >> the PHP/Rails community a

Re: [openstack-dev] [all][oslo.db] Repeatable Read considered harmful

2015-03-31 Thread Mike Bayer
Eugene Nikanorov wrote: > Hi Matthew, > > I'll add just 2c: > > We've tried to move from repeatable-read to read committed in Neutron project. > This change actually has caused multiple deadlocks during regular tempest > test run. > That is a known problem (the issue with eventlet and currec

[openstack-dev] [all] SQLAlchemy-related topics in the Vancouver summit

2015-03-31 Thread Mike Bayer
hey all - Just a heads up that I am booked to attend the Vancouver summit.And I have almost nothing to do. So please reach out and invite me to your database-related design sessions, so that I can help out with SQLAlchemy, Alembic/Migrate, and oslo.db feature support (with props to dogpile

Re: [openstack-dev] [Keystone] SQLite support (migrations, work-arounds, and more), is it worth it?

2015-04-06 Thread Mike Bayer
On 4/6/15 12:06 PM, Clint Byrum wrote: Excerpts from Boris Bobrov's message of 2015-04-03 18:29:08 -0700: On Saturday 04 April 2015 03:55:59 Morgan Fainberg wrote: I am looking forward to the Liberty cycle and seeing the special casing we do for SQLite in our migrations (and elsewhere). My in

Re: [openstack-dev] [Keystone] SQLite support (migrations, work-arounds, and more), is it worth it?

2015-04-06 Thread Mike Bayer
mysqld service running is the lesser of two evils and you get a lot more code coverage by going all the way out to the DB. On Mon, Apr 6, 2015, 12:42 Morgan Fainberg <mailto:morgan.fainb...@gmail.com>> wrote: > On Apr 6, 2015, at 09:20, Mike Bayer mailto:mba...@redhat.com

Re: [openstack-dev] [Keystone] SQLite support (migrations, work-arounds, and more), is it worth it?

2015-04-06 Thread Mike Bayer
On 4/6/15 2:52 PM, Morgan Fainberg wrote: On Apr 6, 2015, at 11:41, Mike Bayer <mailto:mba...@redhat.com>> wrote: On 4/6/15 12:49 PM, David Stanek wrote: Exactly. This is the direction I have been going. Functional tests are written using the public APIs using the client.

[openstack-dev] Re: [oslo] eventlet 0.17.3 is now fully Python 3 compatible

2015-04-13 Thread Mike Bayer
On 4/13/15 8:54 AM, Jeremy Stanley wrote: On 2015-04-13 04:03:49 -0400 (-0400), Victor Stinner wrote: Great. Did you notice a performance regression? Nope. Worth noting, we implemented it primarily for its lack of compiled extensions, and to a lesser because it supports Python 3.x. I suspect

Re: [openstack-dev] [Heat] moving sqlite migration scripts to tests

2015-04-17 Thread Mike Bayer
On 4/17/15 1:29 PM, Zane Bitter wrote: On 16/04/15 04:05, Anant Patil wrote: Hi, Sometime back we had a discussion on IRC regarding sqlite migration scripts. Since sqlite is mostly used for testing, we were thinking about moving the sqlite migration related code to tests folder and keep the m

Re: [openstack-dev] [glance] Why no DB index on sort parameters

2015-04-21 Thread Mike Bayer
On 4/21/15 2:47 AM, Ajaya Agrawal wrote: Hi All, I see that glance supports arbitrary sort parameters and the default is "created_at" while listing images. Is there any reason why we don't have index over these fields? If we have an index over these fields then we would avoid a full table s

[openstack-dev] Re: [oslo] eventlet 0.17.3 is now fully Python 3 compatible

2015-04-22 Thread Mike Bayer
On 4/22/15 7:19 AM, Victor Sergeyev wrote: Hi, All, My 2c are: - yes, oslo.db supports python 3 (unittests passes, at least :) ) - MySQL-python still default MySQL DB driver in OpenStack, but at the moment the only DB driver for MySQL in python3 environment is PyMySQL, so I think, it's ok t

Re: [openstack-dev] [nova] Port Nova to Python 3

2015-04-24 Thread Mike Bayer
On 4/24/15 4:18 PM, Sean Toner wrote: On Friday, April 24, 2015 11:20:03 AM Joshua Harlow wrote: Sean Toner wrote: If written to use python 3, I hope it will use all the new features of python 3.4 moving forward. For example, argument annotations, coroutines, asyncio, etc. At my last workpl

Re: [openstack-dev] [all] Replace mysql-python with mysqlclient

2015-04-30 Thread Mike Bayer
On 4/30/15 11:00 AM, Victor Stinner wrote: Hi, I propose to replace mysql-python with mysqlclient in OpenStack applications to get Python 3 support, bug fixes and some new features (support MariaDB's libmysqlclient.so, support microsecond in TIME column). It is not feasible to use MySQLcli

Re: [openstack-dev] [all] Replace mysql-python with mysqlclient

2015-04-30 Thread Mike Bayer
On 4/30/15 11:16 AM, Dan Smith wrote: There is an open discussion to replace mysql-python with PyMySQL, but PyMySQL has worse performance: https://wiki.openstack.org/wiki/PyMySQL_evaluation My major concern with not moving to something different (i.e. not based on the C library) is the thread

Re: [openstack-dev] [all] Replace mysql-python with mysqlclient

2015-05-05 Thread Mike Bayer
On 5/4/15 6:48 PM, Thomas Goirand wrote: I don't see what it would break. If I do: Package: python-mysqlclient Breaks: python-mysqldb Replaces: python-mysqldb Provides: python-mysqldb everything is fine, and python-mysqlclient becomes another implementation of the same thing. Then I believe

Re: [openstack-dev] [all] Replace mysql-python with mysqlclient

2015-05-05 Thread Mike Bayer
On 5/5/15 1:11 PM, Thomas Goirand wrote: On 04/30/2015 05:00 PM, Victor Stinner wrote: Hi, I propose to replace mysql-python with mysqlclient in OpenStack applications to get Python 3 support, bug fixes and some new features (support MariaDB's libmysqlclient.so, support microsecond in TIM

Re: [openstack-dev] [all] Replace mysql-python with mysqlclient

2015-05-05 Thread Mike Bayer
On 5/5/15 3:07 PM, Mike Bayer wrote: On 5/5/15 1:11 PM, Thomas Goirand wrote: On 04/30/2015 05:00 PM, Victor Stinner wrote: Hi, I propose to replace mysql-python with mysqlclient in OpenStack applications to get Python 3 support, bug fixes and some new features (support MariaDB&#

[openstack-dev] Re: [oslo] Adding Mehdi Abaakouk (sileht) to oslo-core

2015-05-07 Thread Mike Bayer
On 5/7/15 11:01 AM, Joshua Harlow wrote: +1 for Mehdi, hooray to that! http://gph.is/19n19VQ (haha), -Josh +1, welcome aboard ozamiatin wrote: +1 for adding Mehdi to oslo-core! Thanks, Oleksii Zamiatin 07.05.15 17:36, Davanum Srinivas пишет: Dear Oslo folks, I'd like to propose ad

Re: [openstack-dev] [all] Replace mysql-python with mysqlclient

2015-05-07 Thread Mike Bayer
On 5/7/15 5:32 PM, Thomas Goirand wrote: If there are really fixes and features we need in Py2K then of course we have to either convince MySQLdb to merge them or switch to mysqlclient. Given the "no reply in 6 months" I think that's enough to say it: mysql-python is a dangerous package wit

Re: [openstack-dev] [all] Replace mysql-python with mysqlclient

2015-05-08 Thread Mike Bayer
: http://ronaldbradford.com <http://ronaldbradford.com/> LinkedIn: http://www.linkedin.com/in/ronaldbradford Twitter: @RonaldBradford <http://twitter.com/ronaldbradford> Skype: RonaldBradford GTalk: Ronald.Bradford On Thu, May 7, 2015 at 9:39 PM, Mike Bayer mail

Re: [openstack-dev] [all] Replace mysql-python with mysqlclient

2015-05-09 Thread Mike Bayer
On 5/9/15 6:45 AM, John Garbutt wrote: I am leaning towards us moving to making DB calls with a thread pool and some fast C based library, so we get the 'best' performance. Is that a crazy thing to be thinking? What am I missing here? Thanks, John I'd like to do that but I want the whole Op

Re: [openstack-dev] [all] Replace mysql-python with mysqlclient

2015-05-11 Thread Mike Bayer
all] Replace mysql-python with mysqlclient On 30 April 2015 at 18:54, Mike Bayer wrote: On 4/30/15 11:16 AM, Dan Smith wrote: There is an open discussion to replace mysql-python with PyMySQL, but PyMySQL has worse performance: https://wiki.openstack.org/wiki/PyMySQL_evaluation My major concern with not

Re: [openstack-dev] [all] Replace mysql-python with mysqlclient

2015-05-11 Thread Mike Bayer
On 5/11/15 2:02 PM, Attila Fazekas wrote: Not just with local database connections, the 10G network itself also fast. Is is possible you spend more time even on the kernel side tcp/ip stack (and the context switch..) (Not in physical I/O wait) than in the actual work on the DB side. (Check ne

Re: [openstack-dev] [all] Replace mysql-python with mysqlclient

2015-05-11 Thread Mike Bayer
On 5/11/15 5:25 PM, Robert Collins wrote: Details: Skip over this bit if you know it all already. The GIL plays a big factor here: if you want to scale the amount of CPU available to a Python service, you have two routes: A) move work to a different process through some RPC - be that DB's usi

Re: [openstack-dev] [all] Replace mysql-python with mysqlclient

2015-05-12 Thread Mike Bayer
On 5/11/15 9:17 PM, Robert Collins wrote: On 12 May 2015 at 10:44, Mike Bayer wrote: What we have today in our standard architecture for OpenStack is optimised for IO bound workloads: waiting on the network/subprocesses/disk/libvirt etc. Running high numbers of eventlet handlers in a single

Re: [openstack-dev] [nova][heat] sqlalchemy-migrate tool to alembic

2015-05-14 Thread Mike Bayer
On 5/14/15 5:44 AM, John Garbutt wrote: On 14 May 2015 at 07:18, Manickam, Kanagaraj wrote: Hi Nova team, This mail is regarding an help required on the migration from sqlalchemy migration tool to alembic tool. Heat is currently using sqlalchemy-migration tool and In liberty release, we ar

Re: [openstack-dev] [nova][heat] sqlalchemy-migrate tool to alembic

2015-05-14 Thread Mike Bayer
On 5/14/15 11:40 AM, Mike Bayer wrote: Assuming I can get that done in the next few months, the next step would be that the migration streams can be broken into branches, e.g. juno, kilo, liberty, etc. so that we can easily add new migration files that are backportable in place to a

[openstack-dev] [oslo][requirements][all] requesting assistance to unblock SQLAlchemy 1.1 from requirements

2017-03-14 Thread Mike Bayer
Hello all - As mentioned previously, SQLAlchemy 1.1 has now been released for about six months. My work now is on SQLAlchemy 1.2 which should hopefully see initial releases in late spring.SQLAlchemy 1.1 includes tons of features, bugfixes, and improvements, and in particular the most rec

Re: [openstack-dev] [oslo][requirements][all] requesting assistance to unblock SQLAlchemy 1.1 from requirements

2017-03-15 Thread Mike Bayer
On 03/15/2017 07:30 AM, Sean Dague wrote: The problem was the original patch kept a cap on SQLA, just moved it up to the next pre-release, not realizing the caps in general are the concern by the requirements team. So instead of upping the cap, I just removed it entirely. (It also didn't help

Re: [openstack-dev] [oslo][requirements][all] requesting assistance to unblock SQLAlchemy 1.1 from requirements

2017-03-15 Thread Mike Bayer
ts? On Wed, Mar 15, 2017 at 5:20 PM, Sean Dague wrote: On 03/15/2017 10:38 AM, Mike Bayer wrote: On 03/15/2017 07:30 AM, Sean Dague wrote: The problem was the original patch kept a cap on SQLA, just moved it up to the next pre-release, not realizing the caps in general are the concern by the r

Re: [openstack-dev] [Zun]Use 'uuid' instead of 'id' as object ident in data model

2017-04-06 Thread Mike Bayer
On 04/05/2017 11:00 AM, Monty Taylor wrote: On 04/05/2017 09:39 AM, Akihiro Motoki wrote: I noticed this thread by Monty's reply. Sorry for my late :( I think we need to think 'id' separately for API modeling and DB modeling. In the API perspective, one of the important things is that 'id' i

Re: [openstack-dev] [Zun]Use 'uuid' instead of 'id' as object ident in data model

2017-04-06 Thread Mike Bayer
On 04/05/2017 11:02 AM, gordon chung wrote: On 05/04/17 09:00 AM, Monty Taylor wrote: Please do NOT use uuid as a primary key in MySQL: * UUID has 36 characters which makes it bulky. you can store it as a binary if space is a concern. this is highly inconvenient from a datadump / MySQL

Re: [openstack-dev] [tc] revised Postgresql support status patch for governance

2017-05-18 Thread Mike Bayer
On 05/17/2017 02:38 PM, Sean Dague wrote: Some of the concerns/feedback has been "please describe things that are harder by this being an abstraction", so examples are provided. so let's go through this list: - OpenStack services taking a more active role in managing the DBMS , "managi

Re: [openstack-dev] [oslo] Can we stop global requirements update?

2017-05-18 Thread Mike Bayer
On 05/16/2017 05:42 AM, Julien Danjou wrote: On Wed, Apr 19 2017, Julien Danjou wrote: So Gnocchi gate is all broken (agan) because it depends on "pbr" and some new release of oslo.* depends on pbr!=2.1.0. Same things happened today with Babel. As far as Gnocchi is concerned, we're goin

<    1   2   3   4   >