Re: [openstack-dev] [oslo.db]A proposal for DB read/write separation

2014-08-08 Thread Mike Bayer
On Aug 8, 2014, at 12:03 AM, Li Ma skywalker.n...@gmail.com wrote: So, I'd like to propose a transparent read/write separation method for oslo.db that every project may happily takes advantage of it without any code modification. A single transaction begins, which is to emit a series of

Re: [openstack-dev] [oslo.db]A proposal for DB read/write separation

2014-08-10 Thread Mike Bayer
and integrating that into oslo.db or sqlalchemy. Mike Bayer has some thoughts on that[1] and there are other approaches around that can be copied/learned from. These sorts of things are clear to me and while moving towards more transparency for the developer, still require context. Please, share

Re: [openstack-dev] [oslo.db]A proposal for DB read/write separation

2014-08-10 Thread Mike Bayer
On Aug 10, 2014, at 11:17 AM, Li Ma skywalker.n...@gmail.com wrote: How about Galera multi-master cluster? As Mike Bayer said, it is virtually synchronous by default. It is still possible that outdated rows are queried that make results not stable. not sure if I said that :). I know

Re: [openstack-dev] [nova][core] Expectations of core reviewers

2014-08-13 Thread Mike Bayer
On Aug 13, 2014, at 1:44 PM, Russell Bryant rbry...@redhat.com wrote: On 08/13/2014 01:09 PM, Dan Smith wrote: Expecting cores to be at these sorts of things seems pretty reasonable to me, given the usefulness (and gravity) of the discussions we've been having so far. Companies with more

[openstack-dev] [all] Acceptable methods for establishing per-test-suite behaviors

2014-08-22 Thread Mike Bayer
Hi all - I’ve spent many weeks on a series of patches for which the primary goal is to provide very efficient patterns for tests that use databases and schemas within those databases, including compatibility with parallel tests, transactional testing, and scenario-driven testing (e.g. a test

Re: [openstack-dev] [Openstack-stable-maint] [Neutron][stable] How to backport database schema fixes

2014-08-29 Thread Mike Bayer
On Aug 29, 2014, at 7:23 AM, Alan Pevec ape...@gmail.com wrote: It seems that currently it's hard to backport any database schema fix to Neutron [1] which uses alembic to manage db schema version. Nova has the same issue before and a workaround is to put some placeholder files before each

Re: [openstack-dev] Kilo Cycle Goals Exercise

2014-09-08 Thread Mike Bayer
On Sep 7, 2014, at 9:27 PM, Anita Kuno ante...@anteaya.info wrote: On 09/07/2014 09:12 PM, Angus Salkeld wrote: Lets prevent blogs like this: http://jimhconsulting.com/?p=673 by making users happy. I don't understand why you would encourage writers of blog posts you disagree with by sending

Re: [openstack-dev] Kilo Cycle Goals Exercise

2014-09-08 Thread Mike Bayer
On Sep 7, 2014, at 8:14 PM, Monty Taylor mord...@inaugust.com wrote: 2. Less features, more win In a perfect world, I'd argue that we should merge exactly zero new features in all of kilo, and instead focus on making the ones we have work well. Some of making the ones we have work

Re: [openstack-dev] Kilo Cycle Goals Exercise

2014-09-08 Thread Mike Bayer
On Sep 8, 2014, at 11:30 AM, Anita Kuno ante...@anteaya.info wrote: Wow, we are really taking liberties with my question today. What part of any of my actions current or previous have led you to believe that I want to now or ever have silenced anyone? I am curious what led you to believe

Re: [openstack-dev] memory usage in devstack-gate (the oom-killer strikes again)

2014-09-08 Thread Mike Bayer
Hi All - Joe had me do some quick memory profiling on nova, just an FYI if anyone wants to play with this technique, I place a little bit of memory profiling code using Guppy into nova/api/__init__.py, or anywhere in your favorite app that will definitely get imported when the thing first

Re: [openstack-dev] memory usage in devstack-gate (the oom-killer strikes again)

2014-09-09 Thread Mike Bayer
of them, and then add it to our standard toolset. On Sep 9, 2014, at 10:35 AM, Doug Hellmann d...@doughellmann.com wrote: On Sep 8, 2014, at 8:12 PM, Mike Bayer mba...@redhat.com wrote: Hi All - Joe had me do some quick memory profiling on nova, just an FYI if anyone wants to play

Re: [openstack-dev] [all] i need some help on this bug Bug #1365892

2014-09-10 Thread Mike Bayer
On Sep 10, 2014, at 4:11 AM, Li Tianqing jaze...@163.com wrote: After some research, i find the reason for the cycle reference. In closure, the _fix_paswords.func_closre reference the _fix_passwords. So the cycle reference happened. And in

Re: [openstack-dev] [neutron][all] switch from mysqldb to another eventlet aware mysql client

2014-09-12 Thread Mike Bayer
On Sep 12, 2014, at 7:20 AM, Sean Dague s...@dague.net wrote: Because we are in Feature Freeze. Now is the time for critical bug fixes only, as we start to stabalize the tree. Releasing dependent libraries that can cause breaks, for whatever reason, should be soundly avoided. If this was

[openstack-dev] battling stale .pyc files

2014-09-12 Thread Mike Bayer
I’ve just found https://bugs.launchpad.net/nova/+bug/1368661, Unit tests sometimes fail because of stale pyc files”. The issue as stated in the report refers to the phenomenon of .pyc files that remain inappropriately, when switching branches or deleting files. Specifically, the kind of

Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini

2014-09-12 Thread Mike Bayer
On Sep 12, 2014, at 7:39 AM, Sean Dague s...@dague.net wrote: I assume you, gentle OpenStack developers, often find yourself in a hair tearing out moment of frustration about why local unit tests are doing completely insane things. The code that it is stack tracing on is no where to be

Re: [openstack-dev] [neutron][all] switch from mysqldb to another eventlet aware mysql client

2014-09-12 Thread Mike Bayer
On Sep 12, 2014, at 10:40 AM, Ihar Hrachyshka ihrac...@redhat.com wrote: Signed PGP part On 12/09/14 16:33, Mike Bayer wrote: I agree with this, changing the MySQL driver now is not an option. That was not the proposal. The proposal was to introduce support to run against something

Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini

2014-09-12 Thread Mike Bayer
On Sep 12, 2014, at 11:24 AM, Sean Dague s...@dague.net wrote: On 09/12/2014 11:21 AM, Mike Bayer wrote: On Sep 12, 2014, at 7:39 AM, Sean Dague s...@dague.net wrote: I assume you, gentle OpenStack developers, often find yourself in a hair tearing out moment of frustration about why

Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini

2014-09-12 Thread Mike Bayer
On Sep 12, 2014, at 11:33 AM, Mike Bayer mba...@redhat.com wrote: On Sep 12, 2014, at 11:24 AM, Sean Dague s...@dague.net wrote: On 09/12/2014 11:21 AM, Mike Bayer wrote: On Sep 12, 2014, at 7:39 AM, Sean Dague s...@dague.net wrote: I assume you, gentle OpenStack developers, often

Re: [openstack-dev] [neutron][all] switch from mysqldb to another eventlet aware mysql client

2014-09-12 Thread Mike Bayer
On Sep 12, 2014, at 11:56 AM, Johannes Erdfelt johan...@erdfelt.com wrote: On Fri, Sep 12, 2014, Doug Hellmann d...@doughellmann.com wrote: I don’t think we will want to retroactively change the migration scripts (that’s not something we generally like to do), We don't allow semantic

Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini

2014-09-12 Thread Mike Bayer
On Sep 12, 2014, at 12:03 PM, Sean Dague s...@dague.net wrote: On 09/12/2014 11:33 AM, Mike Bayer wrote: On Sep 12, 2014, at 11:24 AM, Sean Dague s...@dague.net wrote: On 09/12/2014 11:21 AM, Mike Bayer wrote: On Sep 12, 2014, at 7:39 AM, Sean Dague s...@dague.net wrote: I assume

Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini

2014-09-12 Thread Mike Bayer
On Sep 12, 2014, at 12:13 PM, Jeremy Stanley fu...@yuggoth.org wrote: On 2014-09-12 11:36:20 -0400 (-0400), Mike Bayer wrote: [...] not to mention PYTHONHASHSEED only works on Python 3. What is the issue in tox you’re referring to ? Huh? The overrides we added to numerous projects

Re: [openstack-dev] [all] PYTHONDONTWRITEBYTECODE=true in tox.ini

2014-09-12 Thread Mike Bayer
On Sep 12, 2014, at 12:29 PM, Daniel P. Berrange berra...@redhat.com wrote: On Fri, Sep 12, 2014 at 04:23:09PM +, Jeremy Stanley wrote: On 2014-09-12 17:16:11 +0100 (+0100), Daniel P. Berrange wrote: [...] Agreed, the problem with stale .pyc files is that it never occurs to developers

Re: [openstack-dev] [Octavia] Which threading library?

2014-09-13 Thread Mike Bayer
On Sep 12, 2014, at 5:53 PM, Eichberger, German german.eichber...@hp.com wrote: Hi, I think the “standard” threading library for OpenStack is eventlet – however, it seems that Oslo is spearheading efforts to get to a more compatible one (see

Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)

2014-09-17 Thread Mike Bayer
On Sep 17, 2014, at 2:46 PM, Clint Byrum cl...@fewbar.com wrote: Excerpts from Davanum Srinivas's message of 2014-09-17 10:15:29 -0700: I was trying request-ifying oslo.vmware and ran into this as well: https://review.openstack.org/#/c/121956/ And we don't seem to have urllib3 in

Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)

2014-09-17 Thread Mike Bayer
On Sep 17, 2014, at 3:42 PM, Ian Cordasco ian.corda...@rackspace.com wrote: Circling back to the issue of vendoring though: it’s a conscious decision to do this, and in the last two years there have been 2 CVEs reported for requests. There have been none for urllib3 and none for chardet.

Re: [openstack-dev] Please do *NOT* use vendorized versions of anything (here: glanceclient using requests.packages.urllib3)

2014-09-17 Thread Mike Bayer
On Sep 17, 2014, at 4:31 PM, Ian Cordasco ian.corda...@rackspace.com wrote: Project X pins a version of requests. Alice doesn’t know anything about requests and does pip install X. Until Alice takes a more active role in the development of Project X and looks into requests, she will never

Re: [openstack-dev] [neutron][db] Ensuring db isolation between tests

2014-09-18 Thread Mike Bayer
I’ve done a lot of work on this issue and from my perspective, the code is mostly ready to go, however we’re in an extended phase of getting folks to sign off as well as that I’m waiting for some last-minute fixup from Robert Collins. Patch: [1] Blueprint, which is to be moved to Kilo: [2]

Re: [openstack-dev] [all] [glance] python namespaces considered harmful to development, lets not introduce more of them

2014-09-23 Thread Mike Bayer
On Sep 23, 2014, at 7:03 PM, Robert Collins robe...@robertcollins.net wrote: On 29 August 2014 04:42, Sean Dague s...@dague.net wrote: On 08/28/2014 12:22 PM, Doug Hellmann wrote: ... The problem is that the setuptools implementation of namespace packages breaks in a way that is repeatable

Re: [openstack-dev] Gate proposal - drop Postgresql configurations in the gate

2014-06-12 Thread Mike Bayer
On 6/12/14, 8:26 AM, Julien Danjou wrote: On Thu, Jun 12 2014, Sean Dague wrote: That's not cacthable in unit or functional tests? Not in an accurate manner, no. Keeping jobs alive based on the theory that they might one day be useful is something we just don't have the liberty to do any

Re: [openstack-dev] [Oslo][DB] oslo.db released

2014-06-19 Thread Mike Bayer
On 6/19/14, 3:56 PM, Doug Hellmann wrote: On Thu, Jun 19, 2014 at 3:20 PM, Victor Sergeyev vserge...@mirantis.com wrote: Hello, Folks! Please, be informed, that the oslo.db library has been released and is available on PyPi. See [1] for the source code, [2] for the documentation. There

[openstack-dev] [oslo] Openstack and SQLAlchemy

2014-06-30 Thread Mike Bayer
Hi all - For those who don't know me, I'm Mike Bayer, creator/maintainer of SQLAlchemy, Alembic migrations and Dogpile caching. In the past month I've become a full time Openstack developer working for Red Hat, given the task of carrying Openstack's database integration story forward

Re: [openstack-dev] [oslo] Openstack and SQLAlchemy

2014-07-02 Thread Mike Bayer
to solve). https://wiki.openstack.org/wiki/OpenStack_and_SQLAlchemy#MySQLdb_.2B_eventlet_.3D_sad On 6/30/14, 12:56 PM, Mike Bayer wrote: Hi all - For those who don't know me, I'm Mike Bayer, creator/maintainer of SQLAlchemy, Alembic migrations and Dogpile caching. In the past month I've become

Re: [openstack-dev] [oslo] Asyncio and oslo.messaging

2014-07-07 Thread Mike Bayer
On 7/4/14, 4:45 AM, Julien Danjou wrote: On Thu, Jul 03 2014, Mark McLoughlin wrote: We're attempting to take baby-steps towards moving completely from eventlet to asyncio/trollius. The thinking is for Ceilometer to be the first victim. Thumbs up for the plan, that sounds like a good

Re: [openstack-dev] [oslo] Openstack and SQLAlchemy

2014-07-07 Thread Mike Bayer
On 7/7/14, 3:57 PM, Matt Riedemann wrote: Regarding the eventlet + mysql sadness, I remembered this [1] in the nova.db.api code. I'm not sure if that's just nova-specific right now, I'm a bit too lazy at the moment to check if it's in other projects, but I'm not seeing it in neutron, for

Re: [openstack-dev] [nova] fastest way to run individual tests ?

2014-07-10 Thread Mike Bayer
On 7/9/14, 4:51 PM, Matt Riedemann wrote: On 6/12/2014 6:17 AM, Daniel P. Berrange wrote: On Thu, Jun 12, 2014 at 07:07:37AM -0400, Sean Dague wrote: On 06/12/2014 06:59 AM, Daniel P. Berrange wrote: Does anyone have any tip on how to actually run individual tests in an efficient manner.

Re: [openstack-dev] Where should a test for eventlet and oslo.db interaction go?

2014-07-10 Thread Mike Bayer
On 7/10/14, 7:47 AM, Sean Dague wrote: Honestly, that seems weird to me. oslo.db is built as a common layer for OpenStack services. eventlet is used by most OpenStack services. There are lots of known issues with eventlet vs. our db access patterns. Knowing that the db layer works in the

Re: [openstack-dev] [nova] fastest way to run individual tests ?

2014-07-10 Thread Mike Bayer
On 7/10/14, 12:08 PM, Chris Dent wrote: On Thu, 10 Jul 2014, Mike Bayer wrote: I typically never use tox or testtools at the commandline until I'm ready to commit and want to see what the jenkins builds will see. I start up the whole thing and then it's time to take a break while

Re: [openstack-dev] [neutron][all] switch from mysqldb to another eventlet aware mysql client

2014-07-11 Thread Mike Bayer
On 7/9/14, 10:59 AM, Roman Podoliaka wrote: Hi all, Not sure what issues you are talking about, but I just replaced mysql with mysql+mysqlconnector in my db connection string in neutron.conf and neutron-db-manage upgrade head worked like a charm for an empty schema. Ihar, could please

Re: [openstack-dev] [neutron][all] switch from mysqldb to another eventlet aware mysql client

2014-07-11 Thread Mike Bayer
On 7/11/14, 7:26 PM, Carl Baldwin wrote: On Jul 11, 2014 5:32 PM, Vishvananda Ishaya vishvana...@gmail.com mailto:vishvana...@gmail.com wrote: I have tried using pymysql in place of mysqldb and in real world concurrency tests against cinder and nova it performs slower. I was inspired by

Re: [openstack-dev] [neutron][all] switch from mysqldb to another eventlet aware mysql client

2014-07-11 Thread Mike Bayer
On 7/11/14, 11:26 PM, Jay Pipes wrote: Yep, couldn't agree more. Frankly, the steps you outline in the wiki above are excellent examples of where we can make significant gains in both performance and scalability. In addition to those you listed, the underlying database schemas themselves,

Re: [openstack-dev] [oslo] Asyncio and oslo.messaging

2014-07-14 Thread Mike Bayer
On Jul 14, 2014, at 9:46 AM, Jay Pipes jaypi...@gmail.com wrote: The point of eventlet, I thought, was to hide the low-level stuff so that developers could focus on higher-level (and more productive) abstractions. Introducing asyncio contructs into the higher level code like Nova and

Re: [openstack-dev] [all] switch from mysqldb to another eventlet aware mysql client -- status of postgresql drivers?

2014-07-14 Thread Mike Bayer
On Jul 14, 2014, at 12:29 PM, Chris Friesen chris.frie...@windriver.com wrote: On 07/09/2014 05:17 AM, Ihar Hrachyshka wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA512 Hi all, Multiple projects are suffering from db lock timeouts due to deadlocks deep in mysqldb library that we

Re: [openstack-dev] [all] switch from mysqldb to another eventlet aware mysql client -- status of postgresql drivers?

2014-07-14 Thread Mike Bayer
On Jul 14, 2014, at 1:02 PM, Chris Friesen chris.frie...@windriver.com wrote: On 07/14/2014 10:41 AM, Mike Bayer wrote: On Jul 14, 2014, at 12:29 PM, Chris Friesen chris.frie...@windriver.com wrote: On 07/09/2014 05:17 AM, Ihar Hrachyshka wrote: -BEGIN PGP SIGNED MESSAGE- Hash

Re: [openstack-dev] [Infra] MyISAM as a default storage engine for MySQL in the gate

2014-07-21 Thread Mike Bayer
OK, so, we aren’t generally running neutron tests w/ MySQL + InnoDB, right? I happen to be running them locally against a MySQL that defaults to InnoDB. And I’m trying to see if it’s deadlocking or not as I’m not able to get through them. All the eventlet + MySQLdb deadlock issues won’t

Re: [openstack-dev] [Oslo] [Ironic] DB migration woes

2014-06-07 Thread Mike Bayer
) As far as I can tell, this is all coming from: https://github.com/openstack/oslo-incubator/blob/master/openstack/common/db/sqlalchemy/test_migrations.py#L86;L111 Hello - Just an introduction, I’m Mike Bayer, the creator of SQLAlchemy and Alembic migrations. I’ve just joined on as a full

Re: [openstack-dev] [Oslo] [Ironic] DB migration woes

2014-06-08 Thread Mike Bayer
On Jun 8, 2014, at 11:46 AM, Roman Podoliaka rpodoly...@mirantis.com wrote: Overall, the approach with executing a test within a transaction and then emitting ROLLBACK worked quite well. The only problem I ran into were tests doing ROLLBACK on purpose. But you've updated the recipe since

Re: [openstack-dev] [Neutron] One performance issue about VXLAN pool initiation

2014-06-08 Thread Mike Bayer
On Jun 7, 2014, at 4:38 PM, Eugene Nikanorov enikano...@mirantis.com wrote: Hi folks, There was a small discussion about the better way of doing sql operations for vni synchronization with the config. Initial proposal was to handle those in chunks. Carl also suggested to issue a single

Re: [openstack-dev] [Oslo] [Ironic] DB migration woes

2014-06-09 Thread Mike Bayer
commits the current transaction implicitly on MySQL and SQLite AFAIK). Thanks, Roman [1] https://review.openstack.org/#/c/33236/ [2] https://blueprints.launchpad.net/nova/+spec/db-api-tests-on-all-backends On Sat, Jun 7, 2014 at 10:27 PM, Mike Bayer mba...@redhat.com wrote: On Jun 6

Re: [openstack-dev] [Oslo] [Ironic] DB migration woes

2014-06-09 Thread Mike Bayer
On Jun 9, 2014, at 1:08 PM, Mike Bayer mba...@redhat.com wrote: On Jun 9, 2014, at 12:50 PM, Devananda van der Veen devananda@gmail.com wrote: There may be some problems with MySQL when testing parallel writes in different non-committing transactions, even in READ COMMITTED mode

Re: [openstack-dev] [Neutron][LBaaS] Migrations in feature branch

2014-09-25 Thread Mike Bayer
If Neutron is ready for more Alembic features I could in theory begin work on https://bitbucket.org/zzzeek/alembic/issue/167/multiple-heads-branch-resolution-support .Folks should ping me on IRC regarding this. On Sep 24, 2014, at 5:30 AM, Salvatore Orlando sorla...@nicira.com wrote:

Re: [openstack-dev] [nova][cinder] (OperationalError) (1040, 'Too many connections') None None

2014-09-29 Thread Mike Bayer
On Sep 28, 2014, at 5:56 PM, Nader Lahouti nader.laho...@gmail.com wrote: Hi All, I am seeing 'Too many connections' error in nova api and cinder when when installing openstack using the latest.. The error happens when launching couple of VMs (in this test around 5 VMs). Here are the

Re: [openstack-dev] [nova][cinder] (OperationalError) (1040, 'Too many connections') None None

2014-09-29 Thread Mike Bayer
On Sep 29, 2014, at 12:31 PM, Nader Lahouti nader.laho...@gmail.com wrote: Hi Jay, Thanks for your reply. I'm not able to use mysql command line. $ mysql ERROR 1040 (HY000): Too many connections $ Is there any other way to collect the information? you can try stopping

Re: [openstack-dev] [nova][cinder] (OperationalError) (1040, 'Too many connections') None None

2014-09-29 Thread Mike Bayer
what’s spooking me is the original paste at http://paste.openstack.org/show/116425/ which showed: icehouse: Fri Sep 26 17:00:50 PDT 2014 Number of open TCP:3306 - 58 Number of open TCP:3306 nova-api - 5 Number of open TCP:3306 mysqld - 29 Number of open TCP:8774 - 10 Number of nova-api - 14

Re: [openstack-dev] [Group-based Policy] Database migration chain

2014-10-04 Thread Mike Bayer
On Oct 4, 2014, at 1:10 AM, Kevin Benton blak...@gmail.com wrote: Does sqlalchemy have good support for cross-database foreign keys? I was under the impression that they cannot be implemented with the normal syntax and semantics of an intra-database foreign-key constraint. cross

Re: [openstack-dev] [Group-based Policy] Database migration chain

2014-10-04 Thread Mike Bayer
On Oct 4, 2014, at 11:24 AM, Clint Byrum cl...@fewbar.com wrote: Excerpts from Mike Bayer's message of 2014-10-04 08:10:38 -0700: On Oct 4, 2014, at 1:10 AM, Kevin Benton blak...@gmail.com wrote: Does sqlalchemy have good support for cross-database foreign keys? I was under the

Re: [openstack-dev] [all][oslo][db][docs] RFC: drop support for libpq 9.1

2014-10-06 Thread Mike Bayer
On Oct 6, 2014, at 9:56 AM, Ihar Hrachyshka ihrac...@redhat.com wrote: But we can do better. We should also enforce utf8 on client side, so that there is no way to run with a different encoding, and so that we may get rid of additional options in sql connection strings. I've sent a

Re: [openstack-dev] [all][oslo][db][docs] RFC: drop support for libpq 9.1

2014-10-07 Thread Mike Bayer
On Oct 7, 2014, at 8:29 AM, Ihar Hrachyshka ihrac...@redhat.com wrote: That said, I wonder how we're going to manage cases when those *global* settings for the whole server should be really limited to specific databases. Isn't it better to enforce utf8 on service side, since we already

Re: [openstack-dev] [oslo] Meeting time

2014-10-07 Thread Mike Bayer
On Oct 7, 2014, at 3:15 PM, Doug Hellmann d...@doughellmann.com wrote: I remember talking about that at some point, but I don’t remember why we decided against it. Since we have several options for earlier in the week using the existing meeting rooms I think it’s safe to pick one of

Re: [openstack-dev] [all] [oslo] Acceptable methods for establishing per-test-suite behaviors

2014-10-07 Thread Mike Bayer
we put it, but in reality I think it’s fine that it would be placed in more database-test-specific locations. I’d really like to get these patches in so please feel free to review the spec as well as the patches. - mike On Aug 22, 2014, at 3:35 PM, Mike Bayer mba...@redhat.com wrote: Hi all

[openstack-dev] [all] [oslo] Proposed database connectivity patterns

2014-10-08 Thread Mike Bayer
Hi all - I’ve drafted up my next brilliant idea for how to get Openstack projects to use SQLAlchemy more effectively. The proposal here establishes significant detail on what’s wrong with the current state of things, e.g. the way I see EngineFacade, get_session() and get_engine() being used,

Re: [openstack-dev] [all] [oslo] Proposed database connectivity patterns

2014-10-09 Thread Mike Bayer
with different context identities in a single call stack, it should raise an exception - not that we can’t support that, but whether it means we should push new state onto the “stack” or not is ambiguous at the moment so we should refuse to guess. On Oct 8, 2014, at 5:07 PM, Mike Bayer mba

Re: [openstack-dev] [all] [oslo] Proposed database connectivity patterns

2014-10-10 Thread Mike Bayer
On Oct 10, 2014, at 6:13 AM, Ihar Hrachyshka ihrac...@redhat.com wrote: Signed PGP part On 09/10/14 21:29, Mike Bayer wrote: So so far, everyone seems really positive and psyched about the proposal. It looks like providing some options for how to use would be best, that is provide

Re: [openstack-dev] [all] [oslo] Proposed database connectivity patterns

2014-10-10 Thread Mike Bayer
On Oct 10, 2014, at 11:41 AM, Mike Bayer mba...@redhat.com wrote: I’ve been asking a lot about “hey are people cool with thread locals?” and have been waiting for what the concerns are. Since I wrote that email I’ve shifted, and I’ve been considering only: @sql.reader def

[openstack-dev] [neutron] [oslo.db] model_query() future and neutron specifics

2014-10-20 Thread Mike Bayer
As I’ve established oslo.db blueprints which will roll out new SQLAlchemy connectivity patterns for consuming applications within both API [1] and tests [2], one of the next big areas I’m to focus on is that of querying. If one looks at how SQLAlchemy ORM queries are composed across

[openstack-dev] [oslo.db] Add long-lived-transactionalized-db-fixtures - can we comment ?

2014-10-21 Thread Mike Bayer
Hi all - Thanks for all the responses I got on my Make EngineFacade a Facade” spec - plenty of people have commented, pretty much all positively so I’m pretty confident we can start building the basic idea of that out into a new review. I want to point out that there is another, closely

Re: [openstack-dev] [neutron] [oslo.db] model_query() future and neutron specifics

2014-10-25 Thread Mike Bayer
On Oct 23, 2014, at 11:27 AM, Kyle Mestery mest...@mestery.com wrote: Mike, first, thanks for sending out this detailed analysis. I'm hoping that some of the DB experts from the Neutron side have read this. Would it make sense to add this to our weekly meeting [1] for next week and discuss

[openstack-dev] [oslo][all] Alembic migrations for SQLite

2014-11-10 Thread Mike Bayer
Bonjour openstackers - While you were all sipping champagne on the Champs-Élysées, I took some time to tackle one of the two most critically wanted features in Alembic, which is that of being able to migrate tables on a SQLite database with some degree of sanity. My immediate focus on

Re: [openstack-dev] [nova] Consistency, efficiency, and safety of NovaObject.save()

2014-11-12 Thread Mike Bayer
On Nov 12, 2014, at 12:45 PM, Dan Smith d...@danplanet.com wrote: I personally favour having consistent behaviour across the board. How about updating them all to auto-refresh by default for consistency, but adding an additional option to save() to disable it for particular calls? I

Re: [openstack-dev] [nova] Consistency, efficiency, and safety of NovaObject.save()

2014-11-12 Thread Mike Bayer
On Nov 12, 2014, at 10:56 AM, Matthew Booth mbo...@redhat.com wrote: For brevity, I have conflated what happens in object.save() with what happens in db.api. Where the code lives isn't relevant here: I'm only looking at what happens. Specifically, the following objects refresh themselves

Re: [openstack-dev] [nova] Consistency, efficiency, and safety of NovaObject.save()

2014-11-12 Thread Mike Bayer
On Nov 12, 2014, at 6:23 PM, Mike Bayer mba...@redhat.com wrote: If I may inquire as to the irrelevant complexity, I’m trying to pinpoint where you see this happening. When we talk about updating a ComputeNode, because I’m only slightly familiar with Nova’s codebase, I assume we

Re: [openstack-dev] [nova] Consistency, efficiency, and safety of NovaObject.save()

2014-11-13 Thread Mike Bayer
On Nov 13, 2014, at 3:52 AM, Nikola Đipanov ndipa...@redhat.com wrote: On 11/13/2014 02:45 AM, Dan Smith wrote: I’m not sure if I’m seeing the second SELECT here either but I’m less familiar with what I’m looking at. compute_node_update() does the one SELECT as we said, then it doesn’t

Re: [openstack-dev] [nova] Consistency, efficiency, and safety of NovaObject.save()

2014-11-13 Thread Mike Bayer
On Nov 13, 2014, at 5:47 AM, Matthew Booth mbo...@redhat.com wrote: Unfortunately this model doesn't apply to Nova objects, which are persisted remotely. Unless I've missed something, SQLA doesn't run on Nova Compute at all. Instead, when Nova Compute calls object.save() this results in an

Re: [openstack-dev] oslo.db 1.1.0 released

2014-11-17 Thread Mike Bayer
2075074 Add history/changelog to docs c9e5fdf Add run_cross_tests.sh script Thanks Andreas Jaeger, Ann Kamyshnikova, Christian Berendt, Davanum Srinivas, Doug Hellmann, Ihar Hrachyshka, James Carey, Joshua Harlow, Mike Bayer, Oleksii Chuprykov, Roman Podoliaka for contributing to this release

Re: [openstack-dev] oslo.db 1.1.0 released

2014-11-18 Thread Mike Bayer
On Nov 18, 2014, at 11:47 AM, Sean Dague s...@dague.net wrote: Also can I request that when deprecating methods in oslo libraries we use a standard deprecation mechanism so that warnings are emitted when this method is used. +1 for DeprecationWarnings, I noticed oslo.db doesn’t seem to

Re: [openstack-dev] [Neutron] DB: transaction isolation and related questions

2014-11-19 Thread Mike Bayer
On Nov 18, 2014, at 1:38 PM, Eugene Nikanorov enikano...@mirantis.com wrote: Hi neutron folks, There is an ongoing effort to refactor some neutron DB logic to be compatible with galera/mysql which doesn't support locking (with_lockmode('update')). Some code paths that used locking in

Re: [openstack-dev] [oslo.db][nova] NovaObject.save() needs its own DB transaction

2014-11-19 Thread Mike Bayer
On Nov 19, 2014, at 11:46 AM, Matthew Booth mbo...@redhat.com wrote: We currently have a pattern in Nova where all database code lives in db/sqla/api.py[1]. Database transactions are only ever created or used in this module. This was an explicit design decision:

Re: [openstack-dev] [oslo.db][nova] NovaObject.save() needs its own DB transaction

2014-11-19 Thread Mike Bayer
On Nov 19, 2014, at 12:59 PM, Boris Pavlovic bpavlo...@mirantis.com wrote: Matthew, LOL ORM on top of another ORM https://img.neoseeker.com/screenshots/TW92aWVzL0RyYW1h/inception_image33.png https://img.neoseeker.com/screenshots/TW92aWVzL0RyYW1h/inception_image33.png I know

Re: [openstack-dev] [Neutron] DB: transaction isolation and related questions

2014-11-19 Thread Mike Bayer
multiple create_engine() calls, since we need to communicate directly to multiple database servers. From there it gets really crazy. Where’s all that ? Ryan Moats Mike Bayer mba...@redhat.com wrote on 11/19/2014 12:05:35 PM: From: Mike Bayer mba...@redhat.com To: OpenStack Development

Re: [openstack-dev] [Neutron] DB: transaction isolation and related questions

2014-11-19 Thread Mike Bayer
On Nov 19, 2014, at 2:58 PM, Jay Pipes jaypi...@gmail.com wrote: In other words, the retry logic like the following will not work: def allocate_obj(): with session.begin(subtrans=True): for i in xrange(n_retries): obj =

Re: [openstack-dev] [Neutron] DB: transaction isolation and related questions

2014-11-19 Thread Mike Bayer
On Nov 19, 2014, at 3:47 PM, Ryan Moats rmo...@us.ibm.com wrote: BTW, I view your examples from oslo as helping make my argument for me (and I don't think that was your intent :) ) I disagree with that as IMHO the differences in producing MM in the app layer against arbitrary backends

Re: [openstack-dev] [Neutron] DB: transaction isolation and related questions

2014-11-19 Thread Mike Bayer
On Nov 19, 2014, at 4:14 PM, Clint Byrum cl...@fewbar.com wrote: One simply cannot rely on multi-statement transactions to always succeed. agree, but the thing you want is that the transaction either succeeds or explicitly fails, the latter hopefully in such a way that a retry can be

Re: [openstack-dev] [Neutron] DB: transaction isolation and related questions

2014-11-21 Thread Mike Bayer
a way to bring retry logic upper to the stack of nesting transactions seems more appropriate. Thanks, Eugene. Cheers, -jay Also, thanks Clint for clarification about example scenario described by Mike Bayer. Initially the issue was discovered

[openstack-dev] Alembic 0.7.0 - hitting Pypi potentially Sunday night

2014-11-21 Thread Mike Bayer
Hi all - I’d like to announce / give a prominent heads up that Alembic 0.7.0 is ready for release. Over the past several weeks I’ve completed the initial implementations for two critical features, SQLite migrations and multiple branch/ merge support, as well as merged over a dozen bug fixes

Re: [openstack-dev] Alembic 0.7.0 - hitting Pypi potentially Sunday night

2014-11-21 Thread Mike Bayer
On Nov 21, 2014, at 7:35 PM, Kevin Benton blak...@gmail.com wrote: This is great! I'm not sure if you have been following some of the discussion about the separation of vendor drivers in Neutron, but one of the things we decided was to leave the vendor data models in the main repo so we

Re: [openstack-dev] Alembic 0.7.0 - hitting Pypi potentially Sunday night

2014-11-22 Thread Mike Bayer
On Nov 21, 2014, at 8:07 PM, Mike Bayer mba...@redhat.com wrote: On Nov 21, 2014, at 7:35 PM, Kevin Benton blak...@gmail.com mailto:blak...@gmail.com wrote: This is great! I'm not sure if you have been following some of the discussion about the separation of vendor drivers in Neutron

Re: [openstack-dev] [oslo] Add a new aiogreen executor for Oslo Messaging

2014-11-23 Thread Mike Bayer
On Nov 23, 2014, at 6:13 PM, Robert Collins robe...@robertcollins.net wrote: So - the technical bits of the plan sound fine. On WSGI - if we're in an asyncio world, *looks around*, we are? when did that happen?Assuming we’re talking explicit async. Rewriting all our code as

Re: [openstack-dev] [oslo] Add a new aiogreen executor for Oslo Messaging

2014-11-23 Thread Mike Bayer
On Nov 23, 2014, at 6:35 PM, Donald Stufft don...@stufft.io wrote: For whatever it’s worth, I find explicit async io to be _way_ easier to understand for the same reason I find threaded code to be a rats nest. web applications aren’t explicitly “threaded”. You get a request, load some

Re: [openstack-dev] [oslo] Add a new aiogreen executor for Oslo Messaging

2014-11-23 Thread Mike Bayer
On Nov 23, 2014, at 7:30 PM, Donald Stufft don...@stufft.io wrote: On Nov 23, 2014, at 7:21 PM, Mike Bayer mba...@redhat.com wrote: Given that, I’ve yet to understand why a system that implicitly defers CPU use when a routine encounters IO, deferring to other routines, is relegated

Re: [openstack-dev] [oslo] Add a new aiogreen executor for Oslo Messaging

2014-11-23 Thread Mike Bayer
On Nov 23, 2014, at 8:23 PM, Donald Stufft don...@stufft.io wrote: I don’t really take performance issues that seriously for CPython. If you care about performance you should be using PyPy. I like that argument though because the same argument is used against the GCs which you like to use

Re: [openstack-dev] [oslo] Add a new aiogreen executor for Oslo Messaging

2014-11-24 Thread Mike Bayer
On Nov 23, 2014, at 9:24 PM, Donald Stufft don...@stufft.io wrote: There’s a long history of implicit context switches causing buggy software that breaks. As far as I can tell the only downsides to explicit context switches that don’t stem from an inferior interpreter seem to be “some

Re: [openstack-dev] [oslo] Add a new aiogreen executor for Oslo Messaging

2014-11-24 Thread Mike Bayer
On Nov 24, 2014, at 9:23 AM, Adam Young ayo...@redhat.com wrote: For pieces such as the Nova compute that talk almost exclusively on the Queue, we should work to remove Monkey patching and use a clear programming model. If we can do that within the context of Eventlet, great. If we

Re: [openstack-dev] [oslo] Add a new aiogreen executor for Oslo Messaging

2014-11-24 Thread Mike Bayer
On Nov 24, 2014, at 12:40 PM, Doug Hellmann d...@doughellmann.com wrote: This is a good point. I’m not sure we can say “we’ll only use explicit/implicit async in certain cases because most of our apps actually mix the cases. We have WSGI apps that send RPC messages and we have other

Re: [openstack-dev] [Nova] Handling soft delete for instance rows in a new cells database

2014-11-24 Thread Mike Bayer
On Nov 24, 2014, at 7:32 PM, Michael Still mi...@stillhq.com wrote: Interesting. I hadn't seen consistency between the two databases as trumping doing this less horribly, but it sounds like its more of a thing that I thought. it really depends on what you need to do. if you need to get a

Re: [openstack-dev] [Nova] Handling soft delete for instance rows in a new cells database

2014-11-25 Thread Mike Bayer
On Nov 25, 2014, at 8:15 PM, Ahmed RAHAL ara...@iweb.com wrote: Hi, Le 2014-11-24 17:20, Michael Still a écrit : Heya, This is a new database, so its our big chance to get this right. So, ideas welcome... Some initial proposals: - we do what we do in the current nova database

Re: [openstack-dev] [Nova] Handling soft delete for instance rows in a new cells database

2014-11-26 Thread Mike Bayer
Precisely. Why is the RDBMS the thing that is used for archival/audit logging? Why not a NoSQL store or a centralized log facility? All that would be needed would be for us to standardize on the format of the archival record, standardize on the things to provide with the archival record

Re: [openstack-dev] sqlalchemy-migrate call for reviews

2014-11-30 Thread Mike Bayer
I’ve +2’ed it, it was caused by https://review.openstack.org/#/c/81955/. On Nov 29, 2014, at 9:54 PM, Davanum Srinivas dava...@gmail.com wrote: Looks like there is a review in the queue - https://review.openstack.org/#/c/111485/ -- dims On Sat, Nov 29, 2014 at 6:28 PM, Jeremy Stanley

[openstack-dev] [neutron] alembic 0.7.1 will break neutron's heal feature which assumes a fixed set of potential autogenerate types

2014-12-01 Thread Mike Bayer
hey neutron - Just an FYI, I’ve added https://review.openstack.org/#/c/137989/ / https://launchpad.net/bugs/1397796 to refer to an issue in neutron’s “heal” script that is going to start failing when I put out Alembic 0.7.1, which is potentially later today / this week. The issue is pretty

Re: [openstack-dev] [neutron] alembic 0.7.1 will break neutron's heal feature which assumes a fixed set of potential autogenerate types

2014-12-03 Thread Mike Bayer
://git.openstack.org/cgit/openstack/neutron/tree/neutron/db/migration/alembic_migrations/heal_script.py#n205 http://git.openstack.org/cgit/openstack/neutron/tree/neutron/db/migration/alembic_migrations/heal_script.py#n205 On 1 December 2014 at 20:35, Mike Bayer mba...@redhat.com mailto:mba

Re: [openstack-dev] [Nova] sqlalchemy-migrate vs alembic for new database

2014-12-05 Thread Mike Bayer
@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev I don't have experience with Alembic but I'd think we should use Alembic for the new database unless there is a compelling reason not to. Maybe we need Mike Bayer (or other oslo.db people) to give us

[openstack-dev] [all] [ha] potential issue with implicit async-compatible mysql drivers

2014-12-05 Thread Mike Bayer
Hey list - I’m posting this here just to get some ideas on what might be happening here, as it may or may not have some impact on Openstack if and when we move to MySQL drivers that are async-patchable, like MySQL-connector or PyMySQL. I had a user post this issue a few days ago which I’ve

  1   2   3   4   >