Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-25 Thread Duncan Thomas
The only thing I'd argue with here is the log level, Robert. Logstash on the gate doesn't index trace/debug, so info or above would be far more helpful, so that we can have a logstash query for the issue On 25 February 2015 at 01:20, Robert Collins robe...@robertcollins.net wrote: On 23

Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-24 Thread Robert Collins
On 23 February 2015 at 13:54, Michael Bayer mba...@redhat.com wrote: Correct me if I'm wrong but the register_after_fork seems to apply only to the higher level Process abstraction. If someone calls os.fork(), as is the case now, there's no hook to use. Hence the solution I have in place

Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-22 Thread Yuriy Taraday
On Sun Feb 22 2015 at 6:27:16 AM Michael Bayer mba...@redhat.com wrote: On Feb 21, 2015, at 9:49 PM, Joshua Harlow harlo...@outlook.com wrote: Some comments/questions inline... Mike Bayer wrote: Yuriy Taradayyorik@gmail.com wrote: On Fri Feb 20 2015 at 9:14:30 PM Joshua

Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-22 Thread Michael Bayer
On Feb 22, 2015, at 10:20 AM, Yuriy Taraday yorik@gmail.com wrote: On Sun Feb 22 2015 at 6:27:16 AM Michael Bayer mba...@redhat.com wrote: On Feb 21, 2015, at 9:49 PM, Joshua Harlow harlo...@outlook.com wrote: Some comments/questions inline... Mike Bayer wrote:

Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-21 Thread Michael Bayer
On Feb 21, 2015, at 9:49 PM, Joshua Harlow harlo...@outlook.com wrote: Some comments/questions inline... Mike Bayer wrote: Yuriy Taradayyorik@gmail.com wrote: On Fri Feb 20 2015 at 9:14:30 PM Joshua Harlowharlo...@outlook.com wrote: This feels like something we could do in

Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-21 Thread Mike Bayer
Yuriy Taraday yorik@gmail.com wrote: On Fri Feb 20 2015 at 9:14:30 PM Joshua Harlow harlo...@outlook.com wrote: This feels like something we could do in the service manager base class, maybe by adding a post fork hook or something. +1 to that. I think it'd be nice to have the

Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-21 Thread Joshua Harlow
Some comments/questions inline... Mike Bayer wrote: Yuriy Taradayyorik@gmail.com wrote: On Fri Feb 20 2015 at 9:14:30 PM Joshua Harlowharlo...@outlook.com wrote: This feels like something we could do in the service manager base class, maybe by adding a post fork hook or something. +1

Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-21 Thread Yuriy Taraday
On Fri Feb 20 2015 at 9:14:30 PM Joshua Harlow harlo...@outlook.com wrote: This feels like something we could do in the service manager base class, maybe by adding a post fork hook or something. +1 to that. I think it'd be nice to have the service __init__() maybe be something like:

Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-20 Thread Victor Stinner
Hi, Davanum Srinivas wrote: +1 to fix Oslo's service module any ways, irrespective of this bug. By the way, the Service class is a blocker point for the implementation of asyncio and threads specs: https://review.openstack.org/#/c/153298/ https://review.openstack.org/#/c/156711/ We may

Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-20 Thread Doug Hellmann
On Thu, Feb 19, 2015, at 09:45 PM, Mike Bayer wrote: Doug Hellmann d...@doughellmann.com wrote: 5) Allow this sort of connection sharing to continue for a deprecation period with apppropriate logging, then make it a hard failure. This would provide services time to find and fix

Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-20 Thread Davanum Srinivas
+1 to fix Oslo's service module any ways, irrespective of this bug. +1 to The db library shouldn't be concerned with whether or not it's in a forked process -- that's not its job -- dims On Fri, Feb 20, 2015 at 10:17 AM, Doug Hellmann d...@doughellmann.com wrote: On Thu, Feb 19, 2015, at

Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-20 Thread Mike Bayer
Doug Hellmann d...@doughellmann.com wrote: And I don't think we want the database library doing anything with this case at all. Recovery code is tricky, and often prevents valid use cases (perhaps the parent *meant* for the child to reuse the open connection and isn't going to continue

Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-20 Thread Doug Hellmann
On Fri, Feb 20, 2015, at 11:28 AM, Mike Bayer wrote: Doug Hellmann d...@doughellmann.com wrote: And I don't think we want the database library doing anything with this case at all. Recovery code is tricky, and often prevents valid use cases (perhaps the parent *meant* for the

Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-20 Thread Joshua Harlow
Doug Hellmann wrote: On Thu, Feb 19, 2015, at 09:45 PM, Mike Bayer wrote: Doug Hellmannd...@doughellmann.com wrote: 5) Allow this sort of connection sharing to continue for a deprecation period with apppropriate logging, then make it a hard failure. This would provide services time to

Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-20 Thread Mike Bayer
Doug Hellmann d...@doughellmann.com wrote: On Fri, Feb 20, 2015, at 11:28 AM, Mike Bayer wrote: Doug Hellmann d...@doughellmann.com wrote: And I don't think we want the database library doing anything with this case at all. Recovery code is tricky, and often prevents valid use cases

Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-20 Thread Joshua Harlow
I started https://etherpad.openstack.org/p/service-py-replacements with some ideas/thoughts/notes; feel free to add any on u want (or adjust it accordingly with better ideas). -Josh Victor Stinner wrote: Hi, Davanum Srinivas wrote: +1 to fix Oslo's service module any ways, irrespective of

Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-19 Thread Doug Hellmann
On Thu, Feb 19, 2015, at 01:09 PM, Ben Nemec wrote: Hi, Mike Bayer recently tracked down an issue with database errors in Cinder to a single database connection being shared over multiple processes. This is not something that should happen, and it turns out to cause intermittent failures

[openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-19 Thread Ben Nemec
Hi, Mike Bayer recently tracked down an issue with database errors in Cinder to a single database connection being shared over multiple processes. This is not something that should happen, and it turns out to cause intermittent failures in the Cinder volume service. Full details can be found in

Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-19 Thread Mike Bayer
Joshua Harlow harlo...@outlook.com wrote: Doug Hellmann wrote: On Thu, Feb 19, 2015, at 01:09 PM, Ben Nemec wrote: Hi, Mike Bayer recently tracked down an issue with database errors in Cinder to a single database connection being shared over multiple processes. This is not something

Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-19 Thread Joshua Harlow
Doug Hellmann wrote: On Thu, Feb 19, 2015, at 01:09 PM, Ben Nemec wrote: Hi, Mike Bayer recently tracked down an issue with database errors in Cinder to a single database connection being shared over multiple processes. This is not something that should happen, and it turns out to cause

Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-19 Thread Carl Baldwin
On Thu, Feb 19, 2015 at 7:45 PM, Mike Bayer mba...@redhat.com wrote: So, option “A”, they call engine.dispose() the moment they’re in a fork, the activity upon requesting a connection from the pool is: look in pool, no connections present, create a connection and return it. FWIW, this is

Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-19 Thread Mike Bayer
Doug Hellmann d...@doughellmann.com wrote: 5) Allow this sort of connection sharing to continue for a deprecation period with apppropriate logging, then make it a hard failure. This would provide services time to find and fix any sharing problems they might have, but would delay the