Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-25 Thread Duncan Thomas
The only thing I'd argue with here is the log level, Robert. Logstash on the gate doesn't index trace/debug, so info or above would be far more helpful, so that we can have a logstash query for the issue On 25 February 2015 at 01:20, Robert Collins robe...@robertcollins.net wrote: On 23

Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-24 Thread Robert Collins
On 23 February 2015 at 13:54, Michael Bayer mba...@redhat.com wrote: Correct me if I'm wrong but the register_after_fork seems to apply only to the higher level Process abstraction. If someone calls os.fork(), as is the case now, there's no hook to use. Hence the solution I have in place

Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-22 Thread Yuriy Taraday
On Sun Feb 22 2015 at 6:27:16 AM Michael Bayer mba...@redhat.com wrote: On Feb 21, 2015, at 9:49 PM, Joshua Harlow harlo...@outlook.com wrote: Some comments/questions inline... Mike Bayer wrote: Yuriy Taradayyorik@gmail.com wrote: On Fri Feb 20 2015 at 9:14:30 PM Joshua

Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-22 Thread Michael Bayer
On Feb 22, 2015, at 10:20 AM, Yuriy Taraday yorik@gmail.com wrote: On Sun Feb 22 2015 at 6:27:16 AM Michael Bayer mba...@redhat.com wrote: On Feb 21, 2015, at 9:49 PM, Joshua Harlow harlo...@outlook.com wrote: Some comments/questions inline... Mike Bayer wrote:

Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-21 Thread Michael Bayer
On Feb 21, 2015, at 9:49 PM, Joshua Harlow harlo...@outlook.com wrote: Some comments/questions inline... Mike Bayer wrote: Yuriy Taradayyorik@gmail.com wrote: On Fri Feb 20 2015 at 9:14:30 PM Joshua Harlowharlo...@outlook.com wrote: This feels like something we could do in

Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-21 Thread Mike Bayer
Yuriy Taraday yorik@gmail.com wrote: On Fri Feb 20 2015 at 9:14:30 PM Joshua Harlow harlo...@outlook.com wrote: This feels like something we could do in the service manager base class, maybe by adding a post fork hook or something. +1 to that. I think it'd be nice to have the

Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-21 Thread Joshua Harlow
Some comments/questions inline... Mike Bayer wrote: Yuriy Taradayyorik@gmail.com wrote: On Fri Feb 20 2015 at 9:14:30 PM Joshua Harlowharlo...@outlook.com wrote: This feels like something we could do in the service manager base class, maybe by adding a post fork hook or something. +1

Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-21 Thread Yuriy Taraday
On Fri Feb 20 2015 at 9:14:30 PM Joshua Harlow harlo...@outlook.com wrote: This feels like something we could do in the service manager base class, maybe by adding a post fork hook or something. +1 to that. I think it'd be nice to have the service __init__() maybe be something like:

Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-20 Thread Victor Stinner
Hi, Davanum Srinivas wrote: +1 to fix Oslo's service module any ways, irrespective of this bug. By the way, the Service class is a blocker point for the implementation of asyncio and threads specs: https://review.openstack.org/#/c/153298/ https://review.openstack.org/#/c/156711/ We may

Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-20 Thread Doug Hellmann
On Thu, Feb 19, 2015, at 09:45 PM, Mike Bayer wrote: Doug Hellmann d...@doughellmann.com wrote: 5) Allow this sort of connection sharing to continue for a deprecation period with apppropriate logging, then make it a hard failure. This would provide services time to find and fix

Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-20 Thread Davanum Srinivas
+1 to fix Oslo's service module any ways, irrespective of this bug. +1 to The db library shouldn't be concerned with whether or not it's in a forked process -- that's not its job -- dims On Fri, Feb 20, 2015 at 10:17 AM, Doug Hellmann d...@doughellmann.com wrote: On Thu, Feb 19, 2015, at

Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-20 Thread Mike Bayer
Doug Hellmann d...@doughellmann.com wrote: And I don't think we want the database library doing anything with this case at all. Recovery code is tricky, and often prevents valid use cases (perhaps the parent *meant* for the child to reuse the open connection and isn't going to continue

Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-20 Thread Joshua Harlow
Doug Hellmann wrote: On Thu, Feb 19, 2015, at 09:45 PM, Mike Bayer wrote: Doug Hellmannd...@doughellmann.com wrote: 5) Allow this sort of connection sharing to continue for a deprecation period with apppropriate logging, then make it a hard failure. This would provide services time to

Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-20 Thread Mike Bayer
Doug Hellmann d...@doughellmann.com wrote: On Fri, Feb 20, 2015, at 11:28 AM, Mike Bayer wrote: Doug Hellmann d...@doughellmann.com wrote: And I don't think we want the database library doing anything with this case at all. Recovery code is tricky, and often prevents valid use cases

Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-20 Thread Joshua Harlow
I started https://etherpad.openstack.org/p/service-py-replacements with some ideas/thoughts/notes; feel free to add any on u want (or adjust it accordingly with better ideas). -Josh Victor Stinner wrote: Hi, Davanum Srinivas wrote: +1 to fix Oslo's service module any ways, irrespective of

Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-19 Thread Doug Hellmann
On Thu, Feb 19, 2015, at 01:09 PM, Ben Nemec wrote: Hi, Mike Bayer recently tracked down an issue with database errors in Cinder to a single database connection being shared over multiple processes. This is not something that should happen, and it turns out to cause intermittent failures

Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-19 Thread Mike Bayer
Joshua Harlow harlo...@outlook.com wrote: Doug Hellmann wrote: On Thu, Feb 19, 2015, at 01:09 PM, Ben Nemec wrote: Hi, Mike Bayer recently tracked down an issue with database errors in Cinder to a single database connection being shared over multiple processes. This is not something

Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-19 Thread Joshua Harlow
Doug Hellmann wrote: On Thu, Feb 19, 2015, at 01:09 PM, Ben Nemec wrote: Hi, Mike Bayer recently tracked down an issue with database errors in Cinder to a single database connection being shared over multiple processes. This is not something that should happen, and it turns out to cause

Re: [openstack-dev] [all][oslo] Dealing with database connection sharing issues

2015-02-19 Thread Mike Bayer
Doug Hellmann d...@doughellmann.com wrote: 5) Allow this sort of connection sharing to continue for a deprecation period with apppropriate logging, then make it a hard failure. This would provide services time to find and fix any sharing problems they might have, but would delay the