[openstack-dev] [cinder] cinder is broken until someone fixes the forking code
Hello Cinder - I’d like to note that for issue https://bugs.launchpad.net/oslo.db/+bug/1417018, no solution that actually solves the problem for Cinder is scheduled to be committed anywhere. The patch I proposed for oslo.db is on hold, and the patch proposed for oslo.incubator in the service code will not fix this issue for Cinder, it will only make it fail harder and faster. I’ve taken myself off as the assignee on this issue, as someone on the Cinder team should really propose the best fix of all which is to call engine.dispose() when first entering a new child fork. Related issues are already being reported, such as https://bugs.launchpad.net/cinder/+bug/1430859. Right now Cinder is very unreliable on startup and this should be considered a critical issue. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [cinder] cinder is broken until someone fixes the forking code
On Wed, Mar 11, 2015, at 01:47 PM, Mike Bayer wrote: Hello Cinder - I’d like to note that for issue https://bugs.launchpad.net/oslo.db/+bug/1417018, no solution that actually solves the problem for Cinder is scheduled to be committed anywhere. The patch I proposed for oslo.db is on hold, and the patch proposed for oslo.incubator in the service code will not fix this issue for Cinder, it will only make it fail harder and faster. I’ve taken myself off as the assignee on this issue, as someone on the Cinder team should really propose the best fix of all which is to call engine.dispose() when first entering a new child fork. Related issues are already being reported, such as https://bugs.launchpad.net/cinder/+bug/1430859. Right now Cinder is very unreliable on startup and this should be considered a critical issue. That service code is actually from the Oslo incubator, so we should fix it there. If we can always call engine.dispose(), then let's put that into the incubated module and see how it works in cinder (submit the patch to the incubator, then update cinder's copy and submit a patch with Depends-On set to point to the patch to the incubator so the copy in cinder won't be updated until we land the fix in the incubator). Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [cinder] cinder is broken until someone fixes the forking code
On 03/11/2015 03:37 PM, Mike Bayer wrote: Mike Perez thin...@gmail.com wrote: On 11:49 Wed 11 Mar , Walter A. Boring IV wrote: We have this patch in review currently. I think this one should 'fix' it no? Please review. https://review.openstack.org/#/c/163551/ Looks like it to me. Would appreciate a +1 from Mike Bayer before we push this through. Thanks for all your time on this Mike. I have a question there, since I don’t know the scope of “Base”, that this “Base” constructor is generally called once per Python process. It’s OK if it’s called a little more than that, but if it’s called on like every service request or something, then those engine.dispose() calls are not the right approach, you’d instead just turn off pooling altogether, because otherwise you’re spending tons of time creating and destroying connection pools that aren’t even used as pools. you want the “engine” to be re-used across requests and everything else as much as possible, *except* across process boundaries. I don't see it used anywhere that isn't a long-standing service, it's only used by service and API managers, and BackupDrivers. So should be ok in this regard. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [cinder] cinder is broken until someone fixes the forking code
Joshua Harlow harlo...@outlook.com wrote: I've also got the following up: https://review.openstack.org/#/c/162781/ Which tries the force file descriptors to be closed; although I'm unsure where the tempest results went (and am rechecking that); maybe it breaks so badly that tempest doesn't even run? OK, I guess if the descriptors are closed then the pool, because we have a lot of reconnection stuff set up in oslo.db, will figure it out, catch the error, and try again, without actually propagating the error. So it would sort of “fix” the problem, rather than fail hard. But Cinder should really at least prevent against this whole condition from occurring anyway. -Josh Mike Bayer wrote: Hello Cinder - I’d like to note that for issue https://bugs.launchpad.net/oslo.db/+bug/1417018, no solution that actually solves the problem for Cinder is scheduled to be committed anywhere. The patch I proposed for oslo.db is on hold, and the patch proposed for oslo.incubator in the service code will not fix this issue for Cinder, it will only make it fail harder and faster. I’ve taken myself off as the assignee on this issue, as someone on the Cinder team should really propose the best fix of all which is to call engine.dispose() when first entering a new child fork. Related issues are already being reported, such as https://bugs.launchpad.net/cinder/+bug/1430859. Right now Cinder is very unreliable on startup and this should be considered a critical issue. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [cinder] cinder is broken until someone fixes the forking code
We have this patch in review currently. I think this one should 'fix' it no? Please review. https://review.openstack.org/#/c/163551/ Walt On 03/11/2015 10:47 AM, Mike Bayer wrote: Hello Cinder - I’d like to note that for issue https://bugs.launchpad.net/oslo.db/+bug/1417018, no solution that actually solves the problem for Cinder is scheduled to be committed anywhere. The patch I proposed for oslo.db is on hold, and the patch proposed for oslo.incubator in the service code will not fix this issue for Cinder, it will only make it fail harder and faster. I’ve taken myself off as the assignee on this issue, as someone on the Cinder team should really propose the best fix of all which is to call engine.dispose() when first entering a new child fork. Related issues are already being reported, such as https://bugs.launchpad.net/cinder/+bug/1430859. Right now Cinder is very unreliable on startup and this should be considered a critical issue. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [cinder] cinder is broken until someone fixes the forking code
On 11:49 Wed 11 Mar , Walter A. Boring IV wrote: We have this patch in review currently. I think this one should 'fix' it no? Please review. https://review.openstack.org/#/c/163551/ Looks like it to me. Would appreciate a +1 from Mike Bayer before we push this through. Thanks for all your time on this Mike. -- Mike Perez __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [cinder] cinder is broken until someone fixes the forking code
I've also got the following up: https://review.openstack.org/#/c/162781/ Which tries the force file descriptors to be closed; although I'm unsure where the tempest results went (and am rechecking that); maybe it breaks so badly that tempest doesn't even run? -Josh Mike Bayer wrote: Hello Cinder - I’d like to note that for issue https://bugs.launchpad.net/oslo.db/+bug/1417018, no solution that actually solves the problem for Cinder is scheduled to be committed anywhere. The patch I proposed for oslo.db is on hold, and the patch proposed for oslo.incubator in the service code will not fix this issue for Cinder, it will only make it fail harder and faster. I’ve taken myself off as the assignee on this issue, as someone on the Cinder team should really propose the best fix of all which is to call engine.dispose() when first entering a new child fork. Related issues are already being reported, such as https://bugs.launchpad.net/cinder/+bug/1430859. Right now Cinder is very unreliable on startup and this should be considered a critical issue. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
