Re: [openstack-dev] Swift-backed volume backups are still breaking the gate
On Thu, Jan 25, 2018 at 7:01 PM, Matt Riedemann wrote: > Is ThreadSafeSysLogHandler something that could live in oslo.log so we > don't have to whack this mole everywhere at random times? That might make sense, unless we can get eventlet's monkey patching of the logging module to do something similar... FWIW, Swift doesn't use oslo.log and has it's own crufty logging issues: https://bugs.launchpad.net/swift/+bug/1380815 -Clay __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Swift-backed volume backups are still breaking the gate
On 1/25/2018 6:41 PM, Clay Gerrard wrote: Does it help that swift also had to fix this? https://github.com/openstack/swift/blob/6d2503652b5f666275113cf9f3e185a2d9b3a121/swift/common/utils.py#L4415 The interesting/useful bit is where we replace our primary loghandlers createLock method to use one of these [Green|OS]-thread-safe PipeMutex lock things... Is ThreadSafeSysLogHandler something that could live in oslo.log so we don't have to whack this mole everywhere at random times? -- Thanks, Matt __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Swift-backed volume backups are still breaking the gate
Does it help that swift also had to fix this? https://github.com/openstack/swift/blob/6d2503652b5f666275113cf9f3e185a2d9b3a121/swift/common/utils.py#L4415 The interesting/useful bit is where we replace our primary loghandlers createLock method to use one of these [Green|OS]-thread-safe PipeMutex lock things... -Clay On Thu, Jan 25, 2018 at 1:12 PM, Matt Riedemann wrote: > We thought things were fixed with [1] but it turns out that swiftclient > logs requests and responses at DEBUG level, so we're still switching thread > context during a backup write and failing the backup operation, causing > copious amounts of pain in the gate and piling up the rechecks. > > I've got a workaround here [2] which will hopefully be good enough to > stabilize things for awhile, but there is probably not much point in > rechecking a lot of patches, at least ones that run through the integrated > gate, until that is merged. > > [1] https://review.openstack.org/#/c/537437/ > [2] https://review.openstack.org/#/c/538027/ > > -- > > Thanks, > > Matt > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] Swift-backed volume backups are still breaking the gate
We thought things were fixed with [1] but it turns out that swiftclient logs requests and responses at DEBUG level, so we're still switching thread context during a backup write and failing the backup operation, causing copious amounts of pain in the gate and piling up the rechecks. I've got a workaround here [2] which will hopefully be good enough to stabilize things for awhile, but there is probably not much point in rechecking a lot of patches, at least ones that run through the integrated gate, until that is merged. [1] https://review.openstack.org/#/c/537437/ [2] https://review.openstack.org/#/c/538027/ -- Thanks, Matt __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev