Re: [openstack-dev] Swift-backed volume backups are still breaking the gate

[Clay Gerrard]

On Thu, Jan 25, 2018 at 7:01 PM, Matt Riedemann  wrote:

> Is ThreadSafeSysLogHandler something that could live in oslo.log so we
> don't have to whack this mole everywhere at random times?


That might make sense, unless we can get eventlet's monkey patching of the
logging module to do something similar...

FWIW, Swift doesn't use oslo.log and has it's own crufty logging issues:

https://bugs.launchpad.net/swift/+bug/1380815

-Clay
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Swift-backed volume backups are still breaking the gate

[Matt Riedemann]

On 1/25/2018 6:41 PM, Clay Gerrard wrote:

Does it help that swift also had to fix this?

https://github.com/openstack/swift/blob/6d2503652b5f666275113cf9f3e185a2d9b3a121/swift/common/utils.py#L4415

The interesting/useful bit is where we replace our primary loghandlers 
createLock method to use one of these [Green|OS]-thread-safe PipeMutex 
lock things...


Is ThreadSafeSysLogHandler something that could live in oslo.log so we 
don't have to whack this mole everywhere at random times?


--

Thanks,

Matt

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] Swift-backed volume backups are still breaking the gate

[Clay Gerrard]

Does it help that swift also had to fix this?

https://github.com/openstack/swift/blob/6d2503652b5f666275113cf9f3e185a2d9b3a121/swift/common/utils.py#L4415

The interesting/useful bit is where we replace our primary loghandlers
createLock method to use one of these [Green|OS]-thread-safe PipeMutex lock
things...

-Clay

On Thu, Jan 25, 2018 at 1:12 PM, Matt Riedemann  wrote:

> We thought things were fixed with [1] but it turns out that swiftclient
> logs requests and responses at DEBUG level, so we're still switching thread
> context during a backup write and failing the backup operation, causing
> copious amounts of pain in the gate and piling up the rechecks.
>
> I've got a workaround here [2] which will hopefully be good enough to
> stabilize things for awhile, but there is probably not much point in
> rechecking a lot of patches, at least ones that run through the integrated
> gate, until that is merged.
>
> [1] https://review.openstack.org/#/c/537437/
> [2] https://review.openstack.org/#/c/538027/
>
> --
>
> Thanks,
>
> Matt
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] Swift-backed volume backups are still breaking the gate

[Matt Riedemann]

We thought things were fixed with [1] but it turns out that swiftclient 
logs requests and responses at DEBUG level, so we're still switching 
thread context during a backup write and failing the backup operation, 
causing copious amounts of pain in the gate and piling up the rechecks.


I've got a workaround here [2] which will hopefully be good enough to 
stabilize things for awhile, but there is probably not much point in 
rechecking a lot of patches, at least ones that run through the 
integrated gate, until that is merged.


[1] https://review.openstack.org/#/c/537437/
[2] https://review.openstack.org/#/c/538027/

--

Thanks,

Matt

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev