Re: [openstack-dev] [requirements] race in keystone unit tests
On 08/03/2016 12:26 PM, Lance Bragstad wrote: > Sending a follow-up because I think we ended up finding something > relevant to this discussion. > > As keystone moves towards making fernet the default, one of our work > items was to mock the system clock in tests. This allows us to advance > the clock by one second where we need to avoid sub-second race > conditions. To do this we used freezegun [0]. We recently landed a bunch > of fixes to do this. > > It turns out that there is a possible race between when freezegun > patches it's modules and when the test runs. This turned up in a patch I > was working on locally and I noticed certain clock operations weren't > using the fake time object from freezegun. As a work-around, we can > leverage the set_time_override() method from oslo_utils.timeutils to > make sure we are using the fake time from within the frozen time > context. In my testing locally this worked. > > If keystone requires a hybrid approach to patching > (oslo_utils.timeutils.set_time_override() + freezegun), we should build > it into a well documented hybrid context manager so that's its more > apparent why we need it. > > Sean, I can start working on this to see if it starts mitigating the > races you're seeing. > > [0] https://pypi.python.org/pypi/freezegun Lance, thanks for digging into this! I think using the oslo set_time_override sounds like the best approach, that's what I remember other places to test time boundaries like this. -Sean -- Sean Dague http://dague.net __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [requirements] race in keystone unit tests
Excerpts from Lance Bragstad's message of 2016-08-03 11:26:56 -0500: > Sending a follow-up because I think we ended up finding something relevant > to this discussion. > > As keystone moves towards making fernet the default, one of our work items > was to mock the system clock in tests. This allows us to advance the clock > by one second where we need to avoid sub-second race conditions. To do this > we used freezegun [0]. We recently landed a bunch of fixes to do this. > > It turns out that there is a possible race between when freezegun patches > it's modules and when the test runs. This turned up in a patch I was > working on locally and I noticed certain clock operations weren't using the > fake time object from freezegun. As a work-around, we can leverage the > set_time_override() method from oslo_utils.timeutils to make sure we are > using the fake time from within the frozen time context. In my testing > locally this worked. Supporting mocking time operations in tests is the primary reason some of those functions exist at all. Doug > > If keystone requires a hybrid approach to patching > (oslo_utils.timeutils.set_time_override() + freezegun), we should build it > into a well documented hybrid context manager so that's its more apparent > why we need it. > > Sean, I can start working on this to see if it starts mitigating the races > you're seeing. > > [0] https://pypi.python.org/pypi/freezegun > > On Tue, Aug 2, 2016 at 9:21 AM, Lance Bragstad wrote: > > > Hi Sean, > > > > Thanks for the information. This obviously looks Fernet-related and I > > would be happy to spend some cycles on it. We recently landed a bunch of > > refactors in keystone to improve Fernet test coverage. This could be > > related to those refactors. Just double checking - but you haven't opened a > > bug in launchpad for this yet have you? > > > > Thanks for the heads up! > > > > On Tue, Aug 2, 2016 at 5:32 AM, Sean Dague wrote: > > > >> One of my concerns about stacking up project unit tests in the > >> requirements jobs, is the unit tests aren't as free of races as you > >> would imagine. Because they only previously impacted the one project > >> team, those teams are often just fast to recheck instead of get to the > >> bottom of it. Cross testing with them in a voting way changes their > >> impact. > >> > >> The keystone unit tests have a existing race condition in them, which > >> recently failed an unrelated requirements bump - > >> > >> http://logs.openstack.org/50/348250/6/check/gate-cross-keystone-python27-db-ubuntu-xenial/962327d/console.html#_2016-08-02_03_52_14_537923 > >> > >> I'm not fully sure where to go from here. But wanted to make sure that > >> data is out there. Any keystone folks who can dive into and sort it out > >> would be highly appreciated. > >> > >> -Sean > >> > >> -- > >> Sean Dague > >> http://dague.net > >> > >> __ > >> OpenStack Development Mailing List (not for usage questions) > >> Unsubscribe: > >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> > > > > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [requirements] race in keystone unit tests
Sending a follow-up because I think we ended up finding something relevant to this discussion. As keystone moves towards making fernet the default, one of our work items was to mock the system clock in tests. This allows us to advance the clock by one second where we need to avoid sub-second race conditions. To do this we used freezegun [0]. We recently landed a bunch of fixes to do this. It turns out that there is a possible race between when freezegun patches it's modules and when the test runs. This turned up in a patch I was working on locally and I noticed certain clock operations weren't using the fake time object from freezegun. As a work-around, we can leverage the set_time_override() method from oslo_utils.timeutils to make sure we are using the fake time from within the frozen time context. In my testing locally this worked. If keystone requires a hybrid approach to patching (oslo_utils.timeutils.set_time_override() + freezegun), we should build it into a well documented hybrid context manager so that's its more apparent why we need it. Sean, I can start working on this to see if it starts mitigating the races you're seeing. [0] https://pypi.python.org/pypi/freezegun On Tue, Aug 2, 2016 at 9:21 AM, Lance Bragstad wrote: > Hi Sean, > > Thanks for the information. This obviously looks Fernet-related and I > would be happy to spend some cycles on it. We recently landed a bunch of > refactors in keystone to improve Fernet test coverage. This could be > related to those refactors. Just double checking - but you haven't opened a > bug in launchpad for this yet have you? > > Thanks for the heads up! > > On Tue, Aug 2, 2016 at 5:32 AM, Sean Dague wrote: > >> One of my concerns about stacking up project unit tests in the >> requirements jobs, is the unit tests aren't as free of races as you >> would imagine. Because they only previously impacted the one project >> team, those teams are often just fast to recheck instead of get to the >> bottom of it. Cross testing with them in a voting way changes their >> impact. >> >> The keystone unit tests have a existing race condition in them, which >> recently failed an unrelated requirements bump - >> >> http://logs.openstack.org/50/348250/6/check/gate-cross-keystone-python27-db-ubuntu-xenial/962327d/console.html#_2016-08-02_03_52_14_537923 >> >> I'm not fully sure where to go from here. But wanted to make sure that >> data is out there. Any keystone folks who can dive into and sort it out >> would be highly appreciated. >> >> -Sean >> >> -- >> Sean Dague >> http://dague.net >> >> __ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [requirements] race in keystone unit tests
Hi Sean, Thanks for the information. This obviously looks Fernet-related and I would be happy to spend some cycles on it. We recently landed a bunch of refactors in keystone to improve Fernet test coverage. This could be related to those refactors. Just double checking - but you haven't opened a bug in launchpad for this yet have you? Thanks for the heads up! On Tue, Aug 2, 2016 at 5:32 AM, Sean Dague wrote: > One of my concerns about stacking up project unit tests in the > requirements jobs, is the unit tests aren't as free of races as you > would imagine. Because they only previously impacted the one project > team, those teams are often just fast to recheck instead of get to the > bottom of it. Cross testing with them in a voting way changes their impact. > > The keystone unit tests have a existing race condition in them, which > recently failed an unrelated requirements bump - > > http://logs.openstack.org/50/348250/6/check/gate-cross-keystone-python27-db-ubuntu-xenial/962327d/console.html#_2016-08-02_03_52_14_537923 > > I'm not fully sure where to go from here. But wanted to make sure that > data is out there. Any keystone folks who can dive into and sort it out > would be highly appreciated. > > -Sean > > -- > Sean Dague > http://dague.net > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [requirements] race in keystone unit tests
One of my concerns about stacking up project unit tests in the requirements jobs, is the unit tests aren't as free of races as you would imagine. Because they only previously impacted the one project team, those teams are often just fast to recheck instead of get to the bottom of it. Cross testing with them in a voting way changes their impact. The keystone unit tests have a existing race condition in them, which recently failed an unrelated requirements bump - http://logs.openstack.org/50/348250/6/check/gate-cross-keystone-python27-db-ubuntu-xenial/962327d/console.html#_2016-08-02_03_52_14_537923 I'm not fully sure where to go from here. But wanted to make sure that data is out there. Any keystone folks who can dive into and sort it out would be highly appreciated. -Sean -- Sean Dague http://dague.net __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev