Re: [openstack-dev] [requirements] race in keystone unit tests

2016-08-03 Thread Sean Dague 
On 08/03/2016 12:26 PM, Lance Bragstad wrote:
> Sending a follow-up because I think we ended up finding something
> relevant to this discussion.
> 
> As keystone moves towards making fernet the default, one of our work
> items was to mock the system clock in tests. This allows us to advance
> the clock by one second where we need to avoid sub-second race
> conditions. To do this we used freezegun [0]. We recently landed a bunch
> of fixes to do this.
> 
> It turns out that there is a possible race between when freezegun
> patches it's modules and when the test runs. This turned up in a patch I
> was working on locally and I noticed certain clock operations weren't
> using the fake time object from freezegun. As a work-around, we can
> leverage the set_time_override() method from oslo_utils.timeutils to
> make sure we are using the fake time from within the frozen time
> context. In my testing locally this worked.
> 
> If keystone requires a hybrid approach to patching
> (oslo_utils.timeutils.set_time_override() + freezegun), we should build
> it into a well documented hybrid context manager so that's its more
> apparent why we need it.
> 
> Sean, I can start working on this to see if it starts mitigating the
> races you're seeing.
> 
> [0] https://pypi.python.org/pypi/freezegun

Lance, thanks for digging into this! I think using the oslo
set_time_override sounds like the best approach, that's what I remember
other places to test time boundaries like this.

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [requirements] race in keystone unit tests

2016-08-03 Thread Doug Hellmann 
Excerpts from Lance Bragstad's message of 2016-08-03 11:26:56 -0500:
> Sending a follow-up because I think we ended up finding something relevant
> to this discussion.
> 
> As keystone moves towards making fernet the default, one of our work items
> was to mock the system clock in tests. This allows us to advance the clock
> by one second where we need to avoid sub-second race conditions. To do this
> we used freezegun [0]. We recently landed a bunch of fixes to do this.
> 
> It turns out that there is a possible race between when freezegun patches
> it's modules and when the test runs. This turned up in a patch I was
> working on locally and I noticed certain clock operations weren't using the
> fake time object from freezegun. As a work-around, we can leverage the
> set_time_override() method from oslo_utils.timeutils to make sure we are
> using the fake time from within the frozen time context. In my testing
> locally this worked.

Supporting mocking time operations in tests is the primary reason some
of those functions exist at all.

Doug

> 
> If keystone requires a hybrid approach to patching
> (oslo_utils.timeutils.set_time_override() + freezegun), we should build it
> into a well documented hybrid context manager so that's its more apparent
> why we need it.
> 
> Sean, I can start working on this to see if it starts mitigating the races
> you're seeing.
> 
> [0] https://pypi.python.org/pypi/freezegun
> 
> On Tue, Aug 2, 2016 at 9:21 AM, Lance Bragstad  wrote:
> 
> > Hi Sean,
> >
> > Thanks for the information. This obviously looks Fernet-related and I
> > would be happy to spend some cycles on it. We recently landed a bunch of
> > refactors in keystone to improve Fernet test coverage. This could be
> > related to those refactors. Just double checking - but you haven't opened a
> > bug in launchpad for this yet have you?
> >
> > Thanks for the heads up!
> >
> > On Tue, Aug 2, 2016 at 5:32 AM, Sean Dague  wrote:
> >
> >> One of my concerns about stacking up project unit tests in the
> >> requirements jobs, is the unit tests aren't as free of races as you
> >> would imagine. Because they only previously impacted the one project
> >> team, those teams are often just fast to recheck instead of get to the
> >> bottom of it. Cross testing with them in a voting way changes their
> >> impact.
> >>
> >> The keystone unit tests have a existing race condition in them, which
> >> recently failed an unrelated requirements bump -
> >>
> >> http://logs.openstack.org/50/348250/6/check/gate-cross-keystone-python27-db-ubuntu-xenial/962327d/console.html#_2016-08-02_03_52_14_537923
> >>
> >> I'm not fully sure where to go from here. But wanted to make sure that
> >> data is out there. Any keystone folks who can dive into and sort it out
> >> would be highly appreciated.
> >>
> >> -Sean
> >>
> >> --
> >> Sean Dague
> >> http://dague.net
> >>
> >> __
> >> OpenStack Development Mailing List (not for usage questions)
> >> Unsubscribe:
> >> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>
> >
> >

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [requirements] race in keystone unit tests

2016-08-03 Thread Lance Bragstad 
Sending a follow-up because I think we ended up finding something relevant
to this discussion.

As keystone moves towards making fernet the default, one of our work items
was to mock the system clock in tests. This allows us to advance the clock
by one second where we need to avoid sub-second race conditions. To do this
we used freezegun [0]. We recently landed a bunch of fixes to do this.

It turns out that there is a possible race between when freezegun patches
it's modules and when the test runs. This turned up in a patch I was
working on locally and I noticed certain clock operations weren't using the
fake time object from freezegun. As a work-around, we can leverage the
set_time_override() method from oslo_utils.timeutils to make sure we are
using the fake time from within the frozen time context. In my testing
locally this worked.

If keystone requires a hybrid approach to patching
(oslo_utils.timeutils.set_time_override() + freezegun), we should build it
into a well documented hybrid context manager so that's its more apparent
why we need it.

Sean, I can start working on this to see if it starts mitigating the races
you're seeing.

[0] https://pypi.python.org/pypi/freezegun

On Tue, Aug 2, 2016 at 9:21 AM, Lance Bragstad  wrote:

> Hi Sean,
>
> Thanks for the information. This obviously looks Fernet-related and I
> would be happy to spend some cycles on it. We recently landed a bunch of
> refactors in keystone to improve Fernet test coverage. This could be
> related to those refactors. Just double checking - but you haven't opened a
> bug in launchpad for this yet have you?
>
> Thanks for the heads up!
>
> On Tue, Aug 2, 2016 at 5:32 AM, Sean Dague  wrote:
>
>> One of my concerns about stacking up project unit tests in the
>> requirements jobs, is the unit tests aren't as free of races as you
>> would imagine. Because they only previously impacted the one project
>> team, those teams are often just fast to recheck instead of get to the
>> bottom of it. Cross testing with them in a voting way changes their
>> impact.
>>
>> The keystone unit tests have a existing race condition in them, which
>> recently failed an unrelated requirements bump -
>>
>> http://logs.openstack.org/50/348250/6/check/gate-cross-keystone-python27-db-ubuntu-xenial/962327d/console.html#_2016-08-02_03_52_14_537923
>>
>> I'm not fully sure where to go from here. But wanted to make sure that
>> data is out there. Any keystone folks who can dive into and sort it out
>> would be highly appreciated.
>>
>> -Sean
>>
>> --
>> Sean Dague
>> http://dague.net
>>
>> __
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [requirements] race in keystone unit tests

2016-08-02 Thread Lance Bragstad 
Hi Sean,

Thanks for the information. This obviously looks Fernet-related and I would
be happy to spend some cycles on it. We recently landed a bunch of
refactors in keystone to improve Fernet test coverage. This could be
related to those refactors. Just double checking - but you haven't opened a
bug in launchpad for this yet have you?

Thanks for the heads up!

On Tue, Aug 2, 2016 at 5:32 AM, Sean Dague  wrote:

> One of my concerns about stacking up project unit tests in the
> requirements jobs, is the unit tests aren't as free of races as you
> would imagine. Because they only previously impacted the one project
> team, those teams are often just fast to recheck instead of get to the
> bottom of it. Cross testing with them in a voting way changes their impact.
>
> The keystone unit tests have a existing race condition in them, which
> recently failed an unrelated requirements bump -
>
> http://logs.openstack.org/50/348250/6/check/gate-cross-keystone-python27-db-ubuntu-xenial/962327d/console.html#_2016-08-02_03_52_14_537923
>
> I'm not fully sure where to go from here. But wanted to make sure that
> data is out there. Any keystone folks who can dive into and sort it out
> would be highly appreciated.
>
> -Sean
>
> --
> Sean Dague
> http://dague.net
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev