Re: [openstack-dev] [ironic][infra][qa] Jobs failing; pep8 not found

[Jim Rollenhagen]

>
> In the short term we only need to fix the few projects with conflicting
> requirements. In the longer term we could have a concerted effort to
> move those dependencies. Someone creative might even be able to script
> it, since we do have a list of the blacklisted items.
>

Agree this is something we should do eventually.

For now, rloo came up with a better solution - remove ironic's pycodestyle
pin. We had backported the pin, then fixed and unpinned in master, but
didn't backport the unpin. After backporting the patch to unpin it, we're
all green again.

// jim
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic][infra][qa] Jobs failing; pep8 not found

[Doug Hellmann]

Excerpts from Jim Rollenhagen's message of 2018-04-20 09:05:23 -0400:
> On Fri, Apr 20, 2018 at 7:33 AM, Jim Rollenhagen 
> wrote:
> 
> > On Thu, Apr 19, 2018 at 3:21 PM, Doug Hellmann 
> > wrote:
> >>
> >>
> >> Reading through that log more carefully, I see an early attempt to pin
> >> pycodestyle <= 2.3.1 [1], followed later by pycodestyle == 2.4.0 being
> >> pulled in as a dependency of flake8-import-order==0.12 when neutron's
> >> test-requirements.txt is installed [2]. Then later when ironic's
> >> test-requirements.txt is installed pycodestyle is downgraded to 2.3.1
> >> [3].
> >>
> >> Reproducing those install & downgrade steps, I see that pycodestyle
> >> 2.4.0 claims to own pep8.py but pycodestyle 2.3.1 does not [4]. So that
> >> explains why pep8 is not re-installed when pycodestyle is downgraded.
> >>
> >
> > Aha, interesting! That's a fun one. :)
> >
> > I think the real problem here is that we have linter dependencies listed
> >> in the test-requirements.txt files for our projects, and they are
> >> somehow being installed without the constraints.
> >
> >
> > This is because they're in the blacklist, right?
> >
> >
> >> I don't think they need
> >> to be installed for devstack at all, so one way to fix it would be to
> >> move those dependencies to the tox.ini section for running pep8, or to
> >> have devstack look at the blacklisted packages and skip installing them.
> >>
> >
> > Yeah, seems like either would work. With the latter, would devstack edit
> > these out of test-requirements.txt before installing, I presume? The former
> > seems less hacky, I'll proceed with that unless folks have objections.
> >
> 
> Although... this would need to be done in every project installed from
> source during the devstack run. I'm going to look into doing this in
> devstack instead to avoid spending all day moving patches.

In the short term we only need to fix the few projects with conflicting
requirements. In the longer term we could have a concerted effort to
move those dependencies. Someone creative might even be able to script
it, since we do have a list of the blacklisted items.

> 
> // jim

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic][infra][qa] Jobs failing; pep8 not found

[Doug Hellmann]

Excerpts from Jim Rollenhagen's message of 2018-04-20 07:33:51 -0400:
> On Thu, Apr 19, 2018 at 3:21 PM, Doug Hellmann 
> wrote:
> >
> >
> > Reading through that log more carefully, I see an early attempt to pin
> > pycodestyle <= 2.3.1 [1], followed later by pycodestyle == 2.4.0 being
> > pulled in as a dependency of flake8-import-order==0.12 when neutron's
> > test-requirements.txt is installed [2]. Then later when ironic's
> > test-requirements.txt is installed pycodestyle is downgraded to 2.3.1
> > [3].
> >
> > Reproducing those install & downgrade steps, I see that pycodestyle
> > 2.4.0 claims to own pep8.py but pycodestyle 2.3.1 does not [4]. So that
> > explains why pep8 is not re-installed when pycodestyle is downgraded.
> >
> 
> Aha, interesting! That's a fun one. :)
> 
> I think the real problem here is that we have linter dependencies listed
> > in the test-requirements.txt files for our projects, and they are
> > somehow being installed without the constraints.
> 
> 
> This is because they're in the blacklist, right?

Yes, that's probably it.

> > I don't think they need
> > to be installed for devstack at all, so one way to fix it would be to
> > move those dependencies to the tox.ini section for running pep8, or to
> > have devstack look at the blacklisted packages and skip installing them.
> >
> 
> Yeah, seems like either would work. With the latter, would devstack edit
> these out of test-requirements.txt before installing, I presume? The former
> seems less hacky, I'll proceed with that unless folks have objections.

I like updating the tox.ini, too, since it has the added benefit of
putting the linter (and other blacklisted) dependencies in a file the
requirements check job ignores.

> 
> Thanks for the help, Doug! :)
> 
> // jim

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic][infra][qa] Jobs failing; pep8 not found

[Jim Rollenhagen]

On Fri, Apr 20, 2018 at 7:33 AM, Jim Rollenhagen 
wrote:

> On Thu, Apr 19, 2018 at 3:21 PM, Doug Hellmann 
> wrote:
>>
>>
>> Reading through that log more carefully, I see an early attempt to pin
>> pycodestyle <= 2.3.1 [1], followed later by pycodestyle == 2.4.0 being
>> pulled in as a dependency of flake8-import-order==0.12 when neutron's
>> test-requirements.txt is installed [2]. Then later when ironic's
>> test-requirements.txt is installed pycodestyle is downgraded to 2.3.1
>> [3].
>>
>> Reproducing those install & downgrade steps, I see that pycodestyle
>> 2.4.0 claims to own pep8.py but pycodestyle 2.3.1 does not [4]. So that
>> explains why pep8 is not re-installed when pycodestyle is downgraded.
>>
>
> Aha, interesting! That's a fun one. :)
>
> I think the real problem here is that we have linter dependencies listed
>> in the test-requirements.txt files for our projects, and they are
>> somehow being installed without the constraints.
>
>
> This is because they're in the blacklist, right?
>
>
>> I don't think they need
>> to be installed for devstack at all, so one way to fix it would be to
>> move those dependencies to the tox.ini section for running pep8, or to
>> have devstack look at the blacklisted packages and skip installing them.
>>
>
> Yeah, seems like either would work. With the latter, would devstack edit
> these out of test-requirements.txt before installing, I presume? The former
> seems less hacky, I'll proceed with that unless folks have objections.
>

Although... this would need to be done in every project installed from
source during the devstack run. I'm going to look into doing this in
devstack instead to avoid spending all day moving patches.

// jim
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic][infra][qa] Jobs failing; pep8 not found

[Jim Rollenhagen]

On Thu, Apr 19, 2018 at 3:21 PM, Doug Hellmann 
wrote:
>
>
> Reading through that log more carefully, I see an early attempt to pin
> pycodestyle <= 2.3.1 [1], followed later by pycodestyle == 2.4.0 being
> pulled in as a dependency of flake8-import-order==0.12 when neutron's
> test-requirements.txt is installed [2]. Then later when ironic's
> test-requirements.txt is installed pycodestyle is downgraded to 2.3.1
> [3].
>
> Reproducing those install & downgrade steps, I see that pycodestyle
> 2.4.0 claims to own pep8.py but pycodestyle 2.3.1 does not [4]. So that
> explains why pep8 is not re-installed when pycodestyle is downgraded.
>

Aha, interesting! That's a fun one. :)

I think the real problem here is that we have linter dependencies listed
> in the test-requirements.txt files for our projects, and they are
> somehow being installed without the constraints.


This is because they're in the blacklist, right?


> I don't think they need
> to be installed for devstack at all, so one way to fix it would be to
> move those dependencies to the tox.ini section for running pep8, or to
> have devstack look at the blacklisted packages and skip installing them.
>

Yeah, seems like either would work. With the latter, would devstack edit
these out of test-requirements.txt before installing, I presume? The former
seems less hacky, I'll proceed with that unless folks have objections.

Thanks for the help, Doug! :)

// jim
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic][infra][qa] Jobs failing; pep8 not found

[Doug Hellmann]

Excerpts from Jim Rollenhagen's message of 2018-04-18 13:44:08 -0400:
> Hi all,
> 
> We have a number of stable branch jobs failing[0] with an error about pep8
> not being importable[1], when it's clearly installed[2]. We first saw this
> when installing networking-generic-switch on queens in our multinode
> grenade job. We hacked a fix there[3], as we couldn't figure it out and
> thought it was a fluke. Now it's showing up elsewhere.
> 
> I suspected a new pycodestyle was the culprit (maybe it kills off the pep8
> package somehow?) but pinning pycodestyle back a version didn't seem to
> help.
> 
> Any ideas what might be going on here? I'm completely lost.
> 
> P.S. if anyone has the side question of why pep8 is being imported at
> install time, it seems that pbr iterates over any entry points under
> 'distutils.commands' for any installed package. flake8 has one of these
> which must import pep8 to be resolved. I'm not sure *why* pbr needs to do
> this, but I'll assume it's necessary.
> 
> [0] https://review.openstack.org/#/c/557441/
> [1]
> http://logs.openstack.org/41/557441/1/gate/ironic-tempest-dsvm-ironic-inspector-queens/5a4a6c9/logs/devstacklog.txt.gz#_2018-04-16_15_48_01_508
> [2]
> http://logs.openstack.org/41/557441/1/gate/ironic-tempest-dsvm-ironic-inspector-queens/5a4a6c9/logs/devstacklog.txt.gz#_2018-04-16_15_47_40_822
> [3] https://review.openstack.org/#/c/561358/
> 
> // jim

Reading through that log more carefully, I see an early attempt to pin
pycodestyle <= 2.3.1 [1], followed later by pycodestyle == 2.4.0 being
pulled in as a dependency of flake8-import-order==0.12 when neutron's
test-requirements.txt is installed [2]. Then later when ironic's
test-requirements.txt is installed pycodestyle is downgraded to 2.3.1
[3].

Reproducing those install & downgrade steps, I see that pycodestyle
2.4.0 claims to own pep8.py but pycodestyle 2.3.1 does not [4]. So that
explains why pep8 is not re-installed when pycodestyle is downgraded.

I think the real problem here is that we have linter dependencies listed
in the test-requirements.txt files for our projects, and they are
somehow being installed without the constraints. I don't think they need
to be installed for devstack at all, so one way to fix it would be to
move those dependencies to the tox.ini section for running pep8, or to
have devstack look at the blacklisted packages and skip installing them.

Doug

[1] 
http://logs.openstack.org/41/557441/1/gate/ironic-tempest-dsvm-ironic-inspector-queens/5a4a6c9/logs/devstacklog.txt.gz#_2018-04-16_15_39_00_392
[2] 
http://logs.openstack.org/41/557441/1/gate/ironic-tempest-dsvm-ironic-inspector-queens/5a4a6c9/logs/devstacklog.txt.gz#_2018-04-16_15_44_56_527
[3] 
http://logs.openstack.org/41/557441/1/gate/ironic-tempest-dsvm-ironic-inspector-queens/5a4a6c9/logs/devstacklog.txt.gz#_2018-04-16_15_47_40_120
[4] http://paste.openstack.org/show/719580/

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic][infra][qa] Jobs failing; pep8 not found

[Julia Kreger]

>
> This looks like some behavior that has been pulled out as part of pbr 4
> (version 3 is being used in the stable branch). Perhaps we want to
> update the pbr constraint there to use the newer version?
>

And it looks like doing that for stable/queens, at least from an
ironic-inspector point of view[1], fixes the issue for the branch. The
funny thing is, our ironic-inspector stable/pike -> stable/queens test
job fails on stable/pike as well now, with the same failure [2]. That
being said, we did observe during troubleshooting this issue last week
that the pep8 dist-info was present, however the actual module
contents were not present, which is why we worked around the issue
forcing the module to be re-installed.

We also had this occur today on an ironic stable/queens backport
triggered grenade job when keystone was being upgraded [3].

If the answer is update the upper constraint, from my point of view, I
suspect we're going to want to consider doing it across the board.

Of course, the real question is what changed, that is causing test
machines to think pep8 is present... :(

[1]: https://review.openstack.org/#/c/562384/
[2]: 
http://logs.openstack.org/84/562384/2/check/ironic-inspector-grenade-dsvm/59f0605/logs/grenade.sh.txt.gz#_2018-04-18_21_53_20_527
[3]: 
http://logs.openstack.org/14/562314/1/check/ironic-grenade-dsvm/2227c41/logs/grenade.sh.txt.gz#_2018-04-18_16_55_00_456

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic][infra][qa] Jobs failing; pep8 not found

[Doug Hellmann]

Excerpts from Jim Rollenhagen's message of 2018-04-18 13:44:08 -0400:
> Hi all,
> 
> We have a number of stable branch jobs failing[0] with an error about pep8
> not being importable[1], when it's clearly installed[2]. We first saw this
> when installing networking-generic-switch on queens in our multinode
> grenade job. We hacked a fix there[3], as we couldn't figure it out and
> thought it was a fluke. Now it's showing up elsewhere.
> 
> I suspected a new pycodestyle was the culprit (maybe it kills off the pep8
> package somehow?) but pinning pycodestyle back a version didn't seem to
> help.
> 
> Any ideas what might be going on here? I'm completely lost.
> 
> P.S. if anyone has the side question of why pep8 is being imported at
> install time, it seems that pbr iterates over any entry points under
> 'distutils.commands' for any installed package. flake8 has one of these
> which must import pep8 to be resolved. I'm not sure *why* pbr needs to do
> this, but I'll assume it's necessary.
> 
> [0] https://review.openstack.org/#/c/557441/
> [1]
> http://logs.openstack.org/41/557441/1/gate/ironic-tempest-dsvm-ironic-inspector-queens/5a4a6c9/logs/devstacklog.txt.gz#_2018-04-16_15_48_01_508
> [2]
> http://logs.openstack.org/41/557441/1/gate/ironic-tempest-dsvm-ironic-inspector-queens/5a4a6c9/logs/devstacklog.txt.gz#_2018-04-16_15_47_40_822
> [3] https://review.openstack.org/#/c/561358/
> 
> // jim

This looks like some behavior that has been pulled out as part of pbr 4
(version 3 is being used in the stable branch). Perhaps we want to
update the pbr constraint there to use the newer version?

Doug

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev