Robert Collins wrote:
Most production systems I know don't run with open ended dependencies.
One of our contributing issues IMO is that we have the requirements
duplicated everywhere - and then ignore them for many of our test runs
(we deliberately override the in-tree ones with global requirements).
Particularly, since the only reason unified requirements matter is for
distro packages, and they ignore our requirements files *anyway*, I'm
not sure our current aggregate system is needed in that light.

That said, making requirements be capped and auto adjust upwards would
be extremely useful IMO, but its a chunk of work;
  - we need the transitive dependencies listed, not just direct dependencies

Wouldn't a pip install of the requirements.txt from the requirements repo itself get this? That would tell pip to download all the things and there transitive dependencies (aka step #1).

  - we need a thing to find possible upgrades and propose bumps

This is an analysis of the $ pip freeze after installing into that virtualenv (aka step #2)?

  - we would need to very very actively propogate those out from global

Sounds like an enhanced that uses the output from step #2?

For now I think making 'react to the situation faster and easier' is a
good thing to push on.

One question I have is that not all things specify all there dependencies, since some of them are pluggable (for example kombu can use couchdb, or a transport exists that seems like it could, yet kombu doesn't list that dependency in its requirements (it gets listed in under 'extra_requires' though); I'm sure other pluggable libraries (sqlalchemy, taskflow, tooz...) are similar in this regard so I wonder how those kind of libraries would work with this kind of proposal.


On 18 November 2014 12:02, Sean Dague<>  wrote:
As we're dealing with the fact that testtools 1.4.0 apparently broke
something with attribute additions to tests (needed by tempest for
filtering), it raises an interesting problem.

Our current policy on requirements is to leave them open ended, this
lets us take upstream fixes. It also breaks us a lot. But our max
version of dependencies happens with 0 code review or testing.

However, fixing these things takes a bunch of debug, code review, and
test time. Seen by the fact that the testtools 1.2.0 block didn't even
manage to fully merge this weekend.

This is an asymetric break/fix path, which I think we need a better plan
for. If fixing is more expensive than breaking, then you'll tend to be
in a broken state quite a bit. We really actually want the other
asymetry if we can get it.

There are a couple of things we could try here:

== Cap all requirements, require code reviews to bump maximums ==

Benefits, protected from upstream breaks.

Down sides, requires active energy to move forward. The SQLA 0.8
transition took forever.

== Provide Requirements core push authority ==

For blocks on bad versions, if we had a fast path to just merge know
breaks, we could right ourselves quicker. It would have reasonably
strict rules, like could only be used to block individual versions.
Probably that should also come with sending email to the dev list any
time such a thing happened.

Benefits, fast to fix

Down sides, bypasses our testing infrastructure. Though realistically
the break bypassed it as well.


There are probably other ways to make this more symetric. I had a grand
vision one time of building a system that kind of automated the
requirements bump, but have other problems I think need to be addressed
in OpenStack.

The reason I think it's important to come up with a better way here is
that making our whole code gating system lock up for 12+ hrs because of
an external dependency that we are pretty sure is the crux of our break
becomes very discouraging for developers. They can't get their code
merged. They can't get accurate test results. It means that once we get
the fix done, everyone is rechecking their code, so now everyone is
waiting extra long for valid test results. People don't realize their
code can't pass and just keep pushing patches up consuming resources
which means that parts of the project that could pass tests, is backed
up behind 100% guarunteed failing parts. All in all, not a great system.


Sean Dague

OpenStack-dev mailing list

OpenStack-dev mailing list

Reply via email to