Re: Juju's dependencies, juju/utils and a road forward

[roger peppe]

I agree wholeheartedly about splitting up the utils package.

We talked about this potential issue when it was initially created
(and named "trivial", for trivial functionality with no better place
and no dependencies).

The dependency graph is crucial - one of the main reasons that utils
is problematic is because it has so many external dependencies.

> I'd like to move juju to only use "stable" dependency branches. No more
> '-unstable'.

It's perhaps worth observing that "unstable" is the default for any
package that's not under gopkg.in. At current count there are 99 such
packages in juju's dependencies, all potentially unstable, so I'm not
sure it's worth picking out the
explicit gopkg.in unstable paths for blame here.

Of course, this depends on how careful package authors are to maintain
backward compatibility. Some are pretty good, some not so much.

That said, I agree that using explicitly stable dependencies is a
worthwhile goal. In particular, the ongoing move towards using
bakery.v2-unstable is probably not a great direction, because that
uses a particular commit that just happens to be the one that the
charm store currently uses. Given that the charmstore dependency is
only there for some tests, I'd suggest that the effort might be better
expended in changing those tests so that they don't need to import the
charmstore code directly (for example by writing a fake charmstore
server, or making the tests talk to an external charmstore server -
or, ideally, both). Then it becomes more feasible to make the Juju
code use bakery.v2.

What I'd really like is for us to fast forward a year so that we can
use the the upcoming vgo dependency stuff, but time travel isn't
currently an option, sadly.

One important thing is to try to commit to API stability as much as
possible. A large number (maybe most) of the issues come from places
where APIs have been changed without regard for backward
compatibility. Using semantic versions to tag releases is also a good
idea in general.



On 4 April 2018 at 01:52, Tim Penhey  wrote:
> Hi folks,
>
> Juju has a problem with its dependencies. We have been trying to upgrade
> a number of our dependencies recently have have been hitting road
> blocks. Mostly because of changes to common libraries that are
> incompatible with other libraries that we use.
>
> I'm pretty sure that the root of all these problems is the juju/utils
> package. It has become a dumping ground for any common functions that
> packages may want to share. However these common functions are changed
> arbitrarily with only thoughts around how they impact a subset of the
> downstreams of juju/utils.
>
> I think that the only sane way forward is to move all the functionality
> we care about out of juju/utils into more cohesive, smaller, packages
> that have meaningful names and you should be able to determine what they do.
>
> Where the current clashes came from were a few places.
>  1) we wanted to change our mgo dependency
>  2) a new file path utility function was wanted
>  3) the debugstatus package inside utils was updated in a backward
> incompatible way
>
> The root problem here is that the utils package tries to do too many
> things. Any package called "utils" should be a warning, and we should
> have listened to Dave Cheney when he brought this up originally.
>
> I think the way forward is to break up the utils package. I'd like to
> propose that we don't add anything new to the utils package, and only
> deletions are allowed. We create new packages that contain the
> functionality we require, and have them be much more controlled on their
> own dependencies.
>
> Thoughts?
>
> Tim
>
> --
> Juju-dev mailing list
> Juju-dev@lists.ubuntu.com
> Modify settings or unsubscribe at: 
> https://lists.ubuntu.com/mailman/listinfo/juju-dev

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Juju's dependencies, juju/utils and a road forward

[John Meinel]

I agree with breaking up utils. I've certainly read similar comments
elsewhere that calling something 'utils' inherently causes issues like
this, and it seems to fit with our experience.

I also agree that X stable depending on Y unstable doesn't seem right.
Though we're also very bad at ever making a genuine release of
dependencies. (it's a lot of overhead if nobody else is actually consuming
them)

There are probably other small things like it is "mgo/bson" where bson
should really be it's own package.

Also some questions about how well packages avoid leaking. (charmstore uses
mgo as a backend. if it doesn't expose those types then we probably don't
care if some of our tree depends on it while the other part doesn't. At
least as a transition phase.)

John
=:->

On Apr 4, 2018 06:19, "Tim Penhey"  wrote:

Which then begs the question...

Why do we have stable gopkg.in branches depending on unstable branches?

i.e. gopkg.in/juju/charmstore.v5 depending on
gopkg.in/macaroon-bakery.v2-unstable?

This seems to smell bad.



Tim


On 04/04/18 14:17, Tim Penhey wrote:
> As a follow on...
>
> I'd like to move juju to only use "stable" dependency branches. No more
> '-unstable'.
>
> Tim
>
> On 04/04/18 12:52, Tim Penhey wrote:
>> Hi folks,
>>
>> Juju has a problem with its dependencies. We have been trying to upgrade
>> a number of our dependencies recently have have been hitting road
>> blocks. Mostly because of changes to common libraries that are
>> incompatible with other libraries that we use.
>>
>> I'm pretty sure that the root of all these problems is the juju/utils
>> package. It has become a dumping ground for any common functions that
>> packages may want to share. However these common functions are changed
>> arbitrarily with only thoughts around how they impact a subset of the
>> downstreams of juju/utils.
>>
>> I think that the only sane way forward is to move all the functionality
>> we care about out of juju/utils into more cohesive, smaller, packages
>> that have meaningful names and you should be able to determine what they
do.
>>
>> Where the current clashes came from were a few places.
>>  1) we wanted to change our mgo dependency
>>  2) a new file path utility function was wanted
>>  3) the debugstatus package inside utils was updated in a backward
>> incompatible way
>>
>> The root problem here is that the utils package tries to do too many
>> things. Any package called "utils" should be a warning, and we should
>> have listened to Dave Cheney when he brought this up originally.
>>
>> I think the way forward is to break up the utils package. I'd like to
>> propose that we don't add anything new to the utils package, and only
>> deletions are allowed. We create new packages that contain the
>> functionality we require, and have them be much more controlled on their
>> own dependencies.
>>
>> Thoughts?
>>
>> Tim
>>
>

--
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at:
https://lists.ubuntu.com/mailman/listinfo/juju-dev
-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Juju's dependencies, juju/utils and a road forward

[Tim Penhey]

Which then begs the question...

Why do we have stable gopkg.in branches depending on unstable branches?

i.e. gopkg.in/juju/charmstore.v5 depending on
gopkg.in/macaroon-bakery.v2-unstable?

This seems to smell bad.


Tim

On 04/04/18 14:17, Tim Penhey wrote:
> As a follow on...
> 
> I'd like to move juju to only use "stable" dependency branches. No more
> '-unstable'.
> 
> Tim
> 
> On 04/04/18 12:52, Tim Penhey wrote:
>> Hi folks,
>>
>> Juju has a problem with its dependencies. We have been trying to upgrade
>> a number of our dependencies recently have have been hitting road
>> blocks. Mostly because of changes to common libraries that are
>> incompatible with other libraries that we use.
>>
>> I'm pretty sure that the root of all these problems is the juju/utils
>> package. It has become a dumping ground for any common functions that
>> packages may want to share. However these common functions are changed
>> arbitrarily with only thoughts around how they impact a subset of the
>> downstreams of juju/utils.
>>
>> I think that the only sane way forward is to move all the functionality
>> we care about out of juju/utils into more cohesive, smaller, packages
>> that have meaningful names and you should be able to determine what they do.
>>
>> Where the current clashes came from were a few places.
>>  1) we wanted to change our mgo dependency
>>  2) a new file path utility function was wanted
>>  3) the debugstatus package inside utils was updated in a backward
>> incompatible way
>>
>> The root problem here is that the utils package tries to do too many
>> things. Any package called "utils" should be a warning, and we should
>> have listened to Dave Cheney when he brought this up originally.
>>
>> I think the way forward is to break up the utils package. I'd like to
>> propose that we don't add anything new to the utils package, and only
>> deletions are allowed. We create new packages that contain the
>> functionality we require, and have them be much more controlled on their
>> own dependencies.
>>
>> Thoughts?
>>
>> Tim
>>
> 

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev

Re: Juju's dependencies, juju/utils and a road forward

[Tim Penhey]

As a follow on...

I'd like to move juju to only use "stable" dependency branches. No more
'-unstable'.

Tim

On 04/04/18 12:52, Tim Penhey wrote:
> Hi folks,
> 
> Juju has a problem with its dependencies. We have been trying to upgrade
> a number of our dependencies recently have have been hitting road
> blocks. Mostly because of changes to common libraries that are
> incompatible with other libraries that we use.
> 
> I'm pretty sure that the root of all these problems is the juju/utils
> package. It has become a dumping ground for any common functions that
> packages may want to share. However these common functions are changed
> arbitrarily with only thoughts around how they impact a subset of the
> downstreams of juju/utils.
> 
> I think that the only sane way forward is to move all the functionality
> we care about out of juju/utils into more cohesive, smaller, packages
> that have meaningful names and you should be able to determine what they do.
> 
> Where the current clashes came from were a few places.
>  1) we wanted to change our mgo dependency
>  2) a new file path utility function was wanted
>  3) the debugstatus package inside utils was updated in a backward
> incompatible way
> 
> The root problem here is that the utils package tries to do too many
> things. Any package called "utils" should be a warning, and we should
> have listened to Dave Cheney when he brought this up originally.
> 
> I think the way forward is to break up the utils package. I'd like to
> propose that we don't add anything new to the utils package, and only
> deletions are allowed. We create new packages that contain the
> functionality we require, and have them be much more controlled on their
> own dependencies.
> 
> Thoughts?
> 
> Tim
> 

-- 
Juju-dev mailing list
Juju-dev@lists.ubuntu.com
Modify settings or unsubscribe at: 
https://lists.ubuntu.com/mailman/listinfo/juju-dev