[go-nuts] Re: Module dependency cycles

[Andy Bursavich]

On Wednesday, February 24, 2021 at 7:42:24 PM UTC-8 Bryan C. Mills wrote:

> That said, that kind of coordinated release is *really* unpleasant, so 
> this is one of the motivating problems for https://golang.org/issue/36460 
> (which I've been actively working on since basically the start of 2020). 
> I'm fairly confident that it will finally land in Go 1.17. After that — 
> when the dependencies of the affected modules become lazy, all of the 
> irrelevant dependencies of older-than-selected versions will just melt 
> away! (A gleaming future lies ahead!)
>

I just saw that the new lazy code paths were enabled in tip. I'm sure 
there's still more to do before 1.17, but I know it's been a lot of work 
already and I wanted to say congrats and thanks... So, Bryan, congrats on 
landing this ~18 month behemoth and thanks for the gleaming future! 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/73ec684e-c5e1-46d2-a3a8-96200520f3d4n%40googlegroups.com.

Re: [go-nuts] Re: Module dependency cycles

[Bart Grantham]

Our solution to this was, against advice otherwise, to put go.sum in 
.gitignore and to rev all the co-dependent modules together 
simultaneously.  As you said, it *was* really unpleasant.  In the 3 days 
since I posted I've since had to do it again.

Is there a "Go modules therapy group" (or at least "Go modules haters 
club")?  I struggled to get our adoption of modules finalized for almost 9 
straight days.  Now I'm exhausted and have a lot of very unhealthy negative 
energy that I need to vent regarding modules.  Its cast quite a shadow on 
my enjoyment of the language.  If I'm honest with myself, I wince every 
time I use it now.

Today I went to do a basic 'go run foobar.go' of a piece of prototype test 
code in a bare directory I was working with before upgrading to 1.16 and it 
gave me a "working directory is not part of a module" error because it 
imports an external package.  I nearly lost my mind.

Bart


On Monday, March 1, 2021 at 11:39:34 AM UTC-8 Bryan C. Mills wrote:

> On Thu, Feb 25, 2021 at 7:43 PM Bart Grantham  
> wrote:
>
>> If I understand the post here, it seems we're also struggling with this 
>> issue with our private repos.  For us it means that in the go.sum of the 
>> highest level repos there's references for everything, all the way back to 
>> the initial v0.0.0 versions of every package.  AFAICT, this is happening 
>> with public modules as well.  I tried using modgraphviz to visualize our 
>> graph, it was hilariously large and it even generated a warning that it had 
>> to be scaled down in order for cairo to draw it.
>>
>> If I can make an observation, and I may be totally wrong, this is a 
>> symptom of the linker's dependency graph and the module dependency graph 
>> being defined differently.  For the purposes of keeping go.sum tidy the 
>> linker's relaxed dependency graph is what I would expect, but I haven't 
>> really thought that through.  Looking at the proposal, it is clear that 
>> this is a very hairy problem.
>>
>> In the meantime our go.sum's are growing monotonically.  Does this mean 
>> our ~/go/pkg/mod/ directories are also growing monotonically?  Mine is 
>> looking pretty heavy these days.
>>
>
> Yes, if you go.sum file is growing monotonically then your module cache is 
> probably also growing monotonically.
>
> That said, for requirement cycles the only checksums (and the only module 
> cache contents) that should actually be needed are for the …/go.mod paths, 
> which contain the checksums for the go.mod files only (not the complete 
> source code). So while it's true that your module cache is “growing 
> monotonically”, the coefficient of growth should be small.
>
> (Even a small coefficient can be a problem over the long term, of course — 
> that's why we're also working on a long-term fix in the `go` command. )
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/c949bba9-4455-4f01-9538-688fb9fca6ebn%40googlegroups.com.

Re: [go-nuts] Re: Module dependency cycles

['Bryan C. Mills' via golang-nuts]

On Thu, Feb 25, 2021 at 7:43 PM Bart Grantham  wrote:

> If I understand the post here, it seems we're also struggling with this
> issue with our private repos.  For us it means that in the go.sum of the
> highest level repos there's references for everything, all the way back to
> the initial v0.0.0 versions of every package.  AFAICT, this is happening
> with public modules as well.  I tried using modgraphviz to visualize our
> graph, it was hilariously large and it even generated a warning that it had
> to be scaled down in order for cairo to draw it.
>
> If I can make an observation, and I may be totally wrong, this is a
> symptom of the linker's dependency graph and the module dependency graph
> being defined differently.  For the purposes of keeping go.sum tidy the
> linker's relaxed dependency graph is what I would expect, but I haven't
> really thought that through.  Looking at the proposal, it is clear that
> this is a very hairy problem.
>
> In the meantime our go.sum's are growing monotonically.  Does this mean
> our ~/go/pkg/mod/ directories are also growing monotonically?  Mine is
> looking pretty heavy these days.
>

Yes, if you go.sum file is growing monotonically then your module cache is
probably also growing monotonically.

That said, for requirement cycles the only checksums (and the only module
cache contents) that should actually be needed are for the …/go.mod paths,
which contain the checksums for the go.mod files only (not the complete
source code). So while it's true that your module cache is “growing
monotonically”, the coefficient of growth should be small.

(Even a small coefficient can be a problem over the long term, of course —
that's why we're also working on a long-term fix in the `go` command. )

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAKWVi_S3jSWZfMFUE8P1n-MN9pSdXkXsEhYQCRFVrain00DCuA%40mail.gmail.com.

[go-nuts] Re: Module dependency cycles

[Bart Grantham]

If I understand the post here, it seems we're also struggling with this 
issue with our private repos.  For us it means that in the go.sum of the 
highest level repos there's references for everything, all the way back to 
the initial v0.0.0 versions of every package.  AFAICT, this is happening 
with public modules as well.  I tried using modgraphviz to visualize our 
graph, it was hilariously large and it even generated a warning that it had 
to be scaled down in order for cairo to draw it.

If I can make an observation, and I may be totally wrong, this is a symptom 
of the linker's dependency graph and the module dependency graph being 
defined differently.  For the purposes of keeping go.sum tidy the linker's 
relaxed dependency graph is what I would expect, but I haven't really 
thought that through.  Looking at the proposal, it is clear that this is a 
very hairy problem.

In the meantime our go.sum's are growing monotonically.  Does this mean our 
~/go/pkg/mod/ directories are also growing monotonically?  Mine is looking 
pretty heavy these days.

Bart



On Wednesday, February 24, 2021 at 7:42:24 PM UTC-8 Bryan C. Mills wrote:

> In general the way to cut cycles is to cut a set of versions of the 
> affected modules such that each module in the set requires only the other 
> versions in that set. (So, a new `thanos` v0.19.0, a new `cortex` v1.8.0, 
> and a new `alertmanager` v0.22.0, I think?)
> The trick is to publish all of them more-or-less simultaneously. The new 
> version of the first module to be published will be broken until the other 
> versions in the set are published, but once *everything* is published 
> those versions will settle out again, and you can leave behind all of the 
> cruft underneath them.
>
> That said, that kind of coordinated release is *really* unpleasant, so 
> this is one of the motivating problems for https://golang.org/issue/36460 
> (which I've been actively working on since basically the start of 2020).
> I'm fairly confident that it will finally land in Go 1.17. After that — 
> when the dependencies of the affected modules become lazy, all of the 
> irrelevant dependencies of older-than-selected versions will just melt 
> away! (A gleaming future lies ahead!)
>
> We also made some more targeted fixes to `exclude` directives in Go 1.16, 
> so that the excluded dependencies are completely expunged from the module 
> dependency graph rather than dynamically upgraded to the next-higher 
> release. So for now, the best workaround is probably: (1) upgrade to Go 
> 1.16, (2) notch out the problematic dependency with an `exclude` directive, 
> and (3) wait for the long-term fix to (finally!) land in the `go` command.
>
> I understand that this is all very inconvenient, and I'm sorry that we 
> weren't able to land the long-term fix in 1.16. 
>
> On Wednesday, February 24, 2021 at 2:11:53 PM UTC-5 Andy Bursavich wrote:
>
>> To summarize the concrete issue:
>>
>> An old version of Prometheus pulled in 
>> k8s.io/clie...@v12.0.0+incompatible 
>> .
>> An old version of Alertmanager pulled in that version of Prometheus.
>> An old version of Cortex pulled in that version of Alertmanager.
>> An old version of Thanos pulled in that version of Cortex
>> A newer version of Cortex pulled in that version of Thanos
>> A newer version of Thanos pulled in that version of Cortex
>> ...
>> A newer version of Cortex pulled in that version of Thanos
>> A newer version of Thanos pulled in that version of Cortex
>> A recent version of Cortex pulled in that version of Thanos
>> A recent version of Thanos pulled in that version of Cortex
>>
>> Newer versions of Prometheus have moved to k8s.io/clie...@v0.20.2 
>> .
>> Newer versions of Alertmanager don't depend on Prometheus.
>> Newer versions of Cortex have moved to newer versions of Alertmanager.
>> Newer versions of Thanos have moved to newer versions of Cortex.
>> Newer versions of Cortex have moved to newer versions of Thanos.
>>
>> Because the k8s release tagging conventions predate go modules, go treats 
>> k8s.io/clie...@v12.0.0+incompatible 
>>  as newer than 
>> k8s.io/clie...@v0.20.2 , and the cycle 
>> between Thanos and Cortex keeps their oldest history together alive in the 
>> dependency graph, if you depend on Thanos then you get stuck with  
>> k8s.io/clie...@v12.0.0+incompatible 
>>  and must use replace 
>> directives.
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/f3a6b4a3-0169-496d-8acc-8a0d9b661605n%40googlegroups.com.

[go-nuts] Re: Module dependency cycles

['Bryan C. Mills' via golang-nuts]

In general the way to cut cycles is to cut a set of versions of the 
affected modules such that each module in the set requires only the other 
versions in that set. (So, a new `thanos` v0.19.0, a new `cortex` v1.8.0, 
and a new `alertmanager` v0.22.0, I think?)
The trick is to publish all of them more-or-less simultaneously. The new 
version of the first module to be published will be broken until the other 
versions in the set are published, but once *everything* is published those 
versions will settle out again, and you can leave behind all of the cruft 
underneath them.

That said, that kind of coordinated release is *really* unpleasant, so this 
is one of the motivating problems for https://golang.org/issue/36460 (which 
I've been actively working on since basically the start of 2020).
I'm fairly confident that it will finally land in Go 1.17. After that — 
when the dependencies of the affected modules become lazy, all of the 
irrelevant dependencies of older-than-selected versions will just melt 
away! (A gleaming future lies ahead!)

We also made some more targeted fixes to `exclude` directives in Go 1.16, 
so that the excluded dependencies are completely expunged from the module 
dependency graph rather than dynamically upgraded to the next-higher 
release. So for now, the best workaround is probably: (1) upgrade to Go 
1.16, (2) notch out the problematic dependency with an `exclude` directive, 
and (3) wait for the long-term fix to (finally!) land in the `go` command.

I understand that this is all very inconvenient, and I'm sorry that we 
weren't able to land the long-term fix in 1.16. 

On Wednesday, February 24, 2021 at 2:11:53 PM UTC-5 Andy Bursavich wrote:

> To summarize the concrete issue:
>
> An old version of Prometheus pulled in k8s.io/clie...@v12.0.0+incompatible 
> .
> An old version of Alertmanager pulled in that version of Prometheus.
> An old version of Cortex pulled in that version of Alertmanager.
> An old version of Thanos pulled in that version of Cortex
> A newer version of Cortex pulled in that version of Thanos
> A newer version of Thanos pulled in that version of Cortex
> ...
> A newer version of Cortex pulled in that version of Thanos
> A newer version of Thanos pulled in that version of Cortex
> A recent version of Cortex pulled in that version of Thanos
> A recent version of Thanos pulled in that version of Cortex
>
> Newer versions of Prometheus have moved to k8s.io/clie...@v0.20.2 
> .
> Newer versions of Alertmanager don't depend on Prometheus.
> Newer versions of Cortex have moved to newer versions of Alertmanager.
> Newer versions of Thanos have moved to newer versions of Cortex.
> Newer versions of Cortex have moved to newer versions of Thanos.
>
> Because the k8s release tagging conventions predate go modules, go treats 
> k8s.io/clie...@v12.0.0+incompatible 
>  as newer than 
> k8s.io/clie...@v0.20.2 , and the cycle 
> between Thanos and Cortex keeps their oldest history together alive in the 
> dependency graph, if you depend on Thanos then you get stuck with  
> k8s.io/clie...@v12.0.0+incompatible 
>  and must use replace 
> directives.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/18a050d5-9642-4cd0-b25b-000e947f546en%40googlegroups.com.

[go-nuts] Re: Module dependency cycles

[Andy Bursavich]

To summarize the concrete issue:

An old version of Prometheus pulled in k8s.io/client-go@v12.0.0+incompatible
.
An old version of Alertmanager pulled in that version of Prometheus.
An old version of Cortex pulled in that version of Alertmanager.
An old version of Thanos pulled in that version of Cortex
A newer version of Cortex pulled in that version of Thanos
A newer version of Thanos pulled in that version of Cortex
...
A newer version of Cortex pulled in that version of Thanos
A newer version of Thanos pulled in that version of Cortex
A recent version of Cortex pulled in that version of Thanos
A recent version of Thanos pulled in that version of Cortex

Newer versions of Prometheus have moved to k8s.io/client-go@v0.20.2.
Newer versions of Alertmanager don't depend on Prometheus.
Newer versions of Cortex have moved to newer versions of Alertmanager.
Newer versions of Thanos have moved to newer versions of Cortex.
Newer versions of Cortex have moved to newer versions of Thanos.

Because the k8s release tagging conventions predate go modules, go treats 
k8s.io/client-go@v12.0.0+incompatible as newer than 
k8s.io/client-go@v0.20.2, and the cycle between Thanos and Cortex keeps 
their oldest history together alive in the dependency graph, if you depend 
on Thanos then you get stuck with  k8s.io/client-go@v12.0.0+incompatible and 
must use replace directives.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/f56e3d1c-620c-4209-9c19-dd0be0c9b16bn%40googlegroups.com.

[go-nuts] Re: Module dependency cycles

[Andy Bursavich]

s/transient/transitive/

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/a420a11d-123c-4b5e-bd91-e4d4051f2484n%40googlegroups.com.