Re: [go-nuts] Re: Changing PWD ruins test caching

[Kevin Burke]

You're right. I ended up inserting a shim to replace os/env.go in the
standard library, that would dump the stack trace of anything that called
os.Getenv("PWD"). It turns out that a popular testing library was doing
this - https://github.com/onsi/ginkgo/issues/1355.

On Thu, Feb 8, 2024 at 3:19 PM 'Sean Liao' via golang-nuts <
golang-nuts@googlegroups.com> wrote:

> Using a basic example which just imports os
> https://go.dev/play/p/BmX0ds901-4 (the rest copied from gobyexample)
> copied to 2 different directories, I do see cache hits, so I don't
> think it's from just the import?
>
> 23:12:37 ~/tmp/testrepo0530/a 0:00:00
> main » go1.22.0 test .
> testcache: example: test ID 7554777255756c714a6773663261576871416373
> => d9016fd4db0cc529026c3bdf959d21e8b37da70f61b3b9bd14cebcf4bcc41e0c
> testcache: example: input list not found: cache entry not found: open
>
> /home/user/.cache/go-build/d9/d9016fd4db0cc529026c3bdf959d21e8b37da70f61b3b9bd14cebcf4bcc41e0c-a:
> no such file or directory
> testcache: example: test ID 695f4e6944335a64497850503861366643364c48
> => e7d8d7272b091d2d5d155ff077e599c1e7c31eaad2d9bf3549e2d03149e94357
> testcache: example: input list not found: cache entry not found: open
>
> /home/user/.cache/go-build/e7/e7d8d7272b091d2d5d155ff077e599c1e7c31eaad2d9bf3549e2d03149e94357-a:
> no such file or directory
> testcache: example: save test ID
> d9016fd4db0cc529026c3bdf959d21e8b37da70f61b3b9bd14cebcf4bcc41e0c =>
> input ID 8671bffc7a3ec5201bb26a0eccecb5599259f0d0e63624375378b61cc00a0733
> => 516ef3f5e0d83ea893583f43f60e654775b3769b5f99f2de7b99487eae637298
> testcache: example: save test ID
> e7d8d7272b091d2d5d155ff077e599c1e7c31eaad2d9bf3549e2d03149e94357 =>
> input ID 8671bffc7a3ec5201bb26a0eccecb5599259f0d0e63624375378b61cc00a0733
> => 42e8604e2c4807cd097efad167887d4d0ddfc76afda71e21f8145604d93ebce5
> ok  example 0.001s
>
> 23:12:42 ~/tmp/testrepo0530/a 0:00:00
> main » cd ../b
>
> 23:12:46 ~/tmp/testrepo0530/b 0:00:00
> main » go1.22.0 test .
> testcache: example: test ID 7554777255756c714a6773663261576871416373
> => d9016fd4db0cc529026c3bdf959d21e8b37da70f61b3b9bd14cebcf4bcc41e0c
> testcache: example: test ID
> d9016fd4db0cc529026c3bdf959d21e8b37da70f61b3b9bd14cebcf4bcc41e0c =>
> input ID 8671bffc7a3ec5201bb26a0eccecb5599259f0d0e63624375378b61cc00a0733
> => 516ef3f5e0d83ea893583f43f60e654775b3769b5f99f2de7b99487eae637298
> ok  example (cached)
>
>
> - sean
>
> On Thu, Feb 8, 2024 at 12:13 AM Kevin Burke  wrote:
> >
> > Sorry for slow reply. We are using Alpine, which I don't think is either
> of those flavors.
> >
> > I discovered this by enabling GODEBUG=gocachehash=1, running the tests
> twice, and then checking the diff between the two test runs.
> >
> > On Tue, Nov 28, 2023 at 11:57 AM 'Bryan C. Mills' via golang-nuts <
> golang-nuts@googlegroups.com> wrote:
> >>
> >> As far as I can tell the code in question is only built on AIX and
> OpenBSD — are you using one of those two platforms?
> >> If not, perhaps the caching problem is coming from somewhere else.
> >>
> >> That said, it does appear that calls to `os.Chdir` result in a spurious
> dependency on the PWD variable:
> >>
> https://cs.opensource.google/go/go/+/master:src/os/file.go;l=340-345;drc=d961b12be9001cf8dbf8f52847607dbf84d94f8d
> >>
> >> On Thursday, November 23, 2023 at 1:50:04 PM UTC-5 Kevin Burke wrote:
> >>>
> >>> We have some tests that are pretty slow. I would like to use Go's test
> caching to skip them in our CI environment, if nothing in the code or the
> environment has changed. Our CI environment has a set of _agents_, each of
> which can run multiple jobs simultaneously. As a result, they check out
> code into a different directory each time they run a job.
> >>>
> >>> Go's test caching checks every env var loaded by the test program. If
> any of them change, then Go assumes that the cache is busted and the test
> must be run.
> >>>
> >>> Any program that imports "os" hits this logic in the os package, which
> checks the value of the $PWD environment variable:
> >>>
> >>> // We query the working directory at init, to use it later to search
> for the
> >>> // executable file
> >>> // errWd will be checked later, if we need to use initWd
> >>> var initWd, errWd = Getwd()
> >>>
> >>> So checking code out to different directories means that any program
> that imports "os" cannot have test caching. This seems like a shame because
> the code is all laid out in the same place in the working directory.
> >>>
> >>> Has anyone tried to fix this issue? Do you think this is worth trying
> to patch or modify the test caching behavior in Go itself? I could solve
> this by running a chroot or another layer of Docker, of course, but I'd
> prefer not to do these because of the difficulty of getting data in and out
> of each one, communicating with other Docker containers, etc.
> >>
> >> --
> >> You received this message because you are subscribed to a topic in the
> Google Groups "golang-nuts" group.
> >> To unsubscribe from this 

Re: [go-nuts] Re: Changing PWD ruins test caching

[Kevin Burke]

Sorry for slow reply. We are using Alpine, which I don't think is either of
those flavors.

I discovered this by enabling GODEBUG=gocachehash=1, running the tests
twice, and then checking the diff between the two test runs.

On Tue, Nov 28, 2023 at 11:57 AM 'Bryan C. Mills' via golang-nuts <
golang-nuts@googlegroups.com> wrote:

> As far as I can tell the code in question is only built on AIX and OpenBSD
> — are you using one of those two platforms?
> If not, perhaps the caching problem is coming from somewhere else.
>
> That said, it does appear that calls to `os.Chdir` result in a spurious
> dependency on the PWD variable:
>
> https://cs.opensource.google/go/go/+/master:src/os/file.go;l=340-345;drc=d961b12be9001cf8dbf8f52847607dbf84d94f8d
>
> On Thursday, November 23, 2023 at 1:50:04 PM UTC-5 Kevin Burke wrote:
>
>> We have some tests that are pretty slow. I would like to use Go's test
>> caching to skip them in our CI environment, if nothing in the code or the
>> environment has changed. Our CI environment has a set of _agents_, each of
>> which can run multiple jobs simultaneously. As a result, they check out
>> code into a different directory each time they run a job.
>>
>> Go's test caching checks every env var loaded by the test program. If any
>> of them change, then Go assumes that the cache is busted and the test must
>> be run.
>>
>> Any program that imports "os" hits this logic in the os package, which
>> checks the value of the $PWD environment variable:
>>
>> // We query the working directory at init, to use it later to search for
>> the
>> // executable file
>> // errWd will be checked later, if we need to use initWd
>> var initWd, errWd = Getwd()
>>
>> So checking code out to different directories means that any program that
>> imports "os" cannot have test caching. This seems like a shame because the
>> code is all laid out in the same place in the working directory.
>>
>> Has anyone tried to fix this issue? Do you think this is worth trying to
>> patch or modify the test caching behavior in Go itself? I could solve this
>> by running a chroot or another layer of Docker, of course, but I'd prefer
>> not to do these because of the difficulty of getting data in and out of
>> each one, communicating with other Docker containers, etc.
>>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "golang-nuts" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/golang-nuts/74iG66JVA9s/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/9507e736-9d8a-4929-9812-3a89358a1cf3n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAKcy5egYecGK8oSaB1xA439BPQ-BnCKi9vHpafe_qzRtQ_D%2B2w%40mail.gmail.com.

[go-nuts] Re: Changing PWD ruins test caching

['Bryan C. Mills' via golang-nuts]

As far as I can tell the code in question is only built on AIX and OpenBSD 
— are you using one of those two platforms?
If not, perhaps the caching problem is coming from somewhere else.

That said, it does appear that calls to `os.Chdir` result in a spurious 
dependency on the PWD variable:
https://cs.opensource.google/go/go/+/master:src/os/file.go;l=340-345;drc=d961b12be9001cf8dbf8f52847607dbf84d94f8d

On Thursday, November 23, 2023 at 1:50:04 PM UTC-5 Kevin Burke wrote:

> We have some tests that are pretty slow. I would like to use Go's test 
> caching to skip them in our CI environment, if nothing in the code or the 
> environment has changed. Our CI environment has a set of _agents_, each of 
> which can run multiple jobs simultaneously. As a result, they check out 
> code into a different directory each time they run a job.
>
> Go's test caching checks every env var loaded by the test program. If any 
> of them change, then Go assumes that the cache is busted and the test must 
> be run.
>
> Any program that imports "os" hits this logic in the os package, which 
> checks the value of the $PWD environment variable:
>
> // We query the working directory at init, to use it later to search for 
> the
> // executable file
> // errWd will be checked later, if we need to use initWd
> var initWd, errWd = Getwd()
>
> So checking code out to different directories means that any program that 
> imports "os" cannot have test caching. This seems like a shame because the 
> code is all laid out in the same place in the working directory.
>
> Has anyone tried to fix this issue? Do you think this is worth trying to 
> patch or modify the test caching behavior in Go itself? I could solve this 
> by running a chroot or another layer of Docker, of course, but I'd prefer 
> not to do these because of the difficulty of getting data in and out of 
> each one, communicating with other Docker containers, etc.
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/9507e736-9d8a-4929-9812-3a89358a1cf3n%40googlegroups.com.

[go-nuts] Re: Changing PWD ruins test caching

[Kevin Chowski]

I haven't had this problem so this is a complete guess. But if PWD is the 
only env variable causing a problem, you could try hard coding it to a 
constant value (which would never match any real directory). That would 
allow the PWD to always be the same no matter which directory it was 
actually running, and briefly looking through the code it seems that the 
PWD variable is only used it if actually matches "."

Sorry for the noise if this isn't helpful - I'm on mobile so I can't test 
right now.

On Thursday, November 23, 2023 at 1:50:04 PM UTC-5 Kevin Burke wrote:

> We have some tests that are pretty slow. I would like to use Go's test 
> caching to skip them in our CI environment, if nothing in the code or the 
> environment has changed. Our CI environment has a set of _agents_, each of 
> which can run multiple jobs simultaneously. As a result, they check out 
> code into a different directory each time they run a job.
>
> Go's test caching checks every env var loaded by the test program. If any 
> of them change, then Go assumes that the cache is busted and the test must 
> be run.
>
> Any program that imports "os" hits this logic in the os package, which 
> checks the value of the $PWD environment variable:
>
> // We query the working directory at init, to use it later to search for 
> the
> // executable file
> // errWd will be checked later, if we need to use initWd
> var initWd, errWd = Getwd()
>
> So checking code out to different directories means that any program that 
> imports "os" cannot have test caching. This seems like a shame because the 
> code is all laid out in the same place in the working directory.
>
> Has anyone tried to fix this issue? Do you think this is worth trying to 
> patch or modify the test caching behavior in Go itself? I could solve this 
> by running a chroot or another layer of Docker, of course, but I'd prefer 
> not to do these because of the difficulty of getting data in and out of 
> each one, communicating with other Docker containers, etc.
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/dfaee5c7-a11e-42be-8637-b8e155e776b0n%40googlegroups.com.