Re: [go-nuts] Re: Struggling with go mod, micro-repos and private forks

2019-10-25 Thread 'Bryan Mills' via golang-nuts 
On Friday, October 25, 2019 at 10:19:03 AM UTC-4, Shaun Crampton wrote:
>
>
>
> We have https://golang.org/issue/26232 open for 2FA workflows in general.
>>
>> In the meantime, you may need to configure a Personal Access Token 
>> 
>>  to 
>> get HTTPS to work.
>>
>
>
> We use 2FA for accessing the github UI but we tend to use SSH for push and 
> pull.  Would be easier if it tried ssh rather than bailing out.
>

 Ah, that one is https://golang.org/issue/26134.


>>>- Another team member said they tried using 
>>>v0.0.0-0001010100- as version but it got replaced.
>>>
>>> That probably means that you have a higher version requirement from some 
>> other dependency. (The `go` command automatically updates the `go.mod` file 
>> to maintain consistency.)
>>
>>>
>>>
> But then we go round in circles, right?  The start of this discussion was 
> that I wanted a "nil" version to put in my require statement so that I 
> could replace it with exactly the commit that I need using a "replace".  
> Now, if the tool helpfully updates the require then that defeats the point 
> of having a "nil" version in the first place.
>

If you don't care what version you're replacing, use a `replace` directive 
without a version and don't worry about the version that the `go` command 
adds in the `require` directive. 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/009b0f6d-4925-4e2c-9b50-c3efdd981198%40googlegroups.com.

Re: [go-nuts] Re: Struggling with go mod, micro-repos and private forks

2019-10-25 Thread Shaun Crampton 


We have https://golang.org/issue/26232 open for 2FA workflows in general.
>
> In the meantime, you may need to configure a Personal Access Token 
> 
>  to 
> get HTTPS to work.
>


We use 2FA for accessing the github UI but we tend to use SSH for push and 
pull.  Would be easier if it tried ssh rather than bailing out.

 

>
>>- Another team member said they tried using 
>>v0.0.0-0001010100- as version but it got replaced.
>>
>> That probably means that you have a higher version requirement from some 
> other dependency. (The `go` command automatically updates the `go.mod` file 
> to maintain consistency.)
>
>>
>>
But then we go round in circles, right?  The start of this discussion was 
that I wanted a "nil" version to put in my require statement so that I 
could replace it with exactly the commit that I need using a "replace".  
Now, if the tool helpfully updates the require then that defeats the point 
of having a "nil" version in the first place.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/7360a2f8-7787-4392-8077-7c4790c55d3e%40googlegroups.com.

Re: [go-nuts] Re: Struggling with go mod, micro-repos and private forks

2019-10-24 Thread 'Bryan C. Mills' via golang-nuts 
On Thu, Oct 24, 2019 at 1:07 PM Shaun Crampton  wrote:

> I circed some of the suggestions round my team.  Sounds like others had
> already tried some of the suggestions with mixed results:
>
>
>- Go v1.13 still has trouble authenticating to github without an
>"insteadof" in the config.  We use 2FA on github, which seems to make HTTPS
>fail in a way that doesn't fall back to git+ssh mode?
>
> We have https://golang.org/issue/26232 open for 2FA workflows in general.

In the meantime, you may need to configure a Personal Access Token

to
get HTTPS to work.


>- Another team member said they tried using
>v0.0.0-0001010100- as version but it got replaced.
>
> That probably means that you have a higher version requirement from some
other dependency. (The `go` command automatically updates the `go.mod` file
to maintain consistency.)


On Thursday, October 24, 2019 at 10:35:32 AM UTC+1, Shaun Crampton wrote:
>>
>>
>>> The good news is that we're aware of (and planning to address) most of
>>> these pain points; the bad news is that we haven't been able to get to most
>>> of them yet.
>>>
>>>
>> Great to see, thanks!
>>
>>
>>
>>>
 I think the biggest problem we have is when working with private
 repos.   What I want to express to the tool is

 My module requires commit abcd1234 (or version v1.2.3) of dependency
 x/y/z

 Look for any instances of dependency x/y/z in git repo 
 g...@github.com:ourfork
 instead of the default.


>>> I believe this is https://golang.org/issue/28176.
>>>
>>> It's an intriguing idea, but problematic if the dependencies are
>>> specified as pseudo-versions, because the commit hashes in a fork in
>>> general will not match the hashes in the upstream module. We're still
>>> exploring alternatives for this use-case.
>>>
>>
>>
>> So, the problem that pseudo-versions are trying to solve is ordering in
>> the face of unordered commit hashes?  For my use case, the way I want this
>> to work is that, once a dependency is pinned to my private fork, it's as if
>> the public repo didn't exist.  In 99% of cases, all the pins that I'm
>> working with are my organisation's repos anyway so once we pin to a private
>> fork that's what we want to use everywhere, transitively, even in
>> downstream repos.  In extreme cases, even the version numbering scheme in
>> the private fork may be different so there's not necessarily any relation
>> between the open source v2.3.0 and the private v2.3.0 tags.
>>
>> Our minority use case is forking some public repo to make a fix.  In that
>> case, a perfect tool would have a way to say "we need a version >= open
>> source version that must have (fix) commit abcd in its commit history"
>> Then, once our fix is upstreamed, the open source version will become a
>> candidate.  I suspect that's too much to ask though!
>>
>>
>>
>>>
>>>
>>> However, what I can express to the tool is

 My module requires version ??? of dependency x/y/z

 Replace  x/y/z @ version ??? with  @ abcd1234




>>> Currently it should be `v0.0.0-0001010100-`.
>>> We're also rethinking this behavior for `replace` in general; see
>>> https://golang.org/issue/33370.
>>>
>>>
>>
>> Good to know that there's a good value to put there.  Would be nice if it
>> could be made shorter, though, I'll have to look that up.
>>
>>
- There's nowhere to specify the details of the repo (e.g.
connection/auth type), all that has to magically work according to my 
 git
settings and the defaults aren't great for private repos (which we 
 access
over ssh).

 See https://golang.org/cmd/go/#hdr-Remote_import_paths.
>>>
>>> In general we expect that either you have an HTTPS server to locate the
>>> repo for a given import path (credentials can be stored in a `.netrc` file;
>>> see also https://golang.org/issue/26232), or you include an explicit
>>> VCS extension somewhere in the import path (and have corresponding
>>> credentials configured per-user in your VCS).
>>>
>>
>>
>> Maybe I'm just out-of-date, go now tries git+ssh as well as https://,
>> was that new in v1.13?  Since we only use github, that should be good
>> enough for me.
>>
>>
>>
>>>
>>> We also just ran into the new GOPRIVATE env var and had to update all
 our builds to use that.  Couldn't that just fall back to the private
 behaviour if it gets a cache miss, it seems over-engineered to require an
 explicit exception!

>>>
>>> The checksum database and proxy fallback are designed to provide
>>> integrity by default.
>>>
>>> Without making the opt-out explicit, an origin could serve one set of
>>> sources (and checksum) to you, and another version to someone else, and
>>> simply refuse to serve anything at all in response to queries from the
>>> checksum database

Re: [go-nuts] Re: Struggling with go mod, micro-repos and private forks

2019-10-24 Thread 'Bryan C. Mills' via golang-nuts 
On Thu, Oct 24, 2019 at 5:36 AM Shaun Crampton  wrote:

> I think the biggest problem we have is when working with private repos.
>>>  What I want to express to the tool is
>>>
>>> My module requires commit abcd1234 (or version v1.2.3) of dependency
>>> x/y/z
>>>
>>> Look for any instances of dependency x/y/z in git repo 
>>> g...@github.com:ourfork
>>> instead of the default.
>>>
>>>
>> I believe this is https://golang.org/issue/28176.
>>
>> It's an intriguing idea, but problematic if the dependencies are
>> specified as pseudo-versions, because the commit hashes in a fork in
>> general will not match the hashes in the upstream module. We're still
>> exploring alternatives for this use-case.
>>
>
>
> So, the problem that pseudo-versions are trying to solve is ordering in
> the face of unordered commit hashes?
>

Yes.


For my use case, the way I want this to work is that, once a dependency is
> pinned to my private fork, it's as if the public repo didn't exist.  In 99%
> of cases, all the pins that I'm working with are my organisation's repos
> anyway so once we pin to a private fork that's what we want to use
> everywhere, transitively, even in downstream repos.  In extreme cases, even
> the version numbering scheme in the private fork may be different so
> there's not necessarily any relation between the open source v2.3.0 and the
> private v2.3.0 tags.
>

In that case, you want a `replace` directive without a version on the left
side. It will prune out more of your transitive dependency graph than is
strictly necessary, but that's not usually a major problem.

(I'm hoping we can offer something better in the future, but there isn't
enough time for it to make 1.14.)


Our minority use case is forking some public repo to make a fix.  In that
> case, a perfect tool would have a way to say "we need a version >= open
> source version that must have (fix) commit abcd in its commit history"
> Then, once our fix is upstreamed, the open source version will become a
> candidate.  I suspect that's too much to ask though!
>

If you expect the upstream fix to be accepted before the next patch
release, generally the procedure there is:

require example.com/some/dependency 
replace example.com/some/dependency  => my.fork 

Then when you upgrade, you'll get a version higher than 
and your `replace` directive will no longer apply.

If the patch doesn't come through, then you update  in the
`replace` directive and try again next time.


However, what I can express to the tool is
>>>
>>> My module requires version ??? of dependency x/y/z
>>>
>>> Replace  x/y/z @ version ??? with  @ abcd1234
>>>
>>>
>>>
>>>
>> Currently it should be `v0.0.0-0001010100-`.
>> We're also rethinking this behavior for `replace` in general; see
>> https://golang.org/issue/33370.
>>
>
> Good to know that there's a good value to put there.  Would be nice if it
> could be made shorter, though, I'll have to look that up.
>

That's the version that the `go` command automatically inserts when it sees
a `replace` directive without an explicit version.
Ideally we shouldn't need to insert a version at all, though.


>>>- There's nowhere to specify the details of the repo (e.g.
>>>connection/auth type), all that has to magically work according to my git
>>>settings and the defaults aren't great for private repos (which we access
>>>over ssh).
>>>
>>> See https://golang.org/cmd/go/#hdr-Remote_import_paths.
>>
>> In general we expect that either you have an HTTPS server to locate the
>> repo for a given import path (credentials can be stored in a `.netrc` file;
>> see also https://golang.org/issue/26232), or you include an explicit VCS
>> extension somewhere in the import path (and have corresponding credentials
>> configured per-user in your VCS).
>>
>
> Maybe I'm just out-of-date, go now tries git+ssh as well as https://, was
> that new in v1.13?  Since we only use github, that should be good enough
> for me.
>

I'm not sure. I do know that 1.12 had a bug in its `.netrc` handling, which
is fixed in 1.13 and might let you proceed past some step that failed
previously.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAKWVi_Qrk9aa39qje9nUkWHWXR0bCyuuvBtLJUW4aJhB_or-tw%40mail.gmail.com.

[go-nuts] Re: Struggling with go mod, micro-repos and private forks

2019-10-24 Thread Shaun Crampton 
I circed some of the suggestions round my team.  Sounds like others had 
already tried some of the suggestions with mixed results:


   - Go v1.13 still has trouble authenticating to github without an 
   "insteadof" in the config.  We use 2FA on github, which seems to make HTTPS 
   fail in a way that doesn't fall back to git+ssh mode?
   - Another team member said they tried using 
   v0.0.0-0001010100- as version but it got replaced.



On Thursday, October 24, 2019 at 10:35:32 AM UTC+1, Shaun Crampton wrote:
>
>
>> The good news is that we're aware of (and planning to address) most of 
>> these pain points; the bad news is that we haven't been able to get to most 
>> of them yet.
>>
>>
> Great to see, thanks!
>
>  
>
>>
>>> I think the biggest problem we have is when working with private repos.  
>>>  What I want to express to the tool is 
>>>
>>> My module requires commit abcd1234 (or version v1.2.3) of dependency 
>>> x/y/z
>>>
>>> Look for any instances of dependency x/y/z in git repo 
>>> g...@github.com:ourfork instead of the default.
>>>
>>>  
>> I believe this is https://golang.org/issue/28176.
>>
>> It's an intriguing idea, but problematic if the dependencies are 
>> specified as pseudo-versions, because the commit hashes in a fork in 
>> general will not match the hashes in the upstream module. We're still 
>> exploring alternatives for this use-case.
>>
>
>
> So, the problem that pseudo-versions are trying to solve is ordering in 
> the face of unordered commit hashes?  For my use case, the way I want this 
> to work is that, once a dependency is pinned to my private fork, it's as if 
> the public repo didn't exist.  In 99% of cases, all the pins that I'm 
> working with are my organisation's repos anyway so once we pin to a private 
> fork that's what we want to use everywhere, transitively, even in 
> downstream repos.  In extreme cases, even the version numbering scheme in 
> the private fork may be different so there's not necessarily any relation 
> between the open source v2.3.0 and the private v2.3.0 tags.
>
> Our minority use case is forking some public repo to make a fix.  In that 
> case, a perfect tool would have a way to say "we need a version >= open 
> source version that must have (fix) commit abcd in its commit history"  
> Then, once our fix is upstreamed, the open source version will become a 
> candidate.  I suspect that's too much to ask though!
>
>  
>
>>
>>
>> However, what I can express to the tool is
>>>
>>> My module requires version ??? of dependency x/y/z
>>>
>>> Replace  x/y/z @ version ??? with  @ abcd1234
>>>
>>>  
>>>
>>>
>> Currently it should be `v0.0.0-0001010100-`.
>> We're also rethinking this behavior for `replace` in general; see 
>> https://golang.org/issue/33370.
>>
>>
>
> Good to know that there's a good value to put there.  Would be nice if it 
> could be made shorter, though, I'll have to look that up.  
>
>
>>>- There's nowhere to specify the details of the repo (e.g. 
>>>connection/auth type), all that has to magically work according to my 
>>> git 
>>>settings and the defaults aren't great for private repos (which we 
>>> access 
>>>over ssh).
>>>
>>> See https://golang.org/cmd/go/#hdr-Remote_import_paths.
>>
>> In general we expect that either you have an HTTPS server to locate the 
>> repo for a given import path (credentials can be stored in a `.netrc` file; 
>> see also https://golang.org/issue/26232), or you include an explicit VCS 
>> extension somewhere in the import path (and have corresponding credentials 
>> configured per-user in your VCS).
>>
>
>
> Maybe I'm just out-of-date, go now tries git+ssh as well as https://, was 
> that new in v1.13?  Since we only use github, that should be good enough 
> for me.
>
>
>
>>
>> We also just ran into the new GOPRIVATE env var and had to update all our 
>>> builds to use that.  Couldn't that just fall back to the private behaviour 
>>> if it gets a cache miss, it seems over-engineered to require an explicit 
>>> exception!
>>>
>>
>> The checksum database and proxy fallback are designed to provide 
>> integrity by default.
>>
>> Without making the opt-out explicit, an origin could serve one set of 
>> sources (and checksum) to you, and another version to someone else, and 
>> simply refuse to serve anything at all in response to queries from the 
>> checksum database (so that the entry would always be missing), and no one 
>> would be the wiser.
>>
>
>
> Fair enough, I suppose.  From a selfish point of view, making the go 
> tooling auto-trust github.com private repos would be nice :-)  I can't 
> easily do that with a single entry in GOPRIVATE since whether a repo is 
> private or not can only be guessed by whether you need to use credentials 
> or ssh to access it.
>
>  
>
>>
>>
>> Another issue we've had is making a build mode where we can build against 
>>> local copies of the dependencies for quick development.  We can add replace 
>>>

[go-nuts] Re: Struggling with go mod, micro-repos and private forks

2019-10-24 Thread Shaun Crampton 

>
>
> The good news is that we're aware of (and planning to address) most of 
> these pain points; the bad news is that we haven't been able to get to most 
> of them yet.
>
>
Great to see, thanks!

 

>
>> I think the biggest problem we have is when working with private repos.  
>>  What I want to express to the tool is 
>>
>> My module requires commit abcd1234 (or version v1.2.3) of dependency x/y/z
>>
>> Look for any instances of dependency x/y/z in git repo 
>> g...@github.com:ourfork instead of the default.
>>
>>  
> I believe this is https://golang.org/issue/28176.
>
> It's an intriguing idea, but problematic if the dependencies are specified 
> as pseudo-versions, because the commit hashes in a fork in general will not 
> match the hashes in the upstream module. We're still exploring alternatives 
> for this use-case.
>


So, the problem that pseudo-versions are trying to solve is ordering in the 
face of unordered commit hashes?  For my use case, the way I want this to 
work is that, once a dependency is pinned to my private fork, it's as if 
the public repo didn't exist.  In 99% of cases, all the pins that I'm 
working with are my organisation's repos anyway so once we pin to a private 
fork that's what we want to use everywhere, transitively, even in 
downstream repos.  In extreme cases, even the version numbering scheme in 
the private fork may be different so there's not necessarily any relation 
between the open source v2.3.0 and the private v2.3.0 tags.

Our minority use case is forking some public repo to make a fix.  In that 
case, a perfect tool would have a way to say "we need a version >= open 
source version that must have (fix) commit abcd in its commit history"  
Then, once our fix is upstreamed, the open source version will become a 
candidate.  I suspect that's too much to ask though!

 

>
>
> However, what I can express to the tool is
>>
>> My module requires version ??? of dependency x/y/z
>>
>> Replace  x/y/z @ version ??? with  @ abcd1234
>>
>>  
>>
>>
> Currently it should be `v0.0.0-0001010100-`.
> We're also rethinking this behavior for `replace` in general; see 
> https://golang.org/issue/33370.
>
>

Good to know that there's a good value to put there.  Would be nice if it 
could be made shorter, though, I'll have to look that up.  


>>- There's nowhere to specify the details of the repo (e.g. 
>>connection/auth type), all that has to magically work according to my git 
>>settings and the defaults aren't great for private repos (which we access 
>>over ssh).
>>
>> See https://golang.org/cmd/go/#hdr-Remote_import_paths.
>
> In general we expect that either you have an HTTPS server to locate the 
> repo for a given import path (credentials can be stored in a `.netrc` file; 
> see also https://golang.org/issue/26232), or you include an explicit VCS 
> extension somewhere in the import path (and have corresponding credentials 
> configured per-user in your VCS).
>


Maybe I'm just out-of-date, go now tries git+ssh as well as https://, was 
that new in v1.13?  Since we only use github, that should be good enough 
for me.



>
> We also just ran into the new GOPRIVATE env var and had to update all our 
>> builds to use that.  Couldn't that just fall back to the private behaviour 
>> if it gets a cache miss, it seems over-engineered to require an explicit 
>> exception!
>>
>
> The checksum database and proxy fallback are designed to provide integrity 
> by default.
>
> Without making the opt-out explicit, an origin could serve one set of 
> sources (and checksum) to you, and another version to someone else, and 
> simply refuse to serve anything at all in response to queries from the 
> checksum database (so that the entry would always be missing), and no one 
> would be the wiser.
>


Fair enough, I suppose.  From a selfish point of view, making the go 
tooling auto-trust github.com private repos would be nice :-)  I can't 
easily do that with a single entry in GOPRIVATE since whether a repo is 
private or not can only be guessed by whether you need to use credentials 
or ssh to access it.

 

>
>
> Another issue we've had is making a build mode where we can build against 
>> local copies of the dependencies for quick development.  We can add replace 
>> directives to point to local directories, which is part of the puzzle, but 
>> it's hard to do that "just for one build".  Not sure what we're looking for 
>> here; maybe an optional go.mod override file that can be passed in for just 
>> one build?
>>
>
> You mean like https://golang.org/issue/34506?  
>
>  
>

Sounds like just what we need, thanks! 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit

[go-nuts] Re: Struggling with go mod, micro-repos and private forks

2019-10-23 Thread 'Bryan Mills' via golang-nuts 
Thanks for the feedback.

The good news is that we're aware of (and planning to address) most of 
these pain points; the bad news is that we haven't been able to get to most 
of them yet.

Detailed responses inline.

On Tuesday, October 22, 2019 at 6:42:05 AM UTC-4, Shaun Crampton wrote:
>
> Hi All,
>
> My team own a large product that sprawls across a lot of repos.  We also 
> manage a private, commercial fork and we occasionally do new development in 
> private and then merge across to open source.  While a monorepo might have 
> been easier to manage, we're stuck with what we've got.  We've recently 
> moved to go mod and it just seems like we're constantly fighting the tool.  
> I'm hoping you can either suggest some good workflows or maybe improve the 
> tool!
>
> I think the biggest problem we have is when working with private repos.  
>  What I want to express to the tool is 
>
> My module requires commit abcd1234 (or version v1.2.3) of dependency x/y/z
>
> Look for any instances of dependency x/y/z in git repo 
> g...@github.com:ourfork instead of the default.
>
>  
I believe this is https://golang.org/issue/28176.

It's an intriguing idea, but problematic if the dependencies are specified 
as pseudo-versions, because the commit hashes in a fork in general will not 
match the hashes in the upstream module. We're still exploring alternatives 
for this use-case.


However, what I can express to the tool is
>
> My module requires version ??? of dependency x/y/z
>
> Replace  x/y/z @ version ??? with  @ abcd1234
>
>  
>
> This throws up a couple of problems:
>
>- What should version ??? be?  It's only there to be replaced, which 
>seems like a bit of a smell.  
>   - If I set it to the commit ID it gets resolved and I have to 
>   change three places in the file each time I move the pin.
>   - If I set it to a particular version, that seems misleading.  
>   - I guess I can set it to v0.0.0, but again that seems misleading.
>
>
Currently it should be `v0.0.0-0001010100-`.
We're also rethinking this behavior for `replace` in general; 
see https://golang.org/issue/33370.



>- There's nowhere to specify the details of the repo (e.g. 
>connection/auth type), all that has to magically work according to my git 
>settings and the defaults aren't great for private repos (which we access 
>over ssh).
>
> See https://golang.org/cmd/go/#hdr-Remote_import_paths.

In general we expect that either you have an HTTPS server to locate the 
repo for a given import path (credentials can be stored in a `.netrc` file; 
see also https://golang.org/issue/26232), or you include an explicit VCS 
extension somewhere in the import path (and have corresponding credentials 
configured per-user in your VCS).


We also just ran into the new GOPRIVATE env var and had to update all our 
> builds to use that.  Couldn't that just fall back to the private behaviour 
> if it gets a cache miss, it seems over-engineered to require an explicit 
> exception!
>

The checksum database and proxy fallback are designed to provide integrity 
by default.

Without making the opt-out explicit, an origin could serve one set of 
sources (and checksum) to you, and another version to someone else, and 
simply refuse to serve anything at all in response to queries from the 
checksum database (so that the entry would always be missing), and no one 
would be the wiser.


Another issue we've had is making a build mode where we can build against 
> local copies of the dependencies for quick development.  We can add replace 
> directives to point to local directories, which is part of the puzzle, but 
> it's hard to do that "just for one build".  Not sure what we're looking for 
> here; maybe an optional go.mod override file that can be passed in for just 
> one build?
>

You mean like https://golang.org/issue/34506?  

 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/68003fda-f411-4e5f-9657-15b774c176d0%40googlegroups.com.