Re: [go-nuts] Re: Web access to golang-dev is down

[thepudds1460]

Hi Ian,

I can now access golang-dev again via the web interface, whereas previously 
I could not.

(Hopefully that is not a coincidence).

Thanks for looking into this.

Regards,
thepudds



On Tuesday, May 7, 2019 at 1:39:50 PM UTC-4, Ian Lance Taylor wrote:
>
> On Tue, May 7, 2019 at 8:01 AM Marcin Romaszewicz  > wrote: 
> > 
> > The same for me. golang-dev is not accessible to me, nor does it show up 
> in a group search, meaning it's not visible to me. I'm not a registered 
> member of it, so it appears to no longer be a public group. 
>
>
> Thanks for all the info. 
>
> I've found that golang-nuts had checked 
>
> List this group in the directory 
> Listing a group in the directory will make it easier for people to 
> find your group. When listed in the directory your group's name, email 
> and description will be visible and searchable. 
>
> where golang-dev had not.  I doubt this changed recently, but I 
> checked the box on golang-dev to make it the same as golang-nuts. 
> Does that help people who were seeing problems before? 
>
> Ian 
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/255d9f19-1c3a-444f-8094-a250d5d214f1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Working outside of VCS - parent/child modules

[thepudds1460]

You do have the option to nest one module within another module in the same 
repo, but from what I have seen, that is usually not what one wants to do, 
and especially not to start. It can be tricky to set up correctly, and 
usually is more work on an on-going basis.

The clearest statement I've seen might be Russ Cox wrote in #26664:

   "For all but power users, you probably want to adopt the usual 
convention 
that one repo = one module. It's important for long-term evolution of 
code 
storage options that a repo can contain multiple modules, but it's 
almost 
certainly not something you want to do by default."

In terms of your question about each child module being able to specify 
it's own dependencies, that is a more nuanced question, but hopefully this 
at least conveys the gist...

If you have a single go.mod with multiple packages, that go.mod will track 
the dependencies across all packages, which usually works reasonably well 
in practice for most projects.

If we take the example where package foo and bar reside in a single module, 
it is usually not a problem for example if:

 1. foo and bar both require the same v1+ major version of some dependency 
X (including given X should be compatible within a v1+ major version, 
according to semver).
 
 2. foo and bar require different major versions of dependency Y (including 
given Y can be separately tracked within a single go.mod as require Y/v2 
and require Y/v3).

It would however be a problem if:

 3. package foo requires a v0 version of some dependency Z that is 
incompatible with the v0 or v1 version of Z required by bar (because at 
most one v0/v1 version of dependency Z would be tracked within a single 
go.mod).
 
 4. if foo and bar require different incompatible versions of a shared 
dependency that does not follow https://semver.org. (Following semver is a 
requirement for any code opting in to modules, but pre-existing code will 
not have always followed semver).

Perhaps your logging example would fall into the second case of needing to 
use different major versions in a single build? If so, modules allow that.

If you weigh the pros and cons and really do feel the need to nest modules 
within a single repo, the modules wiki has an entire section dedicated to 
that, which I would encourage you to read carefully if you do go down that 
path. As you'll note, though, there is some nuance there:

   https://github.com/golang/go/wiki/Modules#faqs--multi-module-repositories
   
Regards,
thepudds

On Saturday, April 20, 2019 at 8:57:09 AM UTC-4, whiteh...@googlemail.com 
wrote:
>
>
> https://github.com/golang/go/wiki/Modules#can-i-work-entirely-outside-of-vcs-on-my-local-filesystem
>
> For a simple parent/child module relationship this seems to work, since 
> the 'replace' work-around is in the parent go.mod.  
> But when the childA depends on childB, then the 'replace' directive in 
> childA's go.mod is ignored, and the build fails.  According to the above 
> information this is working as designed.
>
> Since there is no error feedback that the 'replace' directive has been 
> ignored, I eventually stumbled over a couple of similar reported issues.  
> Which is how I also found the information that sub-module 'replace' 
> directives are being ignored.
>
> https://github.com/golang/go/issues/29376
> https://github.com/golang/go/issues/31345
>
> Since I'm just learning Go I would like to understand the approach 
> better.  I come from a Maven background for dependency management.  Here it 
> is a common pattern to have a parent/child module design.  Each child 
> module can specify it's own dependencies.  And in complex projects two 
> different sub-modules or sub-sub-modules may even have requirements on 
> different versions of the same library (Logging I'm looking at you!).
>
> So this question is also applicable when running under VCS. ie that Go 
> sub-modules may want to specify a specific version of a library that they 
> depend upon using the replace directive.
>
> Anyway I am left with the impression that this parent/child module pattern 
> might be considered non idiomatic in the Go world?  ie Go is targeting a 
> 1-to-1 mapping of VCS project to Module.  Is that fair to say?  
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: Packaging a Go module for Nix

[thepudds1460]

Hi Wael,

Sorry, I am not quite following what you have and have not tried yet, and 
which issues you have hit with which techniques.

Is one of the issues that the 'vcs' cache directory changes, even if the 
actual code you need for your build has not changed?

If so, I wonder if you might be able use a filesystem-based module cache 
via 'GOPROXY=file:///file/path', which could avoid using the 'vcs' 
directory at build time?

It sounds like you have looked into multiple options at this point, so this 
might be well known to you at this point, but:

 * The module download cache location is controlled by GOPATH. In 
particular, 'go mod download', 'go build', etc. populate the module cache 
in GOPATH/pkg/mod.
 * In addition, when you want to use a particular module cache, you can 
tell the 'go' command to use a local module cache by setting 
'GOPROXY=file:///file/path' environment variable.
 * You can put those two things together:

 # Populate a module download cache in /tmp/gopath-for-cache
 $ GOPATH=/tmp/gopath-for-cache  go mod download

 # Build using the contents of the module download cache in 
/tmp/gopath-for-cache
 $ GOPROXY=file:///tmp/gopath-for-cache/pkg/mod/cache/download  go build

Note that /tmp/gopath-for-cache/pkg/mod/cache/download would not contain 
the 'vcs' directory.

I understand those are not the exact steps you would follow, but perhaps 
something along those lines could be adapted within your constraints?

Also, note that even though you are setting the GOPROXY environment 
variable in the steps above, there is no actual proxy process involved, and 
everything is just being read directly from the local filesystem. An even 
more detailed example is here in this "Go Modules by Example" walk-through: 
https://github.com/go-modules-by-example/index/tree/master/012_modvendor

Sorry if anything here is off-base, or if this is not helpful.

Regards, 
thepudds

On Monday, March 11, 2019 at 1:01:04 PM UTC-4, Manlio Perillo wrote:
>
> On Monday, March 11, 2019 at 4:22:04 PM UTC+1, Wael Nasreddine wrote:
>
> > [...]
>  
>
>> On Sunday, March 10, 2019 at 6:59:07 PM UTC-7, Manlio Perillo wrote:
>
> On Monday, March 11, 2019 at 12:30:02 AM UTC+1, Wael Nasreddine wrote:
>>
>> TL;DR Given a Go module, assuming that I have already done `go mod 
>> download`: Is it possible to prevent network access if I delete the 
>> entire 
>> `$GOPATH/pkg/mod/cache`?
>>
>>
> Another solution is to delete only $GOPATH/pkg/mod/cache/vcs.  This is 
> the directory that takes more disk space since it contains the full 
> history 
> of the vcs.
>
  
 That was my first attempt, however it did not work as expect because 
 the contents of $GOPATH/pkg/mod/cache/download does change anytime a new 
 version is created upstream. My hash failed to match multiple times a day!

>>>
>>> It may be caused by the missing pkg/mod/cache/vcs, but it seems unusual.
>>> I was assuming GOPATH was on a temporary directory.
>>> And indeed you wrote that GOPATH is set to $NIX_BUILD_TOP/go, but later 
>>> you wrote that it is set to a temporary directory.
>>>
>>>
>> I am removing the entire cache folder, as the content changes when the 
>> git repository of any of the dependencies change. So if I would hash the 
>> cache folder to A and one of the dependencies get a pull request merged 
>> upstream, even though it does not technically change the version that Go is 
>> building with you will still get B when you hash it. This break the concept 
>> of packaging :(
>>
>>
> Do you perhaps have the same requirements as in the thread
> https://groups.google.com/forum/#!topic/golang-dev/DD88cds-LuI
> as reported by Nicolas Mailhot?
>
> That is, you need to patch the upstream source but keep the same version, 
> because you can't (or don't want to) update all the versions of the 
> required modules.
>
> In this case vendoring is, IMHO, currently the only solution, because the 
> go tool ignores the go.sum files in this case.
> You can try to open a bug report about the incorrect behavior of `go mod 
> vendor` when using cgo.
>
> Or you can patch cmd/go.  However instead of writing a patch specific to 
> Nix, I suggest to write a more generic patch.
>
> You can add a new module download mode, e.g. `-mod trust` to instruct 
> cmd/go to not check go.sum.
> The future notary can be disabled with GONOVERIFY, so its not a problem.
> Maybe you can use GONOVERIFY to decide if the checksum should be ignored.
>
> Finally, you can try to coordinate with other package managers, since this 
> seems to be a shared problem.
>
> Here is a patch.  It seems to work but one the Go test fails, and it 
> should probably be updated:
> https://gist.github.com/perillo/813b52dcff8824da3a2859828961d08b
>
> An alternative is to check for trust mode in the initGoSum function:
> https://gist.github.com/perillo/1b3d3216c59f689e52ee7357d2a99136
>
> Finally the last alternative is to check 

Re: [go-nuts] Allow go mod download to store in subdirectory of current working directory

[thepudds1460]

Jorrit,

The simplest solution might be 'go mod vendor' to populate a 'vendor' 
directory, and then set '-mod=vendor' or 'GOFLAGS=-mod=vendor' when 
building.

If that is not workable for some reason, here is an alternative that is 
more similar to your request for a 'go mod download -dir module_cache':

The module download cache location is controlled by GOPATH. In particular, 
'go mod download', 'go build', etc. populate the module cache in 
GOPATH/pkg/mod (or ~/go/pkg/mod if $GOPATH is not set).

In addition, when you want to use a particular module cache, you can tell 
the 'go' command to use a local module cache by setting 
GOPROXY=file:///file/path.

You can put those two things together, which I think gives you the control 
you were asking about:
 
# Populate a module download cache in /tmp/gopath-for-cache
$ GOPATH=/tmp/gopath-for-cache  go mod download

# Build using the contents of the module download cache in 
/tmp/gopath-for-cache
$ GOPROXY=file:///tmp/gopath-for-cache/pkg/mod/cache/download  go build

Note that even though you are setting the GOPROXY environment variable, 
there is no actual proxy process involved, and everything is just being 
read directly from the local filesystem.
 
Here is a more complete "Go Modules by Example" walk-through of that 
technique:

https://github.com/go-modules-by-example/index/tree/master/012_modvendor

Regards,
thepudds

On Monday, March 4, 2019 at 12:50:51 PM UTC-5, Marcin Romaszewicz wrote:
>
> Can you accomplish what you want with module vendoring? Before I set up my 
> own Athens proxy, I was using that to avoid DDOSing github in my build 
> automation, and it seemed to work fine. Your first pipeline step could 
> vendor in all the modules, and subsequent pipeline steps would use those.
>
> -- Marcin
>
>
> On Mon, Mar 4, 2019 at 8:49 AM > wrote:
>
>> I'd like to be able to use go mod download to download packages to 
>> another directory than GOPATH/pkg/mod and automatically have all go 
>> commands be aware of this new location.
>>
>>
>> This is specifically for CI/CD to ensure I can download modules in one 
>> stage and use them in subsequent stages.
>>
>>
>> I'm the author of https://estafette.io/, and just like Bitbucket 
>> Pipelines and Drone.io it executes steps in (public) docker containers, but 
>> the only data passed from stage to stage are the files inside the working 
>> directory. So downloading modules to a directory outside of the 
>> working directory makes them get downloaded again in subsequent stages.
>>
>>
>> NPM tackles this by having the node_modules directory as a local 
>> subdirectory of your repository, but of course this loses you the 
>> performance advantage of having a shared cache for all your repositories.
>>
>>
>> Nuget uses a global cache as well, but they allow you to specify an 
>> alternative location that gets used by their other commands by running 
>> dotnet 
>> restore --packages .nuget/packages.
>>
>>
>> A similar approach as dotnet takes would be best for all go module 
>> related commands to be able to prevent CI/CD from downloading the same 
>> modules multiple times. This could look something like go mod download 
>> -dir module_cache.
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: How do I update all direct and indirect dependencies in go.mod

[thepudds1460]

Hi Francis,

To ask that your direct dependencies be upgraded to their latest available 
versions, you can do the following in Go 1.11 or 1.12:

  go get $(go list -f '{{if not (or .Main .Indirect)}}{{.Path}}{{end}}' -m 
all)

Your indirect dependencies will be updated as needed according to the 
requirements of your direct dependencies.

There is a proposal to make that easier here, along with some related 
discussion:

https://github.com/golang/go/issues/28424

Regards,
thepudds

On Tuesday, February 26, 2019 at 7:43:18 PM UTC-5, Francis Chuang wrote:
>
> I have dependencies (direct and some indirect) listed in my go.mod file. 
> This was initially populated using `go ./...` after creating the go.mod 
> using `go mod init`.
>
> I now want to upgrade the dependencies and the indirect dependencies in 
> the go.mod file to their latest versions.
>
> If I run `go get -u`, it pulls in the latest versions of the dependencies 
> as well as all the dependencies for running their tests on all platforms. 
> In effect, my go.mod file has more then tripled in size. 
>
> Is there any way to update my direct (and some required indirect) 
> dependencies required just to build my project in go.mod?
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] why go get does not work outside a module?

[thepudds1460]

The good news is that this is addressed for 1.12:

  #24250 cmd/go: allow "go get" when outside a module in module mode
  https://github.com/golang/go/issues/24250

It would be worthwhile to try the 1.12 release candidate if you can:

  https://groups.google.com/forum/#!topic/golang-announce/r0R2jijOjBo
  
My understanding is that in the lead-up to 1.11, it was a bit debatable how 
it should work, so the thinking was that it would be better to first get 
some more experience as part of the initial modules experiment in 1.11.
  
You can do 'go get' for a binary outside of a module in 1.11, but it is not 
as natural as it is in Go 1.12.  There are a few options outlined here in 
this FAQ (along with some more background, a sample helper script, etc.):

  
https://github.com/golang/go/wiki/Modules#why-does-installing-a-tool-via-go-get-fail-with-error-cannot-find-main-module

That FAQ is a little broader than your specific question, but you can look 
over the 1.11 workarounds listed in that FAQ, which hopefully gives a more 
complete description than my quick comment above.

One option in a particular that is mentioned in that FAQ is 
https://github.com/myitcv/gobin, which is a community tool targeting this 
area of functionality that many people are using now. 'gobin' has a bit 
more flexibility, which you can read about at that repo.

Regards,
thepudds

On Saturday, February 16, 2019 at 7:03:03 AM UTC-5, Manlio Perillo wrote:
>
> Well, the problem is that module aware go get will **still** download 
> commands in GOBIN and **will** cache downloaded modules in
> GOPATH/pkg/mod.
>
> So I don't see a valid reason why it should not work outside a module.
>
> Thanks
> Manlio Perillo
>
> On Saturday, February 16, 2019 at 1:08:08 AM UTC+1, Tyler Compton wrote:
>>
>> It sounds to me like you would want to set GO111MODULE to "auto". My 
>> understanding is that then "go get" will behave like it used to when run 
>> inside of the GOPATH or in a directory tree without a go.mod file.
>>
>> Here's a bit of information about it on the wiki: 
>> https://github.com/golang/go/wiki/Modules#when-do-i-get-old-behavior-vs-new-module-based-behavior
>>
>>
>> On Fri, Feb 15, 2019 at 6:23 AM Manlio Perillo  
>> wrote:
>>
>>> I have started to use go modules recently, and I have set GO111MODULE=on.
>>>
>>> It was unexpected to found that go get does not work outside a module.
>>> As an example:
>>> $ go get github.com/davecheney/godoc2md
>>> go: cannot determine module path for source directory /home/manlio 
>>> (outside GOPATH, no import comments)
>>>
>>> When I used go get inside a module I found that:
>>>
>>> 1. It saves the downloaded packages in the go mod cache
>>> 2. It Installs the command in GOBIN
>>>
>>> These two points can be also done outside a module.
>>>
>>> 3. Add an (indirect) entry in the require section in the go.mod file
>>>
>>> The new entry will, however, be removed by go mod tidy.
>>>
>>> Is it really necessary that go get must only work inside a module?
>>>
>>>
>>> Thanks
>>> Manlio Perillo
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to golang-nuts...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: Why is "v" prefix for VCS tag required for ?

[thepudds1460]

To give a quick concrete example for the point below about still being able 
to make use of an existing tag that does not have a leading "v"...

If you do the following from within a module (note no "v" in "0.0.3"):

  $ go get github.com/jondot/groundcontrol@0.0.3

The result is recorded in the go.mod file as the following pseudo-version, 
which includes the date and commit hash:

  github.com/jondot/groundcontrol v0.0.0-20130702202810-bf0b23022af7 

This behavior is covered in more detail here:

  https://golang.org/cmd/go/#hdr-Module_queries

All that said, a module should still be tagged in VCS with a leading "v" 
when released in order for the 'go' tool to be able to properly interpret 
the tag as a semantic version.

Regards,
thepudds

On Monday, February 4, 2019 at 5:39:25 PM UTC-5, thepud...@gmail.com wrote:
>
> As far as I understand it, there is a distinction drawn between:
>
>   1. a "semantic version" (where a leading "v" is not part of a "semantic 
> version"), vs. 
>   2. the mechanism for encoding a "semantic version" into a VCS tag (where 
> a leading "v" is allowed)
>
> In other words, under that interpretation, a leading "v" in a VCS tags is 
> a common and allowed way to encode a semantic version into a VCS tag, 
> although not a universal choice, and not required.
>
> However, I have also seen reasonable people disagree on that 
> interpretation.
>
> One hint is the master branch for github.com/semver/semver contains this 
> statement as part of an FAQ that was added a few years ago:
>
> 
> prefixing a semantic version with a "v" is a common way (in English) to 
> indicate it is a version number. Abbreviating "version" as "v" is often 
> seen with version control. Example: git tag v1.2.3 -m "Release version 
> 1.2.3", in which case "v1.2.3" is a tag name and the semantic version is 
> "1.2.3".
> 
>
> (From https://github.com/semver/semver/blob/master/semver.md)
>
> However, that FAQ is only on master as far as I am aware, and I have also 
> seen reasonable people come to different conclusions even after reading 
> that particular FAQ.
>
> Setting aside for the moment what the best encoding might be, it seems 
> reasonable to me that the 'go' tool has made a choice about a canonical 
> encoding of semantic versions into VCS tags.
>
> Finally, note that tags like "1.2.3" (without a "v") can still be used, 
> but they will not be treated as semantic versions by the 'go' tool. 
> Instead, "go get foo@1.2.3" (note no "v") and "go get foo@sometag" are 
> treated as "module queries", and are often recorded as "pseudo-versions". 
> There is a bit more about those two features of modules here:
>
>https://golang.org/cmd/go/#hdr-Module_queries
>https://golang.org/cmd/go/#hdr-Pseudo_versions
>
> Regards,
> thepudds
>
> On Monday, February 4, 2019 at 2:55:35 PM UTC-5, Ian Denhardt wrote:
>>
>> Also, on a more pragmatic level, what do you do if a repository has both 
>> tags: is the "real" version "v2.0.0" or "2.0.0"? 
>>
>> But I generally agree with the other arguments; machines should be 
>> opinionated about the color of the shed, so humans can argue about 
>> things that actually matter (see also gofmt). 
>>
>> -Ian 
>>
>> Quoting gud...@gmail.com (2019-02-04 13:24:55) 
>> >If you start with two options you open up a flood of endless 
>> > 
>> >discussions about why just these two and not that other 
>> > 
>> >too. 
>> > 
>> >� 
>> >No, it won't. Current version of semver convention (which is 2.0.0 
>> ATM 
>> >and not 1.0.0)� says (as I mentioned) that any prefix violates 
>> (mildly 
>> >though) semver. 
>> >And they mentioned explicitly only "v" prefix. No "r", no any other 
>> >ridiculous ones since it's semver, not semrelease or semútgáfa. 
>> >Other than that is just a faulty generalization and shouldn't be 
>> used 
>> >against this point. 
>> >I actually had nothing against� this particular prefix. 
>> >It just shouldn't be the only option since semver itself doesn't 
>> have 
>> >requirements on tagging notation (at least anymore) and even states 
>> >against that. 
>> > 
>> >  And the v prefix is pretty common 
>> > 
>> >Yep, and that's as common as no-prefix format. 
>> > 
>> >  And retagging a "3.1.4" as "v3.1.4" is dead simple. 
>> > 
>> >Also changing go-semver standard to support non-prefixed tags 
>> without 
>> >losing any backward compatibility. 
>> >Current standard which meant to be implemented currently not 
>> >implementing semver as it is. 
>> >On Monday, February 4, 2019 at 11:31:58 AM UTC+3, Volker Dobler 
>> wrote: 
>> > 
>> >Well, it is good to have exactly one valid format. 
>> >With two options like "vN.M.L" and "N.M.L" confusion starts 
>> >to spread and why not allow a capital "V" too and a "ú" too 
>> >for "útgáfa" and "r" for "release" should be allowed too. 
>> >If you start with two options you open up 

Re: [go-nuts] Re: Why is "v" prefix for VCS tag required for ?

[thepudds1460]

As far as I understand it, there is a distinction drawn between:

  1. a "semantic version" (where a leading "v" is not part of a "semantic 
version"), vs. 
  2. the mechanism for encoding a "semantic version" into a VCS tag (where 
a leading "v" is allowed)

In other words, under that interpretation, a leading "v" in a VCS tags is a 
common and allowed way to encode a semantic version into a VCS tag, 
although not a universal choice, and not required.

However, I have also seen reasonable people disagree on that interpretation.

One hint is the master branch for github.com/semver/semver contains this 
statement as part of an FAQ that was added a few years ago:


prefixing a semantic version with a "v" is a common way (in English) to 
indicate it is a version number. Abbreviating "version" as "v" is often 
seen with version control. Example: git tag v1.2.3 -m "Release version 
1.2.3", in which case "v1.2.3" is a tag name and the semantic version is 
"1.2.3".


(From https://github.com/semver/semver/blob/master/semver.md)

However, that FAQ is only on master as far as I am aware, and I have also 
seen reasonable people come to different conclusions even after reading 
that particular FAQ.

Setting aside for the moment what the best encoding might be, it seems 
reasonable to me that the 'go' tool has made a choice about a canonical 
encoding of semantic versions into VCS tags.

Finally, note that tags like "1.2.3" (without a "v") can still be used, but 
they will not be treated as semantic versions by the 'go' tool. Instead, 
"go get foo@1.2.3" (note no "v") and "go get foo@sometag" are treated as 
"module queries", and are often recorded as "pseudo-versions". There is a 
bit more about those two features of modules here:

   https://golang.org/cmd/go/#hdr-Module_queries
   https://golang.org/cmd/go/#hdr-Pseudo_versions

Regards,
thepudds

On Monday, February 4, 2019 at 2:55:35 PM UTC-5, Ian Denhardt wrote:
>
> Also, on a more pragmatic level, what do you do if a repository has both 
> tags: is the "real" version "v2.0.0" or "2.0.0"? 
>
> But I generally agree with the other arguments; machines should be 
> opinionated about the color of the shed, so humans can argue about 
> things that actually matter (see also gofmt). 
>
> -Ian 
>
> Quoting gud...@gmail.com  (2019-02-04 13:24:55) 
> >If you start with two options you open up a flood of endless 
> > 
> >discussions about why just these two and not that other 
> > 
> >too. 
> > 
> >� 
> >No, it won't. Current version of semver convention (which is 2.0.0 
> ATM 
> >and not 1.0.0)� says (as I mentioned) that any prefix violates 
> (mildly 
> >though) semver. 
> >And they mentioned explicitly only "v" prefix. No "r", no any other 
> >ridiculous ones since it's semver, not semrelease or semútgáfa. 
> >Other than that is just a faulty generalization and shouldn't be used 
> >against this point. 
> >I actually had nothing against� this particular prefix. 
> >It just shouldn't be the only option since semver itself doesn't have 
> >requirements on tagging notation (at least anymore) and even states 
> >against that. 
> > 
> >  And the v prefix is pretty common 
> > 
> >Yep, and that's as common as no-prefix format. 
> > 
> >  And retagging a "3.1.4" as "v3.1.4" is dead simple. 
> > 
> >Also changing go-semver standard to support non-prefixed tags without 
> >losing any backward compatibility. 
> >Current standard which meant to be implemented currently not 
> >implementing semver as it is. 
> >On Monday, February 4, 2019 at 11:31:58 AM UTC+3, Volker Dobler 
> wrote: 
> > 
> >Well, it is good to have exactly one valid format. 
> >With two options like "vN.M.L" and "N.M.L" confusion starts 
> >to spread and why not allow a capital "V" too and a "ú" too 
> >for "útgáfa" and "r" for "release" should be allowed too. 
> >If you start with two options you open up a flood of endless 
> >discussions about why just these two and not that other 
> >too. 
> >So the whole question is "why v.N.M.M and not N.M.L ?" 
> >This might be a question of taste and thus unanswerable. 
> >I personally think the v prefix makes it clear it is a version tag. 
> >Even Semver 1.0.0 suggested to prefix the version number 
> >with v in the VCS. 
> >And the v prefix is pretty common: Docker, Kubernetes, CoreOS. 
> >And retagging a "3.1.4" as "v3.1.4" is dead simple. 
> >In summary. There is one format and that one is v; 
> >it is a standard and standards are to be implemented. 
> >V.� 
> >On Monday, 4 February 2019 07:52:09 UTC+1, Maxim S wrote: 
> > 
> >Now go mod requires to have prefix "v" for it's tag in VCS and it's 
> >mandatory so you can't use tags in form of "A.B.C" in your workflow. 
> >But this limitation has little to zero reason to exist so why is it 
> >strict 

[go-nuts] Re: performance of go mod verify

[thepudds1460]

Hi Joe,

Sorry, I realized I pasted in the wrong quote from the documentation.

I meant to include this quote instead 
(from https://golang.org/cmd/go/#hdr-Module_downloading_and_verification):

"The go command maintains, in the main module's root directory alongside 
go.mod, a file named go.sum containing the expected cryptographic checksums 
of the content of specific module versions. Each time a dependency is used, 
its checksum is added to go.sum if missing or else required to match the 
existing entry in go.sum.

The go command maintains a cache of downloaded packages and computes and 
records the cryptographic checksum of each package at download time. In 
normal operation, the go command checks these pre-computed checksums 
against the main module's go.sum file, instead of recomputing them on each 
command invocation. The 'go mod verify' command checks that the cached 
copies of module downloads still match both their recorded checksums and 
the entries in go.sum."

Regards,
thepudds

On Monday, January 28, 2019 at 2:53:17 PM UTC-5, thepud...@gmail.com wrote:
>
> Hi Joe,
>
> As far as I know, I think it is computing the SHA256 of the dependencies:
>
>
> https://github.com/golang/go/blob/release-branch.go1.11/src/cmd/go/internal/dirhash/hash.go#L25
>
> Personally, I wouldn't expect that to get pathologically worse with more 
> dependencies.
>
> Also, just as you can often sequence 'go mod download' so that you can 
> often take advantage of cached dependencies in CI, there is some chance you 
> could do something similar with 'go mod verify' to avoid that cost every 
> time?
>
> In terms of how it all works, there is a bit more in the documentation 
> here:
>
> https://golang.org/cmd/go/#hdr-Modules_and_vendoring
>
> "By default, the go command satisfies dependencies by downloading modules 
> from their sources and using those downloaded copies (after verification, 
> as described in the previous section). To allow interoperation with older 
> versions of Go, or to ensure that all files used for a build are stored 
> together in a single file tree, 'go mod vendor' creates a directory named 
> vendor in the root directory of the main module and stores there all the 
> packages from dependency modules that are needed to support builds and 
> tests of packages in the main module."
>
> Regards,
> thepudds
>
> On Monday, January 28, 2019 at 1:14:53 PM UTC-5, Joseph Lorenzini wrote:
>>
>> All: 
>>
>> I'd like to run go mod verify as part of the CI/CD. If the verify fails, 
>> the build fails. However, the amount of time it takes to run for a small 
>> project is making me question whether that's viable. I don't actually know 
>> of any documentation that explains how slow or fast one should expect a go 
>> mod verify to take to run nor how much time it will increase as you add 
>> packages (e.g as you add packages is the increase linear or is it more 
>> complicated than that?).
>>
>> I have a project with a go.mod with 113 require statements. Here's the 
>> file.
>>
>> https://gist.github.com/jaloren/564d55145699c2a933798aec334d7ee9
>>
>> I am go 1.11.1 on Mac OS. I've already run go build and primed the local 
>> cache. When I run go mod verify, it completes in 25 seconds. I have no way 
>> to know if that's slow or fast since I don't understand what the the 
>> internals are doing. All I found on modules FAQs is this:
>>
>> "In addition, go mod verify checks that the on-disk cached copies of 
>> module downloads still match the entries in go.sum."
>>
>> But I'd like to find out is if this is going to get pathologically worse 
>> as packages are added. A minute or two is an easy sell. Past 5 minutes 
>> becomes a much harder one.
>>
>> Joe  
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: performance of go mod verify

[thepudds1460]

Hi Joe,

As far as I know, I think it is computing the SHA256 of the dependencies:

https://github.com/golang/go/blob/release-branch.go1.11/src/cmd/go/internal/dirhash/hash.go#L25

Personally, I wouldn't expect that to get pathologically worse with more 
dependencies.

Also, just as you can often sequence 'go mod download' so that you can 
often take advantage of cached dependencies in CI, there is some chance you 
could do something similar with 'go mod verify' to avoid that cost every 
time?

In terms of how it all works, there is a bit more in the documentation here:

https://golang.org/cmd/go/#hdr-Modules_and_vendoring

"By default, the go command satisfies dependencies by downloading modules 
from their sources and using those downloaded copies (after verification, 
as described in the previous section). To allow interoperation with older 
versions of Go, or to ensure that all files used for a build are stored 
together in a single file tree, 'go mod vendor' creates a directory named 
vendor in the root directory of the main module and stores there all the 
packages from dependency modules that are needed to support builds and 
tests of packages in the main module."

Regards,
thepudds

On Monday, January 28, 2019 at 1:14:53 PM UTC-5, Joseph Lorenzini wrote:
>
> All: 
>
> I'd like to run go mod verify as part of the CI/CD. If the verify fails, 
> the build fails. However, the amount of time it takes to run for a small 
> project is making me question whether that's viable. I don't actually know 
> of any documentation that explains how slow or fast one should expect a go 
> mod verify to take to run nor how much time it will increase as you add 
> packages (e.g as you add packages is the increase linear or is it more 
> complicated than that?).
>
> I have a project with a go.mod with 113 require statements. Here's the 
> file.
>
> https://gist.github.com/jaloren/564d55145699c2a933798aec334d7ee9
>
> I am go 1.11.1 on Mac OS. I've already run go build and primed the local 
> cache. When I run go mod verify, it completes in 25 seconds. I have no way 
> to know if that's slow or fast since I don't understand what the the 
> internals are doing. All I found on modules FAQs is this:
>
> "In addition, go mod verify checks that the on-disk cached copies of 
> module downloads still match the entries in go.sum."
>
> But I'd like to find out is if this is going to get pathologically worse 
> as packages are added. A minute or two is an easy sell. Past 5 minutes 
> becomes a much harder one.
>
> Joe  
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Go Modules And Sub-Packages

[thepudds1460]

Assuming your repository is “github.com/user/package1”, your module is already 
on v3.1.9, you have a single go.mod file, and the go.mod file is in the root of 
the repository, then that would mean the first line in your go.mod file 
typically would be:

module github.com/user/package1/v3

To import package2 from code in package1, it would then be the following import 
statement: 

import "github.com/user/package1/v3/package2"

In other words, your middle option, I think. 

You can see more here, for example:

https://golang.org/cmd/go/#hdr-Module_compatibility_and_semantic_versioning

...which includes:

“In semantic versioning, changing the major version number indicates a lack of 
backwards compatibility with earlier versions. To preserve import 
compatibility, the go command requires that modules with major version v2 or 
later use a module path with that major version as the final element. For 
example, version v2.0.0 of example.com/m must instead use module path 
example.com/m/v2, and packages in that module would use that path as their 
import path prefix, as in example.com/m/v2/sub/pkg. Including the major version 
number in the module path and import paths in this way is called semantic 
import versioning”

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: ...cannot find module for path...

[thepudds1460]

Hi,

You might be seeing an issue due to using an absolute path as your module 
path when you did 'go mod init /Users/me/goO/mycat'.

Usually the module path should start with a hostname, including a dot (and 
I think that is a requirement if you with to publish the module and have it 
work with 'go get).

I think I have seen a situation where someone hit some unexpected errors 
when using an absolute path for the module path, so that might explain what 
you are seeing, or might not, but I would advise trying to fix that to see 
if it helps. If it does help, I would advise filing an issue to have a 
clearer error message in the situation you hit. 

Alternatively, you could try with the latest 1.12 beta, which includes 
better error messages in many cases.

Regards,
thepudds

On Sunday, January 27, 2019 at 12:55:38 AM UTC-5, 
advor...@emeraldcloudlab.com wrote:
>
> Hi, I'm playing with modules.
>
> Something doesn't works for me.
>
>
> I have this code in main.go outside of GOPATH directory in 
> */Users/me/goO/mycat:*
>
> package main
>
>   
>
> import (
>
> "io"
>
> "os"
>
>
> "github.com/sirupsen/logrus"
>
> )
>
>
> func main() {
>
> _, err := io.Copy(os.Stdout, os.Stdin)
>
> if err != nil {
>
> logrus.Fatal(err)
>
> }
>
> }
>
>
>
>  
>
> I ran *go mod init /Users/me/goO/mycat*
>
> file* go.mod *created.
>
> No I'm running *go build*, which I believe should return list of 
> packages. But instead it returns following message:
>
> *build /Users/me/goO/src/mycat: cannot find module for path 
> github.com/sirupsen/logrus *
>
> *mac os mojave*
> *go v1.11.5*
>
> What I'm missing here?
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: pkg folder

[thepudds1460]

Hi Alex,

I think this quote from the documentation is intended to explain what you 
are seeing:

   "In module-aware mode, GOPATH no longer defines the meaning of imports 
during a build, but it still stores downloaded dependencies (in 
GOPATH/pkg/mod) and installed commands (in GOPATH/bin, unless GOBIN is 
set)."

>From https://golang.org/cmd/go/#hdr-Modules__module_versions__and_more

Regards,
thepudds

On Sunday, January 27, 2019 at 12:14:33 PM UTC-5, Alex Dvoretskiy wrote:
>
> Hi, do I have to have "$GOPATH/pkg" folder in Go v1.11.5? Looks like go 
> modules save versions of pakages in this folder. But where packages stored 
> if I use $GO111MODULE=on? $GOPATH should be ignored in this case so do pkg 
> folder. Or I'm missing something?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] modules: incrementing major version when adopting modules with v2+ packages

[thepudds1460]

A brief PSA:

If you are adopting modules with an existing v2+ package, the recommended 
best practice for most cases is to increment the major version when first 
adopting modules. 

For example, if you are on already on v2.2.0, then you would use v3.0.0 for 
your first release with modules. 

Part of the rationale is that it makes the resulting behavior easier to 
understand for both the consumer and publisher. See 
https://github.com/golang/go/issues/25967#issuecomment-422828770 for more 
discussion, if interested.

Some related community tools and references:

1. https://github.com/marwan-at-work/mod is a community tool that automates 
the process of adopting modules for a v2+ package. It also automates any 
subsequent upgrade of a module's major version. In both cases, it 
automatically adjusts go.mod files and related import statements in Go 
source code (adding or adjusting the '/vN' as needed). A "Go Modules by 
Example" walkthrough demonstrates using it: 


https://github.com/go-modules-by-example/index/blob/master/015_semantic_import_versioning/README.md

2. Mark Bates recently put out a nicely done video that describes how to 
adopt modules from the perspective of a package maintainer. He runs through 
several options and examples. One of the examples he uses is how things are 
more confusing if the above best practice above is not followed, and how 
things are easier to understand if instead that best practice is followed. 
The video is here, and I recommend it if you are a package maintainer 
considering modules:
   
https://www.gopherguides.com/videos/go-modules-package-maintainers
  
3. The "Modules" wiki hits on these points as well, including with some 
additional rationale and discussion in these two sections:

https://github.com/golang/go/wiki/Modules#releasing-modules-v2-or-higher
https://github.com/golang/go/wiki/Modules#semantic-import-versioning
   
 Finally, I am just a member of the community sharing my personal 
understanding. I am more than happy to learn more if people have a 
different understanding.
 
 Regards,
 thepudds

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: go modules go build fails - normal go build with $GOPATH set works

[thepudds1460]

Hi Robert,

Just to echo what Sam said, there is a healthy chance this could be 
explained by a version mismatch, although maybe it could be something else.

When you enable modules, by default it will look for the latest valid 
semver tag (https://semver.org) to satisfy any dependencies. For example, 
from the 'go' command documentation:

--
https://golang.org/cmd/go/#hdr-Module_aware_go_get
--
For each named package or package pattern, get must decide which version of 
the corresponding module to use. By default, get chooses the latest tagged 
release version, such as v0.4.5 or v1.2.3. If there are no tagged release 
versions, get chooses the latest tagged prerelease version, such as 
v0.0.1-pre1. If there are no tagged versions at all, get chooses the latest 
known commit.
--

If you haven't already, I would recommend running 'go list -m all' to view 
the final versions that will be used in your build for all direct and 
indirect dependencies, and ideally share the results back here.  

Could you share the exact line of code that the compiler seems to be 
complaining about?  What package is 'random.go' in?  Is there a second 
package involved on the line of code that is triggering the compile error 
(and if so, what is the name of the second package)?

One possible explanation could be that some second package has a version 
under modules with a semver tag, whereas in GOPATH mode you might be ending 
up with a different version of that second package (e.g., you might be 
getting the latest commit in master in GOPATH mode, or perhaps you have an 
older version sitting in your GOPATH, etc.), and those two versions might 
not be compatible.

Best,
--thepudds


On Wednesday, December 26, 2018 at 7:05:57 AM UTC-5, Robert Recchia wrote:
>
> and the repo that is failing is my own repo where we do not use 
> versions/releases 
>
> On Tuesday, 25 December 2018 20:22:22 UTC-5, Robert Recchia wrote:
>>
>> I have been testing out the new go modules feature.  Our application 
>> compiles/installs without problems using the normal $GOPATH way and have 
>> been using this for a while now.   trying go modules produces the same 
>> error over and over 
>>
>> Steps I followed
>>
>> unset $GOPATH just to be safe 
>>
>> 1) git clone github project 
>>
>> 2) go mod init 
>>
>> 3) go build 
>>
>>
>> and after some output where I can see it getting all the dependencies I 
>> keep getting 
>>
>> random.go:14:10: assignment mismatch: 2 variables but 1 values
>> random.go:29:10: assignment mismatch: 2 variables but 1 values
>>
>> Again reverting back to the normal $GOPATH way everything compiles fine 
>> and I can start the binary. 
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Go offline development recommendations

[thepudds1460]

Hi Sandro,

Vendoring is another approach that can work here.

In a pre-modules world, vendoring is fairly well known.

In a modules world, vendoring is still an option. For example, you can see 
this FAQ here that touches on using 'go mod vendor' with modules:

  
  
https://github.com/golang/go/wiki/Modules#how-do-i-use-vendoring-with-modules-is-vendoring-going-away

An alternative approach in a modules world is to use the module cache. It 
can end up with similar benefits as traditional vendoring (and in some ways 
ends up with a higher fidelity copy), but uses a different technique. This 
approach is explained as a "Go Modules by Example" walkthrough here:

  
  
https://github.com/go-modules-by-example/index/blob/master/012_modvendor/README.md

Best,
thepudds

On Wednesday, December 12, 2018 at 3:52:58 PM UTC-5, snmed wrote:
>
> Thank you for you reply,
>
> yes i have already read about that project, but as far as I see, there is 
> no offline loading implemented.
> But I'm sure it would be doable with some customisation. I wondering if 
> there is another approach for an offline scenario.
>
> Some other ideas or suggestions?
>
> Thanks in advance
> Sandro
>
> Am Mittwoch, 12. Dezember 2018 21:04:07 UTC+1 schrieb Burak Serdar:
>>
>> On Wed, Dec 12, 2018 at 1:00 PM snmed  wrote: 
>> > 
>> > Hi all 
>> > 
>> > Our customer demands an offline development environment with no 
>> internet connection, is there any best practices to handle package download 
>> and project setup for such an use case? And how would the new go modules 
>> fit in in such an environment? 
>>
>> Somebody just mentioned this today. It looks like it is doing what 
>> you're asking for: 
>>
>> https://github.com/gomods/athens 
>>
>>
>> > 
>> > Any advise will be most appreciated. 
>> > 
>> > Cheers Sandro 
>> > 
>> > -- 
>> > You received this message because you are subscribed to the Google 
>> Groups "golang-nuts" group. 
>> > To unsubscribe from this group and stop receiving emails from it, send 
>> an email to golang-nuts...@googlegroups.com. 
>> > For more options, visit https://groups.google.com/d/optout. 
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Go += Package Versioning

[thepudds1460]

Hi Göcs,

Could you expand on your question, including a bit more about the scenario 
you are in, what you are seeing currently, and what you would like to see 
instead?

It would probably help to include a simplified example that is 
representative of your question or issue. For example (and making something 
up):

  "I have module 'foo' located in a remote git server, and the latest 
commit is tagged v1.2.3. I just did a fresh git clone to 
/tmp/scratchpad/foo, and now I want to do ... but instead I am seeing ..."

Also, it might be good to start a new thread with a new subject, which will 
make it easier for people to find your question. (This thread has many 
people on it, and was primarily about reaction to the initial versioned 
modules blog series from back in February).

Best,
--thepudds

On Tuesday, November 27, 2018 at 9:22:17 AM UTC-5, Göcs Jëss wrote:
>
> Is it possible to restrain of having 2 files in my workspace/project while 
> keeping 2 versions of the same repo?
>
> On Wednesday, February 21, 2018 at 1:20:54 AM UTC+8, Russ Cox wrote:
>>
>> Hi everyone,
>>
>> I have a new blog post you might be interested in.
>> https://research.swtch.com/vgo.
>>
>> I'll try to watch this thread to answer any questions.
>>
>> Best,
>> Russ
>>
>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] New Modules and git clones

[thepudds1460]

Russel,

Just two add a quick doc reference: 

The general capability being discussed here is called "module queries".

This section of the doc is probably worth a quick read or re-read:

 https://golang.org/cmd/go/#hdr-Module_queries

Module queries provide a fair amount of flexibility, and allow you to do 
something like 'go get foo@sometag' or 'go get foo@somebranch' (or 
equivalently, have 'require foo sometag'  or 'require foo somebranch' in 
your go.mod).

However, your module query will be resolved and recorded in go.mod in a 
canonical form (usually a semver tag or a pseudo-version that includes a 
commit hash).  

My understanding is that at least part of the intent behind the resolution 
to a canonical form is related to the desire to maintain a total ordering 
over recorded versions (so that it is clear for example which recorded 
versions are "later" than other recorded versions).

In any event, hope the doc reference helps,

--thepudds

On Thursday, November 15, 2018 at 5:39:10 AM UTC-5, Paul Jolly wrote:
>
> > > The `dev` in that documentation is intended to be a branch name. If 
> that 
> > > module doesn't actually *have* a branch named `dev`, it won't work. 
>
> Thanks, Bryan. 
>
> > ... 
> > 
> > Hopefully there is a way of this getting updated as master/HEAD gets 
> more 
> > commits. 
>
> Not automatically, no. go get X@master will bring you up-to-date. 
>
> https://github.com/golang/go/issues/26964 is about the ability to 
> track a branch in a more systematic way. 
>
> > Now I just have to find out why: 
> > 
> >module arcamclient 
> > 
> > is causing the error: 
> > 
> > can't load package: package arcamclient: unknown import path 
> "arcamclient": cannot find module providing package arcamclient 
>
> See https://github.com/golang/go/issues/27503#issuecomment-437052752 
>
>
> Paul 
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Modules upgrade none transitive modules (aka direct modules)

[thepudds1460]

Hi Jerome,

Are you asking how to just upgrade your direct dependencies (and then your 
indirect dependencies are upgraded according to the various 'require' 
statements in the various 'go.mod' files)?

If so, could you say a bit more about why you are interested in doing that?

Also, you might be interested in this suggestion to make it easier to just 
upgrade your direct dependencies:

#28424 - cmd/go: consider easier separation of upgrades to direct vs. 
indirect dependencies, including to help with 'high-fidelity builds'
https://github.com/golang/go/issues/28424

In terms of what can be done today, it might not feel completely natural, 
but I think the following does what you are asking about (conceptually `go 
get direct@latest`, if that was supported):

# Dry run: list your direct dependencies:
go list -f '{{if not (or .Main .Indirect)}}{{.Path}}{{end}}'

# Upgrade your direct dependencies to their latest release 
go get $(go list -f '{{if not (or .Main .Indirect)}}{{.Path}}{{end}}' 
-m all)```

In general, it seems upgrading direct dependencies has a different risk 
profile than upgrading indirect dependencies, including because people 
usually have more familiarity with their direct dependencies than possibly 
deeply nested indirect dependencies.

Especially compared to doing a simple 'go get -u' or 'go get -u=patch', 
making it easier to just upgrade direct dependencies would I think allow 
people to more easily retain more benefits of the module system's 
philosophy of 'High-Fidelity Builds' (by at least having the option to more 
easily apply that philosophy during an upgrade).

In any event, I'd be curious to hear more about your use case...

Best,
thepudds

On Sunday, November 4, 2018 at 4:32:06 AM UTC-5, Jérôme LAFORGE wrote:
>
> Hello all,
> I am looking for a way to upgrade all none transitive modules (and 
> eventually upgrade transitive modules according to go.mod from upgraded 
> none transitive modules).
>
> Beause go get -u, upgrade all modules (transitive and none transitive 
> modules).
>
> Thx in adv
> Jerome
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: handling unexpected module path

[thepudds1460]

Joe,

There's a good chance your diagnosis is correct -- my understanding is the 
path used in an import statement is required to match up with the module 
path in the 'module' directive in the imported module's go.mod file.

In general, I think some projects have switched back and forth between 
wanting to officially be known as something like 'gopkg.in/foo/bar.v1' vs. 
'github/foo/bar', which then causes mismatches if clients are out of sync, 
which might be what is going on here.

One thing you could try is something like 'go mod graph | grep 
github.com/go-resty/resty', or alternatively manually review the output of 
'go mod graph' to see if you can find someone importing the github import 
path.

(Longer discussion of the whether or not to use `module gopkg.in/foo.v1` as 
the name of your module here:
 https://github.com/russross/blackfriday/issues/491#issuecomment-425585276 
) 

--thepudds

On Wednesday, October 24, 2018 at 7:01:04 PM UTC-4, Joseph Lorenzini wrote:
>
>
> Hi all,
>
> I have a deeply nested dependency somewhere that depends on go-resty. When 
> I do a go list in my module, I see this error.
>
> go: github.com/go-resty/resty@v1.10.0: parsing go.mod: unexpected module 
> path "gopkg.in/resty.v1"
> go: error loading module requirements
>
> I checked out goresty and it does have a go.mod:
>
> https://github.com/go-resty/resty/blob/v1.10.0/go.mod
>
> The module name is gopkg.in/resty.v1
>
> So what I am guessing is that some packages have imported 
> github.com/go-resty/resty while others have imported gopkg.in/resty.v1. 
> If every package imported gopkg.in/resty.v1 then i suspect this would 
> just work since that's module name. I actually am not sure where in the 
> dependency tree this is occurring. It could be I am using package A which 
> depends on package B which requires package C and Package C is the one 
> importing github.com/go-resty/resty. 
>
> I tried a replace directive and that did not work.
>
> replace (
> github.com/go-resty/resty => gopkg.in/resty.v1 v1.10.0
> )
>
> Is there a recommended way to handle this? 
>
> Thanks,
> Joe 
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: go src organization question

[thepudds1460]

Hi Sankar,

How many repos are you directly modifying?

Most projects are likely following the simplest approach of using a single 
module per repository, which typically would mean creating one go.mod file 
located in the root directory of each repository. (The official modules 
proposal predicts that is what most projects will do: 
https://go.googlesource.com/proposal/+/master/design/24301-versioned-go.md 
).

Separately, I believe you are correct that relative import paths are not 
supported with modules. See this FAQ on the modules wiki: 
https://github.com/golang/go/wiki/Modules#do-modules-work-with-relative-imports-like-import-subdir
 
and the related issue 
https://github.com/golang/go/issues/26645#issuecomment-408572701, which 
includes this comment:

   > In modules, there finally is a name for the subdirectory. If the 
parent directory says "module m" then the subdirectory is imported as 
"m/subdir", no longer "./subdir".

If you end up with one module per repo, a related question would be how 
many modules are you editing at roughly the same time? If that is part of 
your concern or question, then there are a few different options there. 
I'll include a few pointers here for things to review and consider, but 
then after looking over those, you might want to circle back here with 
additional questions or comments on your use case.

For editing multiple modules at the same time, one approach is to use the 
'replace' directive to point to local copies. You can read a bit more about 
that in this FAQ on the modules wiki:
  "FAQ: When should I use the replace directive?"
  
 
https://github.com/golang/go/wiki/Modules#when-should-i-use-the-replace-directive
   
A related but more automated approach is github.com/rogpeppe/gohack, which 
is a new community tool to automate and simplify 'replace' and multi-module 
workflows, including allowing you to easily modify one of your 
dependencies. For example, 'gohack example.com/some/dependency' 
automatically clones the appropriate repository and adds the necessary 
replace directives to your 'go.mod' so that you can edit that dependency 
and build using those edits, all locally and without requiring you to 
commit anything yet. You can read more about 'gohack' at the repo, or you 
can also see a worked example of using 'gohack' at the 'Go Modules by 
Example' site here: 
https://github.com/go-modules-by-example/index/blob/master/011_using_gohack/README.md
   
There are several more options as well for multi-module workspaces, 
including the core go tooling seems to have been built to make it possible 
to build more specific tools on top. There's an overview of several of the 
options in the following thread, including possibly using commit hooks, or 
using pre-release semver tags, etc., as well as some pointers to related 
discussion issues in the tracker:
  https://groups.google.com/d/msg/golang-nuts/0KQ4ZuSpzy8/5m4Ek7q2BgAJ

Sorry that is probably not 100% answering your question, but perhaps you 
can say a bit more about your specifics, which then likely will trigger 
some more specific comments from this list.

--thepudds

On Sunday, October 21, 2018 at 12:31:34 PM UTC-4, Sankar wrote:
>
> Hi,
>
> We were using go 1.7 until recently and now want to switch to 1.11 and 
> start using go modules.
>
> Currently our source organization is as follows:
>
> ~/go/src/gitlab.com/example/
>|
>|
>example/
> |
> |
> |api/
> |   |
> |   main.go (imports library1, 
> library2 and some 3rd party libs)
> |library1/
> |   |
> |   lib1.go
> |library2/
> |  |
> |  lib2.go
>
> In our main.go, We were importing packages like:
>
> ```
> "gitlab.com/example/example/library1"
> "gitlab.com/example/example/library2"
> "gitHUB.com/3rdpary/library"
> ```
>
> We actually wanted to import library1 and library2 via relative path 
> ("../library[12]") but since the go compiler did not support relative 
> imports then, we used the absolute imports. We did not use any dependency 
> management tool (like dep, glide etc.).
>
> The libraries that we write (library1 and library2) need not be versioned 
> and will always be used in a reliable state in our api/main.go
>
> We however need versioning support for our 3rd party libraries.
>
> For such a hybrid requirement, what is the most recommended way to import 
> and structure our libraries and the source directories, to work with go 
> modules ? I tried reading through 
> 

Re: [go-nuts] Go modules and replace

[thepudds1460]

Hi all,

See some related discussion here regarding dots in import paths and modules:

   https://github.com/golang/go/issues/27503

including this comment from bcmills:

  "Dotless paths in general are reserved for the standard library; go get 
has (to my knowledge) never worked with them, but go get is also the main 
entry point for working with versioned modules."

Best,
thepudds

On Friday, October 19, 2018 at 7:59:58 PM UTC-4, Paul Jolly wrote:
>
> Actually, now that I think about it, this restriction was relaxed. So 
> the dot in the first part of the path is not a requirement. 
>
> It appears however that go mod edit has partially regressed in this 
> respect. 
>
> Please can you raise an issue? That way we can have the behaviour 
> confirmed one way or the other. 
>
> Thanks 
> On Sat, 20 Oct 2018 at 00:44, Mark Volkmann  > wrote: 
> > 
> > I see though that "go mode edit" really wants there to be a dot in the 
> first part of the import path. 
> > Where can I read about that requirement? 
> > 
> > On Fri, Oct 19, 2018 at 6:30 PM Mark Volkmann  > wrote: 
> >> 
> >> Thank you so much! I actually got it to work without having a dot in 
> the first part of the import path. 
> >> It seems the only thing I was missing was this line in mod.go for the 
> demo code: 
> >> require foo/bar v0.0.0 
> >> I just had the replace directive line without a corresponding require 
> directive. 
> >> 
> >> On Fri, Oct 19, 2018 at 6:13 PM Paul Jolly  > wrote: 
> >>> 
> >>> Hi Mark, 
> >>> 
> >>> When importing a module package, the first element in the path must 
> >>> contain a ".". Hence "foo" is invalid. Here is a working example: 
> >>> 
> >>> $ cd $HOME 
> >>> $ mkdir bar 
> >>> $ cd bar 
> >>> $ go mod init example.com/bar 
> >>> go: creating new go.mod: module example.com/bar 
> >>> $ cat  >>> package bar 
> >>> import "fmt" 
> >>> func Hello() { 
> >>> fmt.Println("Hello from bar!") 
> >>> } 
> >>> EOD 
> >>> $ cd $HOME 
> >>> $ mkdir foo 
> >>> $ cd foo 
> >>> $ go mod init example.com/foo 
> >>> go: creating new go.mod: module example.com/foo 
> >>> $ cat  >>> package main 
> >>> 
> >>> import "example.com/bar" 
> >>> 
> >>> func main() { 
> >>> bar.Hello() 
> >>> } 
> >>> EOD 
> >>> $ go mod edit -require=example.com/bar@v0.0.0 -replace=
> example.com/bar=$HOME/bar 
> >>> $ cat go.mod 
> >>> module example.com/foo 
> >>> 
> >>> require example.com/bar v0.0.0 
> >>> 
> >>> replace example.com/bar => /root/bar 
> >>> $ go run . 
> >>> Hello from bar! 
> >>> On Fri, 19 Oct 2018 at 21:42, Mark Volkmann  > wrote: 
> >>> > 
> >>> > I have a simple demo application that wants to use a package that is 
> on my local file system. 
> >>> > The code for the package is in /Users/Mark/foo/bar. 
> >>> > This directory contains the file bar.go which contains: 
> >>> > 
> >>> > package bar 
> >>> > import "fmt" 
> >>> > func Hello() { 
> >>> > fmt.Println("Hello from bar!") 
> >>> > } 
> >>> > 
> >>> > It also contains the file go.mod which just contains: 
> >>> > 
> >>> > module bar 
> >>> > 
> >>> > The demo application in another directory imports this as "foo/bar" 
> in the file main.go. 
> >>> > It has a go.mod file that contains the following: 
> >>> > 
> >>> > module demo 
> >>> > replace foo/bar => /Users/Mark/foo/bar 
> >>> > 
> >>> > When I enter "go run main.go" in the directory of the demo code I 
> get 
> >>> > build demo: cannot find module for path foo/bar 
> >>> > 
> >>> > Is there something wrong with my use of the "replace" directive? 
> >>> > 
> >>> > None of this code is under the directory pointed to by GOPATH 
> because I'm trying to use Go modules for everything in this demo. 
> >>> > 
> >>> > -- 
> >>> > R. Mark Volkmann 
> >>> > Object Computing, Inc. 
> >>> > 
> >>> > -- 
> >>> > You received this message because you are subscribed to the Google 
> Groups "golang-nuts" group. 
> >>> > To unsubscribe from this group and stop receiving emails from it, 
> send an email to golang-nuts...@googlegroups.com . 
> >>> > For more options, visit https://groups.google.com/d/optout. 
> >> 
> >> 
> >> 
> >> -- 
> >> R. Mark Volkmann 
> >> Object Computing, Inc. 
> > 
> > 
> > 
> > -- 
> > R. Mark Volkmann 
> > Object Computing, Inc. 
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Golang Modules/vgo - local changes push, merge and tags

[thepudds1460]

Hi Guy,

Right now, my understanding is there is not a perfect answer to your 
questions.

However, these are very important questions, including I think community 
discussion right now can help establish alternative immediate solutions, as 
well as help shape how things might work in 1.12 and beyond.

If you haven't already, I would suggest reviewing this thread here:
  https://gophers.slack.com/archives/C9BMAAFFB/p1537968235000100
  
That thread includes comments on some possible current options, some 
community tooling that might help, some comments on how one-off tools or 
commit hooks might be put together on top of things like 'go mod edit 
-json', etc. (That thread also includes a summary of some of the live 
discussion issues in the tracker on these topics).

You might then want to comment in that thread (or in this thread) with a 
bit more on what might or might not work for you, ideally including some 
comments like "gohack doesn't help in our case because X", or "go mod edit 
doesn't help us because Y", or "frequent use of semver pre-release tags 
doesn't help us because Z". Or maybe you will post "Our solution is A and 
B".

Also, towards the end of your post in this thread, you are touching briefly 
on some questions of sequencing, and the ordering for publishing two 
modules if one module depends on an update in the other module.  A brief 
comment on that -- one general property of the modules system is 100% 
reproducible builds using the information in 'go.mod'. One example of this 
is if someone has published module A and module B, and A has a 'require B 
v1.2.0', then A will continue to have 100% reproducible builds even if a 
newer v1.3.0 of B is later published. An implication of this is that you 
can safely publish an update to B without breaking A's build, and then 
later publish an update to A that requires the newer B. You can read a bit 
more about that in the "High Fidelity Builds" section of the official 
proposal (  
 
https://github.com/golang/proposal/blob/master/design/24301-versioned-go.md#update-timing--high-fidelity-builds).
  
That might or might not help in your situation, but wanted to at least 
mention this more general behavior.

In any event, I would encourage you to post more on your topics below, 
including because community discussion on these multi-module workflows I 
think are an important aspect of working through the current possible 
approaches and how the tooling could or should be tweaked in the future.

Best,
--thepudds

On Thursday, October 11, 2018 at 11:31:20 AM UTC-4, Guy Brandwine wrote:
>
> We are trying to use Go-Modules
>
>
>
> We have reusable repo : "github/mycompany/funcs" many repo's use this, 
> example : "github/mycompany/eCom/basket.go"
>
>
>
> A new ticket says : "basket should support export to xls" .
>
> In order to do that, we want to : 
>
> 1. Add funcs.go a function "StructToXls(interface{})" 
>
> 2. basket.go should use funcs.StructToXls(myBasket)
>
>
>
> in the current modules format we need to, push and TAG of funcs and then 
> update github/mycompany/eCom dependencies.
>
>
>
> If we have a small bug we would need to repeatedly push and tag funcs, 
> update eCom etc.
>
>
>
> Is there a better way (other than go.mod -> replace which is dangerous and 
> "developer must remember to use it everytime he changes a basic repo") to 
> work locally and when all done push and TAG the 2 repos ?
>
> Moreover, in our current process (git flow style with some modifications) 
> the developer only pushes the code to a branch, whereas the TL, approves 
> and merges the code.
>
> If the developer tags the library (funcs) locally and does not push that 
> (theoretically its not his duty neither he is permitted to do that), that 
> may create a false dependency, moreover, two different developer may do 
> diff changes in funcs both changing ver from 1.2.3 => 1.2.4 since the 
> increment is not centralized (which BTW needs a better tooling anyhow than 
> manually inc.).
>
>
>
> A better suggested flow would be greatly appreciated.
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] `go mod download -json` does not product valid json?

[thepudds1460]

Hi kalekold,

It looks like it is a stream of JSON objects? 

If so, I suspect this is expected?

See related discussion here (for 'go list -m -u -json' in this case), along 
with sample code for parsing:

  https://github.com/golang/go/issues/27655

--thepudds


On Saturday, October 6, 2018 at 12:26:32 PM UTC-4, kalekold wrote:
>
> It should be:
>
> [{
>   "Path": "github.com/fatih/color", "Version": "v1.7.0",
>  "Info": "/media/Data/Projects/Go/pkg/mod/cache/download/
> github.com/fatih/color/@v/v1.7.0.info",
>  "GoMod": "/media/Data/Projects/Go/pkg/mod/cache/download/
> github.com/fatih/color/@v/v1.7.0.mod",
>  "Zip": "/media/Data/Projects/Go/pkg/mod/cache/download/
> github.com/fatih/color/@v/v1.7.0.zip",
>  "Dir": "/media/Data/Projects/Go/pkg/mod/github.com/fatih/color@v1.7.0",
>  "Sum": "h1:DkWD4oS2D8LGGgTQ6IvwJJXSL5Vp2ffcQg58nFV38Ys=",
>  "GoModSum": "h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5KwzbycvMj4="
> },{
> ...
> }]
>
> Notice the array, it's missing in the example above. This means this 
> output can not be parsed.
>
> On Saturday, 6 October 2018 17:24:35 UTC+1, kalekold wrote:
>>
>> I simply run `go mod download -json` and it outputs this (which isn't 
>> valid.)
>>
>> {
>>  "Path": "github.com/fatih/color",
>>  "Version": "v1.7.0",
>>  "Info": "/media/Data/Projects/Go/pkg/mod/cache/download/
>> github.com/fatih/color/@v/v1.7.0.info",
>>  "GoMod": "/media/Data/Projects/Go/pkg/mod/cache/download/
>> github.com/fatih/color/@v/v1.7.0.mod",
>>  "Zip": "/media/Data/Projects/Go/pkg/mod/cache/download/
>> github.com/fatih/color/@v/v1.7.0.zip",
>>  "Dir": "/media/Data/Projects/Go/pkg/mod/github.com/fatih/color@v1.7.0",
>>  "Sum": "h1:DkWD4oS2D8LGGgTQ6IvwJJXSL5Vp2ffcQg58nFV38Ys=",
>>  "GoModSum": "h1:Zm6kSWBoL9eyXnKyktHP6abPY2pDugNf5KwzbycvMj4="
>> }
>> {
>>  "Path": "github.com/mattn/go-colorable",
>>  "Version": "v0.0.9",
>>  "Info": "/media/Data/Projects/Go/pkg/mod/cache/download/
>> github.com/mattn/go-colorable/@v/v0.0.9.info",
>>  "GoMod": "/media/Data/Projects/Go/pkg/mod/cache/download/
>> github.com/mattn/go-colorable/@v/v0.0.9.mod",
>>  "Zip": "/media/Data/Projects/Go/pkg/mod/cache/download/
>> github.com/mattn/go-colorable/@v/v0.0.9.zip",
>>  "Dir": "/media/Data/Projects/Go/pkg/mod/
>> github.com/mattn/go-colorable@v0.0.9",
>>  "Sum": "h1:UVL0vNpWh04HeJXV0KLcaT7r06gOH2l4OW6ddYRUIY4=",
>>  "GoModSum": "h1:9vuHe8Xs5qXnSaW/c/ABM9alt+Vo+STaOChaDxuIBZU="
>> }
>> {
>>  "Path": "github.com/mattn/go-isatty",
>>  "Version": "v0.0.4",
>>  "Info": "/media/Data/Projects/Go/pkg/mod/cache/download/
>> github.com/mattn/go-isatty/@v/v0.0.4.info",
>>  "GoMod": "/media/Data/Projects/Go/pkg/mod/cache/download/
>> github.com/mattn/go-isatty/@v/v0.0.4.mod",
>>  "Zip": "/media/Data/Projects/Go/pkg/mod/cache/download/
>> github.com/mattn/go-isatty/@v/v0.0.4.zip",
>>  "Dir": "/media/Data/Projects/Go/pkg/mod/
>> github.com/mattn/go-isatty@v0.0.4",
>>  "Sum": "h1:bnP0vzxcAdeI1zdubAl5PjU6zsERjGZb7raWodagDYs=",
>>  "GoModSum": "h1:M+lRXTBqGeGNdLjl/ufCoiOlB5xdOkqRJdNxMWT7Zi4="
>> }
>> {
>>  "Path": "golang.org/x/sys",
>>  "Version": "v0.0.0-20181005133103-4497e2df6f9e",
>>  "Info": "/media/Data/Projects/Go/pkg/mod/cache/download/
>> golang.org/x/sys/@v/v0.0.0-20181005133103-4497e2df6f9e.info",
>>  "GoMod": "/media/Data/Projects/Go/pkg/mod/cache/download/
>> golang.org/x/sys/@v/v0.0.0-20181005133103-4497e2df6f9e.mod",
>>  "Zip": "/media/Data/Projects/Go/pkg/mod/cache/download/
>> golang.org/x/sys/@v/v0.0.0-20181005133103-4497e2df6f9e.zip",
>>  "Dir": "/media/Data/Projects/Go/pkg/mod/
>> golang.org/x/sys@v0.0.0-20181005133103-4497e2df6f9e",
>>  "Sum": "h1:EfdBzeKbFSvOjoIqSZcfS8wp0FBLokGBEs9lz1OtSg0=",
>>  "GoModSum": "h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY="
>> }
>>
>>
>> On Saturday, 6 October 2018 16:24:42 UTC+1, Ian Davis wrote:
>>>
>>> On Sat, 6 Oct 2018, at 3:07 PM, 'kalekold' via golang-nuts wrote:
>>>
>>> `go mod download -json` does not product valid json? Is this a known 
>>> issue or am I missing something?
>>>
>>>
>>> Do you have an example that shows the invalid json?
>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] go mod: how to develop in multiple repos?

[thepudds1460]

  > "Is there an issue for this; it continues to fill me with dread that 
temporary edits to committed files are intended to be part of a development 
workflow."

Hi kortschak,

There is some related discussion here, which also includes some pointers to 
some related issues:

   https://groups.google.com/d/msg/golang-nuts/0KQ4ZuSpzy8/5m4Ek7q2BgAJ

--thepudds

On Monday, October 1, 2018 at 6:10:54 PM UTC-4, kortschak wrote:
>
> Is there an issue for this; it continues to fill me with dread that 
> temporary edits to committed files are intended to be part of a 
> development workflow. 
>
> On Fri, 2018-09-28 at 06:24 -0700, Tamás Gulácsi wrote: 
> > Use the replace directive with relative path. 
> > 
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] go mod tidy pulls in too much

[thepudds1460]

Hi Scott,

Regarding your comment, a related issue (which is still open) is this one:

   #26955: “cmd/go: provide straightforward way to see non-test dependencies”
  https://github.com/golang/go/issues/26955

—thepudds 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] go mod tidy pulls in too much

[thepudds1460]

Hi Harmen,

And my first sentence might not have been clear. When I said "even in your 
current situation, 'go build' is still pulling in exactly what it needs", I 
was trying to reference the actual compilation process.

In other words, I was just trying to make it clear that even if you have 
"extra" dependencies appearing in your go.mod file, the resulting binary 
produced by 'go build' does not have anything extra or any unused 
dependencies.

In any event, I wanted to share at least my personal understanding, but of 
course happy to learn more...

--thepudds


On Friday, September 28, 2018 at 1:00:17 PM UTC-4, thepud...@gmail.com 
wrote:
>
>  > "So if consul adds a go.mod file in the root, then `mod tidy` will 
> suddenly 
> behave as I would expect (i.e. not pull in unused dependencies)? "
>
> Hi Harmen,
>
> Just in case this isn't already clear-- note that even in your current 
> situation, 'go build' is still pulling in exactly what it needs (and not 
> pulling in unused dependencies).
>
> I think there is not a significant harm in your go.mod containing these 
> indirect dependencies (aside from of course seeing that longer list, etc.).
>
> And there is some benefit -- this behavior is part of what provides for 
> 100% reproducible builds and tests. The modules system records precise 
> dependency version information, and in your case, that precise dependency 
> version information for some of your indirect dependencies is being 
> recorded in your go.mod (given that it is not yet recorded in the 
> non-existent go.mod of some of your direct dependencies).  
>
> As an example, this behavior helps make sure that `go test all` is 100% 
> reproducible for you (where `go test all` runs tests for both your module, 
> your direct dependencies, and your indirect dependencies, which is valuable 
> as one way of validating that the currently selected packages versions are 
> compatible -- the number of possible version combinations is exponential in 
> the number of modules, so in general you cannot expect your dependencies to 
> have tested against all possible combinations of *their* dependencies).
>
> --thepudds
>
> On Friday, September 28, 2018 at 12:48:20 PM UTC-4, Harmen wrote:
>>
>> On Fri, Sep 28, 2018 at 04:48:32PM +0100, Paul Jolly wrote: 
>> > Hi Harmen 
>> > 
>> > I described the problem on https://github.com/golang/go/issues/27920, 
>> which 
>> > > got 
>> > > closed within three minutes as being "documented", and "works as 
>> > > expected" (which I assume also means "works as intended"). 
>> > > Is this really the intented behaviour? It seems unexpected to me. Or 
>> > > should I 
>> > > simply stay away from `go mod tidy`? 
>> > > 
>> > 
>> > I replied to your issue earlier. But could arguably have been slightly 
>> more 
>> > detailed in my response beyond simply linking to 
>> > 
>> https://github.com/golang/go/wiki/Modules#why-does-go-mod-tidy-record-indirect-and-test-dependencies-in-my-gomod
>>  
>> > 
>> > As you note, github.com/hashicorp/consul/api is a package. It is a 
>> package 
>> > in the module github.com/hashicorp/consul. Despite there being no 
>> go.mod in 
>> > github.com/hashicorp/consul, the go tool simulates it as a module. 
>> > 
>> > Given there are no go.mod files in any subdirectories below 
>> > github.com/hashicorp/consul, then github.com/hashicorp/consul is the 
>> module 
>> > for all the packages github.com/hashicorp/consul/... 
>> > 
>> > Hence go mod tidy (per the link in my response to your issue) is 
>> pulling 
>> > in the transitive dependencies needed for tests in 
>> > github.com/hashicorp/consul/... The reason you see all of these in 
>> your 
>> > go.mod is that github.com/hashicorp/consul has not yet been converted 
>> to a 
>> > module as yet. 
>> > 
>> > Hopefully that gives a bit more colour on what's going on here. 
>>
>> Hi Paul, 
>>
>> thanks for your extended reply. 
>>
>> So if consul adds a go.mod file in the root, then `mod tidy` will 
>> suddenly 
>> behave as I would expect (i.e. not pull in unused dependencies)? And 
>> until they 
>> add it (if ever) I either better not run `mod tidy`, or simply go back to 
>> the 
>> old dep system? 
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] go mod tidy pulls in too much

[thepudds1460]

 > "So if consul adds a go.mod file in the root, then `mod tidy` will 
suddenly 
behave as I would expect (i.e. not pull in unused dependencies)? "

Hi Harmen,

Just in case this isn't already clear-- note that even in your current 
situation, 'go build' is still pulling in exactly what it needs (and not 
pulling in unused dependencies).

I think there is not a significant harm in your go.mod containing these 
indirect dependencies (aside from of course seeing that longer list, etc.).

And there is some benefit -- this behavior is part of what provides for 
100% reproducible builds and tests. The modules system records precise 
dependency version information, and in your case, that precise dependency 
version information for some of your indirect dependencies is being 
recorded in your go.mod (given that it is not yet recorded in the 
non-existent go.mod of some of your direct dependencies).  

As an example, this behavior helps make sure that `go test all` is 100% 
reproducible for you (where `go test all` runs tests for both your module, 
your direct dependencies, and your indirect dependencies, which is valuable 
as one way of validating that the currently selected packages versions are 
compatible -- the number of possible version combinations is exponential in 
the number of modules, so in general you cannot expect your dependencies to 
have tested against all possible combinations of *their* dependencies).

--thepudds

On Friday, September 28, 2018 at 12:48:20 PM UTC-4, Harmen wrote:
>
> On Fri, Sep 28, 2018 at 04:48:32PM +0100, Paul Jolly wrote: 
> > Hi Harmen 
> > 
> > I described the problem on https://github.com/golang/go/issues/27920, 
> which 
> > > got 
> > > closed within three minutes as being "documented", and "works as 
> > > expected" (which I assume also means "works as intended"). 
> > > Is this really the intented behaviour? It seems unexpected to me. Or 
> > > should I 
> > > simply stay away from `go mod tidy`? 
> > > 
> > 
> > I replied to your issue earlier. But could arguably have been slightly 
> more 
> > detailed in my response beyond simply linking to 
> > 
> https://github.com/golang/go/wiki/Modules#why-does-go-mod-tidy-record-indirect-and-test-dependencies-in-my-gomod
>  
> > 
> > As you note, github.com/hashicorp/consul/api is a package. It is a 
> package 
> > in the module github.com/hashicorp/consul. Despite there being no 
> go.mod in 
> > github.com/hashicorp/consul, the go tool simulates it as a module. 
> > 
> > Given there are no go.mod files in any subdirectories below 
> > github.com/hashicorp/consul, then github.com/hashicorp/consul is the 
> module 
> > for all the packages github.com/hashicorp/consul/... 
> > 
> > Hence go mod tidy (per the link in my response to your issue) is pulling 
> > in the transitive dependencies needed for tests in 
> > github.com/hashicorp/consul/... The reason you see all of these in your 
> > go.mod is that github.com/hashicorp/consul has not yet been converted 
> to a 
> > module as yet. 
> > 
> > Hopefully that gives a bit more colour on what's going on here. 
>
> Hi Paul, 
>
> thanks for your extended reply. 
>
> So if consul adds a go.mod file in the root, then `mod tidy` will suddenly 
> behave as I would expect (i.e. not pull in unused dependencies)? And until 
> they 
> add it (if ever) I either better not run `mod tidy`, or simply go back to 
> the 
> old dep system? 
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Help tracking down module dependencies

[thepudds1460]

Hi Justin,

I'm not sure where that gotest.tools is coming from in your particular 
build, and not sure of the root cause of the issue with your http proxy.

However, one thing you could try is a 'replace' directive to try to get 
gotest.tools directly from github. I don't know if that will help your 
particular situation, but you could try adding something like the following 
to the end of your go.mod file:

   replace gotest.tools => github.com/gotestyourself/gotest.tools 
v2.1.0+incompatible

There is some chance that might work around the issue with your http 
proxy.  (And if that starts to complain instead about a failed replace, you 
could also try adding an explicit 'require gotest.tools 
v2.1.0+incompatible' or maybe even 'require gotest.tools v0.0.0' in your 
go.mod file).

--thepudds

On Sunday, September 23, 2018 at 7:59:36 PM UTC-4, Justin Israel wrote:
>
> I'm converting one of my internal projects from glide to a module, after 
> having done two other conversions. But I am hitting a problem that I can't 
> yet solve.
>
> $ GO111MODULE=on go mod tidy -v
>
> Fetching https://gotest.tools?go-get=1
> https fetch failed: Get https://gotest.tools?go-get=1: Forbidden
> go: gotest.tools@v2.1.0+incompatible: unrecognized import path 
> "gotest.tools" (https fetch failed: Get https://gotest.tools?go-get=1: 
> Forbidden)
> go: error loading module requirements
>
> My http proxy won't let me access this, and I can't find usage of it 
> anywhere in my own codebase or in the immediate dependencies (or even my 
> $GOPATH). The mod subcommand isn't being very specific about where this 
> dependency is coming from (even with the graph command). Any pointers would 
> be greatly appreciated. The dependency didn't show up in my glide.lock 
> originally.
>
> My module requirements are:
>
> require (
>   github.com/boltdb/bolt v1.3.1
>   github.com/elazarl/go-bindata-assetfs v1.0.0
>   github.com/gorilla/context v1.1.1
>   github.com/gorilla/mux v1.3.0
>   github.com/pborman/uuid v1.2.0
>   golang.org/x/sys v0.0.0-20170615053224-fb4cac33e319
>   gopkg.in/yaml.v2 v2.2.0
> )
>
>
> Justin
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: Using modules with go test ./...

[thepudds1460]

Hi Scott, all,

   > "also I think they have obligated themselves to maintaining support 
for the current functionality, which some might not
find fitting to the way they work."

Regarding the concern raised in that post -- I think it is definitely not 
too late to give feedback on how modules work, even if the core Go team has 
said they will maintain backwards compatibility for modules.

My understanding of that backwards compatibility is that a module that is 
defined today will be understood by future Go releases, but that the 
statement of backwards compatibility does not imply that the exact behavior 
of modules is frozen (especially for the behavior of the go tool itself, 
what flags it supports, what those flags do, what does './...' mean, 
etc.).  Rather, as far as I understand, the exact behavior is *expected* to 
change based on 1.11 feedback.

For example, from back in February (from the initial detailed vgo blog 
series):

  > "The details here may be revised, but today's go.mod files will be 
understood by any future tooling. Please start tagging your packages with 
release tags; add go.mod files if that makes sense for your project."
  
And from the 1.11 release notes:

   > "Module support is considered experimental. Details are likely to 
change in response to feedback from Go 1.11 users, and we have more tools 
planned. Although the details of module support may change, projects that 
convert to modules using Go 1.11 will continue to work with Go 1.12 and 
later."
   
I read that as saying that the exact behavior of modules may very well 
change before modules become the default behavior, but that modules defined 
and released under vgo or Go 1.11 will still be understood.

In other words, feedback is still very valuable and will influence what the 
behavior is in 1.12 and beyond.

In my personal experience and from what I have observed, the core Go team 
has been very willing not only to discuss changes to modules, but also to 
make actual changes to how modules behave based on community feedback.  
Here is one sample list of changes from the "Modules" wiki page, where 
almost all of these changes were primarily driven by community feedback I 
think:

  
https://github.com/golang/go/wiki/Modules#changes-since-the-initial-vgo-proposal

In terms of the Go 1.11 behavior, there are still open questions, including 
around what should the exact behavior be when modules are enabled but you 
are outside of any particular module, or how to best support simultaneously 
editing multiple modules, or what if any behavior the core tooling might 
pick up from community tooling such as https://github.com/rogpeppe/gohack, 
etc.  In many cases, I believe some of the currently open questions were 
purposefully left as open questions in 1.11, including so that actual 
experience could better inform future approaches.

Filing module issues or commenting on existing issues is of course 
valuable, but the core team has also expressed great interest in people 
filing experience reports on real-world usages of modules. From the 
experience reports wiki page 
(https://github.com/golang/go/wiki/ExperienceReports):

  "The best experience reports tell: (1) what you wanted to do, (2) what 
you actually did, and (3) why that wasn’t great, illustrating those by real 
concrete examples, ideally from production use. Please write these reports 
about the problems most significant to you, post them on your own blog, or 
on Medium, or as a Github Gist (use a .md extension for Markdown), or as a 
publicly-readable Google doc, and then link them here."
  
In any event, as a member of the community, I would certainly encourage 
feedback from other members of the community...

--thepudds

On Friday, September 21, 2018 at 7:57:02 PM UTC-4, Scott Cotton wrote:
>
> Hi all,
>
> I for one would also consider the interface an improvement if as part of 
> making modules work outside of a "main module" included making go test 
> ./... work
> as before.  Perhaps ./... could mean find each top level module in src/ 
> and test.
>
> I am not sure what the modules implementors would prefer w.r.t. such 
> feedback.  On
> the one hand they need it and can't possibly envision how everyone uses 
> the tooling,  and
> have wisely called it preliminary.
>
> On the other, maybe it's not a bug as modules are defined; also I think 
> they have obligated 
> themselves to maintaining support for the current functionality, which 
> some might not
> find fitting to the way they work. For example from golang.org/cmd/go:
> """
> We intend to keep revising this support, while preserving compatibility, 
> until it can be declared official (no longer preliminary), and then at a 
> later point we may remove support for work in GOPATH and the old 'go get' 
> command.
> """
>
>
> This makes it unclear to me at least how to categorise or proceed with 
> feedback like John's,
> where there is an unexpected error.
>
> I would like to suggest that a tendency 

Re: [go-nuts] Re: Using modules with go test ./...

[thepudds1460]

   > "I imagine I might be in the minority in using test this way.  Its 
useful when you have a small monorepo to do tests.  People who run large 
monorepos must have advanced tooling and individual modules won't notice 
the behavior change."
 
Hi John,
  
Stepping back, one side question: how many modules (`go.mod` files) do you 
have in your single repo?  The easiest approach (especially to start) is to 
have a single module / single `go.mod` file at the root of a repo.  That 
simplifies working with modules overall, and the official modules proposal 
predicts that most projects will have a single module per repo.  That might 
or might not help the specific issue you raised in this thread, but it very 
likely would simplify other aspects of your workflow.

In terms of your other comment here:

   > "But this thread was about if this was expected, not if I think it 
should be changed (I can always open a bug, see if anyone else agrees).  It 
turns out I'm not doing something wrong per say, its just that it doesn't 
do what I want.  I need to adjust my expectations."

If you think the current behavior should change, I would encourage you to 
file an issue. Most important probably is to outline your use case and 
where the current behavior falls short. In addition, you could optionally 
suggest some alternatives you might view as better, but most important 
probably is to outline your use case.

Some possible alternatives for module-aware mode might include:

 1. 'go test ./...' could be redefined to always traverse down the 
filesystem directory structure, ignoring module structure completely.
 
 2. behavior #1 could be the behavior when outside of a `go.mod` file tree, 
but the current behavior for 'go test ./...' could be retained when inside 
of a `go.mod` file tree.
 
 3. 'go test ./...' could be redefined to only find packages within the 
current active module (and not dependencies of the current module that 
happen to be in sub-directories). 
 
 4. behavior #1 could be the behavior when outside of a `go.mod` file tree, 
but behavior #3 could be the behavior when inside of a `go.mod` file tree.

There are pros and cons to each of those, but there are also many more 
options that could be possible. Also, it is of course not just about 'go 
test ./...' behavior, but also 'go build ./...', 'go list ./...', etc.
 
--thepudds

On Friday, September 21, 2018 at 3:41:39 PM UTC-4, John wrote:
>
> Thanks for the reply.  Other that the bug I encountered, this seems to be 
> the behavior.
>
> This new "mod" behavior for ./... at least for "go test" seems to be a 
> step backwards.  It assumes a kind of test methodology based around a 
> certain type of file layout.
> If I am at the top of my src/ and I want to test everything underneath, I 
> can't without adding some tooling.  I have to go inside a specific module.  
> It is nice that it will now test dependencies.  I'm somewhat lost to why 
> the go tool would need to change the behavior for "go test ./...", seems 
> like adding go.mod wouldn't require this change.  
>
> I imagine I might be in the minority in using test this way.  Its useful 
> when you have a small monorepo to do tests.  People who run large monorepos 
> must have advanced tooling and individual modules won't notice the behavior 
> change.  
>
> But this thread was about if this was expected, not if I think it should 
> be changed (I can always open a bug, see if anyone else agrees).  It turns 
> out I'm not doing something wrong per say, its just that it doesn't do what 
> I want.  I need to adjust my expectations.
>
>
> On Friday, September 21, 2018 at 11:05:28 AM UTC-7, thepud...@gmail.com 
> wrote:
>>
>> > "What I would expect with 'go test ./...' is behavior similar to 
>> what I had before modules."
>>
>> Hi John, all,
>>
>> Just to expand slightly on the points from Dave, Scott, and Paul...
>>
>> I suspect part of what you are encountering is that in general:
>>
>>   1. Modules are opt-in for Go 1.11, so by design old behavior is 
>> preserved by default. This has a few implications, including you get old 
>> behavior by default inside GOPATH, but that only applies if you haven't 
>> explicitly forced non-default behavior via GO111MODULE. (In your case, you 
>> started with GO111MODULE=on set, so that is part of why you are not seeing 
>> the old behavior you are used to).
>>   
>>   2. Most of the modules-specific behavior requires that you be "inside" 
>> a module (that is, inside a file tree with a go.mod).
>>   
>> A consequence of #2 is that if you explicitly ask for modules-specific 
>> behavior (e.g., via setting GO111MODULE=on) but are *not* inside a file 
>> tree with a 'go.mod' file, then you might see an error message that 
>> effectively tells you need to be inside a module. In your case, I think 
>> that is why you saw the error "go: cannot determine module path for source 
>> directory" when you tried one of your early go commands with GO111MODULE=on 
>> but were 

Re: [go-nuts] Re: Using modules with go test ./...

[thepudds1460]

> "What I would expect with 'go test ./...' is behavior similar to what 
I had before modules."

Hi John, all,

Just to expand slightly on the points from Dave, Scott, and Paul...

I suspect part of what you are encountering is that in general:

  1. Modules are opt-in for Go 1.11, so by design old behavior is preserved 
by default. This has a few implications, including you get old behavior by 
default inside GOPATH, but that only applies if you haven't explicitly 
forced non-default behavior via GO111MODULE. (In your case, you started 
with GO111MODULE=on set, so that is part of why you are not seeing the old 
behavior you are used to).
  
  2. Most of the modules-specific behavior requires that you be "inside" a 
module (that is, inside a file tree with a go.mod).
  
A consequence of #2 is that if you explicitly ask for modules-specific 
behavior (e.g., via setting GO111MODULE=on) but are *not* inside a file 
tree with a 'go.mod' file, then you might see an error message that 
effectively tells you need to be inside a module. In your case, I think 
that is why you saw the error "go: cannot determine module path for source 
directory" when you tried one of your early go commands with GO111MODULE=on 
but were outside of a file tree with a 'go.mod' file.

In addition, I think in a module-world, something like 'go test ./...' only 
tests the active module(s), and won't traverse down into an unrelated 
module that happens to be in a sub-directory of the current module. I 
believe the doc says that explicitly for a '...' pattern appearing in 'go 
list', and I think that is true beyond just 'go list', at least as far as I 
am aware:

>From https://tip.golang.org/cmd/go/#hdr-List_packages_or_modules 'go list' 
documentation:

  "The main module is the module containing the current directory. The 
active modules are the main module and its dependencies."

and

  "The special pattern "all" specifies all the active modules, first the 
main module and then dependencies sorted by module path. A pattern 
containing "..." specifies the active modules whose module paths match the 
pattern."

I think that explains at least one of the examples you sent where 'go test 
./...' did not work as you expected when modules were enabled (because in 
module-mode, './...' won't match modules that are not dependencies of the 
current module, even if they are in sub-directories, because they would not 
be part of the current "active modules").

One related item I'll mention is that a nice part of the modules work is 
that 'go test all' has been re-defined to be more useful to include all the 
packages in the current module, plus all the packages they depend on (in 
other words, all direct and indirect dependencies of the current module).  
If you want to test the current module and all of its dependencies, 'go 
test all' would do that (rather than 'go test ./...').

In any event, I am just a member of the community and we are all still 
learning about modules, so I would be happy to learn if any of what I 
outlined above doesn't line up with what you are seeing...

Finally, I'm not sure what would be going on with the last piece you 
reported where 'go env GOMOD' returned the path to a deleted 'go.mod' file.

Best,
thepudds

On Friday, September 21, 2018 at 11:31:44 AM UTC-4, John wrote:
>
> Paul, thanks for the explanation.  But I think maybe either I'm missing 
> something or I'm not explaining something correctly.
>
> What I would expect with "go test ./..." is behavior similar to what I had 
> before modules.  That behavior would be that the go tool would recursively 
> run all tests from the directory I am in and below it, regardless if there 
> was a package in the current directory. 
>
> I built your example, which works exactly as you expect.  But it doesn't 
> solve my issue, which is around "go test ./..." recursively.  As I go back 
> up the hierarchy, if I do a go test ./..., it fails.  I could always test 
> by just going into the package and running "go test".  I want to be able to 
> recursively run tests as the current behavior allows with GOPATH.  If this 
> recursive use of "./..." is going away with mod files, then I have to 
> adjust to making smart tools.  But I've got to imagine that this isn't the 
> intention or that I am still doing something incorrectly. 
>  
> Additionally, I have also tried to add go.mod files in sub directories 
> that contain no go files between the root and the package.  This did not 
> work either.
>
> Thanks to everyone (Paul, Dave, Scott) who looked at this.
>
> *Sidenote on my original directory(bug?):*
> Interestingly enough, I have made a top level mod file and put a "hello 
> world" main.go file.  That did not make this work.
> But what looks like a bug is that I deleted both of those files, but if I 
> run "go env GOMOD" at src/, I get back a path to a go.mod file that is the 
> one I deleted.
> Running "go mod tidy" gave the same error, so no auto cleanup
> And go init 

Re: [go-nuts] modules and package cyclic dips

[thepudds1460]

Hi Scott, 

Great, it looks like you ran your own test too. 

(We had both posted at approximately the same time a few minutes ago).

--thepudds

On Sunday, September 16, 2018 at 4:05:16 PM UTC-4, thepud...@gmail.com 
wrote:
>
> Hi Scott,
>
> Regarding your question about pre-release tags and 'go get -u', my 
> understanding is that when the documentation says:
>
> 'The -u flag instructs get to update dependencies to use newer minor 
> or patch releases when available.'
>   
> ...the use of the phrase 'newer minor or patch release' is exclusive of 
> prelease tags (e.g., 'v1.2.3-alpha1').
>
> My understanding here is based on the documentation, what I've read 
> elsewhere, and my experience so far.
>
> There are some similar sentiments expressed elsewhere in the 
> documentation, such as in the "Module-aware go get" section:
>
> 'For each named package or package pattern, get must decide which 
> version of the corresponding module to use. By default, get chooses the 
> latest tagged release version, such as v0.4.5 or v1.2.3. If there are no 
> tagged release versions, get chooses the latest tagged prerelease version, 
> such as v0.0.1-pre1. If there are no tagged versions at all, get chooses 
> the latest known commit.'
>
> Or in the "Module queries" section:
>
> 'All queries prefer release versions to pre-release versions. For 
> example, " even though "v1.2.3-pre1" is nearer to the comparison target.'
>
> All that said, as I mentioned before, I think we are all still learning...
>
> --thepudds
>
> On Sunday, September 16, 2018 at 12:33:17 PM UTC-4, Scott Cotton wrote:
>>
>> Thanks again @thepudds for the helpful info.  
>>
>> with module aware go get -u at tip on golang.org, it says
>> """
>> The -u flag instructs get to update dependencies to use newer minor or 
>> patch releases when available. Continuing the previous example, 'go get -u 
>> A' will use the latest A with B v1.3.1 (not B v1.2.3).
>> """
>>
>> For example, if I have a requirements for M vQ.R.S and the latest 
>> available minor or patch release is 
>> M vQ.R.{S+1}-alpha.1, with nothing else after vQ.R.S available.
>>
>> Then does go get -u in module aware mode update to the alpha?  If so, 
>> should it?  (I would think not, 
>> but reading the docs I would think it would update to the alpha)
>>
>> Scott
>>
>> On 16 September 2018 at 16:44,  wrote:
>>
>>> Hi Scott,
>>>
>>> Modules do provide support for semver pre-release notation. You can read 
>>> more about it in these sections of the documentation, for example:
>>>
>>>   https://tip.golang.org/cmd/go/#hdr-Module_queries
>>>   https://tip.golang.org/cmd/go/#hdr-Pseudo_versions
>>>   https://tip.golang.org/cmd/go/#hdr-Module_aware_go_get
>>>
>>> --thepudds
>>>
>>> On Sunday, September 16, 2018 at 10:31:43 AM UTC-4, Scott Cotton wrote:

 Thanks Sameer,

 I have a question about semver and modules, perhaps it was already 
 answered somewhere, but if so I don't know where.

 The question is: semver.org allows for pre-release notation, the 
 document below does not mention pre-release.
 To some extent Go uses pre-releases.

 I, for one, find the idea useful and would like to use pre-release info 
 with modules.

 So can anyone comment, inform, or provide a pointer to some 
 documentation about pre-releases as they relate to modules?

 Best,
 Scott

 On 15 September 2018 at 22:11, Sameer Ajmani  wrote:

> Thanks. The relationship between semantic versioning and Go packages 
> is described in detail here: https://research.swtch.com/vgo-import
> On Sat, Sep 15, 2018 at 3:52 PM  wrote:
>
>> Hi all,
>>
>> One additional set of brief comments.
>>
>> This was a fairly nuanced and long thread, and I'm pretty sure I did 
>> not follow all of it, but I think a question related to the conversation 
>> here is -- what does 'require foo v1.1.1' really mean?
>>
>> Perhaps one way to think about that usage of the verb 'require' is 
>> that it really needs to be read in the context of semantic versioning (
>> https://semver.org).
>>
>> If something states it requires v1.1.1 of foo, in the context of 
>> semantic versioning that really means that particular requirement for 
>> v1.1.1 would be satisfied by foo >= v1.1.1 and < v2, given semver 
>> defines 
>> versions in that interval to be valid and backwards compatible 
>> substitutions for v1.1.1 (and anything v2 or higher is considered 
>> incompatible with any v1 version).
>>
>> And a variation of that first question is -- what does it mean for a 
>> build if you have both a 'require foo v1.1.1' and a 'require foo v1.2.2'?
>>
>> One initial guess might be that the build ends up with two copies of 
>> foo (v1.1.1 and v1.2.2) given there are two 'require' statements with 
>> those 
>> two different versions. That is a reasonable guess, 

Re: [go-nuts] modules and package cyclic dips

[thepudds1460]

Hi Scott,

Regarding your question about pre-release tags and 'go get -u', my 
understanding is that when the documentation says:

'The -u flag instructs get to update dependencies to use newer minor or 
patch releases when available.'
  
...the use of the phrase 'newer minor or patch release' is exclusive of 
prelease tags (e.g., 'v1.2.3-alpha1').

My understanding here is based on the documentation, what I've read 
elsewhere, and my experience so far.

There are some similar sentiments expressed elsewhere in the documentation, 
such as in the "Module-aware go get" section:

'For each named package or package pattern, get must decide which 
version of the corresponding module to use. By default, get chooses the 
latest tagged release version, such as v0.4.5 or v1.2.3. If there are no 
tagged release versions, get chooses the latest tagged prerelease version, 
such as v0.0.1-pre1. If there are no tagged versions at all, get chooses 
the latest known commit.'

Or in the "Module queries" section:

'All queries prefer release versions to pre-release versions. For 
example, "
> Thanks again @thepudds for the helpful info.  
>
> with module aware go get -u at tip on golang.org, it says
> """
> The -u flag instructs get to update dependencies to use newer minor or 
> patch releases when available. Continuing the previous example, 'go get -u 
> A' will use the latest A with B v1.3.1 (not B v1.2.3).
> """
>
> For example, if I have a requirements for M vQ.R.S and the latest 
> available minor or patch release is 
> M vQ.R.{S+1}-alpha.1, with nothing else after vQ.R.S available.
>
> Then does go get -u in module aware mode update to the alpha?  If so, 
> should it?  (I would think not, 
> but reading the docs I would think it would update to the alpha)
>
> Scott
>
> On 16 September 2018 at 16:44, > wrote:
>
>> Hi Scott,
>>
>> Modules do provide support for semver pre-release notation. You can read 
>> more about it in these sections of the documentation, for example:
>>
>>   https://tip.golang.org/cmd/go/#hdr-Module_queries
>>   https://tip.golang.org/cmd/go/#hdr-Pseudo_versions
>>   https://tip.golang.org/cmd/go/#hdr-Module_aware_go_get
>>
>> --thepudds
>>
>> On Sunday, September 16, 2018 at 10:31:43 AM UTC-4, Scott Cotton wrote:
>>>
>>> Thanks Sameer,
>>>
>>> I have a question about semver and modules, perhaps it was already 
>>> answered somewhere, but if so I don't know where.
>>>
>>> The question is: semver.org allows for pre-release notation, the 
>>> document below does not mention pre-release.
>>> To some extent Go uses pre-releases.
>>>
>>> I, for one, find the idea useful and would like to use pre-release info 
>>> with modules.
>>>
>>> So can anyone comment, inform, or provide a pointer to some 
>>> documentation about pre-releases as they relate to modules?
>>>
>>> Best,
>>> Scott
>>>
>>> On 15 September 2018 at 22:11, Sameer Ajmani  wrote:
>>>
 Thanks. The relationship between semantic versioning and Go packages is 
 described in detail here: https://research.swtch.com/vgo-import
 On Sat, Sep 15, 2018 at 3:52 PM  wrote:

> Hi all,
>
> One additional set of brief comments.
>
> This was a fairly nuanced and long thread, and I'm pretty sure I did 
> not follow all of it, but I think a question related to the conversation 
> here is -- what does 'require foo v1.1.1' really mean?
>
> Perhaps one way to think about that usage of the verb 'require' is 
> that it really needs to be read in the context of semantic versioning (
> https://semver.org).
>
> If something states it requires v1.1.1 of foo, in the context of 
> semantic versioning that really means that particular requirement for 
> v1.1.1 would be satisfied by foo >= v1.1.1 and < v2, given semver defines 
> versions in that interval to be valid and backwards compatible 
> substitutions for v1.1.1 (and anything v2 or higher is considered 
> incompatible with any v1 version).
>
> And a variation of that first question is -- what does it mean for a 
> build if you have both a 'require foo v1.1.1' and a 'require foo v1.2.2'?
>
> One initial guess might be that the build ends up with two copies of 
> foo (v1.1.1 and v1.2.2) given there are two 'require' statements with 
> those 
> two different versions. That is a reasonable guess, but not actually what 
> happens.
>
> If you instead read that verb 'require' in the context of semantic 
> versioning, it means that if the build instead picks a _single_ copy of 
> foo 
> at version v1.2.2, that actually would satisfy _both_ of the requirements 
> for v1.1.1 and v1.2.2 (given v1.2.2 is defined by semver to be a 
> backwards 
> compatible substitution for v1.1.1).  And of course, this is what the go 
> build system actually would do in this scenario.
>
> Said another way, when you use modules, the go command _mandates_ that 
> 

Re: [go-nuts] modules and package cyclic dips

[thepudds1460]

Hi Scott,

Modules do provide support for semver pre-release notation. You can read 
more about it in these sections of the documentation, for example:

  https://tip.golang.org/cmd/go/#hdr-Module_queries
  https://tip.golang.org/cmd/go/#hdr-Pseudo_versions
  https://tip.golang.org/cmd/go/#hdr-Module_aware_go_get

--thepudds

On Sunday, September 16, 2018 at 10:31:43 AM UTC-4, Scott Cotton wrote:
>
> Thanks Sameer,
>
> I have a question about semver and modules, perhaps it was already 
> answered somewhere, but if so I don't know where.
>
> The question is: semver.org allows for pre-release notation, the document 
> below does not mention pre-release.
> To some extent Go uses pre-releases.
>
> I, for one, find the idea useful and would like to use pre-release info 
> with modules.
>
> So can anyone comment, inform, or provide a pointer to some documentation 
> about pre-releases as they relate to modules?
>
> Best,
> Scott
>
> On 15 September 2018 at 22:11, Sameer Ajmani  > wrote:
>
>> Thanks. The relationship between semantic versioning and Go packages is 
>> described in detail here: https://research.swtch.com/vgo-import
>> On Sat, Sep 15, 2018 at 3:52 PM > 
>> wrote:
>>
>>> Hi all,
>>>
>>> One additional set of brief comments.
>>>
>>> This was a fairly nuanced and long thread, and I'm pretty sure I did not 
>>> follow all of it, but I think a question related to the conversation here 
>>> is -- what does 'require foo v1.1.1' really mean?
>>>
>>> Perhaps one way to think about that usage of the verb 'require' is that 
>>> it really needs to be read in the context of semantic versioning (
>>> https://semver.org).
>>>
>>> If something states it requires v1.1.1 of foo, in the context of 
>>> semantic versioning that really means that particular requirement for 
>>> v1.1.1 would be satisfied by foo >= v1.1.1 and < v2, given semver defines 
>>> versions in that interval to be valid and backwards compatible 
>>> substitutions for v1.1.1 (and anything v2 or higher is considered 
>>> incompatible with any v1 version).
>>>
>>> And a variation of that first question is -- what does it mean for a 
>>> build if you have both a 'require foo v1.1.1' and a 'require foo v1.2.2'?
>>>
>>> One initial guess might be that the build ends up with two copies of foo 
>>> (v1.1.1 and v1.2.2) given there are two 'require' statements with those two 
>>> different versions. That is a reasonable guess, but not actually what 
>>> happens.
>>>
>>> If you instead read that verb 'require' in the context of semantic 
>>> versioning, it means that if the build instead picks a _single_ copy of foo 
>>> at version v1.2.2, that actually would satisfy _both_ of the requirements 
>>> for v1.1.1 and v1.2.2 (given v1.2.2 is defined by semver to be a backwards 
>>> compatible substitution for v1.1.1).  And of course, this is what the go 
>>> build system actually would do in this scenario.
>>>
>>> Said another way, when you use modules, the go command _mandates_ that 
>>> modules use semantic versioning and expects that the versions accurately 
>>> describe compatibility: it assumes that v1.2.2 is a backwards compatible 
>>> replacement for v1.1.1, and hence v1.2.2 is a valid answer if you have both 
>>> 'require foo v1.1.1' and a 'require foo v1.2.2'.
>>>
>>> In any event, this is all still fairly new, and everyone is still 
>>> learning about the overall system. (The broader community is of course 
>>> learning about the new system, but even the people who implemented this 
>>> system are in a learning phase, including because they are learning about 
>>> how the community will use the new system). 
>>>
>>> --thepudds
>>>
>>> On Saturday, September 15, 2018 at 1:11:49 PM UTC-4, Scott Cotton wrote:

 Thanks!  Very useful.

 +1 for aliasing "go mod list" to "go list -m"





 On 15 September 2018 at 18:59,  wrote:

>>>> "Maybe having a way for "go mod" to output what is used would be 
> less confusing?"
>
> Hi all,
>
> To see a list of the selected module versions, one way is to use use 
> 'go list -m all'.  This shows the actual versions used based on all of 
> the 
> various requirements in a build.  Sample output:
>
>$ go list -m all
>example.com/my/module
>github.com/aws/aws-sdk-go v1.15.35
>github.com/go-ini/ini v1.25.4
>
> Another other useful tool is for inspecting requirements is 'go mod 
> graph' (which prints the module requirement graph), and if you are 
> curious 
> why a particular module is showing up in your go.mod, you can run 'go mod 
> why -m ' to answer that question. 
>
> You can read some more about these here:
>
>
> https://tip.golang.org/cmd/go/#hdr-The_main_module_and_the_build_list
>
> https://tip.golang.org/cmd/go/#hdr-Explain_why_packages_or_modules_are_needed
>https://tip.golang.org/cmd/go/#hdr-Print_module_requirement_graph
>
> 

Re: [go-nuts] modules and package cyclic dips

[thepudds1460]

Hi all,

One additional set of brief comments.

This was a fairly nuanced and long thread, and I'm pretty sure I did not 
follow all of it, but I think a question related to the conversation here 
is -- what does 'require foo v1.1.1' really mean?

Perhaps one way to think about that usage of the verb 'require' is that it 
really needs to be read in the context of semantic versioning 
(https://semver.org).

If something states it requires v1.1.1 of foo, in the context of semantic 
versioning that really means that particular requirement for v1.1.1 would 
be satisfied by foo >= v1.1.1 and < v2, given semver defines versions in 
that interval to be valid and backwards compatible substitutions for v1.1.1 
(and anything v2 or higher is considered incompatible with any v1 version).

And a variation of that first question is -- what does it mean for a build 
if you have both a 'require foo v1.1.1' and a 'require foo v1.2.2'?

One initial guess might be that the build ends up with two copies of foo 
(v1.1.1 and v1.2.2) given there are two 'require' statements with those two 
different versions. That is a reasonable guess, but not actually what 
happens.

If you instead read that verb 'require' in the context of semantic 
versioning, it means that if the build instead picks a _single_ copy of foo 
at version v1.2.2, that actually would satisfy _both_ of the requirements 
for v1.1.1 and v1.2.2 (given v1.2.2 is defined by semver to be a backwards 
compatible substitution for v1.1.1).  And of course, this is what the go 
build system actually would do in this scenario.

Said another way, when you use modules, the go command _mandates_ that 
modules use semantic versioning and expects that the versions accurately 
describe compatibility: it assumes that v1.2.2 is a backwards compatible 
replacement for v1.1.1, and hence v1.2.2 is a valid answer if you have both 
'require foo v1.1.1' and a 'require foo v1.2.2'.

In any event, this is all still fairly new, and everyone is still learning 
about the overall system. (The broader community is of course learning 
about the new system, but even the people who implemented this system are 
in a learning phase, including because they are learning about how the 
community will use the new system). 

--thepudds

On Saturday, September 15, 2018 at 1:11:49 PM UTC-4, Scott Cotton wrote:
>
> Thanks!  Very useful.
>
> +1 for aliasing "go mod list" to "go list -m"
>
>
>
>
>
> On 15 September 2018 at 18:59, > wrote:
>
>>> "Maybe having a way for "go mod" to output what is used would be 
>> less confusing?"
>>
>> Hi all,
>>
>> To see a list of the selected module versions, one way is to use use 'go 
>> list -m all'.  This shows the actual versions used based on all of the 
>> various requirements in a build.  Sample output:
>>
>>$ go list -m all
>>example.com/my/module
>>github.com/aws/aws-sdk-go v1.15.35
>>github.com/go-ini/ini v1.25.4
>>
>> Another other useful tool is for inspecting requirements is 'go mod 
>> graph' (which prints the module requirement graph), and if you are curious 
>> why a particular module is showing up in your go.mod, you can run 'go mod 
>> why -m ' to answer that question. 
>>
>> You can read some more about these here:
>>
>>https://tip.golang.org/cmd/go/#hdr-The_main_module_and_the_build_list
>>
>> https://tip.golang.org/cmd/go/#hdr-Explain_why_packages_or_modules_are_needed
>>https://tip.golang.org/cmd/go/#hdr-Print_module_requirement_graph
>>
>> The first link above also provides a short description of the build list, 
>> which is related to several of the questions raised in this thread:
>>
>>
>>"The set of modules providing packages to builds is called the "build 
>> list". The build list initially contains only the main module. Then the go 
>> command adds to the list the exact module versions required by modules 
>> already on the list, recursively, until there is nothing left to add to the 
>> list. If multiple versions of a particular module are added to the list, 
>> then at the end only the latest version (according to semantic version 
>> ordering) is kept for use in the build."
>>
>>
>> --thepudds
>>
>> On Saturday, September 15, 2018 at 6:34:55 AM UTC-4, Scott Cotton wrote:
>>>
>>> Thanks for the clarification.
>>>
>>> Maybe having a way for "go mod" to output what is used would be less 
>>> confusing?  I would have found so at least.
>>>
>>> Cycles crossing major versions may be more problematic. 
>>>
>>> similarly in the case of  coordinating a collection of modules, 
>>> dependency cycles may lead unintentionally to different used versions
>>> depending on the build context.  While the potential use of multiple 
>>> versions is a useful feature of modules, I am not convinced 
>>> that in this context it is desirable.  Also, if cycles can span 
>>> arbitrarily large sets of uncoordinated modules, then it seems to me that
>>> is not the intended use case for cycles and 

Re: [go-nuts] modules and package cyclic dips

[thepudds1460]

   > "Maybe having a way for "go mod" to output what is used would be less 
confusing?"

Hi all,

To see a list of the selected module versions, one way is to use use 'go 
list -m all'.  This shows the actual versions used based on all of the 
various requirements in a build.  Sample output:

   $ go list -m all
   example.com/my/module
   github.com/aws/aws-sdk-go v1.15.35
   github.com/go-ini/ini v1.25.4

Another other useful tool is for inspecting requirements is 'go mod graph' 
(which prints the module requirement graph), and if you are curious why a 
particular module is showing up in your go.mod, you can run 'go mod why -m 
' to answer that question. 

You can read some more about these here:

   https://tip.golang.org/cmd/go/#hdr-The_main_module_and_the_build_list
  
 https://tip.golang.org/cmd/go/#hdr-Explain_why_packages_or_modules_are_needed
   https://tip.golang.org/cmd/go/#hdr-Print_module_requirement_graph

The first link above also provides a short description of the build list, 
which is related to several of the questions raised in this thread:

   
   "The set of modules providing packages to builds is called the "build 
list". The build list initially contains only the main module. Then the go 
command adds to the list the exact module versions required by modules 
already on the list, recursively, until there is nothing left to add to the 
list. If multiple versions of a particular module are added to the list, 
then at the end only the latest version (according to semantic version 
ordering) is kept for use in the build."
   

--thepudds

On Saturday, September 15, 2018 at 6:34:55 AM UTC-4, Scott Cotton wrote:
>
> Thanks for the clarification.
>
> Maybe having a way for "go mod" to output what is used would be less 
> confusing?  I would have found so at least.
>
> Cycles crossing major versions may be more problematic. 
>
> similarly in the case of  coordinating a collection of modules, dependency 
> cycles may lead unintentionally to different used versions
> depending on the build context.  While the potential use of multiple 
> versions is a useful feature of modules, I am not convinced 
> that in this context it is desirable.  Also, if cycles can span 
> arbitrarily large sets of uncoordinated modules, then it seems to me that
> is not the intended use case for cycles and that might be undesirable or 
> even lead to the inability of a module maintainer in the cycle
> to effectively propagate an update.
>
> If I come across anything more concrete, I'll file an issue.  Just food 
> for thought at this point.
>
> Best
> Scott 
>
> On Saturday, 15 September 2018 02:45:50 UTC+2, Sameer Ajmani wrote:
>>
>> Go modules can use the listed version or later minor versions. So if B 
>> depends on v1.11 of A, then it can also work with v1.12 of A. Therefore a 
>> valid set of versions for the build is A v1.12 and B v1.3.
>>
>> On Mon, Sep 10, 2018 at 1:28 AM Scott Cotton  wrote:
>>
>>> Well, ok.
>>>
>>> Did you mean if A v1.12 of a module depends on v1.3 of B which depends 
>>> on v1.11 of A?
>>>
>>> So go modules don't actually necessarily depend on the listed 
>>> dependencies?  
>>>
>>> That is quite counteruintintive to me.
>>>
>>> It is much simpler in any case to use acyclic dependencies.
>>>
>>> Scott
>>>
>>> On 10 September 2018 at 04:25, Sameer Ajmani  wrote:
>>>
 I think there's a disconnection between how you and I understand this 
 system works. In a given build, there is only a single version of a 
 module; 
 there cannot be multiple copies, let alone many copies.So if v1.11 of 
 module A depends on v1.3 of module B, which in turn depends on v1.12 of 
 module A, then the build will choose A v1.12. The is only one version of A 
 in the build.

 On Sun, Sep 9, 2018 at 1:40 PM Scott Cotton  wrote:

> Hi Sameer,
>
> Thanks for asking, here are some thoughts,
>
> With time, this could create man many many copies of (different 
> versions) of a module.
> this would slow down fetch, storage, compilation, etc potentially a 
> lot, and it would only
> get worse and worse over time.
>
> if a security patch is applied to the most recent version in a 
> version-compatible space of
> a module that depends on an earlier version of itself, then the 
> security hole may still exist.
> Moreover, if the SCC of module dependencies is outside the control of 
> the authors of the 
> module being patched, it seems there is might be no way they could 
> propagate the patch 
> without editing history, which violates the very notion of versioning 
> to begin with.
>
> The notion of software moving forward by versioning is in part an 
> increase in reliability, 
> not just security patches, so I would be frightened to use cyclic 
> modules even in code
> for which I were certain there would be no security patches (like 
> fixed 

Re: [go-nuts] variables name and scope

[thepudds1460]

Some people strongly recommend always including `-shadow` when running `go 
vet`.

Using your example:

   $ go vet -shadow
   # example
   .\main.go:20: declaration of "p" shadows declaration at .\main.go:15

--thepudds

On Thursday, September 13, 2018 at 9:54:05 AM UTC-4, Jan Mercl wrote:
>
> On Thu, Sep 13, 2018 at 3:42 PM > wrote:
>
> > Why Go accepts to have différents variables with the same name ? Is 
> there any good reasons ?
>
> Because Go is a block scoped language like C and others: 
> https://en.wikipedia.org/wiki/Scope_(computer_science)#Block_scope
>
>
> -- 
>
> -j
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: How to setup your development environment using Go modules

[thepudds1460]

Hi all,

Also, for reference, here are three other related issues:

  #26640 "cmd/go: allow go.mod.local to contain replace/exclude lines"
  https://github.com/golang/go/issues/26640

  #26377 "proposal: cmd/go: module semantics in $GOPATH/src"
  https://github.com/golang/go/issues/26377
  
  #27542 "cmd/go: support simultaneous edits of interdependent modules"
  https://github.com/golang/go/issues/27542


--thepudds

On Sunday, September 9, 2018 at 4:57:38 PM UTC-4, thepud...@gmail.com wrote:
>
> Seth and/or Mirko,
>
> What would you think about one of you opening a new issue that covers the 
> possible approach you were outlining earlier in this thread:
>
>"Another idea is to have a sort of "publish local" semantics, where the 
> go tool has support for something like -devel tags, which override 
> the  defined in go.mod.  So then you would "publish" the next 
> version to your local mod cache (just creating/updating the module of the 
> special version), and the go tool would then make use of that."
>
> and
>
>"Publishing to the local cache is how Maven, a build tool for Java, 
> does it. There is the concept of snapshot versions. For Golang maybe 
> stating master (or any other branch) as version would be fitting. Then your 
> CI could use a master checkout as well."
>
> At least, I don't see an issue suggesting that approach... 
>
> It could be worth it to jot down some quick thoughts on how you would 
> envision that working.
>  
> --thepudds
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: How to setup your development environment using Go modules

[thepudds1460]

Seth and/or Mirko,

What would you think about one of you opening a new issue that covers the 
possible approach you were outlining earlier in this thread:

   "Another idea is to have a sort of "publish local" semantics, where the 
go tool has support for something like -devel tags, which override 
the  defined in go.mod.  So then you would "publish" the next 
version to your local mod cache (just creating/updating the module of the 
special version), and the go tool would then make use of that."

and

   "Publishing to the local cache is how Maven, a build tool for Java, does 
it. There is the concept of snapshot versions. For Golang maybe stating 
master (or any other branch) as version would be fitting. Then your CI 
could use a master checkout as well."

At least, I don't see an issue suggesting that approach... 

It could be worth it to jot down some quick thoughts on how you would 
envision that working.
 
--thepudds

On Sunday, September 9, 2018 at 3:19:34 PM UTC-4, thepud...@gmail.com wrote:
>
> Hi all,
>
> This thread is hitting on a fairly important set of topics.
>
> It certainly seems some additional brainstorming and community discussion 
> about how things could or should work regarding multi-module workspaces 
> would be beneficial.
>
> A file tree with a `go.mod` has been described as something like "a little 
> GOPATH", but as has been observed, if you want to edit multiple modules 
> simultaneously, you have to decide how you want to wire together those 
> multiple "little GOPATHs".
>
> Manually adding `replace` directives is one approach to using local copies 
> of related modules that you might be simultaneously editing.
>
> However, that can translate to many manual steps, especially if they are 
> frequently added and removed.
>
> If you haven't already, it is definitely worthwhile to look at 
> https://github.com/rogpeppe/gohack, which is a new community tool to 
> automate and greatly simplify multi-module workflows and related `replace` 
> directive workflows. It allows you to very easily modify one of your 
> dependencies. For example, `gohack github.com/some/dependency` 
>  automatically clones the appropriate 
> repository to your local disk and adds the necessary `replace` directives 
> to your go.mod so that your build picks up the local copy that you can 
> start editing. You could do this as part of a one-off investigatory 
> workflow, or you could do `gohack example.com/foo  example.com/bar  
> example.com/baz`  as part of your normal 
> development setup if that makes sense for your workflow. When done, `gohack 
> -u` removes all gohack replace statements (which can be done prior to 
> committing).
>
> A related note is that community-based tooling for modules in general is 
> starting to emerge. You can see an initial list here:
>   
>   
> https://github.com/golang/go/wiki/Modules#what-community-tooling-exists-for-working-with-modules
>   
> Overall, it seems clear one of the intents of the modules system is to 
> enable one-off or community-based tooling to be built on top of the core 
> modules functionality, and often it can be done in a relatively 
> straightforward manner. For example, there is whole section of the official 
> documentation titled "Edit go.mod from tools or scripts" (
> https://tip.golang.org/cmd/go/#hdr-Edit_go_mod_from_tools_or_scripts).
>
> Some example capabilities for inspecting module configuration (which might 
> be used to encode a sanity check, or to automate further tooling):
>
>   *  `go mod edit -json` outputs the current module's `go.mod` file in 
> JSON format.
>
>   *  `go mod edit -json ` outputs a particular `go.mod` 
> file in JSON format.
>
>   *  `go list -m -json all` gives a JSON representation for the full set 
> of modules available to a build, including the path to each `go.mod` file. 
> (You can then feed each `go.mod` path to `go mod edit -json 
> ` to get the additional details for that `go.mod`).
>
> Some example capabilities for programmatically editing module 
> configuration (which for example could support automating adding/removing 
> `replace` directives to wire together multiple related modules for 
> simultaneous editing of multiple modules):
>
>   *  `go mod edit -replace github.com/my/foo=../foo` 
>  adds a `replace` statement to use a 
> copy of `foo` on the local disk.
>
>   *  `go mod edit -dropreplace=github.com/my/foo` 
>  removes all replacements for `foo` from the 
> current `go.mod`.
>
> One of the concerns raised in this thread is that if you use a `replace` 
> directive to point a module to a development copy of another module, 
> someone might accidentally check that the `go.mod` containing the `replace` 
> directive. 
>
> To help guard against that (and as a simple example of one-off tooling 
> that could be built on top of 1.11 modules), you could consider some type 
> of 

Re: [go-nuts] Re: How to setup your development environment using Go modules

[thepudds1460]

Hi all,

This thread is hitting on a fairly important set of topics.

It certainly seems some additional brainstorming and community discussion 
about how things could or should work regarding multi-module workspaces 
would be beneficial.

A file tree with a `go.mod` has been described as something like "a little 
GOPATH", but as has been observed, if you want to edit multiple modules 
simultaneously, you have to decide how you want to wire together those 
multiple "little GOPATHs".

Manually adding `replace` directives is one approach to using local copies 
of related modules that you might be simultaneously editing.

However, that can translate to many manual steps, especially if they are 
frequently added and removed.

If you haven't already, it is definitely worthwhile to look at 
https://github.com/rogpeppe/gohack, which is a new community tool to 
automate and greatly simplify multi-module workflows and related `replace` 
directive workflows. It allows you to very easily modify one of your 
dependencies. For example, `gohack github.com/some/dependency` 
automatically clones the appropriate repository to your local disk and adds 
the necessary `replace` directives to your go.mod so that your build picks 
up the local copy that you can start editing. You could do this as part of 
a one-off investigatory workflow, or you could do `gohack example.com/foo  
example.com/bar  example.com/baz` as part of your normal development setup 
if that makes sense for your workflow. When done, `gohack -u` removes all 
gohack replace statements (which can be done prior to committing).

A related note is that community-based tooling for modules in general is 
starting to emerge. You can see an initial list here:
  
  
https://github.com/golang/go/wiki/Modules#what-community-tooling-exists-for-working-with-modules
  
Overall, it seems clear one of the intents of the modules system is to 
enable one-off or community-based tooling to be built on top of the core 
modules functionality, and often it can be done in a relatively 
straightforward manner. For example, there is whole section of the official 
documentation titled "Edit go.mod from tools or scripts" 
(https://tip.golang.org/cmd/go/#hdr-Edit_go_mod_from_tools_or_scripts).

Some example capabilities for inspecting module configuration (which might 
be used to encode a sanity check, or to automate further tooling):

  *  `go mod edit -json` outputs the current module's `go.mod` file in JSON 
format.

  *  `go mod edit -json ` outputs a particular `go.mod` 
file in JSON format.

  *  `go list -m -json all` gives a JSON representation for the full set of 
modules available to a build, including the path to each `go.mod` file. 
(You can then feed each `go.mod` path to `go mod edit -json 
` to get the additional details for that `go.mod`).

Some example capabilities for programmatically editing module configuration 
(which for example could support automating adding/removing `replace` 
directives to wire together multiple related modules for simultaneous 
editing of multiple modules):

  *  `go mod edit -replace github.com/my/foo=../foo` adds a `replace` 
statement to use a copy of `foo` on the local disk.

  *  `go mod edit -dropreplace=github.com/my/foo` removes all replacements 
for `foo` from the current `go.mod`.

One of the concerns raised in this thread is that if you use a `replace` 
directive to point a module to a development copy of another module, 
someone might accidentally check that the `go.mod` containing the `replace` 
directive. 

To help guard against that (and as a simple example of one-off tooling that 
could be built on top of 1.11 modules), you could consider some type of 
pre-commit VCS check or CI check to validate that a go.mod does not have 
any development replace directives, such as something along the lines of:
go mod edit -json | python2.7 -c 'import json, sys; assert 
json.load(sys.stdin)["Replace"] == None'
(Sorry for slipping in a python one-liner; that was just to convey a 
simplified example. Obviously could instead be in Go, bash, jq, or whatever 
your preferred method might be).

There are admittedly many different opinions on how to use git, but another 
approach to add and remove development-specific `replace` directives could 
be via something like the git smudge/clean workflow that lets you tweak 
things on checkout (e.g., add `replace` directives) but then automatically 
reverse the tweak when checking back in (to strip out development `replace` 
directives). Or perhaps something similar could be automated however might 
work best for your current workflows...

A completely different approach could be to make the on-disk organization 
of your multiple modules always be the same in their 
development/environment/test/build/CI environment, and wire that together 
via replace directives that you do check in (in contrast to adding and 
removing replace directives for specifically for development). That is not 
something that 

[go-nuts] Re: Fail build on missing dependencies

[thepudds1460]

Hi,

A few quick comments.

There is a there is fair amount of flexibility to adjust the default 
behavior for the go tooling in terms of when it updates `go.mod`, when it 
is allowed to reach out to the network, when it uses the vendor directory, 
etc.

In your case, you could consider something like setting the environment 
variable `GOFLAGS=-mod=readonly` in CI if you want the go tooling to fail 
if it would have otherwise implicitly wanted to update go.mod.

But that said, it really depends on what you want the behavior to be.

I tried to pull together a consolidated list of some related useful knobs 
here:

https://github.com/thepudds/go-module-knobs/blob/master/README.md

You might want to skim that list, and then think about what you want the 
behavior to be, and maybe some set of options there give you the exact 
behavior you want.

Also, you probably should run `go mod tidy` (e.g., before publishing a 
release, or perhaps before pushing a commit, or at least run it once).  One 
of the reasons `go mod tidy` exists is to ensure your current go.mod 
reflects all possible build tags/OS/architecture combinations. (In 
contrast, other commands like `go build` and `go test` only update go.mod 
based on the current build invocation's tags/OS/architecture, as far as I 
understand it). This is covered in a bit more detail here, including links 
to related discussion:

   https://github.com/golang/go/wiki/Modules#how-to-prepare-for-a-release

--PJ

On Friday, August 31, 2018 at 6:12:10 AM UTC-4, distributed wrote:
>
> Consider the following scenario. I commit a version of my project and I'm 
> happy with the dependencies and their versions that are listed in my go.mod 
> file. I push my code to the central repo and the CI starts to build the 
> project with go get or go install. Because the CI builds for a GOOS/GOARCH 
> pair that I did not try On My Machine, the list of .go files to be built 
> changes. One of the newly added files has a dependency on a module that is 
> not yet recorded in the go.mod file. go get/install promptly adds the 
> dependency and fetches it from the internet.
>
> As a result, I get a binary that is built with dependencies that I don't 
> know about, the versions of the dependencies being whatever was freshest 
> that day.
>
> How do I prevent this situation? I do not want to forbid go get/install to 
> fetch modules from the internet with GOPROXY. I'm fine that go get/install 
> fetches the dependencies with versions/hashes recorded in go.mod and 
> go.sum. I would only like for the build to fail if there are missing 
> dependencies.
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: `exec: "gcc": executable file not found in $PATH` when compiling as a module

[thepudds1460]

Hi all,

In a separate (short) forum, Russ wrote:

   "Because the pre-built packages are non-module builds and can’t be 
reused. Sorry. Disable cgo for now or install gcc."

I suspect these are related as well:

   #26993 -- "cmd/go: prebuilt net/http not used"
   https://github.com/golang/go/issues/26993


   #26988 -- cmd/go: GO111MODULE=on go build requires gcc 
   https://github.com/golang/go/issues/26988

But I would be curious to hear any additional commentary...

--thepudds

On Monday, August 27, 2018 at 5:14:18 PM UTC-4, Kevin Bowrin wrote:
>
> https://gist.github.com/kevinbowrin/5f93152fdac23529e06a3a38cde88ce8
>
>
> Compiling this little program:
>
> package main
>
>
> import (
> "fmt"
> "net/http"
> )
>
>
> func main() {
> x, _ := http.NewRequest("GET", "https://google.com;, nil)
> fmt.Println(x)
> }
>
> in the GOPATH works fine.
>
> But if it is compiled as a Go 1.11 module outside the GOPATH, the error:
>
> exec: "gcc": executable file not found in $PATH
>
> is returned. 
>
> I know I can `CGO_ENABLED=0 go build`. I'm more curious why 'mod-ing' this 
> package causes the change in behavior.  
>
> Thanks all.
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Local repository for go modules

[thepudds1460]

   "There is no VCS, only a tree in my own file system."

Hi Ignazio,

I think you could go down the Athens path if you wanted to, but I don't 
think you need to do so.

I suspect the 'replace' directive I described in my earlier post in this 
thread might be sufficient for what you describe, because it lets you map 
from an import path like "example.com/me/foo" to something on your local 
filesystem.

Here is a simple example. This is a 'hello' module that imports a trivial 
'goodbye' module, both of which reside on my local filesystem. Neither is 
checked in to a VCS. The 'hello' module also imports "rsc.io/quote" (just 
to show a normal import grabbed from the Internet).

Here is the file structure on my local system, all outside of GOPATH:

/tmp/playground/hello
|-- go.mod
`-- hello.go
/tmp/playground/goodbye
|-- go.mod
`-- goodbye.go

Here is the 'go.mod' file for the main package showing a sample use of the 
'replace' directive to translate an import path of "example.com/me/goodbye" 
to a relative filesystem path on my local computer:

==> /tmp/playground/hello/go.mod <==

module example.com/me/hello

require (
 example.com/me/goodbye v0.0.0
 rsc.io/quote v1.5.2
)

replace example.com/me/goodbye => ../goodbye

That's the most interesting bit. 

That 'go.mod' also happens shows a require for "rsc.io/quote" because I 
happened to use that as well, where that code is obtained behind the scenes 
from GitHub.

And to round out the example, here are the rest of the files:

==> /tmp/playground/hello/hello.go <==
package main

import (
 "fmt"

 "example.com/me/goodbye"
 "rsc.io/quote"
)

func main() {
 fmt.Println(quote.Hello())
 fmt.Println(goodbye.Goodbye())
}


==> /tmp/playground/goodbye/go.mod <==
module example.com/me/goodbye


==> /tmp/playground/goodbye/goodbye.go <==
package goodbye

func  Goodbye() string {
  return "Goodbye"
}

And running it from  /tmp/playground/hello/hello.go shows:

$ go run .
   Hello, world.
   Goodbye

Hope that helps, or is at least food for thought,
--thepudds


On Thursday, August 16, 2018 at 5:41:01 PM UTC-4, Ignazio Di Napoli wrote:
>
> On Thursday, August 16, 2018 at 8:20:10 PM UTC+2, thepud...@gmail.com 
> wrote:
>>
>> Could you say a few more words about your use case?
>>
>
> Thank you. Looking at design docs, I think Athens can do what I'm looking 
> for, but maybe it is a little "too much", and either docs are incomplete or 
> I'm unable to find them (e.g.: how do I configure the proxy to tell where 
> to find private modules?).
>
> Let me explain my needs better.
> I have developed a fairly big library I can't/don't want to publish. It 
> could be mapped in several modules, maybe twenty. 
> There is no VCS, only a tree in my own file system.
>
> Now, I want the programs I develop to became module based and use those 
> private modules along with other public modules.
>
> So my idea was: I put my modules as they where in non-existant example.com
> .
> Then, configure a proxy so that if host is example.com it returns the 
> local file. If it is from any other url, get the remote file (optionally 
> using a proxy). 
> Is my idea correct? Has anyone already configured something like it?
>
> Another solution could be to run a local webserver serving the files, then 
> configure hosts file adding `127.0.0.1 example.com`, and configure a 
> proxy exception. 
> But I though a more specific solution was available (like, in effect, 
> Athens).
>
> Thank you.
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Local repository for go modules

[thepudds1460]

Hi Ignazio,

Athens is one option, but there are also possible options depending on what 
you are trying to accomplish.

Could you say a few more words about your use case?

If you are doing something simple, like experimenting with creating a hello 
world module in something like /tmp/hello, then I'm not sure you would need 
to do anything special after you cd to /tmp/hello and create the go.mod 
file in /tmp/hello. A module is defined by a tree of Go source files with a 
go.mod file in the tree's root directory, so if you are inside /tmp/hello 
or a subdirectory, the go commands will find that go.mod file, and things 
should 'just work' (where all of that is on your local filesystem, with no 
direct tie to VCS or anything else).

If you are doing something more complex, some related questions might be-- 
are you creating these modules from scratch? If not, where are they coming 
from (GitHub, other VCS, manually copied, ___)? Is there a go 1.10 / GOPATH 
based workflow you already have that you are trying to map to a 
modules-based approach? Roughly how many modules?

The `replace` directive in go.mod is particularly flexible, and might be 
part of the answer. It gives additional control in the top-level `go.mod` 
for what is actually used to satisfy a dependency found in the Go source or 
go.mod files. One sample use case is if you need to fix something in a 
dependency, you can have a local fork and use a `replace 
example.com/original/import/path => your/forked/import/path` in your 
top-level `go.mod` (without needing to update the import paths in the 
actual source code). The `replace` directive allows you to supply another 
import path that might be another module located in VCS (GitHub or 
elsewhere), or you can specify the location to be on your local filesystem, 
which might be what you are after here, but not sure.

In any event, sorry if this is not helpful or not very specific...

--thepudds


On Thursday, August 16, 2018 at 1:40:04 PM UTC-4, Zellyn wrote:
>
> I think Project Athens is what you want? https://github.com/gomods/athens
>
> On Thursday, August 16, 2018 at 12:12:11 PM UTC-4, Ignazio Di Napoli wrote:
>>
>> I'd like to experiment with modules, in a local directory. I know I can 
>> do it setting GOPROXY to a file:/// url. 
>> But what about looking for my own modules in a local folder, and the 
>> others normally (optionally through the proxy)? Is there a simple proxy I 
>> can configure in this way? Thank you.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] modules: significant changes since the first vgo blog series?

[thepudds1460]

Hi all,

It's been a bit tricky to track how vgo/Go 1.11 modules have evolved since 
the first vgo blog series and since the official proposal was published.

To help with that, I tried to put together a list of some of the larger 
user-facing changes in approach that happened after the first vgo blog was 
published. I think there might be > 400 related GitHub issues at this 
point, so I'm not aiming for an exhaustive list, but rather trying to hit 
some of the highlights.

I'm sharing that list with this group in case it is helpful, but I'm also 
hoping for any quick commentary from the community on other large changes 
that might not be listed here yet.  

(Also, there's a better version of this list that I put on the "Modules" 
wiki, where that version of this list has a bunch of hyperlinks to related 
discussions/CLs/issues for each bullet. The HTML version of this list is 
here: 
https://github.com/golang/go/wiki/Modules#changes-since-the-initial-vgo-proposal
 
).

Any quick comments/corrections/additions?


Larger Changes Since Initial vgo Announcement

 1. Top-level vendor support was retained rather than vgo-based builds 
ignoring vendor directories entirely
 2. Backported minimal module-awareness to allow older Go versions 1.9.7+ 
and 1.10.3+ to more easily consume modules for v2+ projects
 3. Allowed consuming pre-existing packages with v2+ tags that don't yet 
have a go.mod (initial vgo would not use v2+ tags on a project without a 
go.mod)
 4. Added support via command go get -u=patch to update all transitive 
dependencies to the latest available patch-level versions on the same minor 
version
 5. Additional control via environmental variables (e.g., GOFLAGS in #26585)
 6. Added more flexible replace directives
 7. Added additional ways to interrogate modules (for human consumption, as 
well as for better editor / IDE integration)
 8. The UX of the go CLI has continued to be refined based on experiences 
so far (e.g., #26581)
 9. Most likely: additional support for warming caches for use cases such 
as CI or docker builds (#26610)
10. Most likely: better support for installing specific versions of 
programs to GOBIN (#24250)
 
 --thepudds

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] implementation of sync.atomic primitives

[thepudds1460]

Hi Ian,

I know you were not giving any type of definitive treatise on how go treats 
atomics across different processors...

but is a related aspect restricting instruction reordering by the compiler 
itself?

I don't know what the modern go compiler does at this point, but I think at 
least circa go 1.5 there was a nop function that seemed to be used to help 
prevent the compiler from inlining and then doing instruction re-ordering 
(first snippet below), and I think I've seen you make related comments more 
recently (e.g., FreeBSD atomics discussion snippet I included at the end of 
this post)?

I haven't followed the more recent atomics related changes (including it 
seems in 1.10 there might have been some work around intrinsics such as CL 
28076: "cmd/compile: intrinsify sync/atomic for amd64"?)...

And yes, on the one hand the answer is "respect the memory model and get a 
clean report from the race detector, etc., etc."... but of course sometimes 
the performance aspect of the current compiler does matter beyond just mere 
natural curiosity about how the go compiler does what it does (where 
performance was the context I had looked at this more closely in the past).

Two related snippets:


from go 1.5 
https://github.com/golang/go/blob/release-branch.go1.5/src/runtime/atomic_amd64x.go#L11

// The calls to nop are to keep these functions from being inlined.
// If they are inlined we have no guarantee that later rewrites of the
// code by optimizers will preserve the relative order of memory accesses.

//go:nosplit
func atomicload(ptr *uint32) uint32 {
nop()
return *ptr
}



Ian Lance Taylor response to question on FreeBSD atomics discussion on 
golang-dev: https://groups.google.com/forum/#!topic/golang-dev/f3PS8hp4Jfs


*> The second issue I have is translating FreeBSD atomic operations to 
runtime *
*> atomic ops. *
*> If I understand it correctly then atomic_load_acq_32 has weaker 
requirements *
*> compared to runtime/internal/atomic.Load. *
*> On x86 the FreeBSD variant is just a compiler barrier to prevent it *
*> re-oredering instructions. *

The Go compiler does reorder instructions.  But it doesn't reorder 
instructions across a non-inlined function call.  On x86 a simple 
memory load suffices for atomic.Load because x86 has a fairly strict 
memory order in any case.  Most other processors are more lenient, and 
require more work in the atomic operation. 



--thepudds

On Monday, March 19, 2018 at 1:55:07 AM UTC-4, Ian Lance Taylor wrote:
>
> On Sun, Mar 18, 2018 at 9:47 PM, shivaram via golang-nuts 
>  wrote: 
> > 
> > I noticed that internally, the language implementation seems to rely on 
> the 
> > atomicity of reads to single-word values: 
> > 
> > 
> https://github.com/golang/go/blob/bd859439e72a0c48c64259f7de9f175aae3b9c37/src/runtime/chan.go#L160
>  
>
> In the machine level, words like "atomicity" are overloaded with 
> different meanings.  I think what you are saying is that the runtime 
> package is assuming that a load of a machine word will never read an 
> interleaving of two different store of a machine word.  It will always 
> read the value written by a single store, though exactly which store 
> it sees is unknown.  This is true on all the processors that Go 
> supports. 
>
>
> > As I understand it, this atomicity is provided by the cache coherence 
> > algorithms of modern architectures. Accordingly, the implementations in 
> > sync.atomic of word-sized loads (e.g., LoadUint32 on 386 and LoadUint64 
> on 
> > amd64) use ordinary MOV instructions: 
> > 
> > 
> https://github.com/golang/go/blob/bd859439e72a0c48c64259f7de9f175aae3b9c37/src/sync/atomic/asm_386.s#L146
>  
> > 
> > 
> https://github.com/golang/go/blob/bd859439e72a0c48c64259f7de9f175aae3b9c37/src/sync/atomic/asm_amd64.s#L103
>  
> > 
> > However, word-sized stores on these architectures use special 
> instructions: 
> > 
> > 
> https://github.com/golang/go/blob/bd859439e72a0c48c64259f7de9f175aae3b9c37/src/sync/atomic/asm_amd64.s#L133
>  
> > 
> > Given that the APIs being implemented don't provide any global ordering 
> > guarantees, what's the reason they can't be implemented solely with MOV? 
>
> You are not giving the correct reason for why atomic.LoadUint32 and 
> LoadUint64 can use ordinary MOV instructions on x86 processors.  The 
> LoadUint32, etc., functions guarantee much more than that they read a 
> value that is not an interleaving a multiple writes.  They are also 
> load-acquire operations, meaning that when the function completes, the 
> caller will see not only the value that was loaded but also all other 
> values that some other processor core wrote before 

Re: [go-nuts] implementation of sync.atomic primitives

[thepudds1460]

Hi Shivaram,

Regarding the memory model definition and sync/atomic, there is this github 
issue that you likely would be interested in:

   issue #5045 "doc: define how sync/atomic interacts with memory model"

Including this comment:

=   
https://github.com/golang/go/issues/5045#issuecomment-252730563
=
rsc commented on Oct 10, 2016:

Yes, I spent a while on this last winter but didn't get a chance to write 
it up properly yet. The short version is that I'm fairly certain the rules 
will be that Go's atomics guarantee sequential consistency among the atomic 
variables (behave like C/C++'s seqconst atomics), and that you shouldn't 
mix atomic and non-atomic accesses for a given memory word.
=

--thepudds

On Monday, March 19, 2018 at 8:46:56 AM UTC-4, Shivaram Lingamneni wrote:
>
> Thanks, this was very helpful. If I understand correctly:
>
> 1. These ordering guarantees are part of the Go memory model? I couldn't 
> find explicit descriptions of them in these pages:
>
> https://golang.org/ref/mem
> https://golang.org/pkg/sync/atomic/
>
> 2. The property that word-sized values are not subject to 
> interleaving/tearing is an implementation detail, rather than a guarantee 
> of the Go memory model?
>
> On Monday, March 19, 2018 at 1:55:07 AM UTC-4, Ian Lance Taylor wrote:
>>
>> On Sun, Mar 18, 2018 at 9:47 PM, shivaram via golang-nuts 
>>  wrote: 
>> > 
>> > I noticed that internally, the language implementation seems to rely on 
>> the 
>> > atomicity of reads to single-word values: 
>> > 
>> > 
>> https://github.com/golang/go/blob/bd859439e72a0c48c64259f7de9f175aae3b9c37/src/runtime/chan.go#L160
>>  
>>
>> In the machine level, words like "atomicity" are overloaded with 
>> different meanings.  I think what you are saying is that the runtime 
>> package is assuming that a load of a machine word will never read an 
>> interleaving of two different store of a machine word.  It will always 
>> read the value written by a single store, though exactly which store 
>> it sees is unknown.  This is true on all the processors that Go 
>> supports. 
>>
>>
>> > As I understand it, this atomicity is provided by the cache coherence 
>> > algorithms of modern architectures. Accordingly, the implementations in 
>> > sync.atomic of word-sized loads (e.g., LoadUint32 on 386 and LoadUint64 
>> on 
>> > amd64) use ordinary MOV instructions: 
>> > 
>> > 
>> https://github.com/golang/go/blob/bd859439e72a0c48c64259f7de9f175aae3b9c37/src/sync/atomic/asm_386.s#L146
>>  
>> > 
>> > 
>> https://github.com/golang/go/blob/bd859439e72a0c48c64259f7de9f175aae3b9c37/src/sync/atomic/asm_amd64.s#L103
>>  
>> > 
>> > However, word-sized stores on these architectures use special 
>> instructions: 
>> > 
>> > 
>> https://github.com/golang/go/blob/bd859439e72a0c48c64259f7de9f175aae3b9c37/src/sync/atomic/asm_amd64.s#L133
>>  
>> > 
>> > Given that the APIs being implemented don't provide any global ordering 
>> > guarantees, what's the reason they can't be implemented solely with 
>> MOV? 
>>
>> You are not giving the correct reason for why atomic.LoadUint32 and 
>> LoadUint64 can use ordinary MOV instructions on x86 processors.  The 
>> LoadUint32, etc., functions guarantee much more than that they read a 
>> value that is not an interleaving a multiple writes.  They are also 
>> load-acquire operations, meaning that when the function completes, the 
>> caller will see not only the value that was loaded but also all other 
>> values that some other processor core wrote before writing to the 
>> address being loaded (assuming the write was done using StoreUint32, 
>> etc.).  It happens that on x86 you can implement load-acquire using a 
>> simple MOV instruction.  Most other multicore processors use a more 
>> complex memory model, and their sync/atomic implementations are 
>> accordingly more complex. 
>>
>> Ian 
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: How to reuse go fix tool code

[thepudds1460]

Hi Anuj,

Two quick comments.

1. FYI, in case you didn't see this in the recent vgo dependency 
management  discussion, go fix might start to be used much more broadly in 
the not-too-distant future across the go community as a way to help 
automate in some cases the ability to update client code as a way to help 
manage API changes, somewhat similar to how go fix was used by the core go 
team before go 1.0. See the snippet pasted below from one of rsc's recent 
blog posts, or visit that link for a longer discussion. Not yet 
implemented, but could be very nice if it happens.

2. I'm mildly curious what type of changes you are trying to do, including 
what aspect of your intended changes makes it such that the simpler 'gofmt 
-r' automated refactoring or the more flexible 'eg' example-based 
refactoring tool (from the go team) aren't applicable.  The 'eg' help is 
mildly annoying to track down via godoc, so forgive the volume of text but 
I pasted in the main help text below from 'eg' at the bottom of this post. 
'eg' is focused on expression-level changes, so perhaps that is one reason 
it might not be applicable for you, but as I said I'm a bit curious about 
the specifics of your case...


from https://research.swtch.com/vgo-import

Before Go 1, we relied heavily on go fix, which users ran after updating to 
a new Go release and finding their programs no longer compiled.

...

The ability to name and work with multiple incompatible versions of a 
package in a single program suggests a possible solution: if a v1 API 
function can be implemented as a wrapper around the v2 API, the wrapper 
implementation can double as the fix specification. For example, suppose v1 
of an API has functions EnableFoo and DisableFoo and v2 replaces the pair 
with a single SetFoo(enabled bool). After v2 is released, v1 can be 
implemented as a wrapper around v2:

package p // v1

import v2 "p/v2"

func EnableFoo() {
//go:fix
v2.SetFoo(true)
}

func DisableFoo() {
//go:fix
v2.SetFoo(false)
}

The special //go:fix comments would indicate to go fix that the wrapper 
body that follows should be inlined into the call site. Then running go fix 
would rewrite calls to v1 EnableFoo to v2 SetFoo(true). The rewrite is 
easily specified and type-checked, since it is plain Go code. 




from golang/tools eg help via 
https://github.com/golang/tools/blob/release-branch.go1.10/refactor/eg/eg.go#L20

This tool implements example-based refactoring of expressions.

The transformation is specified as a Go file defining two functions,
'before' and 'after', of identical types.  Each function body consists
of a single statement: either a return statement with a single
(possibly multi-valued) expression, or an expression statement.  The
'before' expression specifies a pattern and the 'after' expression its
replacement.

package P
  import ( "errors"; "fmt" )
  func before(s string) error { return fmt.Errorf("%s", s) }
  func after(s string)  error { return errors.New(s) }

The expression statement form is useful when the expression has no
result, for example:

  func before(msg string) { log.Fatalf("%s", msg) }
  func after(msg string)  { log.Fatal(msg) }

The parameters of both functions are wildcards that may match any
expression assignable to that type.  If the pattern contains multiple
occurrences of the same parameter, each must match the same expression
in the input for the pattern to match.  If the replacement contains
multiple occurrences of the same parameter, the expression will be
duplicated, possibly changing the side-effects.

The tool analyses all Go code in the packages specified by the
arguments, replacing all occurrences of the pattern with the
substitution.

So, the transform above would change this input:
err := fmt.Errorf("%s", "error: " + msg)
to this output:
err := errors.New("error: " + msg)

Identifiers, including qualified identifiers (p.X) are considered to
match only if they denote the same object.  This allows correct
matching even in the presence of dot imports, named imports and
locally shadowed package names in the input program.

Matching of type syntax is semantic, not syntactic: type syntax in the
pattern matches type syntax in the input if the types are identical.
Thus, func(x int) matches func(y int).

This tool was inspired by other example-based refactoring tools,
'gofmt -r' for Go and Refaster for Java.


LIMITATIONS
===

EXPRESSIVENESS

Only refactorings that replace one expression with another, regardless
of the expression's context, may be expressed.  Refactoring arbitrary
statements (or sequences of statements) is a less well-defined problem
and is less amenable to this approach.

A pattern that contains a function 

[go-nuts] Re: Long running task in select case

[thepudds1460]

Hi all,

In this particular case, this is a toy example of course, but for this toy 
example, it is absolutely a case where the performance of the 
synchronization primitive literally does not matter at all. (If I followed 
here, the intent is seemingly to watch a long running task, and the example 
has a 500ms sleep, etc.).

That said, sometimes performance does matter.

If instead this was a DIFFERENT example that was instead in some tight 
performance critical inner loop, the performance of the synchronization 
primitive for a stop flag can start to be meaningful (which of course 
should first be demonstrated by your benchmarking and/or your profiling of 
your particular program -- "premature optimization is the root of all evil" 
and all that).

Empirically speaking, that is then a situation where I've seen people start 
to get into discussion around "well on amd64 you can rely on X and Y so we 
can get away with a stop flag that doesn't use a mutex or an atomic", and 
then people start talking about benign data races, and then other people 
start talking about "there are no benign data races", and then there's the 
reference to Dmitry Vyukov's article on benign data races, etc.:

  
https://software.intel.com/en-us/blogs/2013/01/06/benign-data-races-what-could-possibly-go-wrong

To be honest I've seen it take up a fair amount of energy just to discuss, 
and again empirically speaking I've seen very smart people make mistakes 
here.

One question I have regarding using an atomic for a stop flag is whether or 
not there is material performance overhead compared to a simple 
unprotected/non-atomic stop flag.

I looked at that question a bit circa go ~1.7, so possibly stale info here, 
and I haven't gone back to look at my more detailed notes, but if I recall 
correctly I think my conclusion at the time was:

1. If your use case is a stop flag that will be checked many times (say, in 
a tight inner loop), and the stop flag only gets set rarely, then the 
performance of the set doesn't matter much, the assembly emitted for an 
atomic load (such as atomic.LoadUint64) seems to be identical for the 
assembly emitted for a simple load (say, *myUnint64Ptr) based on some spot 
checking a while ago (with a sample of assembly from 1.9 pasted in at the 
bottom of this post for comparison purposes).

2. Basic micro benchmarks didn't show a difference between an atomic load 
and an unprotected load for a stop flag.

3. There might be some theoretical possible overhead due to the go compiler 
will do instruction reordering in general, but won't do instruction 
reordering across function calls, so that could be one difference that 
might or might not make a difference depending on your exact code (though 
likely modest impact)?  Regarding this last point, I'm just an interested 
spectator when it comes to the go compiler, so just to be clear I don't 
really know this definitively.

At least for us at the time, the quick test program & comparison of 
assembly was enough to move us past the whole "Well, on amd64 you can get 
away with X" discussion, so I didn't delve too much deeper than that at the 
time.

In any event, sharing this sample assembly with the community in case 
anyone is interested and/or has additional commentary on the particulars 
here in terms of performance impact in go of using an atomic load vs an 
unprotected stop flag for the (admittedly somewhat rare) cases when the 
nanoseconds do indeed matter. (And for me, a clean report from the race 
detector trumps the performance arguments, but that doesn't mean I'm not 
curious about the performance...).

Here is a simple test program:

https://play.golang.org/p/PaCQwb5m9ag

func simpleLoadUint64(inputPtr *uint64) uint64 {
// normal load of *inputPtr
return *inputPtr
}

func atomicLoadUint64(inputPtr *uint64) uint64 {
// atomic.LoadUint64 atomically loads *inputPtr
return atomic.LoadUint64(inputPtr)
}

And here is the corresponding assembly snippets (from go 1.9):

go build -gcflags -S atomic_vs_normal_load.go 

// trivial function with an unprotected load

"".simpleLoadUint64 STEXT nosplit size=14 args=0x10 locals=0x0
0x 0 (atomic_vs_normal_load.go:8)  TEXT
"".simpleLoadUint64(SB), NOSPLIT, $0-16
0x 0 (atomic_vs_normal_load.go:8)  FUNCDATA$0, 
gclocals·aef1f7ba6e2630c93a51843d99f5a28a(SB)
0x 0 (atomic_vs_normal_load.go:8)  FUNCDATA$1, 
gclocals·33cdeebe80329f1fdbee7f5874cb(SB)
0x 0 (atomic_vs_normal_load.go:8)  MOVQ
"".inputPtr+8(SP), AX
0x0005 5 (atomic_vs_normal_load.go:10) MOVQ(AX), AX
0x0008 8 (atomic_vs_normal_load.go:10) MOVQAX, "".~r1+16(SP)
0x000d 00013 (atomic_vs_normal_load.go:10) RET
0x 48 8b 44 24 08 48 8b 00 48 89 44 24 10 c3
H.D$.H..H.D$..

// trivial function with a sync/atomic LoadUint64:

"".atomicLoadUint64 STEXT nosplit size=14 args=0x10 locals=0x0
0x 0 

[go-nuts] Re: Long running task in select case

[thepudds1460]

>
> *> "Here's another way: https://play.golang.org/p/gEDef3LolAZ 
>  "*


Hi all,

I think the second example alternative given (playground link above) has a 
data race?

Sample race detector run just now. (The two reports are inverses of each 
other: read then write vs. write then read).

---
go run -race stop_flag_from_gonuts.go

. . . . . quit sending ...
after quit sent==
.
WARNING: DATA RACE
Write at 0x00c042072000 by goroutine 8:
  main.f.func1()
  C:/cygwin64/bin/gonuts/stop_flag_from_gonuts.go:13 +0x59

Previous read at 0x00c042072000 by goroutine 6:
  main.f()
  C:/cygwin64/bin/gonuts/stop_flag_from_gonuts.go:17 +0x102

Goroutine 8 (running) created at:
  main.f()
  C:/cygwin64/bin/gonuts/stop_flag_from_gonuts.go:11 +0x8a

Goroutine 6 (running) created at:
  main.main()
  C:/cygwin64/bin/gonuts/stop_flag_from_gonuts.go:28 +0x70
==
==
WARNING: DATA RACE
Read at 0x00c042072000 by goroutine 6:
  main.f()
  C:/cygwin64/bin/gonuts/stop_flag_from_gonuts.go:17 +0x102

Previous write at 0x00c042072000 by goroutine 8:
  main.f.func1()
  C:/cygwin64/bin/gonuts/stop_flag_from_gonuts.go:13 +0x59

Goroutine 6 (running) created at:
  main.main()
  C:/cygwin64/bin/gonuts/stop_flag_from_gonuts.go:28 +0x70

Goroutine 8 (finished) created at:
  main.f()
  C:/cygwin64/bin/gonuts/stop_flag_from_gonuts.go:11 +0x8a
==
quit called
Found 2 data race(s)
exit status 66
---

--thepudds

On Friday, March 16, 2018 at 11:04:38 AM UTC-4, matthe...@gmail.com wrote:
>
> While this is running, your select won't be receiving on the quit 
>> channel, even if it is non-nil. 
>> If you want to be able to cancel it, you'll need to make the code in 
>> the loop responsive to the quit channel 
>> (for example, by using a select like you're using in f already). 
>
>
> The default select case does it: https://play.golang.org/p/jlfaXu6TZ8L
>
> Here's another way: https://play.golang.org/p/gEDef3LolAZ
>
> Matt
>
> On Friday, March 16, 2018 at 9:45:00 AM UTC-5, Sathish VJ wrote:
>>
>> All the examples I've seen use some kind of ticker to run various cases 
>> of a select statement.  But how does one run a long running task that is 
>> still cancelable?  
>>
>>
>> In the example below the quit part is never reached.  
>>
>> https://play.golang.org/p/PLGwrUvKaqn  (it does not run properly on 
>> play.golang.org).
>>
>> package main
>>
>>
>> import (
>>  "fmt"
>>  "os"
>>  "time"
>> )
>>
>>
>> func f(quit chan bool) {
>>  for {
>>select {
>>case <-time.After(0 * time.Second):
>>  // start long running task immediately.
>>  for {
>>time.Sleep(500 * time.Millisecond)
>>fmt.Printf(". ")
>>  }
>>case <-quit:
>>  fmt.Println("quit called")
>>  //deallocate resources in other long running task and then return 
>> from function.
>>  os.Exit(0) // or return
>>}
>>  }
>> }
>>
>>
>> func main() {
>>  var quit chan bool
>>  go f(quit)
>>
>>
>>  println("quit sending ... ")
>>  quit <- true
>>  println("after quit sent")
>>
>>
>>  var i chan int
>>  <-i
>> }
>>
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Channels vs Callbacks in API

[thepudds1460]

On a semi-related note, I think there is a fair amount of wisdom in the 
"Synchronous Functions" section of the Code Review Comments golang wiki 
page (including it is not written as an absolute and starts with the word 
"Prefer").

https://github.com/golang/go/wiki/CodeReviewComments#synchronous-functions

 

--
Synchronous Functions
--
Prefer synchronous functions - functions which return their results 
directly or finish any callbacks or channel ops before returning - over 
asynchronous ones.

Synchronous functions keep goroutines localized within a call, making it 
easier to reason about their lifetimes and avoid leaks and data races. 
They're also easier to test: the caller can pass an input and check the 
output without the need for polling or synchronization.

If callers need more concurrency, they can add it easily by calling the 
function from a separate goroutine. But it is quite difficult - sometimes 
impossible - to remove unnecessary concurrency at the caller side.
--


--thepudds

On Friday, December 29, 2017 at 6:41:43 PM UTC-5, dc0d wrote:
>
> 1 - I should emphasize that the API in question is concurrent by nature 
> (inbounds at any time which might need replies).
>
> 2 - All points made are good valid points in general. Thanks!
>
> 3 - With 1 in mind, still IMHO, channels should be preferred. Draining on 
> either side, needs goroutines. But consumer must have better control - not 
> just one goroutine per inbound. But yes, it's a concern.
>
> (I'm on mobile now, but I'll give it more thought).
>
> On Sat, Dec 30, 2017, 02:00 Axel Wagner  > wrote:
>
>> Some counterpoints:
>>
>> * In the question of "Channels vs. Callbacks", the best choice is 
>> "neither". Write a blocking API with a fixed number of items and a 
>> Scanner-like Iterator for an unknown number of items.
>> * It requires spawning a Goroutine even if none is necessary
>> * It adds syntactic overhead to the caller even if none is necessary
>> * If the caller in any way wants to customize the channel (e.g. add 
>> buffering) they have to spawn yet another Goroutine
>> * It requires specifying how the API behaves if the channel isn't 
>> read/quickly enough/whether it needs to be drained or not/when it will be 
>> drained - while an io.Closer already is the common idiom for resources to 
>> be closed
>>
>>>
>>>- Internals of the API might be affected by unknown behavior of 
>>>callbacks 
>>>
>>> Like what? A panic should just crash the program, no affected behavior. 
>> The package providing the API shouldn't have to worry about it. Anything 
>> else is isomorphic to unknown behavior of the channel consumer.
>> That being said, again, the correct choice is neither, which also doesn't 
>> suffer any of these problems. Encapsulate the concurrency.
>>
>>>
>>>- lets the consumer decides how many goroutines is needed to handle 
>>>it
>>>
>>> Just like a Scanner-like iterator.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.