[go-nuts] Re: super large virtual memory allocation for simplest golang apps, why?

[pat2...@gmail.com]

On Thursday, January 14, 2021 at 12:27:25 PM UTC-5 laos...@gmail.com wrote:

> I run the helloworld net/http on 128MB MIPS 32bit cpu, helloworld takes 
> 700MB VSS each, I then run 40 of them(each takes 4M RSS) in parallel, and I 
> got 'can't fork: out of memory' if I set overcommit_memory to 2, change it 
> to 0 made this disappear. However for embedded systems I normally set 
> overcommit as 2 and no swap to avoid OOM in the field.
>

The big footprint is from common libraries and runtime system. I believe 
this is a clear result of the design decision trying to avoid
"DLL hell" that we all lived with in the early Windows era.

If we all ran a good operating system, such as Tenex, most of the libraries 
would be shared by virtual page system. Automatically,
so that even in a small real world memory system, it would be fine. The 
early Tenex systems typically had about 480 K of memory. Back in 1969, that 
was very expensive. Now that I think about it, the memory size and CPU 
speed of those early Tenex systems is far smaller than most embedded 
microcontrollers. See Tenex, by 
BBN. https://en.wikipedia.org/wiki/TENEX_(operating_system)
 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/228faf77-a5d0-4672-a9b0-d843fa38baadn%40googlegroups.com.

[go-nuts] [ANN] go-featureprocessing v1.0.0

[Nikolay Dubina]

*What is this?*

Fast feature preprocessing in Go with feature parity to sklearn

https://github.com/nikolaydubina/go-featureprocessing

*What is new?*

   - Added batch processing
   - Added inplace processing 
   - Added parallel processing
   - Made all inference code to have zero memory allocations
   - Added sklearn Python comparison benchmarks
   - Did a lot of benchmarking
   - Cleaned up API
   - Cleaned up generated code a bit
   - Made more realistic generated tests
   - Made more realistic benchmarks
   - Improved documentation

Thanks! 

- Nikolay

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/899a8eec-2c54-4940-a476-03040fb54a12n%40googlegroups.com.

[go-nuts] Re: Context on one-shot structs

[Ross Light]

The Go CDK stores a Context in a struct while performing I/O: 
https://pkg.go.dev/gocloud.dev/blob#Bucket.NewWriter

It could be argued that this is done for compatibility with the io.Reader 
and io.Writer interfaces. However, I think this pattern, used sparingly, is 
suitable for API interactions where multiple method calls are required for 
a single conceptual task.

-Ross

On Tuesday, January 12, 2021 at 9:29:39 AM UTC-8 Jean de Klerk wrote:

> Er, lifetime of the struct *
>
> On Tue, Jan 12, 2021 at 8:11 AM Jean de Klerk  wrote:
>
>> Hi all,
>>
>> I'm looking into how contexts are used, and wondering: does anybody know 
>> of an idiomatic use of context on a struct for the sake of a one-shot?
>>
>> (a one-shot being some struct that gets used once, like an RPC or http 
>> request)
>>
>> For example, http.Request has a context on its struct. But, I think the 
>> context exists there for backwards compatibility reasons.
>>
>> I'm looking for a struct that has context on it specifically because the 
>> context is scoped to the lifetime of the request.
>>
>> Thanks!
>> Jean
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/0b8690f0-0d73-460c-b944-4b2fe2e6c49dn%40googlegroups.com.

Re: [go-nuts] `go list` across multiple modules?

['Tim Hockin' via golang-nuts]

On Fri, Jan 15, 2021 at 2:17 PM Jay Conrod  wrote:
>
> I was initially going to suggest adding the module subdirectories as 
> requirements to the main go.mod, then replacing those with the subdirectories.
>
> module example.com/m
>
> go 1.15
>
> require (
>   example.com/other1 v0.0.0
>   example.com/other2 v0.0.0
>   example.com/m/submod v0.0.0
> )
>
> replace (
>   example.com/other1 => ./staging/src/example.com/other1
>   example.com/other2 => ./staging/src/example.com/other2
>   example.com/m/submod v0.0.0 => ./submod
> )
>
>
> I think you might have tried this already. It gives the same "main module ... 
> does not contain package" error. I believe that's a bug. I've opened #43733 
> to track it.

Interesting.  If that's a bug, then maybe I'll be able to do what I
need once fixed.

> In general, it should be possible to give 'go list' an absolute or relative 
> path (starting with ./ or ../) to any directory containing a package which is 
> part of any module in the build list. For example, some tools list 
> directories in the module cache to find out what package a .go file belongs 
> to.
>
> As a workaround, you could put a go.mod in an otherwise empty directory (in 
> /tmp or something), then require the relevant modules from the repo and 
> replace them with absolute paths. Then you can run 'go list' in that 
> directory with absolute paths of package directories.

Interesting - is the difference the absolute paths vs relative?

I hoped maybe `-modfile` would do the same trick, but alas not:

```
$ (cd /tmp/gomodhack/; go list /tmp/go-list-modules/submod/used/)
example.com/m/submod/used

$ go list --modfile /tmp/gomodhack/go.mod /tmp/go-list-modules/submod/used/
main module (tmp) does not contain package tmp/submod/used
```

It also fails some cases:

```
 (cd /tmp/gomodhack/; go list /tmp/go-list-modules/submod/used/)
example.com/m/submod/used
thockin@thockin-glaptop4 go-list-modules main /$ (cd /tmp/gomodhack/;
go list /tmp/go-list-modules/staging/src/example.com/other1/used/)
go: finding module for package example.com/m/staging/src/example.com/other1/used
cannot find module providing package
example.com/m/staging/src/example.com/other1/used: unrecognized import
path "example.com/m/staging/src/example.com/other1/used": reading
https://example.com/m/staging/src/example.com/other1/used?go-get=1:
404 Not Found
```

It seems that is because the "main" (top-level dir) go.mod has
`replace` directives with relative paths, which kubernetes really
does.

> Incidentally, golang.org/x/tools/go/packages will call 'go list' under the 
> hood in module mode. go/build might do the same, depending on how it's 
> invoked. 'go list' may be the best thing to use if it gives the information 
> you need.

Yeah, I noticed.  When GO111MODULE=off, everything I am doing is much
faster.  I'm wary of depending on that forever, though.

Stepping back, I fear I am pushing the square peg into a round hole.
Let me restate what I am trying to do.

I want to run a slow codegen process only if the packages it depends
on have ACTUALLY changed (mtime is a good enough proxy) and I don't
know a priori which packages need codegen.  I want to scan the file
tree, find the files that need codegen, check their deps, and only
then run the codegen.

We do this today with `go list` and GO111MODULE=off, but I was advised
at some point that x/tools/go/packages was the future-safe approach.

If there's a better way, I am all ears.

Tim
GO111MODULE=off
> On Fri, Jan 15, 2021 at 11:59 AM 'Tim Hockin' via golang-nuts 
>  wrote:
>>
>> Hi.  This isn't exactly burning urgent, but it is a long-term issue
>> for Kubernetes.  If there's anything I can do to catalyze the
>> discussion - tests to run, info to dig up, etc - please let me know.
>>
>> On Wed, Dec 23, 2020 at 10:48 AM Tim Hockin  wrote:
>> >
>> > Hi Paul!
>> >
>> > On Wed, Dec 23, 2020 at 4:23 AM Paul Jolly  wrote:
>> > >
>> > > > I just can't figure out how to do this.  Maybe it can't be done in `go
>> > > > list` ?  Or maybe we're just missing some detail of go modules..
>> > >
>> > > go list operates in the context of a single module (in the mode you
>> > > are interested in), so you cannot do this with a single command across
>> > > multiple modules.
>> >
>> > This might be a real problem for us.  For this post I am reducing it
>> > to `go list`, but in actuality we have a small program that we wrote
>> > which does what we need in terms of `go/build`.  It works great when
>> > `GO111MODULE=off` but is more than 100x slower normally.  I thought it
>> > was finally time to rewrite it in terms of `go/packages` and get rid
>> > of GO111MODULE=off.  That didn't pan out, hence this post.
>> >
>> > More inline and below
>> >
>> > > > First I do a `find` for any file that has a specific comment tag,
>> > > > indicating that the package needs codegen.  The results span several
>> > > > of the in-repo submodules.
>> > >
>> > > Just to check, I'm assuming the results of this find command are 

Re: [go-nuts] `go list` across multiple modules?

['Jay Conrod' via golang-nuts]

I was initially going to suggest adding the module subdirectories as
requirements to the main go.mod, then replacing those with the
subdirectories.

module example.com/m

go 1.15

require (
  example.com/other1 v0.0.0
  example.com/other2 v0.0.0
  example.com/m/submod v0.0.0
)

replace (
  example.com/other1 => ./staging/src/example.com/other1
  example.com/other2 => ./staging/src/example.com/other2
  example.com/m/submod v0.0.0 => ./submod
)


I think you might have tried this already. It gives the same "main module
... does not contain package" error. I believe that's a bug. I've opened
#43733  to track it.

In general, it should be possible to give 'go list' an absolute or relative
path (starting with ./ or ../) to any directory containing a package which
is part of any module in the build list. For example, some tools list
directories in the module cache to find out what package a .go file belongs
to.

As a workaround, you could put a go.mod in an otherwise empty directory (in
/tmp or something), then require the relevant modules from the repo and
replace them with absolute paths. Then you can run 'go list' in that
directory with absolute paths of package directories.

module tmp

go 1.15

require (
  example.com/other1 v0.0.0
  example.com/other2 v0.0.0
  example.com/m v0.0.0
  example.com/m/submod v0.0.0
)

replace (
  example.com/other1 => /home/gopher/go/src/
example.com/m/staging/src/example.com/other1
  example.com/other2 => /home/gopher/go/src/
example.com/m/staging/src/example.com/other2
  example.com/m v0.0.0 => /home/gopher/go/src/example.com/m
  example.com/m/submod v0.0.0 => /home/gopher/go/src/example.com/m/submod
)


Incidentally, golang.org/x/tools/go/packages will call 'go list' under the
hood in module mode. go/build might do the same, depending on how it's
invoked. 'go list' may be the best thing to use if it gives the information
you need.

On Fri, Jan 15, 2021 at 11:59 AM 'Tim Hockin' via golang-nuts <
golang-nuts@googlegroups.com> wrote:

> Hi.  This isn't exactly burning urgent, but it is a long-term issue
> for Kubernetes.  If there's anything I can do to catalyze the
> discussion - tests to run, info to dig up, etc - please let me know.
>
> On Wed, Dec 23, 2020 at 10:48 AM Tim Hockin  wrote:
> >
> > Hi Paul!
> >
> > On Wed, Dec 23, 2020 at 4:23 AM Paul Jolly  wrote:
> > >
> > > > I just can't figure out how to do this.  Maybe it can't be done in
> `go
> > > > list` ?  Or maybe we're just missing some detail of go modules..
> > >
> > > go list operates in the context of a single module (in the mode you
> > > are interested in), so you cannot do this with a single command across
> > > multiple modules.
> >
> > This might be a real problem for us.  For this post I am reducing it
> > to `go list`, but in actuality we have a small program that we wrote
> > which does what we need in terms of `go/build`.  It works great when
> > `GO111MODULE=off` but is more than 100x slower normally.  I thought it
> > was finally time to rewrite it in terms of `go/packages` and get rid
> > of GO111MODULE=off.  That didn't pan out, hence this post.
> >
> > More inline and below
> >
> > > > First I do a `find` for any file that has a specific comment tag,
> > > > indicating that the package needs codegen.  The results span several
> > > > of the in-repo submodules.
> > >
> > > Just to check, I'm assuming the results of this find command are being
> > > translated to a list of packages? Because the transitive dependencies
> > > of a list of packages within a module can be done via a single go list
> > > command.
> >
> > The trick is "within a module".  I'll update
> > https://github.com/thockin/go-list-modules to reflect the process
> > more.   I've added a
> > get_codegen_deps.sh that models the behavior.  Note that I really want
> > files, not packages, so I can express the dep-graph.
> >
> > What do you mean by "translated to a list of packages" - which specific
> syntax?
> >
> > What I end up with is something like `go list ./path/to/dir1
> > ./path/to/dir2 ./path/to/dir3`.  Any of those dirs might be in
> > different modules.  So `go list` tells me "main module (example.com/m)
> > does not contain package example.com/m/path/to/dir1" and so on.
> > Setting `GO111MODULE=off` does work, but I fear the future of that.
> >
> > > > For each target package, I want to get the list of all deps and
> > > > extract the GoFiles.  Then I can use that to determine if the codegen
> > > > needs to run.
> > >
> > > FWIW I wrote a tool to do just this:
> > >
> https://pkg.go.dev/myitcv.io@v0.0.0-20201125173645-a7167afc9e13/cmd/gogenerate
> > > which might work in your situation.
> >
> > I will take a look - it seems I will need to restructure a bunch of
> > tooling to prove it works for us or doesn't :)
> >
> > > > Where it breaks down is that I can't seem to `go list` all at once:
> > > >
> > > > ```
> > > > # This works within the "root" module
> > > > $ go list -f 

Re: [go-nuts] an open source project to replace SMTP/email

[Liam]

Brian's feedback is here: https://github.com/networkimprov/mnm/issues/5


On Thursday, January 14, 2021 at 9:20:02 AM UTC-8 Brian Candler wrote:

> I do have some feedback, but I think this is off-topic for golang-nuts. 
> Where would you like it to be sent?
>
> My suggestion would be to enable Discussions on the project's github page. 
> This is a relatively new feature in Github, but it's being used by some 
> projects e.g. Netbox 
> .
>
> On Wednesday, 13 January 2021 at 20:45:34 UTC Liam wrote:
>
>> Here's the backstory to mnm, in a draft article not yet published. I'd 
>> love any feedback...
>>
>> https://mnmnotmail.org/volunteered.html
>>
>> On Tue, Jan 5, 2021 at 1:31 PM Liam  wrote:
>>
>>>
>>> The mnm project is building a legitimate email replacement: a client[1], 
>>> a server[2], and a simple protocol[3] between them. (Legitimate, i.e. 
>>> n-identity, decentralized, store-and-forward, open protocol, open source.)
>>>
>>> https://mnmnotmail.org
>>> https://twitter.com/mnmnotmail
>>>
>>> Both client and server are written in Go. The client UI runs in a 
>>> localhost browser tab, rendered by Vue.js. The client has had nine preview 
>>> releases since Spring 2019.
>>>
>>> Contributors welcome!
>>>
>>> [1] https://github.com/networkimprov/mnm-hammer
>>> [2] https://github.com/networkimprov/mnm
>>> [3] https://github.com/networkimprov/mnm/blob/master/Protocol.md
>>>
>>> PS: Ambitious? Yes and no. Establishing a new protocol is a process, but 
>>> email as we know it is headed for extinction, because phishing has made it 
>>> a trivial cybercrime portal.
>>>
>>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/ae914aa2-e9a9-41b3-b531-c8ba42e82ae7n%40googlegroups.com.

Re: [go-nuts] `go list` across multiple modules?

['Tim Hockin' via golang-nuts]

Hi.  This isn't exactly burning urgent, but it is a long-term issue
for Kubernetes.  If there's anything I can do to catalyze the
discussion - tests to run, info to dig up, etc - please let me know.

On Wed, Dec 23, 2020 at 10:48 AM Tim Hockin  wrote:
>
> Hi Paul!
>
> On Wed, Dec 23, 2020 at 4:23 AM Paul Jolly  wrote:
> >
> > > I just can't figure out how to do this.  Maybe it can't be done in `go
> > > list` ?  Or maybe we're just missing some detail of go modules..
> >
> > go list operates in the context of a single module (in the mode you
> > are interested in), so you cannot do this with a single command across
> > multiple modules.
>
> This might be a real problem for us.  For this post I am reducing it
> to `go list`, but in actuality we have a small program that we wrote
> which does what we need in terms of `go/build`.  It works great when
> `GO111MODULE=off` but is more than 100x slower normally.  I thought it
> was finally time to rewrite it in terms of `go/packages` and get rid
> of GO111MODULE=off.  That didn't pan out, hence this post.
>
> More inline and below
>
> > > First I do a `find` for any file that has a specific comment tag,
> > > indicating that the package needs codegen.  The results span several
> > > of the in-repo submodules.
> >
> > Just to check, I'm assuming the results of this find command are being
> > translated to a list of packages? Because the transitive dependencies
> > of a list of packages within a module can be done via a single go list
> > command.
>
> The trick is "within a module".  I'll update
> https://github.com/thockin/go-list-modules to reflect the process
> more.   I've added a
> get_codegen_deps.sh that models the behavior.  Note that I really want
> files, not packages, so I can express the dep-graph.
>
> What do you mean by "translated to a list of packages" - which specific 
> syntax?
>
> What I end up with is something like `go list ./path/to/dir1
> ./path/to/dir2 ./path/to/dir3`.  Any of those dirs might be in
> different modules.  So `go list` tells me "main module (example.com/m)
> does not contain package example.com/m/path/to/dir1" and so on.
> Setting `GO111MODULE=off` does work, but I fear the future of that.
>
> > > For each target package, I want to get the list of all deps and
> > > extract the GoFiles.  Then I can use that to determine if the codegen
> > > needs to run.
> >
> > FWIW I wrote a tool to do just this:
> > https://pkg.go.dev/myitcv.io@v0.0.0-20201125173645-a7167afc9e13/cmd/gogenerate
> > which might work in your situation.
>
> I will take a look - it seems I will need to restructure a bunch of
> tooling to prove it works for us or doesn't :)
>
> > > Where it breaks down is that I can't seem to `go list` all at once:
> > >
> > > ```
> > > # This works within the "root" module
> > > $ go list -f '{{.GoFiles}}' ./subdir
> > > [file.go]
> >
> > This will work.
> >
> > > # This does not work across modules
> > > $ go list -f '{{.GoFiles}}' ./submod/used ./submod/unused
> > > main module (example.com/m) does not contain package 
> > > example.com/m/submod/used
> > > main module (example.com/m) does not contain package 
> > > example.com/m/submod/unused
> >
> > Per above, this will not work across module boundaries.
>
> It works with `GO111MODULE=off` which means that introducing modules
> is a breaking change.  Can I depend on GO111MODULE=off to work the
> same way forever?
>
> > > # Nor does this work, even with module replacements
> > > $ go list -f '{{.GoFiles}}' ./staging/src/example.com/other1/used
> > > ./staging/src/example.com/other1/unused
> > > main module (example.com/m) does not contain package
> > > example.com/m/staging/src/example.com/other1/used
> > > main module (example.com/m) does not contain package
> > > example.com/m/staging/src/example.com/other1/unused
> > > ```
> >
> > With replace directives in place this should work, but you won't be
> > able to use the relative path to the modules (which is in fact
> > interpreted as a directory): it will need to be the full
> > module/package path.
>
> Given a "./path/to/pkg" - how do I convert that to a module/package
> path?  I can run `(cd $dir && go list -m)` but that is super slow.
> Running JUST that for each directory that needs codegen in kubernetes
> takes 20+ seconds.  Is there a better way, short of writing my own
> directory-climb and parsing go.mod?
>
> > > I can run `go list` multiple times, but that's INCREDIBLY slow - most
> > > of these submodules have common deps that are large.  This re-parses
> > > everything over and over.  It takes almost 60 seconds just to do `cd
> > > $dir; go list` (on the real kubernetes repo).
> >
> > Do you have a repro of this taking 60 seconds? Because that really
> > shouldn't be the case with a populated local module cache.
>
> github.com/kubernetes/kubernetes
>
> ```
> $ time \
> find . -type f -name \*.go \
> | xargs grep -l "^// *+k8s:" \
> | xargs -n 1 dirname \
> | sort \
> | uniq \
>  

Re: [go-nuts] return own custom error from template while it executed

[Shulhan]

14 Jan 2021 17:32:37 Vasiliy Tolstov :


Hi. I can't find in google how to solve my problem. Inside
text/template i want to check some passed data and return some
descriptive text if conditions not matching.
I don't have ability to check it before template executed. Does it 

possible?




I am not sure your exact problem. So, you have a function that accept 
data and you want to check the data before passing it to template 
execution? Why you can not check it before template executed?


--
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/cf99443a-9571-4e9e-90a1-5f2e37d7b75f%40gmail.com.