Re: [go-nuts] Re: Unicode variable name error

[Konstantin Khomoutov]

On Sun, Nov 06, 2022 at 01:45:53PM +0530, Nikhilesh Susarla wrote:

>> Per the Go spec[1], an identifier consists of a Unicode letter followed by
>> zero or more Unicode letters or digits. The character పే is in the Unicode
>> category nonspacing mark rather than the category letter.
[...]
> So, if the unicode letters are there in the nonspacing mark as you
> mentioned they can't be used right ?

I sense the source of your misunderstanding might be rooted in your lack of
certain basics about Unicode. You seem to call "a letter" anything which may
appear in a text document (a Go source code file is a text document) but this
it not true. Maybe that's just a terminological problem, but still the fact
is, the Unicode standard calls "letters" a very particular group of things
among those the Unicode standard describes. To give a very simplified example,
in the text string "foo bar" there are six letters (five distinct) and one
space character which is not a letter. The charcter being discussed is not a
letter in Unicode, either.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/20221106105154.xkoemtt6tx25flam%40carbon.

Re: [go-nuts] Refactor Asynchronous JS to Go

[Konstantin Khomoutov]

On Sun, Nov 06, 2022 at 01:33:59AM +0530, Shubh Karman Singh wrote:

> I am refactoring a JS library to Go. I have some questions but I'm not able
> to find some concrete answers for this.
> 1. What's the idiomatic way to refactor Asynchronous JS APIs to Go?
> 2. How to refactor Callbacks from JS to Golang? I have seen some Go code
> which takes function as an input parameter. Is the intent same here?

While my exposure to programming is twice as less than that of Kurtis,
I would like to support his view on this.
The chief idea of Go's concurrency is that it's integrated into the language's
code - as opposed to be bolted on in form of a library, as is the case with
most other popular programming languages out there. This is one of the reasons
why you don't see many callbacks in idomatic Go code: they are either not
needed or their functionality is implemented via sending values via channels
and a set of patterns involving them (see [1, 2, 3]).

For a good and fun overview of that "in core" vs "as a library" distinction
please see [4].

 1. https://go.dev/blog/concurrency-timeouts
 2. https://go.dev/blog/pipelines
 3. https://go.dev/blog/io2013-talk-concurrency
 4. https://journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function/

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/20221106104252.mu5ttxabdhthy26d%40carbon.

Re: [go-nuts] WebAssembly and Filesystem access

[Konstantin Khomoutov]

On Fri, Nov 04, 2022 at 02:08:08PM -0700, 'Kevin Chowski' via golang-nuts wrote:

[...]
> I am on a project which primarily ships a Go command line interface (CLI), 
> but we have aspirations of using the wasm compilation mode to also 
> distribute a simple webapp version of it, while sharing most of the code.
[...]
> The next big hurdle is the fact that we cache things on disk 
> for later reuse, so things are failing out-of-the-box any time we attempt 
> to touch the filesystem. Luckily, in our case, the fact that we are 
> touching the filesystem is a bit incidental (its used as a cache for making 
> consecutive CLI invocations faster), and as long as the system was 
> consistent for the lifetime of the main Go process things will work just 
> fine.
> 
> We /could/ pass a filesystem object across the codebase, and maybe we will 
> one day we will anyway for some other reasons, but I'd rather not have to 
> do that just to get further with this webapp prototype: it's a lot of 
> tedious plumbing, and there is a nontrivial amount of code to touch.
> 
> So instead I decided to try some global-mutable-at-startup variables for 
> things like os.OpenFile, os.Remove, etc, that should be easy to cleanup if 
> we decide not to move forward with the prototype; but even then, there are 
> plenty of random things I have to do to get this to work. I've spent about 
> 2 hours on this direction so far and I keep hitting unexpected roadblocks - 
> thus this email seeking out a new strategy.
> 
> Are there any plans to automatically support filesystems on wasm-compiled 
> Go applications? Even an ephemeral, in-memory filesystem would basically 
> solve all of my problems without having to change any code on my end, which 
> would be nice.

I'm not a Go developer, but I'd say it is unlikely due to the combination of
these two facts:

 - The os package was implemented as a sort-of grab bag of many features
   one expects on a typical OS (hence the name). The fact the filesystem
   operations were implemented in that package directly and not elsewhere
   is the direct manifestation of that approach.
   An environment in which WASM runs in a browser is too different from that
   provided by a typical OS, so I cannot see how one could sensibly implement
   in GOOS=js.

 - Even if some sort of FS emulation were to be implemented for GOOS=js,
   the question is: which one exactly? Keeping stuff in memory in just one
   way of doing things. While I'm not a web developer, I know of at least
   session storage and local storage which provide for two levels of
   persistence beyond keeping stuff in memory. Which one to pick for
   implementing stuff from the os package?

> In lieu of that, does anyone have any tips about how I could go about doing 
> this in a better way?

If you need a hard-and-fast solution, I'd propose you're trying to abstract
your stuff away on a wrong level. I think it would be cleaner to factor out
your persistence code into a set of functions "store this stuff" and "load
that stuff"; the default implementations would call out to the os package, and
the wasm implementation would use either approach you select: keeping things
in memory, using local storage etc.

I mean, trying to implement and pass around through the whole codebase
something like a "filesystem accessor" switchable at runtime is only worth it
if your code makes heavy use of the filesystem in nontrivial way. I'd say that
it's better to abstract away high-level store/load operations.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/20221105115837.zbnyd6kc6xvuistb%40carbon.

Re: [go-nuts] Underscore symbol

[Konstantin Khomoutov]

On Fri, Nov 04, 2022 at 04:58:35AM -0700, Canuto wrote:

> I'm just starting out with go ...
> I have searched for lights on this string but without success.
> What does this sign mean " _, err " , what the underscore symbol means here?  
> 

If you're starting with Go, please start with the Go Tour [1].
For instance, the use of this underscore sign is covered in [2].

 1. https://go.dev/tour/
 2. https://go.dev/tour/moretypes/17

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/20221104210943.w2qgygnchfa2kngj%40carbon.

Re: [go-nuts] fs.Glob, fs.DirWalk, etc. does the pattern support complex regex

[Konstantin Khomoutov]

On Thu, Nov 03, 2022 at 01:20:16PM -0700, pat2...@gmail.com wrote:

> This has to be a FAQ, but my google foo for searching it is not clear
> The docs say: " The syntax of patterns is the same as in path.Match."
> 
> The pattern seems to implement only fairly simple search expressions.
> 
> I'd like to search for "./*.MP4", but of course to be platform independant
> I have to search for both upper and lower case "mp4" and "MP4" 
> and maybe even allow "Mp4".
> 
> This is trivial in most regex, including go's regex
> 
> Can I do this with Glob or DirWalk? Or do I need to just
> get the file names and manually apply the regex?

The latter.

But please note that I, for one, have not heard of any popular filesystem
on a platform Go runs on which would support regexp matching when traversing
its entries, and fs.Glob is largely modelled after what Unix shells implement.
Hence basically if something like fs.Regexp were introduced, it would anyway
need to walk the hierarchy and apply the supplied regexp to each pathname it
visits.

Please also note that using a regexp for what you're after looks like an
overkill to me: a simple

  strings.EqualFold(filepath.Ext(fname), "mp4")

in the callback function to fs.WalkDir wold convey the intent better, IMO.

In any case, such a callback function to find a bunch of files matching their
extensions in a case-insensitive manner would clock at 10-15 lines, I'd say.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/20221104124713.vlglr7fm6azkxpap%40carbon.

Re: [go-nuts] Go Crashes on a long running process (with stack trace)

[Konstantin Khomoutov]

On Tue, Nov 01, 2022 at 10:14:28AM -0700, Vineet Jain wrote:

> We have a long running process that listens to and processes a large number 
> of network packets. It died today with the following stack trace. Anyone 
> seen this before, or have any idea what is causing it?
> 
> Nov 01 12:40:30  sv_svc[100448]: I1101 12:40:30.865268  100448 
> stratconn.go:83] Message Sequence(109845000): Q Length(1)
> Nov 01 12:40:31  sv_svc[100448]: n on lineno rlimitpanicwrap: unexpected 
> string after ty
> Nov 01 12:40:31  sv_svc[100448]: 
> erfreecr1742248535156257105427357601001nTpUpd0x1NFOTyp0x0 pc=0x0]
> Nov 01 12:40:31  sv_svc[100448]: goroutine 4 [wner di]:
> Nov 01 12:40:31  sv_svc[100448]: runtime.throw({0x75b1b0?, 0x0?})
> Nov 01 12:40:31  sv_svc[100448]: /go/src/runtime/panic.go:1047 +0x5d 
> fp=0xcc8080 sp=0xcc8050 pc=0x436a1d
> Nov 01 12:40:31  sv_svc[100448]: runtime: g 4invalid use of 
> scannermalforuntime.sigpanic_ERRORGC scav0x0
[...]

The stack trace looks pretty suspicious because it itself appears to be
corrupted as certain prhases are missing letters in the words or have
different letters instead of "expected" ones.

I can conjure that may be that's just some person typed the text while looking
at the screen - instead of copying and pasting. Or maybe you have run the
"search and replace" operation on the stack trace's text, and it replaced more
stuff than was intended.
But I'm asking because if it really was copied and pasted, I'd propose you
either had a memory corruption in your process - for instance, because of
misbehaving C library code linked to the program or a bug in the code using
the `unsafe` package - directly or indirectly.

Still, these tree lines

| I1101 12:40:30.865268  100448 stratconn.go:83] Message Sequence(109845000): Q 
Length(1)
| n on lineno rlimitpanicwrap: unexpected string after ty
| erfreecr1742248535156257105427357601001nTpUpd0x1NFOTyp0x0 pc=0x0]

(I have stripped the logger prefixes from them for brewity)

look like a manifestation of a "normal" panic from the application's code
or the code of one of the libraries it uses. The clues are:

 - stratconn.go does not appear to be a file from the stdlib "net" package.

 - The "rlimitpanicwrap" cannot be found in the Go runtime's source code.
   Judging from the name, I'd say it's some sort of a handler to catch
   panics and do something; I guess, attach the stack trace to them and
   maybe bubble them up or whatever. It's not really relevant to the issue
   at hand, though.

 - The guru meditation string "Message Sequence(109845000): Q Length(1)"
   appears to be related to parsing of the protocol your application
   may be using.

Based on this evidence, I'd say you have merely found a regular bug in your
code.

Still, I need to confess that

| runtime: g 4invalid use of scannermalforuntime.sigpanic_ERRORGC scav0x0

looks scary and related to the runtime.
The problem is that 1) the message text appearch to be damaged, as I've
already stated, and 2) I fail to find even the piecs of this string in the Go
runtime source code.


That said, asking such question w/o stating the Go flavor used to build the
program, and its exact version is useless. Including at least some details
about the runtime environment (GOOS and GOARCH) would not harm, either.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/20221104110033.yjfmnsh6xvl72rbx%40carbon.

Re: [go-nuts] How does the go compiler treat initialized string variables?

[Konstantin Khomoutov]

On Tue, Nov 01, 2022 at 11:18:48AM -0700, Frank Jüdes wrote:

> I have to write a program that should verify the content of 
> configuration-servers. It will need a lot of pre-initialized data to verify 
> the actual content of a server, so it has to initialize many strings. What 
> i know from other C-style languages is, that code like 
> 
> var MyString *string = 'Hello World!'; 
> 
> Will result in having two identical copies of the string »Hello World!« in 
> the memory: The first one within the program-code itself and a second copy 
> somewhere in the heap-memory.
[...]
> Will there be two copies of the string »Hello World!" in the memory or just 
> one? As said, mMy code will contain a gazillion of pre-defined string 
> variables and having each string allocate two copies of itself in memory 
> would be bad for small systems. 

In addition to what others have said, please note that you can use for
analysis the usual tooling such as `objdump` and `nm` to peek at the generated
executable image file, as well as Go's native `go tool objdump` and
`go build -gcflags=-S` - with the latter producing the assembly code.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/20221103154101.clxygi7iqsvcfegt%40carbon.

Re: [go-nuts] Understanding some gotchas with linking slices together via indexing

[Konstantin Khomoutov]

On Tue, Nov 01, 2022 at 09:38:20PM -0700, Brian, son of Bob wrote:

> Can anyone explain these gotchas (demo )?  
> I've tried reading articles on this [one , 
[...]
> but they don't go into enough detail.


Because that's an intro ;-)

Try reading the next one in the series: https://go.dev/blog/slices

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/20221103144412.7rmrpggsk5gmneir%40carbon.

Re: [go-nuts] Occasional hard lockup w/ 100% CPU usage in go applications

[Konstantin Khomoutov]

On Mon, Oct 31, 2022 at 09:32:21AM -0700, Steven Sokol wrote:

> I tried sending SIGABRT, but the process did not respond - continued 
> running in the runaway state. I can kill it with SIGKILL but that doesn't 
> generate a core. Can you suggest a specific kill command that will? Thanks!

Sending SIGKILL is worthless because this signal is never delivered to the
target process: it's actually a request to the kernel to forcibly bring the
process down without any extra fuss.

To make the process drop the core, you should first make sure that this is
allowed for the process - usually by running

  ulimit -c unlimited

in the shell which starts the process (well, exactly how to make sure this
ulimit parameter is set for your process heavily depends on how it starts up)
and then sending a process any of the signals with "drop the core" default
disposition - see the signal(7) manual page - and not handled by the process
itself using the os/signal package).
On Linux, SIGQUIT is typically the signal you want.
And on a terminal (including virtual terminals such as Linux's virtual
consoles, XTerm, GNOME Terminal, screen, tmux etc) sending of this signal is
bound to Ctrl-\ by default. So, if you can run this program in the foreground
mode - that is, from a shell session attached to a terminal, you can do

  ulimit -c unlimited
  /path/to/the/process
  # idenfity lockup
  # press Ctrl-\ to have the process sent the SIGQUIT

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/20221103091256.7dbi4nbcn3yxcprr%40carbon.

Re: [go-nuts] about upstream?

[Konstantin Khomoutov]

On Wed, Nov 02, 2022 at 02:15:51AM -0700, xie cui wrote:

> there are two micro service writen in go, let's call them A and B. A will 
> call B. In this scene we will call B is the upstream of A. or A is the 
> upstream of B? which one is correct way?

I'm not a native speaker but I would say that using the term "upstream" is not
really fit for this particular case unless A really _streams_ data from B -
that is, opens a long-lived connection with the amount of data to be
transferred not known before making a request, and with the request usually
lasting until B goes offline or A loses interest in receiving the data.
A very good example of streaming is transferring some sort of "events" (for
instance, "realtime" data from a stock exchange, messages in a chat system
etc).

Otherwise, if A makes short-lived requests to B, I would just call B
"a server" and A - "a client". The both parties may be programmed in a way
that B also requests some data from A, and in this case they switch roles;
this is fine.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/20221103090609.tpyftbjxyztoqv63%40carbon.

Re: [go-nuts] Re: Atomic pointers to arrays and sequenced-before guarantees for array elements

[Konstantin Khomoutov]

On Sun, Oct 30, 2022 at 08:47:41AM -0700, jake...@gmail.com wrote:

> I would like to give you a definitive answer, but the same question comes 
> up in https://groups.google.com/g/golang-nuts/c/Gze1TRtdLdc/ and there is 
> much disagreement and no clear resolution. (The initial question is not 
> exactly yours, but if I understand correctly, the discussion quickly gets 
> simplified to exactly yours.) So unless *someone from the go team is 
> willing to clarify,* all you are getting are varying options. Of course, 
> read the thread yourself and maybe you will believe one side was correct. 

Thank you!
Somehow I missed that thread, and it is indeed very close to my situation.
An very interesting read - though not quite satisfying, alas ;-)

> Reading your description, I am also concerned that you could have a logical 
> race. Its possible you already account for this, but if you are setting the 
> 'array pointer' atomically and separately setting the 'count' atomically, 
> the there could be cases where readers read a count not matching the 
> current array. There are many simple fixes, and you hopefully have already 
> taken this into account, but I just wanted to mention it.

Again, thanks!
Yes, there is a logical race, but in the real code the variable holding the
address of an array is written only once: the writer goroutine actually grows
a linked list, and the "active" array is the list's tail - in the same
compound variable with its length. So I _thought_ that any reader who managed
to get to the list's tail had no way reading the length from a different
array. (Trying to explain the whole construction I'm dealing with would
supposedly made it impenetrable for the readers so I've tried to simplify the
things as much as possible.)

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/20221030173354.5325rpxzkzps73in%40carbon.

Re: [go-nuts] Race detector question

[Konstantin Khomoutov]

On Fri, Sep 16, 2022 at 01:26:05PM -0700, 'Keith Randall' via golang-nuts wrote:

> Atomic operations can establish happens-before relationships.
> 
> initially, y==0.
> 
> goroutine 1:
> x = 99
> atomic.Store(, 1)
> 
> goroutine 2:
> if atomic.Load() == 1 {
>println(x)
> }
> 
> This program, if it prints anything, must print 99.
> "x=99" and "atomic.Store(,1)" are ordered by the sequenced-before 
> relationship.
> "atomic.Load()" and "println(x)" are likewise.
> "atomic.Store(,1)" and "atomic.Load()" are ordered by the 
> synchronized-before relationship (if the latter in fact reads the value 
> written by the former).
> So according to the memory model, the x=99 statement happens-before the 
> println(x) statement.
> 
> We use this pattern all the time in the runtime.

But the subtle connection between the x and y in the goroutine 1 is not in any
way explicit, and as I understand, to provide the guarantees you've presented,
the compiler must consider any atomic write-like operation in the code of a
function to be a point at which all earlier stores must have their effects be
observable at any point in a running program which performs an atomic
read-like operation from the same memory location, right?
But I doubt there is any way for such fine-grained approach so basically any
atomic store must work like a full memory fence then.

Another question: if your code were read

| initially, y==0.
| 
| goroutine 1:
| x = 42
| x = 99
| atomic.Store(, 1)
| x = 100
| 
| goroutine 2:
| if atomic.Load() == 1 {
|println(x)
| }

then would println have been guaranteed to still print 99 and not 100?

In other words, do I understand correctly that an atomic write-like operation
guarantees that all stores made by a goroutine before that operation must be
visible, but stores made afterwards may be visible or not visible?
Or, to put it differently, if the read from y in goroutine 2 sees a 1,
there's no way println(x) prints 42, but it may print 99 or 100.
..and also if x is not naturally aligned (I think it's impossible in Go) or
not "hardware atomic" - like a 64-bit integer on a 32-bit platform, -
println(x) could produce some weird value as a result of the torn write/torn
read which may happen when one core updates the x while another one reads it,
right?

[...]

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/20221030171554.gr5u2vjt3mydm5f6%40carbon.

[go-nuts] Atomic pointers to arrays and sequenced-before guarantees for array elements

[Konstantin Khomoutov]

Hi!

I would like to receive clarifications, if possible, on Go memory model
guarantees about non-atomic load and stores when intermixed with atomic load
and stores.

I'm reviewing a piece of code which, simplified, works as follows:

 - There is a single "writer" goroutine which

- Allocates a new array, then uses sync/atomic.StorePointer to save the
  address of this array someplace.

- Writes the array elements one by one.
  After writing each element it - again atomically - updates some memory
  location which contains the number of elements currently written in the
  array.

- When the array fills up, it repeats the routine.

   A point to note is that the writer writes each element only once,
   and after it's done with the array, it never touches this array again.

 - There are multiple "reader" goroutines each of which consumes the data
   being updated by the "writer" in the following way:

- Use sync/atomic.LoadPointer to obtain the address of the array currently
  in use by the "writer" goroutine.

- Atomically read the current number of elements written in the array
  by the "writer" goroutine and then read that many elements
  from the array.

My problem is that I fail to properly apply the text of the Go memory model
to this case to figure out if there is still a data race on the elements
of these arrays between the writer and the readers.

The chief problem I have with my reasoning is that I fail to understand
whether the sequenced-before guarantees provided by atomics apply only to the
atomic operations theirselves - and the memory locations they operate on - or
to the whole sequences of statements executed by goroutines which perform
the atomic operations.

To simplify, and given the description of the algorithm above, I apply the
following reasoning.
A goroutine A performs multiple non-synchronized stores to memory locations X,
Y and Z, and then performs atomic store into memory location M. Naturally, all
store operations done on A up to and including the atomic store to M, are
naturally sequenced - all the non-synchronized stores has happened before the
atomic store.
Now a goroutine B uses atomic load of memory location M and then
performs multiple non-synchronized memory loads from memory locations X, Y
and Z. These non-synchronized loads are naturally sequenced after the atomic
load from M.
("Naturally sequenced" means it's just a regular program flow within a
goroutine.)
Given the above, is it guaranteed that A's stores to X, Y and Z are
synchronized-before B's loads from X, Y and Z?

My gut feeling is that no, Go does not provide this guarantee because of two
reasons:

 - The store performed by A to M does not depend on stores to X, Y and Z,
   and the compiler and/or the hardware are free to reorder them.

 - An atomic store operation is not required to perform a full memory fence,
   so even if no reordering has happened on any layer, the goroutine B
   can still obtain stale data from the CPU cache.

Hence I came to the conclusion that theoretically the code described above is
still prone to data races on the elements of the arrays.

On the other hand, a sample program implementing the algorithm described above
compiled with -race does not fail (8-core i7-8550U, linux/amd64, Go 1.17.13;
also tried other versions). I know that the race detector does only have no
false positives, but the possibility to detect a particular data race heavily
depends on the hardware and relative timings of the operations in a running
program, so the fact it does not see the data race here does not prove it is
not possible.

So, can anyone please help me analyze this case?

(Well, I also should admit the real code is more complicated than that but the
basic idea still holds: the implementor has tries to use atomics in the way
presented in the algorithm described.)

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/20221030132816.zrmrqxucsc2rf7ou%40carbon.

Re: [go-nuts] filepath.walk in Go 1.19

[Konstantin Khomoutov]

On Sat, Oct 29, 2022 at 02:54:48PM -0700, Robert Solomon wrote:

[...]
>>> On ubuntu 22.04, I would like the walk function to NOT follow symlinks to
>>> other filesystems.  The find command uses the -xdev switch to achieve
>>> this.
>>> 
>>> How can I get walk to behave like the -xdev switch to find?
[...]
>> // GetDevice returns the Device ID on which the given file resides.
>> func GetDevice(path string, fi fs.FileInfo) DevId {
>> var stat = fi.Sys().(*syscall.Stat_t)
>> return DevId(stat.Dev)
>> }
[...]
>> Then before calling filepath.Walk, filepath.WalkDir (more efficient), or
>> fs.WalkDir, obtain the device ID of the root.  In the call to WalkDir,
>> pass this device ID to your walk function:
>>
>> err = filepath.WalkDir(root, func(path string, d fs.DirEntry, err
>> error) error { return MyWalkFn(devId, path, d, err) })
>>
>> In MyWalkFn, use d to obtain the device ID of the current path, and
>> return fs.SkipDir if the device IDs do not match.
> I now have this working as I want using filepath/walk.  
> 
> But if I try to use a concurrent walk operation, like the one originally 
> written by Michael T Jones, I get behavior that seems to treat any return 
> of SkipDir as a fatal error and all walking stops.  filepath/walk does not 
> do this; it continues to the next target correctly.
> 
> I feel I'm missing something.  Am I?

It appears you have switched the topic to discuss some custom code which walks
a filesystem which is not in the standard library, have you?

If yes, this is sort of useless without presenting the actual code and the
problem you are having with it. The reason is simple: io/fs.SkipDir is just a
value without any magic attached to it (as there is with certain builtins like
`copy` and `append` or compiler intrinsics like the functions in the
sync/atomic package). That is, returning io/fs.SkipDir from your callback
function can only "work" if the code which calls that callback is prepared to
treat that value in some special manner; if it is not, it will likely treat it
like any other value implementing the standard interface `error`.
This seems to perfectly explain the 

| <...> seems to treat any return of SkipDir as a fatal error <...>

bit.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/20221030115129.pcab4vlswzc3uwwk%40carbon.

[go-nuts] cgo: Is it possible to make exported function have __stdcall calling convention?

[Konstantin Khomoutov]

Hi!

I have a need to create a Windows i386 DLL whose exported functions have
__stdcall calling convention - this is the requirement of the piece of
software which is to load and call this DLL; we cannot change the caller, and
have to adapt to its requirements.

The only approach I was able to find so far is to export the Go function under
a different name and have a separate C file with an adapter function declared
as __stdcall, which calls the target Go function. For instance, this approach
is detailed in [1].

The question: is there a more direct approach? Is it somehow possible to tell
cgo to generate the wrapper for the exported function as using __stdcall
calling convention (as such a wrapper is generated anyway)?

 1. https://github.com/golang/go/issues/32851#issuecomment-507017147

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/20221029122303.o4yl5g33swjectqn%40carbon.

Re: [go-nuts] What's the consensus these days on dealing with defer errors?

[Konstantin Khomoutov]

On Thu, Oct 27, 2022 at 11:01:01AM -0700, Gergely Brautigam wrote:

[...]
> > Since Go 1.13 you can also make chains of errors; say, in your second 
> > example 
> > you could do 
> >
> > err = fmt.Errorf("doSomething failed with %s while handling %w", 
> > tmpErr, err) 
> >
> > and then have callers use errors.Is / errors.As or whatever other approach 
> > to 
> > learn whether the error of their interest is in the error chain. 
> >
> > Note that in ether approach with returning multiple errors, the callers 
> > are to 
> > be prepared to handle multiple errors; otherwise it's engineering for the 
> > sake 
> > of engineering which leads to nothing more than code which is convoluted 
> > for 
> > nothing. 
[...]

> Interesting, the chaining is a nice idea. However, wouldn't that mean that 
> I would still have to errors.Is a specific error. So I would have to handle 
> both. Which is fine ofc. Just saying. :)

Chaining is good in the sense you can still consider a chain to be a single
error, and only inspect the chain when there is need for that.

The usual case for this approach is that when some operation involving
multiple nested calls happens, some deep one fails, then each level wraps the
error in its own context as the error bubbles up the stack.
Then at the call site you have an error with properly nested contexts in the
message - like in "failed to X: error frobincating Y: fizzling of bar
failed: Z: error while fooing quux: ... and so on", and you can just log it
and have a full actionable error message in the logs. And at the same time
you can inspect the chain if you need special processing in case, say,
fooing of quux failed, but not in other cases. With chaining, you can do this,
while still passing the whole chain around where the rest of the code does not
care about whether it is really a chain or a "simple" error.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/20221027181557.tgwbqniwnegusky6%40carbon.

Re: [go-nuts] What's the consensus these days on dealing with defer errors?

[Konstantin Khomoutov]

On Thu, Oct 27, 2022 at 06:23:31AM -0700, Gergely Brautigam wrote:

> I was wondering what's the consensus these days on handling defer errors?
> 
> Is it something like, overwriting the returned value, or using a list of 
> errors?
> 
> Either:
> 
> func test() (err error) {
> defer func() {
> if tmpErr := doSomething(); tmpErr != nil {
> err = tmpErr
> }
>  }()
> 
>  err = doSomethingElse()
>  return err
> }
> 
> Or:
> func test() (errList []error) {
> defer func() {
> if tmpErr := doSomething(); tmpErr != nil {
> errList = append(errList, tmpErr)
> }
>  }()
> 
>  if err := doSomethingElse(); err != nil {
>   errList = append(errList, err)
>  }
>  return errList
> }
> 
> Or even something else entirely? 

I think you're falling into a common trap: there's no need to have any sort of
consensus here because what you have demonstrated are two approaches to tackle
a problem, and they depend on what you're after.

One could easily add other variations.

For instance, often you want to Close() a resource on an error path out;
calling Close might itself return an error, and sometimes this error is less
severe than the "original" one. As an example, consider opening a network
socket and then doing some setup on it: say, socket creation went OK but the
setup failed, so you'd want to return the error about the failed setup
operation, and if closing the socket on the way out fails for some reason,
you could possibly ignore this error. Of may be you can log it.

On the other hand, if, say, you had a file opened for writing, wrote some data
to it, then some error has happened, and you're aborting the operation
returning that error, and closing of the file on the way out fails, this might
be a severe enough error because it might indicate some data which was written
got lost. What to do about it? There's no ready answer. You could construct a
custom error with a compound error message, like "error X has happened while
Y, and also closing of F has failed during cleanup".

You can also return a list of errors.
Or create a custom type implenting the error interface which holds a list of
errors.

Since Go 1.13 you can also make chains of errors; say, in your second example
you could do

  err = fmt.Errorf("doSomething failed with %s while handling %w",
tmpErr, err)

and then have callers use errors.Is / errors.As or whatever other approach to
learn whether the error of their interest is in the error chain.

Note that in ether approach with returning multiple errors, the callers are to
be prepared to handle multiple errors; otherwise it's engineering for the sake
of engineering which leads to nothing more than code which is convoluted for
nothing.

So I'd recommend to start small, use the simplest approach possible until you
see clear need to use a more complex one, and then decide what looks/works
best in a particular case. I know, this is a glib piece of advice, but it's
really how the things are ;-)

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/20221027175042.uhaakvkgnmbvqmh4%40carbon.

Re: [go-nuts] Occasional hard lockup w/ 100% CPU usage in go applications

[Konstantin Khomoutov]

On Wed, Oct 26, 2022 at 08:45:00AM -0500, Robert Engels wrote:

> Trigger a core dump then use gdb on the core file. 

Also, the gdb is usually shipped with the `gcore` helper which can attach to
the specified PID and dump its core.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/20221026150257.rtxcqfb4u2vexwlv%40carbon.

Re: [go-nuts] Analyse postgresql error

[Konstantin Khomoutov]

On Wed, Oct 26, 2022 at 01:38:51PM +0300, David Harel wrote:

[...]
> func errorCheckResult(err error) string {
> if err == nil {
> return ""
> }
> pqerr := err.(*pq.Error)
> switch pqerr.Get('C') {
> case "23505":
> return "Key violation"
> }
> return "Error unknown"
> }
> 
> Now I want to be able to test my code thereby generating this type of error
> in the testing function.
> So I need to create an error variable of the type pq.Error and use it as a
> parameter to db.Mock. Something like:
> 
> mockDB := wrapper.NewMockQuerier(ctrl)
> // The part that breaks
> err := fmt.Errorf("pq: %s", `duplicate key value violates unique constraint
> "treatment_pk"
> {"Severity":"ERROR","Code":"23505","Message":".MoreStuff."}`)
> 
> mockDB.EXPECT().AppointmentUpdate(gomock.Any(), gomock.Any()).Return(err)
> 
> Any idea?

Well, look at what you do:

  fmt.Errorf("pq: %s", `some long text`)

creates a value of some (irrelevant) type implementing the standard interface
error and containing a string which is obtained by calling fmt.Sprintf on the
aguments. The "%s" verb in the format string means "get the string
representation of the matching argument and insert in into the format string
in place of the verb". Your argument is itself a string, and it is completely
opaque to fmt.Errorf - it's taken "as is" and is not interpreted in any way.

Please stop and think of this for a moment: even if the contents of this
string would be somehow interpreted, how would the code in package fmt guess
it has to produce a value of type pg.Error from that string, and how exactly?

So, if you need to create a *pq.Error, go on and do just that in your
mock. Based on [1], you could do something like

  return {
  Code: "23505",
  Severity: "ERROR',
  Message: `duplicate key value violates unique constraint "treatment_pk"`,
  }

 1. 
https://github.com/lib/pq/blob/d65e6ae4bdd1c86b16cd6d2bcff4fe970dc697b4/error.go#L25

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/20221026124535.rlmptmaonqsvtsup%40carbon.

Re: [go-nuts] Analyse postgresql error

[Konstantin Khomoutov]

On Fri, Oct 21, 2022 at 08:17:16AM -0700, David Harel wrote:

> Newbie on golang. Using sqlc: https://github.com/kyleconroy/sqlc in the 
> environment created by Karl Keefer: 
> https://github.com/karlkeefer/pngr/actions/workflows/build.yml, thanks Karl.
> My queries use querier which is auto generated by sqlc.
> 
> Sometime I get an error result in my query such as key violation.
> I want to check the error code of such an error.
> 
> as one of the guys suggested I can look into the object using Marshal
> and I noticed that the error contains a very elaborated, like:
> {"Severity":"ERROR","Code":"23505","Message":"duplicate key value violates
> unique constraint \"treatment_pk\"","Detail":"Key (customer_id, patient_ttz,
> therapist_id, appointment_timestamp)=(1, 060525649, 160, 2022-10-20
> 10:30:00) already
> exists.","Hint":"","Position":"","InternalPosition":"",
> "InternalQuery":"","Where":"","Schema":"public",
> "Table":"treatment","Column":"","DataTypeName":"",
> "Constraint":"treatment_pk","File":"nbtinsert.c","Line":"434",
> "Routine":"_bt_check_unique"}

What is "Marshal" you're referring to here?
Is this encoding/json.Marshal called on some value returned by that
"querier generated by sqlc"?

> Trying to get the same result using straight forward approach like:
> fmt.Printf("%+v\n", err) gives me a much simpler object display.
> 
> How can I get the "smart" content of the error object like the "Code" 
> member?

If my guesseneering is correct, you can just access the Code field of that
value you passed through "Marshal" and analyze it using any convenient method
such as an if or switch statement.

>From [1], I gather that "23505" is indeed a standardized error code for
violation of unique key constraints in PostgreSQL, so basically you could do

  const uniqueKeyViolation = "23505"

  switch result.Code {
  case uniqueKeyViolation:
// Do something sensible
  default:
log.Println("unexpected failure: ", result.Code)
  }

Still, please notice that all of the above is what's called "psychic
debugging" and might miss the point partially or completely because your
problem statement is not an MCVE [2]. I'm not sure you could sensibly create
one (as it could have required posting swaths of generated code or something
like this) so please don't take it as a blame.

 1. https://www.postgresql.org/docs/current/errcodes-appendix.html
 2. https://stackoverflow.com/help/minimal-reproducible-example

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/20221025162116.eao4slua2xihi2ta%40carbon.

Re: [go-nuts] Tracking which connection is used to make an HTTP request

[Konstantin Khomoutov]

On Tue, Oct 25, 2022 at 03:43:21PM +0400, Konstantin Khomoutov wrote:

>> However, that does not work well for TLS. The reason is that the Go HTTP 
>> client has special logic in case a custom dialer returns a *tls.Conn. That 
>> logic will be disabled.
> 
> The Conn funtion of the crypto/tls package allows to "upgrade" a user-supplied

The function is called Client, sorry for the confusion.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/20221025115754.kec4g6mqabx3cvj6%40carbon.

Re: [go-nuts] Tracking which connection is used to make an HTTP request

[Konstantin Khomoutov]

On Wed, Oct 12, 2022 at 09:01:46AM -0700, Christian Worm Mortensen wrote:

(I have rehashed the options you've presented, to help commenting on them.)

[...]

> 2) I can make a my own struct and return from my custom dialer:
> 
> type myConn struct {
>   net.Conn
>   dialInfo string
> }
> 
> However, that does not work well for TLS. The reason is that the Go HTTP 
> client has special logic in case a custom dialer returns a *tls.Conn. That 
> logic will be disabled.

The Conn funtion of the crypto/tls package allows to "upgrade" a user-supplied
net.Conn to a tls.Conn, so you could probably try to modify your dialer so
that it returns myConn which has, say, LocalAddr method returning a custom
type which implements net.Addr but also conveys some information about itself,
like in

  type MyAddr struct {
addr net.TCPAddr
dialInfo string
  }

  ...

  func (a *MyAddr) DialInfo() string { return m.dialInfo }

It's clumsy but should work: you could

 if laddr, my := someTLSConn.LocalAddr().(*whateverpkg.MyAddr); my {
 di := laddr.DialInfo()
 }

or type-assert to an interface containing the single DialInfo() method.

> 1) I can keep a map from net.Conn to information that identifies the dial. 
> But when can I delete entries from this map? Go does not have weak pointers.

Combined with the idea above, each myConn could keep a reference to the
"registry" which maintains the mapping from the conns to the dial information
blocks, an have myConn deregister in its Close() method.

Alternatively you could use runtime.SetFinalizer() on your custom myConn type
- passing it a reference to that "registry" object which had your myConn
registered after creation.

The type of that registry must be safe for concurrent use in both cases.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/20221025114321.exxsktohqjckiyxn%40carbon.

Re: [go-nuts] Zero-sized data type

[Konstantin Khomoutov]

On Thu, Oct 13, 2022 at 10:45:33AM -0700, Richiise Nugraha wrote:

> The field can't be reordered.
> I want to use zero-sized data type to mark offset and has no effect on 
> struct size, I can take the address of it and use it like:
> ```go
> c := ...
> cmd := (*[SIZE]byte)(unsafe.Pointer())
> ```
> I am using this on mmap-ed memory.
> I've found a workaround to do the same thing by adding `c` pointer with 
> it's data size

IIUC, you cannot use flexible arrays in Go - please see [1] and the
Go isse #20275 [2].

1. https://stackoverflow.com/a/73660093/720999
2. https://github.com/golang/go/issues/20275

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/20221020125413.ix6zbbqyq7mbfqoi%40carbon.

Re: [go-nuts] go runtime in shared libs functions called in threads

[Konstantin Khomoutov]

On Wed, Oct 19, 2022 at 06:28:20AM -0700, Peter Galbavy wrote:

> I have built a shared lib in Go to replace an old thing we use to send 
> email - mainly to modernise things and add TLS and authentication. We run 
> each call to the entry point in it's own thread in the main program.
> 
> I am both curious but also concerned about what happens here with go 
> runtimes. I presume each thread has it's own runtime, and on exit it is 
> torn down, but I may be wrong and would like to know more.

As already hinted, there's only a single instance of the Go runtime powering
your shared library after it got loaded and initialized. Basically it's the
same thing as a regular Go program would call into a C library via cgo, and
that library would be using multiple running threads which could access the
data passed from the Go side and even call back to the Go side.

To have less cognitive burden, I would recommend to move one step further and
make your library have a single entry point which would accept a job unit and
process it concurrently with any other job units - that is, defer concurrency
to the Go library.

If you're basically already doing something like this and are only concerned
about what happens when you call to a Go library from multiple threads, then
the answer is - apply the usual logic: all variables which are local to the
function call chain which got called from a thread are safe as they are not
shared from the other call chains. If these call chains happen to share some
state in the library, you have to guared the access to it - in one way or
another - using channels or resorting to mutexes etc.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/20221020112742.n35ewbvzb2t72f7s%40carbon.

[go-nuts] Write heap dump in a forked process - could that be even possible?

[Konstantin Khomoutov]

More than once at my $dayjob we experienced the need to grab the heap dump or
a running process implementing some critical functionality.
The problem is that saving of the dump file requires freezing of the process,
and if its memory consumption is relatively high (in our case it clocked
around 50 GiB), saving of the dump freezes the process for too long to have
negative impact (the service processes "real-time" data).

I were thinking of an approach similar to what Redis uses for one of its
backup modes on Unix (I'm mainly interested in Linux): it periodically forks
itself and the child then saves away the memory it "inherited" from its parent
while the parent continues working. This feature takes advantage of the memory
CoW mechanism implemented on Linux (and I think on most if not all popular
platforms which have fork) to have total memory consumption resonably low, and
allows the main process to have relatively brief pauses - required only to
fork a child copy and be done with it.

Could something like this at leat theoretically be implemented for Go's
runtime/debug.WriteHeapDump?

Well, in order to not create an XY Problem, I should stress that my end goal
would be to have a way to "dump core" of a running Go process with big memory
footprint in a way to not freeze the process for too long (under, say, 2
seconds).  Being able to somehow collect the "classic" core file (using
something what `gcore` does with the help of GDB) would be fine as well, but
`gcore` freezes the process as well.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/20221010183320.rdm4dyijfcl6hj6n%40carbon.

Re: [go-nuts] goroutine private / local var/const

[Konstantin Khomoutov]

On Wed, Jul 20, 2022 at 01:00:59PM +0300, Konstantin Khomoutov wrote:

> > for i=0;i<10;i++ { go func () { /* const c=i OR var v=i */
> >  fmt.Println("f:beg i=",i) // here c or v instead 
> > // action 
> >  fmt.Println("f:end i=",i) // here c or v instead 
> > }}

[...]

>  - Make your anonymous functions not close over that shared variable.
> 
>This can be done by defining one or more arguments in the function's
>definition and pass the values the function has to operate on via that
>parameters when calling it. Since passing of parameters is done via
>copying them, each anonymous function will have their own copies.
> 
>In your simple case you could do what PeterGo proposed - something like:
> 
>go func (int i) { ... }(i)

[...]

While we are at it, pay attention to the fact that variable assignments and
passing parameters to functions always copy values, some types contain
pointers - addresses of other variables - and hence copying of them copies
those pointers, not the memory they point at. Sometimes it is said that such
types have reference semantics.

Examples of such types are slices and maps. For instance, if in our toy
example we'd make our anonymous function receive a slice variable and pass
it a slice value when calling it to execute in a separate goroutine, the
slice itself would be copied but the memory of its backing array would not -
only the address of that memory block. Hence if we were to start multiple
goroutines this way, and at least one of them would attempt to mutate the
slice, that, again, could produce a data race (and generally weird behavior -
if some goroutine were to append to the slice, and it was required to grow).

To recap, when pondering how to make your goroutines not step on each other's
memory toes, you have to consider how the types the variables of which these
goroutines intend to manipulate, are defined, and which properties they have.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/20220720110832.qwuszqutgf7o2bpp%40carbon.

Re: [go-nuts] goroutine private / local var/const

[Konstantin Khomoutov]

On Tue, Jul 19, 2022 at 08:28:02AM -0700, 'andreas graeper' via golang-nuts 
wrote:

> for i=0;i<10;i++ { go func () { /* const c=i OR var v=i */
>  fmt.Println("f:beg i=",i) // here c or v instead 
> // action 
>  fmt.Println("f:end i=",i) // here c or v instead 
> }}
> when this routines get interrupted then beg-i and end-i differ
> now i want at the beginning of the routine a const copy `const c int = i`
> but its not allowed and `var v int = i` does not work, too.
> 
> does an instance of a go-routine has no own stack-frame like a normal 
> function (c i.e) and therein local/private variables ? 

This?

https://github.com/golang/go/wiki/CommonMistakes#using-goroutines-on-loop-iterator-variables

In short:

 - Your goroutines run so-called anonymous functions - those defined "inline"
   and having no name, some folks call them "lambdas".

   Anonymous functions works like "closures": they "close over" any variable
   accessed in such a function but declared in any of the outer scopes
   accessible to the function.

 - A closure captures its external variables via reference - that is, if you
   create multiple closures in the same lexical scope, they all will reference
   the same variables. To say that in other words, they do not receive copies
   of such variables; they reference the original variables.

 - In your code snippet you start 10 goroutines passing each a new closure,
   each referencing the same loop variable, i.

Hence, in your example you create the condition for a data race of 10
goroutines over the same variable i. The behaviour of this code is hence
undefined. If you were to `go build -race` it (or `go run -race`, FWIW),
the runtime would crash your running program.


Conceptually two solutions are possible:

 - Make your anonymous functions not close over that shared variable.

   This can be done by defining one or more arguments in the function's
   definition and pass the values the function has to operate on via that
   parameters when calling it. Since passing of parameters is done via
   copying them, each anonymous function will have their own copies.

   In your simple case you could do what PeterGo proposed - something like:

   go func (int i) { ... }(i)

   If the set of variables to operate is huge so as being impractical to
   be passed via arguments, you usually either define a helper `struct`
   type containing all the required stuff and pass it as a single parameter
   or stop using anonymous functions altogether and define a custom type
   with a method, and to run a gorotine, you construct an instance of this
   type and then call the goroutine to run that method on that instance.

   (To recap: do not be attached to passing anonymous functions to goroutines:
   it indeed is a cool feature, often useful, but it is so widely demonstrated
   by various guides and blog posts that some novice Go programmers do not
   even understand that you can pass anything "callable" to the `go`
   statement, not only an anonymous function. Yes, I've witnessed people with
   this mind wart in interviews at my $dayjob.)

 - Create a copy of the loop iteration variable and capture _it_ in the
   anonymous function - like this:

   for i := 0, i < 10; i++ {
 i := i
 go func() { fmt.Println(i) }()
   }

   This "trick" works because in Go, all loop statements have two scopes: the
   outer scope is where the loop iteration variables live, and the inner one
   is the loop's body; hence in the presented example we create a variable
   named "i" in the inner scope and initialize it with the value currently
   held in the variable named "i" from the outer scope. The "inner" "i" is
   different on each iteration - or, to say that in other words, - is
   recreated anew on each iteration, so it's safe to capture it in a goroutine
   started on that iteration - provided, of course, that it's the sole
   goroutine capturing that variable and started on that iteration.

What approach to pick depends on the use case, personal aesthetics and
whatever the person(s) doing code review on your team prefer.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/20220720100059.44upqj4vcqpnuf7d%40carbon.

Re: [go-nuts] how to convert or typecast windows handle/fd to net.conn

[Konstantin Khomoutov]

On Tue, Dec 04, 2018 at 12:29:03PM -0800, apmat...@gmail.com wrote:

> My problem is following: I need to use windows WSAAccept() instead of 
> normal accept (net package seems to use AcceptEx).
> 
> I'm I missing somethig 
> 
> I do not know a way to extend net-package so only way I know is to use 
> reflect to get the listener Sysfd (I know .. really bad, other options ?) 
> and pass it to WSAAccept().
> 
> Now I get the new handle/fd back and to be able to use rest of the code I 
> would need to convert it to net.conn, how can I do this ?

I think you can call net.TCPListener.SyscallConn() to get the syscall.RawConn
giving you access to the underlying handle.

The problem is: what would you want to do next?
I mean, it will be possible to not call Accept* on the listener and pass the
handle you will fetch from the raw connection to WSAAccept(), but then what
would you do with the handle of the "accepted" socket it will produce?
IIRC, net.FileConn, which is supposed to "wrap" an FD into a Go-native
"object" is not supported on Windows.

> In general, how can we extend net package when all important structures are 
> private ? fd_windows.go:netFD struct includes poll.FD what is even deeper 
> at package chain.

You cannot, but Go is F/OSS, so if there's really no other way to do what you
are after, you can carry a patch or a patch series on top of the branch of the
Go release you're using and use it to build your code.
Sure, this implies you do this for an in-house project but the stated problem
appears to be a really weird one and hence suggests that this is the case.

Still, can you explain what is the real reason for using WSAAccept?
I'm asking to make sure we're not dealing with a so-called "XY Problem".

Since some time the net package in the Go standard library gained the ability
to get a grip on the underlying connection/endpoint FD/handle (partially
explained above) so if you need to set some funky options on the connection,
you can do this w/o resorting to using a different Win32 API call to accept
connections on it.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/20220713171245.6i2ztxdkyptxry54%40carbon.

Re: [go-nuts] The program stucked on __vdso_clock_gettime (x86_64)

[Konstantin Khomoutov]

On Mon, Jul 04, 2022 at 01:57:06AM -0700, Chenhong Liu wrote:

[...]
> After profiling the process with perf command, the perf data shows the 
> program looped to call __vdso_clock_gettime .
> 
> 64.20% controllerd [vdso] [.] __vdso_clock_gettime 
> 8.82% controllerd controllerd [.] runtime.procyield
> 8.82% controllerd controllerd [.] runtime.suspendG 
> 8.01% controllerd controllerd [.] runtime.nanotime1
> 1.47% controllerd [kernel.kallsyms] [k] __enqueue_entity 
> 0.79% controllerd [kernel.kallsyms] [k] system_call_after_swapgs 
> 0.79% controllerd [kernel.kallsyms] [k] set_next_entity 
> 0.68% controllerd [kernel.kallsyms] [k] _raw_gspin_lock
> 0.68% controllerd [kernel.kallsyms] [k] change_pte_range
> 0.65% controllerd [kernel.kallsyms] [k] auditsys
> 0.57% controllerd [kernel.kallsyms] [k] update_curr 
> 0.45% controllerd [kernel.kallsyms] [k] native_sched_clock 
> 0.45% controllerd [kernel.kallsyms] [k] cpuacct_charge 
> 0.45% controllerd [kernel.kallsyms] [k] __x86_indirect_thunk_+rax 
> 0.45% controllerd [kernel.kallsyms] [k] __audit_syscall_exit
> 0.34% controllerd [kernel.kallsyms] [k] pick_next_task_fail
> 0.23% controllerd controllerd [.] runtime.osyield 
> 0.23% controllerd [kernel.kallsyms] [k] native_queuedc_spin_lock_slowpath
> 0.23% controllerd [kernel.kallsyms] [k] dput 
> 0.23% controllerd [kernel.kallsyms] [k] __schedule 
> 0.23% controllerd [kernel.kallsyms] [k] put_prev_task_fair 
> 0.23% controllerd [kernel.kallsyms] [k] yield_task_fair
> 0.22% controllerd [kernel.kallsyms] [k] sys_sched_yield 
> 0.11% controllerd [kernel.kallsyms] [k] clear_buddies 
> 0.11% controllerd [kernel.kallsyms] [k] update_rq_clock.part.78 
> 0.11% controllerd [kernel.kallsyms] [k] update_min_vruntime 
> 0.11% controllerd [kernel.kallsyms] [k] tick_do_update_jiffies64 
> 0.11% controllerd [kernel.kallsyms] [k] system_call 
> 0.11% controllerd [kernel.kallsyms] [k] rb_next 
> 0.11% controllerd [kernel.kallsyms] [k] rb_insert_color 
> 0.00% controllerd controllerd [.] runtime.notesleep 
> 0.00% controllerd [kernel.kallsyms] [k] load_balance 
> 0.00% controllerd controllerd [.] runtime.runggrab 
> 0.00% controllerd controllerd [.] runtime.findrunnable 
> 0.00% controllerd [kernel.kallsyms] [k] update_numa_stats 
> 0.00% controllerd [kernel.kallsyms] [k] __lock_task_sighand
> 0.00% controllerd [kernel.kallsyms] [k] idle_cpu
> 0.00% controllerd [kernel.kallsyms] [k] kmem_cache_free_bulk 
> 0.00% controllerd [kernel.kallsyms] [k] task_numa_find_cpu
> 0.00% controllerd [kernel.kallsyms] [k] __queue_work
> 0.00% controllerd [kernel.kallsyms] [k] __perf_event_task_sched_in
> 0.00% controllerd [kernel.kallsyms] [k] finish_task_switch
> 0.00% controllerd [kernel.kallsyms] [k] perf_ctx_unlock
> 0.00% controllerd [kernel.kallsyms] [k] native_write_msr_safe
> 0.00% controllerd [kernel.kallsyms] [k] __perf_event_enable
[...]

For me, this looks like the program is merely idling, and what you're
observing is reading current time by the scheduler implemented by the Go
runtime.

[...]
>1. Any body know the reason of the problem?

I'm inclined to think that you're merely facing some deadlock or similar
condition which precludes your normal Go code from doing useful progress while
the runtime itself is OK.

>2. What can I do to get more information if this problem happened again?

Do the following:

 - Have a way to collect whatever the process is writing to its stderr.

 - When the process wedges in the same way, send it SIGQUIT or SIGABRT so that
   it dumps the stacks of all the active goroutines to the stderr and exits.

If that service handles HTTP requests, you might enable the handlers of the
core debug/pprof package and when the process it apparently wedged navigate to
its /debug/pprof/goroutine?debug=2 (unless you have grafted the root of
those endpoints somewhere else) to have the stacks sent in the response.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/20220705095534.ay2wdgeinruwuf2w%40carbon.

Re: [go-nuts] ListenAndServeTLS() (pem and key files for private network)

[Konstantin Khomoutov]

On Mon, Jun 27, 2022 at 05:35:38PM -0700, Hugh Myrie wrote:

> I wish to create a secure private network using a self-signed certificate 
> with a Go web server: See the following code block:
> 
> // Code 
> err := http.ListenAndServeTLS(":"+port, "auto.org.pem", 
> "auto.org-key.pem", handler)
> if err != nil {
> 
> logError((err.Error()))
> log.Fatal("ListenAndServe: ", err)
> }
> // End of Code
> 
> Could I auto  generate (and register) the .pem and .key files using GO?  I 
> wish to create a trust certificate if there files do not exist.
> 
> I came across the following website:
> 
> "https://gist.github.com/shaneutt/5e1995295cff6721c89a71d13a71c251;
> 
> I am not sure how to implement this. Your help is appreciated.

I'm afraid there may be a critical flaw in your approach as a concept.
I'll try to explain how I perceive it. I might be wrong in my assessment, and
if yes, please excuse me - I'm just trying to help.

OK, so, TLS has two conceptual facets in the way it implements secure data
exchange tunnels: encryption (information hiding) and mutual authentication.
Based on my experience, people tend to ignore the second one while fixating on
the former. Maybe this comes from the extensive usage of web browsers, in
which using of certificates for authentication most of the time is strictly
one-way - most websites to not require their clients to authenticate on the
TLS level, and authenticating of the websites is well hidden under the hood.

Now consider implementing a custom "secure private network" with the help of
TLS. Say, your server accepts TLS sessions from its clients, and uses
a self-signed certificate and the matching key. Now, have you thought out how
this server will make sure that a client wanting to connect to actually has
the permission to do that? Conversely, how the client knows the server is
legitimate and was not spoofed using a Man-in-the-Middle attack?

To authenticate clients, you might implement some non-TLS method - such as
passwords. This would work, but when architecting a secure communication
system you should apply "security mindset" when thinking: if the client has
set up a TLS session with a rogue server, any information the client sends to
that session must be considered as compromised, and any imformation received
must not be trusted (unless there's a way to reliably verify it). This inclues
the password exchange. You could implement a secure password exchange scheme
which does not result in disclosing the password (only proves its knowledge)
but the rogue server can just tell the client it authenticated OK, and then
start accepting actual data from the client. You could implement the reverse
scheme to also authenticate the server to the client, and this would require
keeping the server's password on each client.

OK, so TLS is already able to authenticate both sides to each other - using
certificates. There are two ways do do it. The "normal" one is to trust a
certificate presented during a TLS handshake exchange by trusting whoever had
issued that certificate (and hence signed it). The "punk" way is to check the
so-called fingerprint - a cryptographic hash calculated on the certificate's
data - to match whatever stored at the authenticating side.

Add to the picture that the server usually wants to have a way to prevent
certain clients - which would otherwise be properly authenticated - from
accessing the server - usually because they have been compromised somehow
(consider a stolen laptop which contains the cert+key used to access the
server). Again, TLS has a way to support this - through the so-called
certificate revocation list, CRL, which can list otherwise valid certificates
which must be considered not eligible for use - "revoked".

So, what I'm leading to, is basically these two things:

 - Proper framework for mutual authentication of the server(s) and the clients
   forming a secure network requires careful planning and implementing.

   An often overlooked aspect of it is managing keys used for authentication.

 - TLS already implements support for both mutual authentication during session
   initiation phase, and for implementing the key management framework.

Not using these features should require careful consideration: security is
notoriously hard to get right, and one has to think twice before forfeiting
tried-and-tested solutions. Autogenerating a self-signed certificate and
sticking it into a library call which starts a HTTPS server looks like merely
looking for TLS-encryption without considering authentication.

OK, so, should you decide to acually rely on TLS to do proper authentication,
you will need to read up on how authentication based on X.509 certificates
actually works, what certification authoriries (CAs) are, and how certificates
are to be issued and maintained and revoked.

Note that it's not required to maintain a full-blown certification authority
(CA) to generate certificates and keys for the servers and the 

Re: [go-nuts] Go binary, that tries to run shell command in ESXI environment is failing

[Konstantin Khomoutov]

On Wed, May 18, 2022 at 09:40:03AM -0700, Ian Lance Taylor wrote:

[...]

> Thanks for including the strace output.  It shows that the clone
> system call is failing with ENOSPC.  There are several reasons that
> clone can fail with ENOSPC.  None of them have anything to do with
> disk space (though shortage of inodes appears to be a possibility).
> ESXI appears to map several internal errors to ENOSPC as well, mostly
> dealing with lack of memory.
> 
> You may need to discuss this with someone familiar with ESXI.

Quick googling turns up [1] which supports what Ian said.

It's interesting, whether there is a ESXi-native tool analogous to strace
but which would provide "native" kernel error codes? That could possibly move
us closer to the root cause of the problem.


By the way, Harsh, in your original SO post you state that you

| have the same program in python and C++, and they both run without any
| issues in the same environment.

Could you please show bits of this code? I'm asking because it's interesting
whether you really do fork+exec (or something) like this in those programs.
I would also be interesting to inspect the traces captured by strace of the
(successful) runs of these programs to see whether the clone(2) syscall gets
performed with the same arguments (I think the "flags" argument is of the most
interest). Of course, this might be a red herring, and the problem might lie
elsewhere, so the ability to somehow obtain the "native" kernel's error code
would probably help a lot.

 1. 
http://virtualization24x7.blogspot.com/2017/03/table-of-all-vmkernel-error-codes-in.html

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/20220519110101.gdbzf7rxf4m65bfu%40carbon.

Re: [go-nuts] when EOF returned from http client.Post, client can continue to be used ?

[Konstantin Khomoutov]

On Sun, Apr 17, 2022 at 06:52:17AM -0700, Toon Knapen wrote:

> It is not clear to me from the documentation whether I can continue to use 
> an http.Client after a POST failed (due to the connection being closed ;  
> In my case the connection closes probably due to rate-limiting).
> 
> The documentation of the http.Transport does mention that, if a network 
> error occurs, it will retry if the request is idempotent. Given I'm doing a 
> basicPOST that is not the case for me. 
> 
> But the http.Client will reestablish the connection on the next call to 
> Post(...) ?

It won't be http.Client who will reestablish the connection, but whatever
implementation supporting the http.RoundTripper interface provided in the
Transport field of the http.Client. If that field is not set, the default
http.DefaultTransport will be used, which is an instance of http.Transport.

Basically the idea is that an http.Transport first tries to pick a live but
idle TCP connection from its pool of idle connections, and if there are none,
it tries to establish a new one.

I'd undeline the idea is that an http.Client is not something akin to a single
invocation of `curl`; instead it's a shared client-side HTTP request
"multiplexer" which is OK to be used for multiple concurrent HTTP requests
wihch will share a single http.RoundTripper configured to be used for that
http.Client (as explained above), and it, in turn, is responsible for
maintaining a pool of ready-to-use connections, any number of which can be
used to carry out HTTP requests concurrently.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/20220418093427.bmdwdtbttppi4aqm%40carbon.

Re: [go-nuts] Getting 502 errors in golang with heavy load

[Konstantin Khomoutov]

On Fri, Aug 06, 2021 at 08:13:46PM +0300, Konstantin Khomoutov wrote:

[...]
>  * Aerospike does not use HTTP as its client protocol; it implements some
>custom protocol over TCP sockets, so that 502

Sorry, fat fingers. I meant to say "... so that those '502 errors' could not
come from Aerospike". At least unless you're accessing it in some interesting
way, but then again - you did not tell us.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/20210806171812.eqfyxmptawq2t2qx%40carbon.

Re: [go-nuts] Getting 502 errors in golang with heavy load

[Konstantin Khomoutov]

On Thu, Aug 05, 2021 at 08:07:58PM -0700, Anand vishnu wrote:

> my code is running absolutely fine with a smaller load, when load is 
> increased it is failing.
> The changes i made in the newer release tag:
> 1. pushing data to aerospike at each and every chunk
> when i turn off this line it is working fine , but when i turn this on it 
> is failing

Please consider the following set of facts:

 * While you did not explain that, "502 errors" appear to mean
   some HTTP requests complete with the server answering with responses
   containing the 502 HTTP status code.

   Sadly, you did not tell us which HTTP requests actually fail this way.

 * Aerospike does not use HTTP as its client protocol; it implements some
   custom protocol over TCP sockets, so that 502

 * HTTP status code 502 is defined to mean "Gateway Timeout".

Based on these facts I could do a guess: you have a setup like this:

 [your clients] → [http proxy] → [your Go program] → [aerospike]

Depending on how your Go server is deployed, you can be easily unaware that an
HTTP proxy - such as nginx - is sitting between your server and your clients.

When the load from the client increases, the request handling code starts to
spend progressively more time in the calls which send data to Aerospike,
and eventually some of them spend so much time doing that they surpass a
maximum time span configured in that HTTP proxy which any request is allowed
to spend. Once a request trips that threshold, the proxy rips it and returns a
response to the client, telling it the service it mediates access to is not
keeping up.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/20210806171346.scqjtaju4ulwibkj%40carbon.

Re: [go-nuts] What's going on while using "append" in memory?

[Konstantin Khomoutov]

On Fri, Aug 06, 2021 at 12:42:34AM -0700, Miraddo wrote:

Hi!

> I can not understand why when we use append, it does not rewrite values in 
> the same address in the memory that it had before, 
> 
> (I'm not sure it "append" problem or the variable |  or just my problem for 
> sure :) ) 
[...]
> I saw all addresses in memory was changed, 
> 
> now my question is why it doesn't rewrite the values in the same address as 
> before?
[...]

Have you read and understood [1] and [2]?
I think the information from these articles will enable you to solve your
problem all by yourself.

I would also argue that rotating an array *in place* by a non-negative value
is "classically" solved by using a temporary variable and moving of the
array elements; the latter can be done by using copy().

 1. https://blog.golang.org/slices-intro
 2. https://blog.golang.org/slices

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/20210806165959.lwyefdc5daxlmp7y%40carbon.

Re: [go-nuts] Re: go install an/import/path@revision and /vendor in Go 1.16

[Konstantin Khomoutov]

On Thu, Aug 05, 2021 at 04:39:00PM -0700, Sean Liao wrote:

> This is intentional and sort of covered by the statement that no module is 
> considered the main module (where replace directives and vendor dirs take 
> effect)

I must admit that sentence regarding the main module has caught my attention
when I was reading that part of the docs but it looks like I have not paid
enough attention to the concept of a main module earlier in the manual.
To be honest, I do not remember I have even seen it, no idea why.

> https://github.com/golang/go/issues/40276 has some more context

Thanks for your explanations.

[...]

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/20210806105206.h4m2azmsocjlxaqq%40carbon.

Re: [go-nuts] Re: go install an/import/path@revision and /vendor in Go 1.16

[Konstantin Khomoutov]

On Fri, Aug 06, 2021 at 12:35:16AM -0700, Brian Candler wrote:

> The key point I just learned from #40276:
> "Vendor directories are not included in module zip files."
> 
> I found this surprising because -mod=vendor is now the default (here 
> ) - therefore, I expected the vendor 
> directory, if present in the source, always to be used unless explicitly 
> disabled.

An interesting point regarding our particular case - which I sadly failed to
mention - is that we do not use any module proxies for own internal projects,
and the hostname of our GitLab instance is listed in the GONOPROXY environment
variable, so technically `go install` did not download a "classic" module,
which is a ZIP archive file, but has rather performed the usual sequence of
issuing a HTTP GET request with the "?go-get=1" query parameter followed by a
shallow clone of the target Git repository.
Hence technically the "vendor" directory with its full contents was available.

But yes, that point of yours is a sensilbe one; it sheds some light on the
reasoning behind the behaviour have observed.

[...]
> It might also be worth a mention at https://golang.org/ref/mod#zip-files 
> that vendor directories are excluded from module zip files (although I 
> wouldn't be reading that section if I was only interested in how vendoring 
> works)

I would say the doc could make use of some cross-referencing: this fact could
be mentioned either there or in the section on vendoring, and the remaining
section could contain a single statement about the fact with the link to a
full explanation in the other section.

What do you think?

[...]

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/20210806104542.2wmd7c3vfn6ohrt6%40carbon.

[go-nuts] go install an/import/path@revision and /vendor in Go 1.16

[Konstantin Khomoutov]

Hi!

We have recently turned one of our internal Go projects into a Go module.
The project was using vendoring and we intend to continue doing so - just
rather with the natural support from `go mod` instead of govendor as before.

An interesting issue came up when we've decided to try the recently added
ability of `go install` to grab a named module and build an executable from
it: we would expect that when we do

  $ go install an/module/import/path@latest

the command would grab the module's source code (it's hosted by a GitLab
instance, so we'd expect the source code to be `git clone`-d) and then behave
as if one would clone the module's repository by hand and then run
`go install` in the local clone's work tree - that is, we'd expect the command
to actually make use of the vendored packages and not attempt to download
anything except the source code of the specified module.

Instead, the command appears to read the module's go.mod file and then
download all the dependencies from the network - completely ignoring the
"vendor" directory of the module.

I have tried to locate a related issue in the bug tracker, but failed;
[1] is the closest I was able to come up with, but it appears to not touch the
very problem I have described.

The documentation also does not seem to specify the expected behaviour.

Hence my question is: is this a problem worth reporting/discussing or just an
undocumented behaviour which is "by design"?

My take on why this behaviour is confusing is that it makes `go install`
behave differently when it's apparently told to do the same thing to the same
module:

 - When I have the module's source code on my filesystem, and I run
   `go install` in that directory, the command uses the vendored dependencies,
   builds and installs the executable provided by the module.

 - When I run `go install import/path/of/that/module@revision`
   I expect the command to just add one extra step - fetch the module's
   source code and then run `go install` in the (temporary) directory
   with the fetched source code.

 1. https://github.com/golang/go/issues/44841

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/20210805193923.6vf7q4cssrdwaxre%40carbon.

Re: [go-nuts] Windows Write Call Error when Writing to Network mapped Folder.

[Konstantin Khomoutov]

On Tue, Sep 22, 2020 at 10:56:43AM -0700, helhadad wrote:

> Hi Folks,
> I am trying to create file and write to it on a *network mapped drive*, 
> which I can access, create, delete and edit files using windows explorer or 
> CMD (Windows 10/Server 2016).  
> 
> You can find all details of the issue and response from other Go-Expert 
> below:
> 
> https://stackoverflow.com/questions/63960049/writing-to-networked-mapped-drive-empty-files-or-failure
> 
> I would appreciate if you enlighten me about the issue and how to solve it.

I would in particular highlight one weirdness discovered in [1]:
when the OP did use forward slashes in their call to os.OpenFile,
the error message read

  {Op:"write", Path:"H://00_SC//Dest01.txt", Err:0x57}

that is, every forward slash in the input has been doubled.

Another issue I can't fully grasp is why creating of the file goes well,
and the error is only returned on the first call to Write on the returned
file handle?  Some folks suggested special dances are required in order to
properly open a file on such a network share but this does not explain why
plain commands like `copy` work in a cmd.exe session with that same resource
attached by the `net use` command.

(Kindly looking for soliciting any insight from Alex Brainman.)

Is there a way to trace what happens on the system call level?

 1. https://stackoverflow.com/questions/63960049#comment113160125_63960049

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/20200924143320.7cdinhe4o3f5rsit%40carbon.

Re: [go-nuts] Regarding time.NewTicker() and monotonic time

[Konstantin Khomoutov]

On Wed, Jun 10, 2020 at 04:03:36PM -0700, Ian Lance Taylor wrote:

[...]
> In the current implementations of Go, Tickers are not garbage
> collected.  They run until they are stopped.  So if you don't stop a
> ticker, it will keep ticking until your program exits.
> 
> (It is possible that future implementations will garbage collect
> Tickers, but it still won't hurt to stop a ticker that you no longer
> need.)

Do I understand correctly, that a stopped Tiker is not garbage
collected, too?  IOW, stopping a Ticker merely makes it not waste CPU
time but it will still float in memory.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/20200617142227.5et3eb2ygi3ngh3c%40carbon.

Re: [go-nuts] json struct tag

[Konstantin Khomoutov]

On Wed, Jan 31, 2018 at 09:26:35AM +1030, Dan Kortschak wrote:

> > > Does the order in which the options in the tag appear matter (e.g. 
> > > `json:"first_name,omitempty"` vs `json:"omitempty,first_name"`) ?
> > 
> > I'd say they should not as their purposes are orthogonal to each
> > other.
> It does matter, and omitempty must have a leading comma; the first
> comma separated term is the JSON field name. From the json docs:
> 
> ```
> The format string gives the name of the field, possibly followed by a
> comma-separated list of options. The name may be empty in order to
> specify options without overriding the default field name.

Thank you! I stand corrected.

I think that's what happens when you get certain knowledge deep under
your skin: sure I habitually used ",option" countless times and surely
knew that it's due to the name field having been skipped but that
knowledge was unconscious. ;-)

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Decoder.DisallowUnknownFields() works when passing *struct but not *map[string]interface{}

[Konstantin Khomoutov]

On Sun, Jan 28, 2018 at 10:59:58AM -0800, Trig wrote:

> This works as intended when you pass it a pointer to a Struct; however, it 
> should also work (you would think, since the Unmarshaller can handle both 
> types) a pointer to a *map[string]interface{}; however, it does not.  Are 
> there any future plans to implement this option on both types?

You're talking about [1], are you?

If yes, the commit [2] which closed it specifically talks about struct
types, so the function behaves as documented.

On the other hand, I have two questions to narrow the scope of your
claim down.

 - To carry out its intended task, DisallowUnknownFields() has to operate
   on a value which it can use to figure out the set of known fields.

   In the case of a map this suggests that the map passed to that method
   must be populated with the keys which would define the names of
   the known fields (and the values assigned to those keys are expected
   to be overwritten by the decoding process). Is this what you're proposing?

 - Why a pointer to a map?

   In my eyes, this suggests that you may legitimately pass a pointer
   to an unitialized map value and expect the decoder to create one for
   you; is that correct?

   If "yes" is the answer to the both points, they cannot be satisfied
   together: to fulfill the first requirement the map value must be non-nil,
   and you cannot expect the decoder to initialize it for you.

All in all, I suggest you to create a proposal in the issue tracker.
Just be sure to be crystal clear, when laying it out -- to save the devs
from guessing what you really meant.

1. https://github.com/golang/go/issues/15314
2. https://github.com/golang/go/commit/2596a0c075aeddec571cd658f748ac7a712a2b69

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] json struct tag

[Konstantin Khomoutov]

On Sun, Jan 28, 2018 at 09:21:12AM -0800, Trig wrote:

> I've searched but can't find any good sources on json struct tag options 
> and rules...

These rules are contained in the pieces of documentation for
encoding/json.Marshal and encoding/json.Unmarshal; studying the output
of `go doc json.Marshal` and `go doc json.Unmarshal` should have you
covered.

> Does the order in which the options in the tag appear matter (e.g. 
> `json:"first_name,omitempty"` vs `json:"omitempty,first_name"`) ?

I'd say they should not as their purposes are orthogonal to each other.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Nested struct types and closing over them in function literals

[Konstantin Khomoutov]

We've just have a discussion with a colleague about the case of what
exactly variable would a function literal close over, if it calls a
method on a field of a struct-typed variable available in the scope of
the enclosing function.

Here's the code:

---8<
package main

import (
"fmt"
)

type Foo struct {
val int
}

func (foo *Foo) Show() {
fmt.Println(foo.val)
}

type Bar struct {
foo *Foo
}

func main() {
foo := {123}
bar := {foo}

fn := func() {
bar.foo.Show()
}

fn()

bar.foo = {456}

fn()
}
---8<

(Playground link: [1]).

and the dispute was about whan variable would fn() close over.

I proposed that it will close over bar.foo since it's the "narrowest"
variable needed to be able to call Show().
As it turned out, I was wrong, and fn() closes over bar.

My line of reasoning stems from the following passage from the spec [2]:

| Structured variables of array, slice, and struct types have elements and
| fields that may be addressed individually. Each such element acts like a
| variable.

which seem to suggest in the case above bar.foo is a variable, and since
bar itself is not needed to call the Show() method provided by the
bar.foo's type, the function literal would close over the field alone
(presumably internally storing a pointer to it).

The part of the spec which discusses function literals [3] states:

| Function literals are closures: they may refer to variables defined in a
| surrounding function. Those variables are then shared between the
| surrounding function and the function literal, and they survive as long
| as they are accessible.

So there is no clearly defined rule of how the compiler decides what to
capture.


I've got hunch that may be [2] really has concurrent access in mind when
it states "Each such element acts like a variable", and this does not
apply to other cases.


Can anyone please explain where there is a flaw in my line of reasoning
which led me to a wrong conclusion?

 1. https://play.golang.org/p/_VfdzQDNZR
 2. https://golang.org/ref/spec#Variables
 3. https://golang.org/ref/spec#Function_literals

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Understanding panic at addr=0x38 when calling a method on a nil interface

[Konstantin Khomoutov]

On Mon, Nov 13, 2017 at 03:58:35PM -0800, keith.rand...@gmail.com wrote:

[...]
 What puzzles me, is that the address it panics is not 0x0 (which I would
 expect from an x86/amd64 H/W platform to stand for nil) but 0x38:
[...]
>> Konstantin, your description is correct.  The code is trying to load a 
>> function pointer out of an itab, but the pointer to the itab is nil.
>> I think this is actually a bug.  If you have an interface with more than 
>> pagesize/ptrsize methods in it, this code might not panic when it should.
> Filed issue https://github.com/golang/go/issues/22703

Thank you! (And @bradfitz for fixing.)

Still interesting why the runtime powering playground crashes at 0x0 —
as Alexander discovered in his message to this thread.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Understanding panic at addr=0x38 when calling a method on a nil interface

[Konstantin Khomoutov]

While debugging a program which had a panic due to an attempt to call a
method on a value of an interface typeš, I came across the behaviour I
find strange, and would like to get help understanding what happens.

The behaviour is exhibited by this simple program:

---8<
 1  package main
 2  
 3  import (
 4  "fmt"
 5  "os"
 6  )
 7  
 8  func main() {
 9  var fi os.FileInfo
10  s := fi.Name()
11  fmt.Println(s)
12  }
---8<

When built by Go 1.8.3 on Linux/amd64 and run on that same system
it expectedly panics at line 10.


What puzzles me, is that the address it panics is not 0x0 (which I would
expect from an x86/amd64 H/W platform to stand for nil) but 0x38:

---8<
$ go run foo.go
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0x47d148]

goroutine 1 [running]:
main.main()
/home/user/foo.go:10 +0x28
exit status 2
---8<


If I run `go tool objdump` on the generated binary, I get this
(instruction codes removed for brewity):

---8<
TEXT main.main(SB) /home/user/foo.go
foo.go:80x47d120FS MOVQ FS:0xfff8, CX   
foo.go:80x47d129CMPQ 0x10(CX), SP   
foo.go:80x47d12dJBE 0x47d1d3
foo.go:80x47d133SUBQ $0x58, SP  
foo.go:80x47d137MOVQ BP, 0x50(SP)   
foo.go:80x47d13cLEAQ 0x50(SP), BP   
foo.go:10   0x47d141MOVQ $0x38, AX  
foo.go:10   0x47d148MOVQ 0(AX), AX  
foo.go:10   0x47d14bMOVQ $0x0, 0(SP)
foo.go:10   0x47d153CALL AX 
foo.go:10   0x47d155MOVQ 0x10(SP), AX   
foo.go:10   0x47d15aMOVQ 0x8(SP), CX
foo.go:11   0x47d15fMOVQ CX, 0x30(SP)   
foo.go:11   0x47d164MOVQ AX, 0x38(SP)   
foo.go:11   0x47d169MOVQ $0x0, 0x40(SP) 
foo.go:11   0x47d172MOVQ $0x0, 0x48(SP) 
foo.go:11   0x47d17bLEAQ 0xf3de(IP), AX 
foo.go:11   0x47d182MOVQ AX, 0(SP)  
foo.go:11   0x47d186LEAQ 0x30(SP), AX   
foo.go:11   0x47d18bMOVQ AX, 0x8(SP)
foo.go:11   0x47d190CALL runtime.convT2E(SB)
foo.go:11   0x47d195MOVQ 0x10(SP), AX   
foo.go:11   0x47d19aMOVQ 0x18(SP), CX   
foo.go:11   0x47d19fMOVQ AX, 0x40(SP)   
foo.go:11   0x47d1a4MOVQ CX, 0x48(SP)   
foo.go:11   0x47d1a9LEAQ 0x40(SP), AX   
foo.go:11   0x47d1aeMOVQ AX, 0(SP)  
foo.go:11   0x47d1b2MOVQ $0x1, 0x8(SP)  
foo.go:11   0x47d1bbMOVQ $0x1, 0x10(SP) 
foo.go:11   0x47d1c4CALL fmt.Println(SB)
foo.go:12   0x47d1c9MOVQ 0x50(SP), BP   
foo.go:12   0x47d1ceADDQ $0x58, SP  
foo.go:12   0x47d1d2RET 
foo.go:80x47d1d3CALL runtime.morestack_noctxt(SB)   
foo.go:80x47d1d8JMP main.main(SB)   
---8<

So, for the call at line 10 we have

MOVQ $0x38, AX
MOVQ 0(AX), AX

which I translate as "load the quad word 0x38 into the register AX
and then load the quad word located at offset 0 in the memory at
the address located in the register AX, into that same register".

That second instruction fails (since IIRC Linux maps a special
sentinel page at address 0x0 to catch problems like this one).


I fail to comprehend why 0x38 appears to be a constant (some magic
number).  Looks like this is an offset of something.  Recalling [1],
I found out Go 1.8.3 defines an Itab as

type itab struct {  

inter  *interfacetype   

_type  *_type   

link   *itab   

Re: [go-nuts] Upcasting/Downcasting in Go

[Konstantin Khomoutov]

On Wed, Nov 08, 2017 at 02:48:06PM -0800, Ian Lance Taylor wrote:

[...]
>> So when should I expect the type casting to work?
[...]
> When thinking about Go it's best to avoid concepts that do not apply,
> like derived class, upcast, and downcast.  You can't write
> d.testUnderlyingTypeAsReceiver() because the method is only defined on
> the type S, and d is type []int, not type S.  Passing d to
> testUnderlyingTypeAsParam works because a value of an unnamed type is
> assignable to a value of a named type when the underlying type of the
> named type is the same as the unnamed type.  There is no upcasting or
> downcasting involved.  There is just simple assignment.

Haiyu Zhen, please also note that there are no such thing as "type casting"
in Go: it only has "type conversions" and "type assertions" (which come in
several forms).

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] XML Parsing Nested Elements

[Konstantin Khomoutov]

On Tue, Nov 07, 2017 at 03:35:45AM -0800, lesm...@gmail.com wrote:
> I am really struggling to access nested elements of an XML string and 
> suspect it is down to the namespaces.  This string is obtained from a 
> larger document and is the "innerXML" of some elements.  A simplified 
> version is at...
> 
> I could probably do this with multiple structs but want to have this in a 
> single struct.
> 
> https://play.golang.org/p/Een-guMNP9
> 
> I can seem to read things at the root but cannot get them using the ">" 
> syntax at all.  What am I doing wrong?  Can I "insert" a namespace element 
> to assist it at all?
> 
> I have manually removed the namespaces from this example to show what I 
> think should happen!?
> https://play.golang.org/p/eCzbzgBYMq

The chief problem with your approach is lack of error checking.
The encoding/xml.Unmarshal() function returns an error value.
Had you checked it for being set (not nil), it would have given you an
immediate idea of what was wrong with your approach.

Regarding namespaces, your hunch is correct: since your XML document is
a fragment extracted from another document by a seemingly "textual"
method, all those "XML namespace prefixes" — parts in the names of the
elements which come before the ':' characters — have no meaning to the
XML parser since they are not defined in the document itself.

Unfortunately, currently there's no way to somehow explicitly define
them anywhere (say, in an instance of encoding/xml.Decoder) before
decoding, so you basically have three options:

- Somehow textually stick their definition on the top element of your
  XML document fragrems, so, say, it reads something like

http://example.com/another/namespace/uri/;
 ...>

  …and then parse the resulting document into a value of a struct
  type the tags on whose fields contain full namespaces in the names
  of the XML elements they're supposed to decode.

- Use iterative approach by creating an instance of encoding/xml.Decoder
  and calling its Token() method.

  When it returns a token of the types StartElement or EndElement,
  their Name property can be examined to see what its "Space" and
  "Local" fields are.

- Ignore the XML namespace prefixes completely.

  In your case this appears to be the simplest solution as the
  names of the elements appear to be unique anyway.

The variant which checks for errors, ignores the XML namespace prefixes
and also defines the field named "XMLName" on the type to check the
name of the element it's supposed to unmarshal can be implemented
as follows:

8<
package main

import (
"encoding/xml"
"log"
)

type TrackInformation struct {
XMLName struct{} `xml:"trackInformation"`

TimeAtPosition string `xml:"timeAtPosition"`
Speed  int`xml:"speed"`

DepApt string `xml:"qualifiedAircraftId>departurePoint>airport"`
ArrApt string `xml:"qualifiedAircraftId>arrivalPoint>airport"`
Gufi   string `xml:"qualifiedAircraftId>gufi"`
}

func main() {

xmlToParse := `


TEST
KR32642300

KJFK


KJFK


245
2017-11-07T11:20:43Z
`

var trackInfo TrackInformation
err := xml.Unmarshal([]byte(xmlToParse), )
if err != nil {
log.Fatal(err)
}
log.Println(trackInfo)
}
8<

Playground [1].


A couple of more notes.

- You can't use namespaces when defining the names of the nested
  elements.  The wording of the documentation is a bit moot but it does
  explicitly state this: «If the XML element contains a sub-element
  whose name matches the prefix of a tag formatted as "a" or "a>b>c"…» —
  notice that "the prefix of a tag" bit which actually means "the local
  name of an element".

  So when you need to match on full names of the elements, you'd have to
  use nested structs so that each field stands for an element without
  nesting, and the nesting is defined via your types rather than
  tags on their fields.

- The XML decoder implements a "strict" mode, which is "on" by default.

  What's interesting about it is that even when it's on, it turns a
  blind eye on undefined XML namespace prefixes: «Strict mode does not
  enforce the requirements of the XML name spaces TR. In particular it
  does not reject name space tags using undefined prefixes. Such tags
  are recorded with the unknown prefix as the name space URL.»

  This means that you can use your undefined namespace prefixes "as is"
  when decoding. [2] demonstrates this approach applied to the top-level
  XML elements.  You can't do this for that "a>b>c" notation in the tags
  but you still can apply it when 

Re: [go-nuts] Short Term DSN for database/sql

[Konstantin Khomoutov]

On Tue, Nov 07, 2017 at 12:06:27AM -0800, agruet...@gmail.com wrote:

> It appears the Generic Interface for SQL (database/sql) does not support 
> the ability to use short lived DSN's for cases like AWS's IAM 
> Authentication to RDS instances. It appears that when doing its connection 
> pooling activities it always uses whatever DSN was passed initially on Open.
> 
> 1) Is this correct.
> 2) Assuming one is correct would the Golang project every consider adding 
> this feature?
> 3) If the answer is yes. How would one go about either helping to get it on 
> the road map or better yet contributing this feature back to the community? 
> What is this process?

Does setting the maximum number of the idle (that is, pooled) database
connections [1] to a value equal or less than zero helps with this?

1. https://golang.org/pkg/database/sql/#DB.SetMaxIdleConns

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Is there a way or lib to read a binary into a sturct{len int32;data [len]int32}

[Konstantin Khomoutov]

On Tue, Oct 31, 2017 at 04:36:20PM +0800, hui zhang wrote:

[...]
> > > Is there a way or lib to read a binary into a this structure in one line
> > > code?
> > >
> > > how about more complicate struct ?
> > > type complicatestrt struct {
> > >   len int32
> > >   strData []variastrt
> > > }
[...]
> > - Write some complicated code which uses reflection -- just like
> >   the stock encoding/binary does -- to understand how to interpret the
> >   fields of any custom struct type based on that type's tags (or just
> >   "shape" -- that is, the types and relative order of its fields).
> >
> > For the latter, take into account that while there's no mention of the
> > format of the field tag in the language spec, the convention for their
> > format is documented at [4]:
> >
> > | By convention, tag strings are a concatenation of optionally
> > | space-separated key:"value" pairs. Each key is a non-empty string
> > | consisting of non-control characters other than space (U+0020 ' '),
> > | quote (U+0022 '"'), and colon (U+003A ':'). Each value is quoted using
> > | U+0022 '"' characters and Go string literal syntax.
> >
> > Hence you could come up with something like:
> >
> >   type variastrt struct {
> >   len  int32   `var:"value:data endian:big"`
> >   data []int32 `var:"endian:little"`
> >   }
> >
> > And then you'd reflect over the type the user supplied to your
> > unmarshaling code, and in that reflection code you'd parse the field's
> > tag, get the value associated with the "var" key and parse that, in
> > turn, to know the name of the field to unmarshal the value into, and the
> > endianness of the elements of that value (if applicable).
[...]
> > 4. https://golang.org/pkg/reflect/#StructTag

> Thank you very much
> I know something like below could do.  The problem has some one did it ? I
> don't know how to use the tag string.
>  type variastrt struct {
>   len  int32   `var:"value:data endian:big"`
>   data []int32 `var:"endian:little"`
>   }

I'm not sure what you're asking about.
As I have said, if you would like to implement the indicated approach,
you would need to write code which obtains tags defined on the fields of
a user-programmed type, parses them and acts based on the results.

I have shown you what is the codified policy on the format of the struct
field tags (they are not defined by the Go language specification but
everyone agrees upon that policy which I cited in my first reply to this
thread).  To inspect at runtime the type of a user-supplied variable,
and then further inspect its properties — such as fields of a struct
type, you need to use the standard package "reflect".

If you want to learn how to reflect over any custom user-defined type
using the standard package "reflect" and deal with the tags defined on
the fields of such a type, I'd suggest looking at the standard package
encoding/json and encoding/xml which make heavy use of these tags.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] sync.Map performance problem

[Konstantin Khomoutov]

On Fri, Oct 20, 2017 at 11:50:42PM -0700, barce show wrote:

> I use sync.Map in my program to support access concurrently, but witin two 
> hours with less than 50 clients, my program run out all of my cpu, so I go 
> back to read the pkg document, and I find this sentence 
> 
> "For use cases that do not share these attributes, it will likely have 
> comparable or worse performance and worse type safety than an ordinary map 
> paired with a read-write mutex."

You seem to omit the key point from the description [1], which is

| It is optimized for use in concurrent loops with keys that are stable
| over time, and either few steady-state stores, or stores localized to
| one goroutine per key.

I dare to "translate" this into more layman's language in the following
way:

  the sync.Map is optimized for the use cases where the following
  two conditions are met:
  
  1. The keys in the map are mostly stable.  That is, there is no
 constant churn of the keys — insertions and deletions, — but rather
 reads and updates.

  2. There is not much stores (into the same keys, mostly, — see above)
 compared to the number of reads
 - or -
 such stores are performed by the same goroutine, — that is, each
 goroutine updates the data under its own key, and no other
 goroutine does that, or does this infrequently.

>From this, we may gather that the implementation of sync.Map supposedly
uses a couple of tricks which are more involved than just having a
single sync.RWMutex under the hood.  The implementation goes to those
lengths to improve the performance for the cases outlined above but, as
with all things related to optimizations, it sucks performance-wise for
the cases which do not fit into the intended usage pattern.

You can see this even with the simple locks: the sync.RWMutex allows
multiple parattel readers and only goes exclusive for a writer but the
payback is the implementation slower than sync.Mutex which is
all-exclusive, and so if you have roughly equal numbers of readers and
writers sync.RWMutex may actually work slower for your case.

> maybe this the my problem, but I don't really understand what it represent
> can anybody help me?

May I ask whether you have actually profiled your problem?
Are you sure that's sync.Map which affects your performance, and not
something else completely?

Go comes packed with a slew of tools to profile your programs.
You should start with the CPU profile and block profile.
There are lots of information on how to start with profiling the Go code
so just use your favourite search engine to get started.

1. https://golang.org/pkg/sync/#Map

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: XML unmarshaller doesn't handle whitespace

[Konstantin Khomoutov]

On Thu, Oct 05, 2017 at 12:48:53AM -0700, Per Persson wrote:

> > Probably it's best to ask here before I try to contact the Go developers.
> >
> > I had a file with space padded numbers (NumberOfPoints="   266703") and 
> > the XML unmarshaller couldn't handle the spaces.
> >
> > Booleans are trimmed 
> > , 
> > so why not numbers?

> Thanks for a good answer and a good suggestion of a solution!

You're welcome!

> Perhaps I should take the question to the developers to have them either 
> give a technical answer or discuss a change.

I'd recommend to just file a bug over there in the issue tracker and
solicit a discussion on the matter.

1. https://github.com/golang/go/issues

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] XML unmarshaller doesn't handle whitespace

[Konstantin Khomoutov]

On Mon, Oct 02, 2017 at 03:10:19AM -0700, Per Persson wrote:

> Probably it's best to ask here before I try to contact the Go developers.
> 
> I had a file with space padded numbers (NumberOfPoints="   266703") and the 
> XML unmarshaller couldn't handle the spaces.
> 
> Booleans are trimmed 
> , 
> so why not numbers?

I'd take this question to be rather philosophical.
The question of whether the string "   266703" represents a valid
integer is hardly decidable in my opinion.  For instance, would the
string "\v\v\v\n\n\n\t\t266703\n\n\n\v\v\v\v\t\x20\t" represent a valid
integer as well?

I don't know why the decoder from encoding/xml is more lax regarding
booleans; may be that's because the string representation of a boolean
value strconv.ParseBool() accepts as valid is pretty lax in itself,
or may be that's simply because when that code was written, that space
trimming was added semi-automatically by the programmer without giving
it much thought. ;-)

All in all, you could use the fact XML decoder from encoding/xml 
checks whether the type of the variable it's going to unmarshal textual
data into implements the encoding.TextUnmarshaler interface [2],
and if it does, the UnmarshalText() method of that type is used to parse
that textual data.

So a way to go in your case is to define a special type to unmarshal
those "whitespace integers" from your XML data, and make that type
implement encoding.TextUnmarshaler:

8<
package main

import (
"bytes"
"encoding/xml"
"fmt"
"strconv"
)

type xmlInt int64

func (xi *xmlInt) UnmarshalText(b []byte) error {
v, err := strconv.ParseInt(string(bytes.TrimSpace(b)), 10, 64)
if err != nil {
return err
}
*xi = xmlInt(v)
return nil
}

type data struct {
NumberOfPoints xmlInt `xml:",attr"`
}

const s = ``

func main() {
var d data
err := xml.Unmarshal([]byte(s), )
if err != nil {
panic(err)
}
fmt.Println(d)
}
8<

Playground link: [1].

1. https://play.golang.org/p/Euy8Sag88P
2. https://golang.org/pkg/encoding/#TextUnmarshaler

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] `ltrace` yields "Couldn't find .dynsym or .dynstr in "/proc/*/exe" with binaries generated by "go build *.go"

[Konstantin Khomoutov]

On Sat, Sep 16, 2017 at 04:51:00AM +0800, Sen Han wrote:
> Thanks for your information. I want to use `ltrace` to find out if the
> golang binary is using
> the __vdso_clock_gettime_sym with CLOCK_MONOTONIC option for measuring time
> interval.
> 
> But it finally turns out currently the `ltrace` cannot works very well with
> go built binaries on
> vdso tracing.
> 
> Is there some exist tools could trace the vdso call in golang? Or is there
> any recommend methods
> could achieve it on the runtime?

Go is free software so instead of trying to trace the binaries its
compiler creates, you could just grep the runtime's source code. ;-)
Especially given the fact that since version 1.5 Go is entirely written
in Go (modulo some assembly code).

I have just grepped the checkout of 1.9 for CLOCK_MONOTONIC, and
I think devel/golang1.9/src/runtime/sys_linux_amd64.s is what you need.
Look for the "nanotime" function.

Note that at least on i386 and amd64, Linux maps the memory ranges
containing the data related to date/time information into the address
space of every process, so reading these data does not involves any
syscalls -- merely direct reads of the data located at a certain memory
address.

IIRC, on Windows, a similar mechanism is used.  Don't know about other
supported kernels.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] `ltrace` yields "Couldn't find .dynsym or .dynstr in "/proc/*/exe" with binaries generated by "go build *.go"

[Konstantin Khomoutov]

On Fri, Sep 15, 2017 at 04:09:24AM -0700, nehs...@gmail.com wrote:

> Hi, I just found `ltrace` would yield "Couldn't find .dynsym or .dynstr in 
> "/proc/*/exe" with executable binary generated by "go build *.go".
> 
> Is there any options in `go build` could solve this problem?
> 
> Thanks a lot.
> 
> PS:
> $go version
> go version go1.9 linux/amd64
> $ cat hi.go 
> package main
> 
> 
> import "fmt"
> 
> 
> func main() {
>  fmt.Println("hi!")
> }
> $ go build hi.go
> $ ltrace ./hi
> Couldn't find .dynsym or .dynstr in "/proc/`*`/exe"

>From the description of the ltrace's Debian package, I gather that

| ltrace is a debugging program which runs a specified command until it
| exits.  While the command is executing, ltrace intercepts and records
| the dynamic library calls which are called by
| the executed process and the signals received by that process.
| It can also intercept and print the system calls executed by the program.

The Go compiler (of the "gc" suite you seem to use) generates statically
compiled executable which link with no dynamic libraries (at least on
Linux/amd64, with the reasonably current Go versions).

Supposedly this explains why you don't see the those ".dyn*" sections in
the generated ELF file.

So what is your goal, exactly?  What do you want to achieve?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Doing "C-like" things in Go

[Konstantin Khomoutov]

On Mon, Sep 11, 2017 at 09:56:59AM -0700, CampNowhere wrote:
> I am a C developer and am trying to pick up Go.
> 
> My question is this. C doesn't "care" about truthfulness, it just cares 
> about zero and non-zero when evaluating a logical AND operator. So 
> something like the following in C is totally kosher:
> 
> int a = 10;
> int b = 20;
> 
> while(a && b)
> {
> do_something();
> }
> 
> However, Go requires blloean values used with the logical AND operator ... 
> so my necessary code change for Go implementation becomes the following: 
> 
> var a,b int = 10,20
> 
> for (a != 0) && (b != 0) {
> do_something()
> }
> 
> Is doing things this way absolutely necessary? It seems a lot clunkier and 
> less elegant.

I disagree.  Go approach is a lot clearer.

Why 10 is true?  Why -42 is true?  Why a float NaN is true?

That can even boild down to reading the code as plain English.
"while a and b do something" is of dubious readability compared to
"while a is not zero and b is not zero do something".

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] A question about evaluation of channel’s send statement

[Konstantin Khomoutov]

On Wed, Sep 06, 2017 at 04:26:09AM -0700, T L wrote:

> > > It is just weird that the evaluation timing of *p is different to other 
> > expressions, such as, *p+0, *p*1, func()int{return *p}m etc. 
> >
> > The value depends on a data race so it's entirely undefined in all cases. 
> > That the actual outcome then depends on trivial differences that may affect 
> > CPU loads/stores, instruction ordering, register usage, etc isn't 
> > surprising. 
> 
> I don't think it is so complex. It is simply that gc adopts a different 
> route for evaluate pointer dereference.

It's not complex, it's a simple data race.

In your code

var num = 10
var p = 

c := make(chan int)

go func() {
c <- func()int{return *p}()
}()

time.Sleep(time.Second)
num++
fmt.Println(<-c)

two goroutines -- the main one and the one you spawned are accessing the
variable "num" concurrently without any synchronization.

Some assorted points to may be make this more clear:

* Sleeping for "just enough" time is not a form of synchronization.

* You might be driven away by using a function literal which is
  immediately executed but that's pretty much equivalent to just doing

c <- *p

  which supposedly makes the data race over the variable "num"
  even more obvious.

* You might keep false assumption that function literals producing
  function values close over values -- that is, they work sort of as
  code templates.  This is wrong: they close over variables, so in your
  example the function value you're constructing before running it on
  a separate goroutine closes over the variable "p", not the value
  obtained by dereferencing it at the time of producing that function
  value.


To explain what happens, step by step, the statement

  go func() {
  c <- func()int{return *p}()
  }()

in your example:

1) Creates a function value which is a closure consisting of the
   anonymous function

 func() {
   c <- func()int{return *p}()
 }

   and a reference to the variable "p" in the scope of the function
   main().

 2) Spawns a goroutine and tells it to execute the function value
created on the previous step.

 3) When run, the function executes the function value produced by
the expression on the right side of the channel send statement.

That function value is itself a closure, closing over the variable
"p" in the scope of its lexical parent function -- the closure
created on step 1.

The pointer gets dereferenced and this action accesses the memory
location "backing" the variable "num".
The same memory is accessed concurrently from the main goroutine.

Since there is no synchronization done for _this_ access, you have
a data race.  The fact you have a channel send/receive has nothing
to do with regard to accessing the memory of the "num" variable
since both goroutines do it before attempting its respective channel
send/receive operations.  During that time, they may have being run on
different hardware CPUs or one of the goroutines might have no chance to
run at all.  Running on different CPUs involves a whole lot of
interesting problems, of which the most glaring are reordering of memory
accesses and coherence problems of CPU caches.

Please read [1, 2] and [3] (which is written by one of the Go core devs).

1. http://preshing.com/20120515/memory-reordering-caught-in-the-act/
2. 
http://preshing.com/20120710/memory-barriers-are-like-source-control-operations/
3. 
https://software.intel.com/en-us/blogs/2013/01/06/benign-data-races-what-could-possibly-go-wrong

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: error handling needs syntactical sugar

[Konstantin Khomoutov]

On Mon, Sep 11, 2017 at 08:49:30PM +1200, Tim Uckun wrote:

> | However, what's great for avoiding straightforward failures becomes a
> | nightmare when your goal is to guarantee that no undefined behaviour
> | happens
> 
> It seems to me that if this is your goal other languages are more suitable.
> Erlang for example is famous for it's error handling capability,
> elm guarantees no runtime errors, and some other functional languages can
> be proven mathematically at compile time.

That's not what the author used the term "no undefined behaviour" for;
he referred to the simpler idea of having all possible code paths in the
program flow being handled.

When you have exceptions, you have two problems:

* When they are handled not right at the spot where they are generated, you
  - Lose track of the points where they may be generated;
  - Have hard time distinguishing between exceptions occured in
different places in the control flow (unless you use super-shallow
and super-wide type hierarchy or encode this information in the
exceptions).
* When new exceptions are added, you have to identify and modify all
  places where they're handled.

That's why typically "exception handling" indeed means having try/catch
somewhere at the top level and displaying an error message dialog box
(or a HTTP 500 Internal Error page) to the user which is not error
handling but "avoiding straightforward failures".

As to your particular examples, I'm not familiar with Elm but Erlang
stands out for two reasons:

- It has the so-called "pattern matching" which, coupled with
  Erlang's typelessness, allows to implement something which works like
  "algebraic types".  That is, it's easy to write code which asserts
  specific "shapes" and contents of the values at runtime, and crashes
  when these expectatiosn are not met.  While this is very neat, pretty
  much of this is covered by Go's static types.

- It implements the so-called "supervision trees" which allow you to
  construct hierarchies of "processes" in which parents get notified
  when the children crash, and policies for restarting those processes
  which crashed.

  While, again, being very neat, their ultimate usefullness is something
  disputable: when you have a panic in your Go code, this means you
  have a bug, and given the fact Go is a compiled language, this may
  mean your whole runtime is an unknown state.

  All in all, implementation of supervision trees for Go do exist.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: error handling needs syntactical sugar

[Konstantin Khomoutov]

On Wed, Sep 06, 2017 at 11:00:08PM -0700, Tim Uckun wrote:

> Totally not a go programmer but here are my worthless two cents.
> 
> I don't see anything wrong with the try catch paradigm, you can choose your 
> granularity. If you want to check every call then you can, if you want to 
> let a few cluster together you can. You guys make is sound like go style 
> error handling is the only way to catch errors at every step. That's not 
> true at all.  
[...]

Actually, I do think there are cases when you don't care which error has
actually happened, and where -- you just want to abort processing and
quit.  Unfortunately, this only works for "batch-style" processing, like
shell scripts -- where you want the whole thing to be done, or to fail
ASAP should any problem be detected (well, sometimes, though quite much
more rarely, you'd even want to merely log errors and continue to chug
away with the operations).

I have a goto writeup on exceptions bookmarked, which I invite you to
read [1].

The key point from it, which pretty much summarizes the idea behind how
error handing is done in Go, is:

| C++ exceptions … are great for guaranteeing that program doesn't
| fail — just wrap the main function in try/catch block and you can
| handle all the errors in a single place.
|
| However, what's great for avoiding straightforward failures becomes a
| nightmare when your goal is to guarantee that no undefined behaviour
| happens. The decoupling between raising of the exception and handling
| it, that makes avoiding failures so easy in C++, makes it virtually
| impossible to guarantee that the program never runs info undefined
| behaviour.

1. http://250bpm.com/blog:4

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] closing a channel to interrupt a send: racy, but incorrect?

[Konstantin Khomoutov]

[...]
> > AFAIK, all operations on a channel happen in-order and are fully 
> > serialized.  That is, close() will wait until the currently active 
> > operation 
> > is active, if any. 
> >
> > In other words, close() on a channel is not like closing an OS file 
> > descriptor -- even though the operation's names are the same. 
> >
> 
> 
> It looks like any send happens after the channel mutex is grabbed here
> https://github.com/golang/go/blob/master/src/runtime/chan.go#L181
> 
> and the release of senders on close also happens after grabbing the channel 
> mutex here
> https://github.com/golang/go/blob/master/src/runtime/chan.go#L330
> 
> so at the moment this appears to be safe.
> 
> I think it might be worth proposing an addition that the language spec (Go 
> 1 or Go 2) to guarantee this is safe on into the future.
> Unless that isn't necessary because some part of the spec already implies 
> this?

Currently the spec states:

| A single channel may be used in send statements, receive operations, and
| calls to the built-in functions cap and len by any number of goroutines
| without further synchronization.

Books and tutorials on Go typically state something like "close() is a send
operation" -- meaning that closing a channel unblocks all the goroutines
currently being blocked on it -- thus clearly having "send" semantics.
There even exist an idiom of using a channel for broadcasting a single
event to a number of goroutines.

Still, I don't see any specific wording regarding this in the spec --
neither in its section on channels nor in its section on the close()
function.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] closing a channel to interrupt a send: racy, but incorrect?

[Konstantin Khomoutov]

On Thu, Aug 31, 2017 at 05:08:23AM -0700, Jason E. Aten wrote:

> A question for those familiar with the runtime internals of channel close 
> and channel receive:
[...]
> Will closing a channel in the middle of a send introduce the possibility of
> data corruption?  The language spec guarantees a panic, and that panics
> can be recovered, but would a *reader* of ch above ever see incorrect data?

AFAIK, all operations on a channel happen in-order and are fully
serialized.  That is, close() will wait until the currently active operation
is active, if any.

In other words, close() on a channel is not like closing an OS file
descriptor -- even though the operation's names are the same.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Building gopath packages

[Konstantin Khomoutov]

> > See github.com/rjeczalik/gobin for updating, but a simple
> > "go install $GOPATH/src/..." may be enough.
> 
> The URL appears to 404.  There are multiple projects called "gobin" on
> Github but neither appear to be concerned with the task we're
> discussing, unfortunately.

Citing the answer Tamás gave to me, the correct URL for `go get` is [1],
and the project itself is [2].

1. github.com/rjeczalik/bin/cmd/gobin
2. https://github.com/rjeczalik/bin/

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Building gopath packages

[Konstantin Khomoutov]

On Wed, Aug 30, 2017 at 10:35:46PM -0700, Tamás Gulácsi wrote:

> See github.com/rjeczalik/gobin for updating, but a simple
> "go install $GOPATH/src/..." may be enough.

The URL appears to 404.  There are multiple projects called "gobin" on
Github but neither appear to be concerned with the task we're
discussing, unfortunately.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] english language help

[Konstantin Khomoutov]

On Wed, Aug 30, 2017 at 07:17:26PM +0600, Oleg Puchinin wrote:

> Hello !
> Anybody speak russian ?
> I work on "Jay" - indexer for Go language (for LiteIDE). Can anybody speak
> my this theme in russian language ?
> 
> Last link:
> svn checkout --username=olegpuchinin svn+ssh://
> olegpuchi...@svn.code.sf.net/p/opjay/code/ opjay-code

Как насчёт https://groups.google.com/d/forum/golang-ru ? ;-)

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: Convert grid of colors to SVG

[Konstantin Khomoutov]

On Mon, Aug 28, 2017 at 06:35:25AM -0700, ajstarks wrote:

> it would be helpful to better understand what you want to do, but:
> 
> https://speakerdeck.com/ajstarks/svgo-code-plus-picture-examples
> 
> includes the code below (with many other examples would should be 
> illustrate the capabilities of the package)

It's pretty much impressive!
Thanks for sharing.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] ANN: xcryptossh: an ssh library with per channel timeouts and context based cancellation

[Konstantin Khomoutov]

On Sat, Aug 26, 2017 at 04:47:04PM -0700, Jason E. Aten wrote:

> https://github.com/glycerine/xcryptossh
> 
> This is an evolution of golang.org/x/crypto/ssh to fix memory leaks, 
> provide for graceful shutdown, and implement idle timeouts for each 
> multiplexed channel. It is not API backwards compatible because it provides 
> context.Context based cancelation.
> 
> Its new so I still consider it experimental. Please try it out and leave 
> feedback on github.

Do you plan intergating this upstream (into golang.org/x/crypto/ssh)?
Or was this proposed upstream and rejected?  (If yes, do you care to
provide links to the relevant discussions, if any)?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Using pprof on publicly accessible servers

[Konstantin Khomoutov]

On Mon, Aug 28, 2017 at 08:25:05AM -0700, st ov wrote:

> It's recommended to add pprof to servers
> 
> import _ "net/http/pprof"
> 
> Is this true even for publicly exposed servers? How do you limit access?

Either through your custom ServeMux or by reverse-proxying to your Go
server via something which has configurable access rules.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Custom ServeMux and pprof

[Konstantin Khomoutov]

On Mon, Aug 28, 2017 at 08:31:11AM -0700, st ov wrote:

> How do you add pprof to a custom ServeMux?
> 
> https://golang.org/pkg/net/http/#ServeMux
> 
> Its my understanding that the following import only adds handlers to the 
> default mux.
> 
> import _ "net/http/pprof"

https://blog.cloudflare.com/exposing-go-on-the-internet/#servemux

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Convert grid of colors to SVG

[Konstantin Khomoutov]

On Fri, Aug 25, 2017 at 10:18:17AM -0700, steve_bagw...@sil.org wrote:

> > There's svgo: https://github.com/ajstarks/svgo 
[...]
> Thanks Andrey,
> SVGO didn't seem to include that particular functionality. Maybe I just 
> didn't look carefully enough?

So, this begs the question: why producing yet another narrow-scoped
library when there exist another one which has pretty much of stuff
already packed in?

Is it possible for you to came in touch with the svgo maintainers and
discuss integrating your code into that library?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Help! Same code, different results

[Konstantin Khomoutov]

On Fri, Aug 25, 2017 at 08:39:29AM -0700, Tong Sun wrote:

> I'm experiencing a *very very* strange problem now -- the same Go code is 
> producing different results *for me*. 
[...]
> Can someone verify for me what you get please? 
> 
> go get github.com/go-dedup/fsimilar
> 
> then 
> 
> cd go-dedup/fsimilar
> go build
> find test/sim -type f | ./fsimilar -i -d 12 -vv 
> 
> and tell me what's the last output that you got please. 
> 
> The problem is that *two *of my machines produce:
> 
> [fsimilar] ## Similar items
>  map[Similars:[map[Hash:6184610222622303958 Dist:0 SizeRef:1 Name:GNU - 
> 2001 - Python Standard Library Ext:.pdf Size:1 Dir:test/sim/] map[Name:GNU 
> - Python Standard Library (2001) Ext:.rar Size:1 Dir:test/sim/ 
> Hash:6184610222622303958 Dist:0 SizeRef:1]]].
> *test/sim/GNU - 2001 - Python Standard Library.pdf*
> *test/sim/GNU - Python Standard Library (2001).rar*
> 
> But another one, the *only one*, produce:
[...]

Two quick points:

* What happens if you copy the "working" binary to the "wrong" machine
  and run it there?

  Does it work?

  I mean, you should rule out possible inconsistencies with build
  systems.  This means both Go and all the graph of the libraries
  your project uses.  If you're building on each machine, try doing
  clean-room building.  This means freshly cloning the Go source code
  of, say, the 1.8 release branch and building it.  Then copying over
  the whole hierarchy of the whole branch of the dependencies from the
  "working" machine into a new directory which must be a Go workspace.
  Then building there.

* This bit [1] smells bad IMO.
  
  From the quick glance I failed to see where it's used but
  why it's there?  What happens if you seed PRNG with a constant?

1. https://github.com/go-dedup/fsimilar/blob/master/fsimilarCLICmd.go#L70

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: Go 1.9 is released

[Konstantin Khomoutov]

On Fri, Aug 25, 2017 at 02:05:52AM -0700, Igor Maznitsa wrote:

> and why the SDK  is not presented in the list 
> https://storage.googleapis.com/golang/ ?
> also under Ubuntu 16.04 LTS I have some error for any attempt to start Go 
> 1.9 from downloaded SDK : 
> /lib/ld-linux-aarch64.so.1: No such file or directory
> I don't have any such error with the 1.8.3, is it normal situation now?

The package search [1] turns up this file is provided by libc6 (for arm64).

So do you really using arm64 (not amd64) and have libc6 installed?
(I doubt the second as this package should be essential on
Debian-derived distros but the former might be the culprit.

1.  
https://packages.ubuntu.com/search?searchon=contents=ld-linux-aarch64.so.1==xenial=any

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Understanding "scheduler latency profile" charts

[Konstantin Khomoutov]

Hi!

We're trying to investigate a case of pathological behaviour of one of
one HTTP services (built with Go 1.8) under heavy load.

The current hypothesis is that under the load being considered,
the scheduler (or the set of available Ps) have hard time serving that
load.  We came to this conclusion by recording the runtime trace and
then analyzing the delays between the pairs of linked events "a goroutine
is unblocked by the netpoller" and "that goroutine starts executing"
(these delays in our case peak to as much as 3-5 seconds).

To verify our observations, we've decided to try looking at the
"scheduler latency profile" which can be obtained from the runtime
trace -- something like:

  $ go tool trace -pprof=sched runtime.trace >sched.pprof
  $ go tool pprof -evince https://groups.google.com/d/optout.


sched.dot.gz
Description: application/gzip


sched.svg.gz
Description: application/gzip

Re: [go-nuts] Initializing Go Struct Literals

[Konstantin Khomoutov]

On Tue, Aug 22, 2017 at 07:38:03AM -0700, Tong Sun wrote:

> How to initialize a go struct like the following?
> 
>  type Server struct {
>  Namestring
>  ID  int32
>  Enabled bool
>  }

This type definition looks pretty much OK.

>  s := struct {
> Servers []Server{
>   {
> Name:"Arslan",
> ID:  123456,
> Enabled: true,
>   },
>   {
> Name:"Arslan",
> ID:  123456,
> Enabled: true,
>   },
> }
>   }

...and this is a very strange idea: struct types contain fixed number
of named fields, so

  s := struct{
// what's this?
  }

> That didn't work so I tried to introduce a new type to capture it: 
> 
>  type Server struct {
>  Namestring
>  ID  int32
>  Enabled bool
>  }
>  type Servers struct {
>  servers []Server
>  }

This is possible but arguably don't needed.
If all you need is merely a slice of Server instances,
just use it.

>  s := {servers: []Server{
>   {
> Name:"Arslan",
> ID:  123456,
> Enabled: true,
>   },
>   {
> Name:"Arslan",
> ID:  123456,
> Enabled: true,
>   },
> }
> 
> but that failed also. 
> 
> What's the correct way?

In the simplest case:

  s := []Server{
Server{
  Name:"Arslan",
  ID:  123456,
  Enabled: true,
},
...
  }

If you need your nested structs, then

  s := Servers{
servers: []Server{
  Server{
Name:"Arslan",
ID:  123456,
Enabled: true,
  },
  ...
}
  }

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Testing that an implementation satisfies an interface in the standard library

[Konstantin Khomoutov]

On Mon, Aug 21, 2017 at 10:05:12PM +0200, 'Axel Wagner' via golang-nuts wrote:

> > Satisfying a standard interface like io.Reader or io.Writer correctly is
> > not just a matter matching the function signatures, but rather adhering to
> > behaviour that is documented in comments
> > 
> >
> > Is there a standard way for me to specify an implementation of such an
> > interface and run an existing suite of tests against it that verify the
> > behaviour is indeed as expected?
> >
> > So, in other words, does something like testing.TestBehaviour(
> > myReader,io.Reader) or testing.SatisfiesIoReader(myReader) exist?
> >
> > To clarify, I'm not asking how to write tests that verify the
> > implementation of a standard interface.
> > Rather, I want to know if there is a way I can reuse tests that already
> > exist for an implementation that I wrote?
> I think in general that will be very hard to do, generically. The reason
> is, that the precise edge cases in which the behavior is critical depend so
> much on your implementation.
> 
> The best I can think of would be to write a wrapper which checks the
> invariants given and run your implementation-specific testsuite using that.
> Or to fuzz it.
> 
> I'm aware that this doesn't answer your question (it's hard to prove a
> negative and AFAIK the answer is "no"), but maybe it explains why the
> problem is non-trivial. FTR, I would be interested in something like that
> too.

I'm afraid, there are no such test exposed to the general audience, but
the standard package testing/iotest [1] might be of at least some help.

1. https://golang.org/pkg/testing/iotest/

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Why constant is not addressable in golang?

[Konstantin Khomoutov]

On Mon, Aug 21, 2017 at 12:56:19PM +, Jan Mercl wrote:

>> That is
>>
>> const onethird = 1 / 3
>>
>> has greater precision than
>>
>> onethird := float64(1) / 3
> 
> Zero does not seem to have greater precision than 0. when
> approximating the real number 1/3.

IMHO, the less smart-ass responses the list would see, the better.

But I see your point, thanks.

To the OP: the first statement should have been

  const onethird = 1.0 / 3

instead.

By the way, currently defined limits the compliant Go compilers must
support with regard to constants, can be found in [1].

1. https://golang.org/ref/spec#Constants

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Why constant is not addressable in golang?

[Konstantin Khomoutov]

On Sun, Aug 20, 2017 at 10:23:27PM -0700, chou wrote:

> I guess that const will be substitute literally at compile time
> for performance improvements. So there is not such variable in
> run time.
> 
> Is it right?

One of the reasons for this is that constants in Go have properties
which set them apart from values:

- For numerics, they are allowed (and enforced to a certain extent)
  to have size and/or precision way larger than of the regular types
  supported by the language.

  That is

const onethird = 1 / 3

  has greater precision than

onethird := float64(1) / 3

  and this may affect calculations done on floating-point numbers.
  See for instance [2] for a recent example on why this matters.
  [3] is an in-depth explanation of those effects, and [4] is a gentler
  one.

- Constant are "almost typeless": they have default type for their
  literal value (say, int for the literal 42 or string for the literal
  "42") but the compiler allows a much greater freedom when you assign
  a constant to a variable (which has exact type).

  Say, in Go, it's impossible to do

i := 0
uint32 j := i

  but it's perfectly valid to do

const i = 0
uint32 j := i

As you can see, having these properties appears to be incompatible
with actually storing the values of the constants in some R/O memory
and making them addressable.

Please read [1] to get full grasp on constants in Go.

1. https://blog.golang.org/constants
2. https://groups.google.com/d/topic/golang-nuts/XP-_nRWjKnM/discussion
3. http://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html
4. http://floating-point-gui.de/

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] WaitGroup should be introduced before "Exercise: Web Crawler" in "A Tour of GO"

[Konstantin Khomoutov]

On Sun, Aug 20, 2017 at 08:20:08AM -0700, 823545...@qq.com wrote:

> I'm a freshman. I went through the Tour of GO smoothly today, until I 
> encountered "Exercise: Web Crawler". The exercise is at here 
> https://tour.golang.org/concurrency/10 .
> This exercise requires me to implement a concurrent crawler. I write a main 
> function like this:
>  
> 
> > func main() {
> > go Crawl("http://golang.org/;, 4, fetcher)
> > }
> 
> 
> Yes, the program terminated just after it created the goroutine, nothing 
> was printed. I thought the program would wait for all goroutines to finish 
> without explicit coding, but it didn't. The canonical way to do this is to 
> use WaitGroup, but the tutorial omitted it. I think it should be added 
> before this exercise.

Supposedly the authors implied you'd use a channel to have individual
crawlers run on separate goroutines to communicate the results of their
work back to the controlling goroutine (the main one) via a channel.

(I also think the "Don't fetch the same URL twice" invariant is supposed
to be kept through using the sync.Mutex introduced in exercise 9).

The question of where to use sync.WaitGroup and where to synchronize
using channels (or sync.Cond or whatever else) is both technical and
philosophical.  Dumping it right onto the heads of freshmen would be
nonsensical, and since channels are viewed as the most natural and
elegant way of making the concurrent workers synchronize their work,
that's what typically gets taught first.

I heartily recommend you to read [1] and then [2], in this order.
[1] is one of the best-written books on CS matters I ever read, and it
introduces concurrency in Go pretty gently.
[2] is more hard-core (as it focuses precisely on concurrency matters),
it's less smooth in presenting its material than the first book, but it
gets its job done well and is definitely worth its price IMO.

1. http://www.gopl.io/
2. http://katherine.cox-buday.com/concurrency-in-go/

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Segmentation fault when linking with Go library from a C++ program

[Konstantin Khomoutov]

On Fri, Aug 18, 2017 at 04:21:27AM -0700, ine...@gmail.com wrote:
> I have a C++ application from which I need to call Go functions. I noticed 
> that the application crashes
> on exit from the `main` function (with only a "Segmentation fault" message, 
> no core dump

Not to answer your question, but these days core dumping is usually
disabled by default through the "soft" limit, but it's possible to
enable it, say, by executing

  ulimit -c unlimited

before starting the problem.

[...]

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Domain-driven design and go

[Konstantin Khomoutov]

On Fri, Aug 18, 2017 at 03:29:01AM -0700, Hugo Torres wrote:

> Recently I've been reading "Domain Driven Design" and I think it has some 
> useful stuff about organizing Go programs.
> 
> I wrote some of my thoughts up on my blog here 
> .
> 
> Would love to know what everybody thinks.

I'm not sure I got the point from reading.

What you have labelled as "Aggregates" is just a conventional
common-sense approach to structuring data: of course, a car "contains"
(hence "aggregates") its wheels.

What was labelled "Factories" are the standard ways to produce values of
types which do not have useful zero-values or just are convenient for
one reason or another.  Examples of what you call "factories" are right
there in the go-to documents for Go beginners [1, 2].

What was labelled "Repositories" is a pretty standard way of decoupling
disparate parts of code by using an interface.  Interfaces naturally
decouple the code which only relies on some behaviour from the code
which defines types implementing that behaviour.

I'm afraid you might be too dragged away by that shallow-water
theoretisation which appear to be done by this book.
As someone commented on Amazon, «Another meta-methodology book filled
with self-evident observations normally learned on the job. A lot of it
is just common sense.» — based on your blog post, it sounds just about
correct, sorry.

1. https://blog.golang.org/package-names
2. https://golang.org/doc/effective_go.html

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] How do you debug Go itself on VM env or Docker container?

[Konstantin Khomoutov]

On Fri, Aug 18, 2017 at 12:26:02AM -0700, 高橋誠二 wrote:

> I'm trying to debug and run `./all.bash` with VM or docker, but they don't 
> work for several reasons.
> 
> On VM, I added box of CentOS and Ubuntu, though, they cancels ./all.bash at 
> `hardlink` testing.

The "VM" is too vague a term which these days may mean a whole lot of
very different things.

I'm pretty confident that if we define "a VM" to mean a fully
virtualized machine running on any popular host such as KVM or
VirtualBox, hardlinks must work there just OK -- of course, provided
you've installed the guest OS on a POSIX-compliant FS (such as ext4 for
the case of CentOS/Ubuntu).

I'm also pretty confident such stuff should work just OK in an LXC
container -- again provided the host os (this time) uses a
POSIX-compatible FS which is the case most of the time.

In any case, the exact error message (with some textual context around
it) could help us try to light more shed on the source of the problem.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] The memory release about Turning C arrays into Go slices

[Konstantin Khomoutov]

On Fri, Aug 18, 2017 at 10:39:29AM +0300, Konstantin Khomoutov wrote:

[...]
> There can't be easy answer to a question like this in C.
> Consider:
> 
>   // This function clearly transfers the ownership of the memory
>   // it returns to the caller.
>   int* allocate(int n)
>   {
> int *mem = (int*)malloc(n * sizeof(int));
> if (mem == NULL) {
>   panic("Out of memory");
> }
>   }

Sorry, I've missed the crucial return statement; the code should have
instead been:

   // This function clearly transfers the ownership of the memory
   // it returns to the caller.
   int* allocate(int n)
   {
 int *mem = (int*)malloc(n * sizeof(int));
 if (mem == NULL) {
   panic("Out of memory");
 }
 return mem;
   }

[...]

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] The memory release about Turning C arrays into Go slices

[Konstantin Khomoutov]

On Thu, Aug 17, 2017 at 06:50:42PM -0700, jianzhang...@gmail.com wrote:

> > > As the instruction 
> > > of https://github.com/golang/go/wiki/cgo#turning-c-arrays-into-go-slices 
> > > shows, the GC of Go will not release it. 
> > > An example as the following, but I have a question about when and how to 
> > > release this `slice`? 
> >
> > You shouldn't: the GC won't touch the memory it does not "own" so it 
> > will eventually collect the slice structure itself but won't do anything 
> > to its underlying array (which value you obtained via that double 
> > type-conversion). 
> >
> > You should supposedly be instead concerned with freeing the array's 
> > memory (via calling C.free or some dedicated function to do this 
> > provided by your C side, if any).  As always with C, you should only do 
> > that if that getTheArray() function's semantics are such that its caller 
> > "owns" the memory the function returns. 
[...]
> Thanks your reply, you mean I should be release the memory of `theCArray` 
> in C side?
> I still have some doubts, and I made a clearer example as the below, any 
> problem with it?

I see no reason to defer calls to C.free().
defer is a costly operation, and it should be only used if the resource
the deferred call is intended to free can be used in arbitrary places
down the code paths after it's allocated; that's not the case there.

> So, the memory of this tmpslice should be release by release the argv on C 
> side?

Yes and no.

Yes, if that memory was produced by a function whose semantics are
"the caller is responsible to freeing the memory of the data I
returned", and no otherwise.

There can't be easy answer to a question like this in C.
Consider:

  // This function clearly transfers the ownership of the memory
  // it returns to the caller.
  int* allocate(int n)
  {
int *mem = (int*)malloc(n * sizeof(int));
if (mem == NULL) {
  panic("Out of memory");
}
  }

  // But this one doesn't.
  int* find_first_non_zero(int *p, int count)
  {
for (i = 0; i < count; i++) {
  if p[i] > 0 {
return [i];
  }
}
return NULL;
  }

I tried to highlight that even though the latter function returns a
pointer to memory, its caller is not supposed to do anything with that
pointer (other than supposedly checking it for being non-NULL and
dereferencing otherwise).

You should apply the same logic to the code on your C side: if the
function which is used to obtain the value for the "argv" argument
of GoStrings() actually allocates the memory and supposes its caller has
to deallocate it theirselves, go on and free it.  Otherwise don't.

> func GoStrings(length int, argv **C.char) []string {
> 
> tmpslice := (*[1 << 30]*C.char)(unsafe.Pointer(argv))[:length:length
> ]
> gostrings := make([]string, length)
> for i, s := range tmpslice {
> gostrings[i] = C.GoString(s)
> defer C.free(unsafe.Pointer(s))
> }
> return gostrings
> }
> 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Calling Julia from Go

[Konstantin Khomoutov]

On Wed, Aug 16, 2017 at 11:57:51PM -0700, mrech...@gmail.com wrote:

> I would like to implement a valuation server. For the whole server 
> infrastructure I would like to use Go, because it feels more natural than 
> any other language for it. For the implementation of the valuations I would 
> like to use Julia. Thus, using the aforementioned languages where (IMHO) 
> their strengths are. Googling turned up nothing so far except for an old 
> post on this list suggesting that it is indeed not too easy: 
> https://groups.google.com/d/msg/golang-nuts/Tl7IMu6UnZ4/PBqA_v4rZhEJ
> Is anyone aware of an existing solution for embedding Julia into Go?

I'm not familiar with Julia, but I'd offer two possible points to
consider:

* If Julia code can be compiled to a library exposing C interface
  (as Go makes possible on some platforms with the compiler's option
  -buildmode=c-archive), you can do just that and the call it from Go
  via the standard Go's cgo facility.

* You may consider an often overlooked approach to just not embed
  anything to anything else and instead employ IPC (inter-process
  communication).

  Say, you could possibly compile your Julia program
  so that it reads data in certain agreed-upon format on its stdin
  and writes the results of processing them to its stdout.

  Your Go server then merely starts your Julia program and wires two
  pipes to those standard I/O streams of the process it created.
  Communication then becomes a matter of writing stuff to one of those
  pipes and then reading the result from another one.

  If your Julia server need to be multithreaded, it's possible to
  use "job identifiers" with each task submitted to that Julia process;
  this would allow the latter to perform the jobs in parallel and
  use those job IDs in its answers which then could come out the result
  pipe in any order.

Certainly the latter approach is slower than the first but it's
definitely easier to implement and to reason about, and it fully
decouples both programs.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] studying golang code

[Konstantin Khomoutov]

On Tue, Aug 15, 2017 at 02:42:45PM -0700, Keith Brown wrote:

[...]
> Now, as I learn golang, are there any worth while projects I can use as 
> reference for writing high quality go code? I am not necessary looking for 
> code standards but more of code setup and quality so I can practice those 
> principles.

The Go standard library.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Receiving Post Using Go

[Konstantin Khomoutov]

On Mon, Aug 14, 2017 at 05:06:08AM -0700, Kennedy Kanyi wrote:

> I would want to inquire how I can receive POST data from a URL using Golang.
> 
> A case study would be where a system sends a username and password via POST 
> and my Go receiving file reads the post and the values.

In addition to what Shawn said, note that "a system sends" is too vague.

The problem is that the HTTP POST method basically defines that the
sender streams some data after the HTTP header; the format of this data
is not defined by the method. And hence it's possible to send data in
absolutely any format using HTTP POST.

In the simplest and oldest form, when a browser "posts" a so-called HTML
form from an HTML document it presented to the user, it formats the data
either using the application/x-www-form-urlencoded format (for simple
forms) or multipart/form-data (when a file upload is used).

But these days, when the so-called "single page web applications" are a
norm, it's customary to send payloads in POST requests formatted using
more "common" approaches such as JSON.

So you should clearly figure out for yourself what generates your data
and what format it has.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] ungetc behavior in go

[Konstantin Khomoutov]

On Mon, Aug 14, 2017 at 05:43:04AM -0700, Doğan Kurt wrote:

> > Consider combining using the bufio package and scanning form a buffered 
> > reader using fmt.Fscanf(). 
> 
> Great worked like a charm. I just thought i couldn't pass *bufio.Reader to 
> Fscanf which takes io.Reader. 

Oh, then that's something which you definitely should make yourself
accustomed more -- this consept is one of the corner stones of Go.

Basically, interfaces in Go are sort-of "backwards" to what they mean in
other, "more mainstream" languages.  The fact fmt.Fscanf() accepts an
io.Reader means it accepts a value of absolulely any type which
implements func Read ([]byte) (int, error).  So it would accept, say, a
network socket just as fine as a bufio.Reader or bytes.Buffer or any
other custom type implementing that method.

[1] might be a good start.

1. https://golang.org/doc/effective_go.html#interfaces_and_types

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] ungetc behavior in go

[Konstantin Khomoutov]

On Mon, Aug 14, 2017 at 05:09:00AM -0700, Doğan Kurt wrote:

> I want to do something analogous to this C code.
> 
> c = getchar()
> 
> if (isdigit(c)) {
> ungetc(c, stdin);
> scanf("%d", );
> }
> 
> In fmt package, fmt.Scanf("%d", ) exists but ungetc doesn't exist. For 
> bufio package, Peek function exists which is great but there is no ReadInt. 

Consider combining using the bufio package and scanning form a buffered
reader using fmt.Fscanf().

Please see the documentation on the fmt package and notice that the
function it exports come in packs where the first letter of the name of
each function in such a pack indicates ...

For instance:
- Print() just prints to os.Stdout, Printf() is "print formatted".
- Fprint() is "file print" -- it outputs to the specified io.Writer.
- Sprint() is "string print" -- it outputs to a string.

The pack of functions to do scanning roughly conforms to the same idea:
- Scan() scans from os.Stdin.
- Scanf() is "scan formatted".
- Fscan() scans from the specified io.Reader.
- Sscan() scans from a string.
...

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] how can I decrypt "ENCRYPTED PRIVATE KEY" using golang

[Konstantin Khomoutov]

On Fri, Aug 11, 2017 at 12:30:56AM -0700, terry.an...@gmail.com wrote:

> My private key is generated by "openssl req -new -x509 -keyout a.key -out 
> a.crt -days 3650" with password.
> 
> And I tried to decrypted it by x509.DecryptPEMBlock(keyBlock, password), 
> but failure.  error message: %!(EXTRA *errors.errorString=x509: no DEK-Info 
> header in block)
> 
> my privatekey is in below, and password is '123456'
> 
> anyone else can help me ? 
> 
> -BEGIN ENCRYPTED PRIVATE KEY-
> MIIFDjBABgkqhkiG9w0BBQ0wMzAbBgkqhkiG9w0BBQwwDgQIQgjjAaLRMbsCAggA
> MBQGCCqGSIb3DQMHBAi4V74SzJ52kwSCBMiZ7gONJxBWnJsqaiyhDBvLrMQW5c/G
[...]

As you can see in [1], typically a PEM-encoded encrypted key indeed
contains a header block detailing how it was encoded, like this:

  -BEGIN RSA PRIVATE KEY-
  Proc-Type: 4,ENCRYPTED
  DEK-Info: DES-EDE3-CBC,02306CD22AAC94CD
  ...

As you can gather from the accepted answer in [1], `openssl req`
uses DES-EDE3-CBC.

I don't know what to do but I see two ways to attack this:

* Decode the PEM (it's just a base64 encoding of a DER-encoded stream
  IIRC) and then try decoding it with some API function which allows
  specifying the encryption type directly.

* Try monkey-patching the PEM data by a bogus DEK-Info header.

1. https://security.stackexchange.com/q/93417

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Check for minimum Go version

[Konstantin Khomoutov]

On Wed, Aug 09, 2017 at 03:11:48PM +0200, Michael Banzon wrote:

> Is there a way to have a (bash) script check if the version of the Go
> compiler installed is a specific minimum version?

In the light of [1], I think you could combine checking of the presense
of the `go` tool itself with build tags.
Say, something like this (untested):

>8
  set -eu
  
  rc=0
  go version >/dev/null || rc=$?
  if [ $rc -ne 0 ]; then
echo "The 'go' tool is not available" >&2
exit 2
  fi
  
  d=`mktemp -d`
  f=`mktemp`
  trap "rm -rf '$d' '$f'" EXIT INT TERM QUIT
  cd "$d"
  
  cat >false.go <<'EOF'
  // +build !go1.7
  package main
  
  import "os"
  
  func main() {
os.Exit(1)
  }
  EOF
  
  cat >true.go <<'EOF'
  // +build go1.7
  package main
  
  import "os"
  
  func main() {
os.Exit(0)
  }
  EOF
  
  go build -o "$f" "$d/*.go"
  
  rc=0
  "$f" || rc=1
  
  if [ $rc -ne 0 ]; then
echo "Insufficient Go version" >&2
exit 2
  fi
  
  exit 0
>8

One possible caveat is that `mktemp` by default creates its filesystem
entries under $TMPDIR or /tmp, and that directory might be mounted with
"noexec" on certain systems.  So if your setup script (or whatever it
is) has a luxury of using its own scratch space, you should probably do
that (like creating a temp. directory using `mktemp -f ~/.cache/XX`
or something like this).

1. https://github.com/golang/go/issues/21207

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Can warnings of race detector be ignored and toleranted?

[Konstantin Khomoutov]

On Tue, Aug 08, 2017 at 07:52:19PM -0700, Cholerae Hu wrote:

> Some of my colleagues think that, in some cases, such as approximately 
> counting the sum total of requests, we don't need to know the accurate 
> value, so we can let several goroutines to access one variable without lock 
> or atomic operation to get the highest performance, and we can ignore the 
> warning of race detector in these cases. Is that true?

Using atomic.AddInt64() won't really affect your performance.
If it will, you should probably consider switching from Go to assembly.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: Go channels overused and hyped?

[Konstantin Khomoutov]

On Tue, Aug 08, 2017 at 03:39:42AM -0700, snmed wrote:

> > There are trade-offs.
> >
> > Channels are easy to use for simple things, but complicated for complected 
> > things.
> >
> > Locking data-structures can easily introduce data-races (see The Little 
> > Book of Semaphores http://greenteapress.com/wp/semaphores/).
[...]
> Thank you for your reply, I myself used channel as a semaphore, i'm not 
> sure if that is a appropriate use case for channels.
[...]

Channels are quite good to implement simple counting semaphores (those
which have more than a single token / permit to provide): the length of
the channel used in this way is the quantity of permits the semaphore
has, a send to such channel is acquisition of a permit, and receiving
from it returning of the acquired permit back to the semaphore.

Acquisition hence naturally blocks if the semaphore has no free permits
and blocks the requesting goroutine until a free permit becomes
available.

You can easily combine this with a timer to get a counting semaphore
with a timeout.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] [CGO] how to pass a 2d slice to C?

[Konstantin Khomoutov]

On Mon, Aug 07, 2017 at 08:20:01PM -0700, jianzhang...@gmail.com wrote:

[...]
> > > > > golevelmatrix := [][]int{{1}, {3, 3}, {3, 3, 2}} 
> > > > >levelmatrix := make([][]C.int, len(golevelmatrix)) 
> > > > >for i, _ := range golevelmatrix { 
> > > > >levelmatrix[i] = make([]C.int, len(golevelmatrix[i])) 
> > > > >for j, _ := range golevelmatrix[i] { 
> > > > >levelmatrix[i][j] = C.int(golevelmatrix[i][j]) 
> > > > >} 
> > > > >} 
> > [...] 
> >
> > I'd like to reiterate more precisely: your client is expecting a 
> > multi-dimensional array which, in C, would be a contiguous region of 
> > memory.  The Go's [][]C.int is a slice of individual []C.int slices. 
> > Hence when you pass [0][0] to your C client, it receives the 
> > address of the first element of the first slice, and the only valid 
> > region of memory to access via that pointer is that element and all the 
> > element following it up to (but not including) the length of the first 
> > slice. 
[...]
> But, if I can not change the C client, how to implement it? the C client as 
> the following. As described on the above, I have to change the ` int**
>  _levelmatrix` argument, right?
> bool test_settopologyresource(bsagomodule* _self, char* _nodename,
>int _resourceindex, char** _nameunits, int** _levelmatrix) {
>printf("\n set topology resource-> node:%s, resource index:%d", _nodename
> , _resourceindex);
>bool result = settopologyresource(_self->client, _nodename, 
> _resourceindex, _nameunits, _levelmatrix);
>return result;
> }

Ah, I see.

So, do I understand correctly that your C side is actually expecting
an "array of arrays" (or "jagged array" as some call them)?
That would explain the int** type your function expects.

If yes, this is doable without changing the C side -- by C.malloc()'ing
each member array and storing the pointer to it in the main array:

  package main
  
  import "unsafe"
  
  /*
  extern int crunch_matrix(int **a);
  */
  import "C"
  
  func main() {
golevelmatrix := [][]int{{1}, {3, 3}, {3, 3, 2}}
  
matrix := make([]*C.int, len(golevelmatrix))
for i, row := range golevelmatrix {
p := (*C.int)(C.malloc(C.size_t(C.sizeof_int * len(row
matrix[i] = p
  
pa := (*[1 << 30]C.int)(unsafe.Pointer(p))
for j, v := range row {
(*pa)[j] = C.int(v)
}
}
C.crunch_matrix([0])
  }

Here, for each row of the source matrix we allocate an array of C.int of
the source row's size, and place a pointer to it into the destination
slice.

To fill the allocated row array with the source values without jumping
through the hoops of direct pointer arithmetics, we type-convert the
value of "p" (which has type *C.int) to the type *[1 << 30]C.int which
is a pointer to a very long array of C.int (read more on this in the
section "Turning C arrays into Go slices" in [1]).

Note two problems with this approach.

First, I made no effort to free the memory obtained by calls to
C.malloc().  That's mostly for brewity, but see below.

Second, as the code stands now, as soon as we allocate an array for the
target row, the information of its size is lost.  We can safely pass the
pointer to it to C.free (because C.malloc stores a special block of
information along with each block it allocates which contains the length
of that block) but I fail to see how your C side is aware of the length
of each row array.  On the Go side, the matrix is a slice of slices, and
since a Go slice is fully self-aware, that's OK but in C, an array is
merely a pointer, and it carries no information of the length of the
memory block pointed to by that pointer.

Hence, unless I fail to spot where you encode this information in
the arguments supplied to test_settopologyresource(), this might
indicate a problem with your design of your C side: you might need to
either use a custom type for the rows of your C matrix (a struct
containing a pointer and a size, for instance) or operate on full square
matrices as in my first example.

1. https://github.com/golang/go/wiki/cgo

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] [CGO] how to pass a 2d slice to C?

[Konstantin Khomoutov]

On Mon, Aug 07, 2017 at 06:25:48AM -0700, jianzhang...@gmail.com wrote:

> Thank you very much for your patience and help. I got it and will try it 
> later. :)

Glad to help!

> > > golevelmatrix := [][]int{{1}, {3, 3}, {3, 3, 2}} 
> > >levelmatrix := make([][]C.int, len(golevelmatrix)) 
> > >for i, _ := range golevelmatrix { 
> > >levelmatrix[i] = make([]C.int, len(golevelmatrix[i])) 
> > >for j, _ := range golevelmatrix[i] { 
> > >levelmatrix[i][j] = C.int(golevelmatrix[i][j]) 
> > >} 
> > >} 
[...]

I'd like to reiterate more precisely: your client is expecting a
multi-dimensional array which, in C, would be a contiguous region of
memory.  The Go's [][]C.int is a slice of individual []C.int slices.
Hence when you pass [0][0] to your C client, it receives the
address of the first element of the first slice, and the only valid
region of memory to access via that pointer is that element and all the
element following it up to (but not including) the length of the first slice.

As soon as the C client attempts to access any memory region other than
that, it may read/write random memory and even invalid memory (at the
addressed which are not allocated/mapped) -- in which case you get
SIGSEGV or the like.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] [CGO] how to pass a 2d slice to C?

[Konstantin Khomoutov]

On Sun, Aug 06, 2017 at 07:08:03PM -0700, jianzhang...@gmail.com wrote:

> Thanks your reply, I also used this way, but it still not work. code as the 
> following.
> 
> gonameunits := []string{"gpu0", "gpu1", "gpu2", "gpu3"}
>nameunits := make([]*C.char, len(gonameunits))
>for i, _ := range gonameunits {
>nameunits[i] = C.CString(gonameunits[i])
>defer C.free(unsafe.Pointer(nameunits[i]))
>}
>fmt.Println("nameunits:", nameunits)

This looks correct...

> golevelmatrix := [][]int{{1}, {3, 3}, {3, 3, 2}}
>levelmatrix := make([][]C.int, len(golevelmatrix))
>for i, _ := range golevelmatrix {
>levelmatrix[i] = make([]C.int, len(golevelmatrix[i]))
>for j, _ := range golevelmatrix[i] {
>levelmatrix[i][j] = C.int(golevelmatrix[i][j])
>}
>}

...but this is not: remember that in Go, the type []T denotes *a slice* of
elements of type T.  A slice is a dynamic construct which is a view into
underlying array, and as such it itself consists of a pointer into that
array's contents, the current length of the slice and the capacity of
it.

>fmt.Println("levelmatrix:", levelmatrix)
> 
> C.test_settopologyresource(mod, C.CString("node1"), C.int(2), (**C.char
> )(unsafe.Pointer([0])), (**C.int)(unsafe.Pointer([0][0
> ])))

That thing about slices means that passing a pointer to a single slice
element to C is OK; passing a pointer to an Nth element of a slice to C
and expecting it to access it and the elements following it up to the
slice's length is also OK but interpreting a slice of slices (which
[][]*C.int really is) is wrong: it's not represented by a contiguous
array: it's a slice of N independent slices -- each backed by its
independent array.

So what you really want is something like:

1. Know the dimensions of your matrix (let's denote them by N and M).
2. Allocate a slice of the length N*M.
3. Fill it up with the values from your source slice of slices.
4. Pass the pointer to the array to C.

Like this (untested, may not even compile):

  n := len(golevelmatrix)
  m := 0
  for _, row := range golevelmatrix {
if len(row) > m {
  m = len(row)
}
  }
  
  a := make([]int, n*m)
  i := 0
  for _, row := range golevelmatrix {
for j, v := range row {
a[i*n+j] = v
}
i++
  }

  C.my_func([0])

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Goroutines chan to get data at specific time

[Konstantin Khomoutov]

On Sun, Aug 06, 2017 at 12:53:08AM -0700, desaiabhi...@gmail.com wrote:

> Can you please help with below code to get output at specific cutoff time 
> and exit
[...]
> c1 := make(chan string)
> 
> go func() { //Sending data after certain time
>   
> c1 <- "result 1"
> 
> time.Sleep(time.Second * 1)
> c1 <- "result 2 afer 1 sec"
> 
> time.Sleep(time.Second * 1)
> c1 <- "result 2 afer 2 sec"
> 
> time.Sleep(time.Second * 1)
> c1 <- "result 2 afer 3 sec"
> 
> time.Sleep(time.Second * 1)
> c1 <- "result 2 afer 4 sec"
> 
> time.Sleep(time.Second * 1)
> c1 <- "result 2 afer 5 sec"
> }()
> 
> select {
> case <-time.After(time.Second * 4): { //cut off 4s and return the 
> value
> res := <-c1
> fmt.Println(res)  // expecting result "result 2 afer 3 sec" but 
> returning "result 1"
> }
> }
> }

You have created a channel of capacity 0, which means that as soon as
some goroutine sends to the channel it blocks until another goroutine
reads from it.  What's more, any other goroutines which attempt to send
to that channel while the first goroutine is blocked waiting for the
value it's sending to be received, gets blocked, too.

So, in your example the goroutine which sends gets blocked in

  c1 <- "result 1"

and is suspended by the scheduler.

The main goroutine which performs select waits for the timer to fire and
then reads from c1 hereby unblocking the background goroutine and
allowing its send operation to proceed.

After that the background goroutine continues to work and then at some
indeterminate point gets reaped by the Go runtime which kills all the
active goroutine when the goroutine running main() exits.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] all goroutines are asleep - deadlock!

[Konstantin Khomoutov]

On Fri, Aug 04, 2017 at 08:11:45AM -0700, prankpla...@gmail.com wrote:

[...]
> func fact_worker(fact_resp_chan chan string, x int) {
> defer wg.Done()
> response := factorial(x)
> fact_resp_chan <-  fmt.Sprintf("Factorial of %d is : %d", x, response)
> }
[...]

Please also consider naming your identifiers consistently with what the
Go itself and the community uses — for instance, [1, 2].

That is, use factWorker and factRespChan.

"Snake case" (foo_bar) is not used in Go code.

1. https://golang.org/doc/effective_go.html#names
2. https://blog.golang.org/package-names

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Size of struct{}?

[Konstantin Khomoutov]

On Fri, Aug 04, 2017 at 02:25:01AM -0700, 樊冰心 wrote:

[...]
> type T struct {
> i int
> x struct{}
> }
> 
> func main() {
> var t T
> fmt.Println(unsafe.Sizeof(t))
> }
> 
> Since size of struct{} is 0, why the result is 16?

Consider also reading
https://dave.cheney.net/2014/03/25/the-empty-struct
and
https://dave.cheney.net/2015/10/09/padding-is-hard

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Calling a C function from Go code in cgo must new a thread

[Konstantin Khomoutov]

On Fri, Aug 04, 2017 at 03:45:49AM -0700, bysui hou wrote:

> I'm going to use CGO to call c++ in my project and do some performance 
> tests. So I do research on cgo, and find that calling a C function from Go 
> code won't block other goroutines from running. I view the cgocall code 
> (runtime/cgocall.go, is there any official doc?), it seems that calling a C 
> function from Go code in cgo would lock g to m and call entersyscall (dose 
> it means new a thread for running ?). If in go code I make lots of C 
> function calling, will the program new a thread for each calling?

It's sort of backwards: when a goroutine makes a cgo call it gets locked
to the thread it currently runs on until the call returns -- just like a
syscall, yes.  This does not prevent other goroutines from running
concurrently, and the Go runtime is free to spawn another OS thread to
max them quantity out up to runtime.GOMAXPROCS because that knob control
the number of goroutines running "normally" -- that is, crunching CPU
and not sleeping in a syscall or a cgo call.

So yes, making lots of cgo calls in parallel might create lots of OS
threads.  So if you think this is an issue, plan on having one or more
"worker" goroutines which receive "tasks" to perform via cgo calls and
communicate their results back.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] [CGO] how to pass a 2d slice to C?

[Konstantin Khomoutov]

On Fri, Aug 04, 2017 at 02:09:14AM -0700, jianzhang...@gmail.com wrote:

> Hey Guys,
> 
> I'm in trouble in the same issue. My code as the following:
> 
> test.go
> ```go
> name := []string{"gpu0", "gpu1", "gpu2", "gpu3"}
> matrix := [3][3]int{{1, 0, 0}, {3, 3, 0}, {3, 3, 2}}
> 
> C.test_settopologyresource(mod, C.CString("node1"), C.int(2), 
> (**C.char)(unsafe.Pointer([0])), 
> (**C.int)(unsafe.Pointer([0][0])))
> ```

I'm afraid that's not going to work: the elements of the "name" slice
are strings, and they can't be coersed to *C.char because a Go string is
internally a struct consisting of a pointer and a length.

So I think you'd need to make a "clone" data struct -- something like
this:

  cnames := make([]*C.Char, len(names))
  for i, name := range names {
cnames[i] = C.CString(name)
  }

and then destroy those objects after returning from the C side:

  for _, p := range cnames {
C.free(p)
  }

[...]

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] accept(2), runtime.LockOSThread and goroutine scheduling

[Konstantin Khomoutov]

We're experiencing a problem with our program which serves HTTP requests.

Its clients have TCP connection timeouts set to 1 second, and under
certain pattern of heavy load the server fails to perform some
net.netFD.Accept() calls in time so a fraction of clients gets I/O
timeouts when attempting to connect.

Thanks to the Go runtime tracing facility, we were able to pinpoint that
those I/O timeouts happen due to the goroutine which does the accept() call
being unblocked by the netpoller but then waits in a run queue to
actually have a chance to run for too long (in our case -- sometimes for
over two seconds).  We hypothesize that this happens due to heavy load on
the goroutine scheduler.

Still, we have to invent a temporary workaround until the real cause of
the degraded performance is understood and fixed.  We were able to come
up with two workarounds:

- Run multiple instances of net/http.Server in several goroutine.

  This appears to alleviate the problem.
  We hypothesize that doing this can be viewed as raising the relative
  number of quanta the scheduler will grant to gorutines serving the
  accept() call.

- Lock the goroutine which runs the net/http.Server instance to its
  underlying OS thread (and may be do a syscall to raise that thread's
  priority above normal).

  We did not yet test this approach but it appears to be better that the
  first as it should lower possible contentions between multiple
  instances of the net/http.Server type, and makes the whole setup
  simpler.

I'd like to solicit insight on whether the latter approach is workable.

I sadly lack full understanding on how actually running of goroutines
and the runtime scheduler really interacts with the scheduling of the
threads done by the OS.  For instance, whould the described setup do
anything to help the goroutine doing accept() have more execution
quanta?

I'm in doubt because of the interaction between the netpoller and the
goroutines.  Say, a goroutine have itself locked to its underlying OS
thread, that thread has its OS-level priority raised. Now suppose that
goroutine does the accept() syscall; the socket backlog is zero, so the
syscall would block and so it goes to the netpoller and the goroutine
gets parked.  Since it has its thread locked, my understanding is such
that thread gets essentially dormant (and the runtime is free to spawn
another one to fullfill GOMAXPROCS).

Now what happens in these two cases:

- A client initiates the connection and the netpoller unblocks the
  goroutine doing accept.

  As I understand, the runtime will merely figure out that that
  particular goroutine must run on its dedicated thread, so it will make
  its P execute it on that thread (hope I got the terminology right).

  Will the fact the goroutine was bound to its own thread help it get
  executed with priority compared to other goroutines (which essencially
  contend for GOMAXPROCS other threads)?

- What happens to the thread, to which a goroutine was locked, while
  that goroutine remains parked?  Is this thread somehow suspended and
  the OS never wakes it until the goroutine gets unblocked?
  Or is the OS free to schedule it, and if yes, what runs on it when it
  gets its execution quantum?

Any help or pointers would be greatly appreciated!

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.