Re: [go-nuts] Possible to make base64 pick the decode encoding automatically?

[hey...@gmail.com]

> So having a “meta/relaxed decoder” usually leads to 
specification/interoperability/security problems down the road
I respectfully disagree. Since it's only relaxed with regard to decoding, 
it follows the robustness principle where you be liberal in what you accept.

Within a system, the encoding should be explicitly defined, but when that 
system has to consume base64 data from outside, being liberal actually 
avoids interoperability problems.
On Wednesday, February 3, 2021 at 1:10:32 PM UTC+8 ren...@ix.netcom.com 
wrote:

> Sorry it just doesn’t “feel right”. There are different encoding scheme as 
> laid out in the RFC. and other RFCs 
>  that cover 
> their uses.
>
> If you have a system that states “send us Base64 data” it is poorly 
> specified - better to state, send us Base64 data according to RFC 4648 
> base64url format or according to RFC-2045.
>
> In fact, the RFC states:
>
> "This encoding may be referred to as "base64url".  This encoding
>should not be regarded as the same as the "base64" encoding and
>should not be referred to as only "base64".  Unless clarified
>otherwise, "base64" refers to the base 64 in the previous section.”
>
> It also states:
>
> "If non-alphabet characters are ignored, instead of causing rejection
>of the entire encoding (as recommended), a covert channel that can be
>used to "leak" information is made possible."
>
> So having a “meta/relaxed decoder” usually leads to 
> specification/interoperability/security problems down the road. I realize 
> that in the “real world” you are often forced to interoperate with these 
> “bad” systems, but as most things in Go, better to be explicit and report 
> errors rather than be clever.
>
>
>
> On Feb 2, 2021, at 12:34 PM, 'Axel Wagner' via golang-nuts <
> golan...@googlegroups.com> wrote:
>
> On Tue, Feb 2, 2021 at 6:43 PM Robert Engels  wrote:
>
>> What “padding” are you referring to? Each must be 2 characters. And there 
>> is a standard that covers this https://tools.ietf.org/html/rfc4648
>>
>
> Yes, there indeed is. Section 5 describes a second encoding scheme, used 
> for URLs and the like. Section 3.2 also talks about the padding I'm 
> referring to (it's defined elsewhere in the standard) and mentions that, in 
> certain situations, it can be omitted. In particular, you can omit padding 
> and, in the decoder, implicitly pad to a multiple of 4 bytes.
>
> I don't really understand what's the argument is here. The question was if 
> it is possible to handle all four encoding schemas 
>  supported by the 
> Go base64 package in one swoop, because as-is, the API requires you to pick 
> one schema and just see if it returns an error. Roger provided, IMO, a 
> pretty good answer to that: You can wrap the io.Reader in one that 
> transparently rewrites any of the four into one well-known one, which can 
> then be handled by the corresponding decoder. His link provides the code 
> for an implementation of such a reader.
>  
>
>>
>> On Feb 2, 2021, at 10:57 AM, 'Axel Wagner' via golang-nuts <
>> golan...@googlegroups.com> wrote:
>>
>> 
>> Rogers approach seems like the best one to me - wrap the input in a 
>> custom `io.Reader` that transparently replaces `-_` with `+/` respectively 
>> (and drop trailing `=`). The bufio approach doesn't work, because there is 
>> no guarantee that one of the distinguishing characters is early in the 
>> stream and the "send it to multiple decoders" approach duplicates effort 
>> and wastes resources.
>>
>> On Tue, Feb 2, 2021 at 5:43 PM Amnon  wrote:
>>
>>> Reading through a bufio.Reader is often useful for these situations.
>>> You can peek the beginning of the input in order to decide which decoder 
>>> to use.
>>>
>>> Another option is to use the io.TeeReader to duplicate the reader,
>>> and then send one copy to each decoder.
>>> One will succeed, and give you the output.
>>> But you will need to drain the one that fails to prevent the TeeReader 
>>> form stalling.
>>>
>>>
>>> On Tuesday, 2 February 2021 at 14:17:19 UTC axel.wa...@googlemail.com 
>>> wrote:
>>>
 This question isn't about the decoded data, but about *which* base64 
 format is used - i.e. if it uses padding or not and what 2 characters are 
 used outside of a-zA-Z0-9. The most common ones use +/ and -_, so it's 
 easy 
 to tell which is used and just accept either (and padding can be viewed as 
 optional during decoding anyway).

 On Tue, Feb 2, 2021 at 2:37 PM Robert Engels  
 wrote:

> Base64 is always ASCII. The encoded data may be in an arbitrary 
> format. You need to pass additional metadata or try and detect its 
> encoding. 
>
> On Feb 2, 2021, at 6:50 AM, roger peppe  wrote:
>
> 
> In case you find it helpful, here's a clone of the base64 command that 
> I wrote in Go. I did it precisely because I wante

[go-nuts] Re: Contributing to golang based project on github - collaboration best practice

[Volker Dobler]

You either have to use the "replace" directive in go.mod, or:
Do not work on the fork. Do your work on a plain _clone_ of the
repo (which works without "replace"ing dependencies).
To create a Github PR: git push to your fork (add it as an additional
git remote) and create the PR. The "fork" is just a vehicle for a
Github PR and nothing you do work on (or try to build).

V.

On Wednesday, 3 February 2021 at 04:17:40 UTC+1 yue.ni...@gmail.com wrote:

> Hi,
>
>   I am relatively new to golang coming from a C++/Python world.
>
>   When contributing to C++/Python projects on GitHub, I usually fork the 
> project, make changes to hit and submit a pull request to the 
> author/maintainer.
>
>   I found that the above didn't work for me.
>
>   I am hoping to contribute to the following project
>
> https://github.com/scipipe/scipipe
>   
>   I am trying to ascertain if my challenges is because Golang has specific 
> ways for contribution or the above project is setup differently. I can't 
> tell as I am new to Golang.
>
>   I clone the forked project to $HOME/go/src/github/scipipe and testing it 
> in another directory. I found that whenever I run my small test program, it 
> pulls from some archive rather than my cloned repository and hence is not 
> able to see the changes I made.
>
>   Is there some special way to import the module so that it picked my 
> clone directory rather than pulling down a zip archive versioned copy ?
>
> ```
> package main
>
> import (
> // Import the SciPipe package, aliased to 'sp'
> sp "github.com/scipipe/scipipe"
> )
>
> func main() {
> // Init workflow with a name, and max concurrent tasks
> wf := sp.NewWorkflow("hello_world", 4)
>
> // Initialize processes and set output file paths
> hello := wf.NewProc("hello", "echo 'Hello ' > {o:out}")
> world := wf.NewProc("world", "echo $(cat {i:in}) World >> {o:out}")
>
> // Connect network
> world.In("in").From(hello.Out("out"))
>
> wf.Nicholas() // Failed to find this new exported function I added
> }
> ```
>
> Cheers
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/16720d3c-b18f-4890-82e5-4d590af2e9d1n%40googlegroups.com.

Re: [go-nuts] Possible to make base64 pick the decode encoding automatically?

[robert engels]

Sorry it just doesn’t “feel right”. There are different encoding scheme as laid 
out in the RFC. and other RFCs 
 that cover their 
uses.

If you have a system that states “send us Base64 data” it is poorly specified - 
better to state, send us Base64 data according to RFC 4648 base64url format or 
according to RFC-2045.

In fact, the RFC states:

"This encoding may be referred to as "base64url".  This encoding
   should not be regarded as the same as the "base64" encoding and
   should not be referred to as only "base64".  Unless clarified
   otherwise, "base64" refers to the base 64 in the previous section.”

It also states:

"If non-alphabet characters are ignored, instead of causing rejection
   of the entire encoding (as recommended), a covert channel that can be
   used to "leak" information is made possible."

So having a “meta/relaxed decoder” usually leads to 
specification/interoperability/security problems down the road. I realize that 
in the “real world” you are often forced to interoperate with these “bad” 
systems, but as most things in Go, better to be explicit and report errors 
rather than be clever.



> On Feb 2, 2021, at 12:34 PM, 'Axel Wagner' via golang-nuts 
>  wrote:
> 
> On Tue, Feb 2, 2021 at 6:43 PM Robert Engels  > wrote:
> What “padding” are you referring to? Each must be 2 characters. And there is 
> a standard that covers this https://tools.ietf.org/html/rfc4648 
> 
> 
> Yes, there indeed is. Section 5 describes a second encoding scheme, used for 
> URLs and the like. Section 3.2 also talks about the padding I'm referring to 
> (it's defined elsewhere in the standard) and mentions that, in certain 
> situations, it can be omitted. In particular, you can omit padding and, in 
> the decoder, implicitly pad to a multiple of 4 bytes.
> 
> I don't really understand what's the argument is here. The question was if it 
> is possible to handle all four encoding schemas 
>  supported by the Go 
> base64 package in one swoop, because as-is, the API requires you to pick one 
> schema and just see if it returns an error. Roger provided, IMO, a pretty 
> good answer to that: You can wrap the io.Reader in one that transparently 
> rewrites any of the four into one well-known one, which can then be handled 
> by the corresponding decoder. His link provides the code for an 
> implementation of such a reader.
>  
> 
>> On Feb 2, 2021, at 10:57 AM, 'Axel Wagner' via golang-nuts 
>> mailto:golang-nuts@googlegroups.com>> wrote:
>> 
>> 
>> Rogers approach seems like the best one to me - wrap the input in a custom 
>> `io.Reader` that transparently replaces `-_` with `+/` respectively (and 
>> drop trailing `=`). The bufio approach doesn't work, because there is no 
>> guarantee that one of the distinguishing characters is early in the stream 
>> and the "send it to multiple decoders" approach duplicates effort and wastes 
>> resources.
>> 
>> On Tue, Feb 2, 2021 at 5:43 PM Amnon > > wrote:
>> Reading through a bufio.Reader is often useful for these situations.
>> You can peek the beginning of the input in order to decide which decoder to 
>> use.
>> 
>> Another option is to use the io.TeeReader to duplicate the reader,
>> and then send one copy to each decoder.
>> One will succeed, and give you the output.
>> But you will need to drain the one that fails to prevent the TeeReader form 
>> stalling.
>> 
>> 
>> On Tuesday, 2 February 2021 at 14:17:19 UTC axel.wa...@googlemail.com 
>>  wrote:
>> This question isn't about the decoded data, but about *which* base64 format 
>> is used - i.e. if it uses padding or not and what 2 characters are used 
>> outside of a-zA-Z0-9. The most common ones use +/ and -_, so it's easy to 
>> tell which is used and just accept either (and padding can be viewed as 
>> optional during decoding anyway).
>> 
>> On Tue, Feb 2, 2021 at 2:37 PM Robert Engels > wrote:
>> Base64 is always ASCII. The encoded data may be in an arbitrary format. You 
>> need to pass additional metadata or try and detect its encoding. 
>> 
>>> On Feb 2, 2021, at 6:50 AM, roger peppe > wrote:
>>> 
>>> 
>>> In case you find it helpful, here's a clone of the base64 command that I 
>>> wrote in Go. I did it precisely because I wanted to be able to decode any 
>>> encoding scheme interchangeably.
>>> 
>>> https://github.com/rogpeppe/misc/blob/master/cmd/base64/base64.go 
>>> 
>>> 
>>> I agree that it might be useful to have some of this functionality 
>>> available in the standard library.
>>> 
>>>   cheers,
>>> rog.
>>> 
>>> On Tue, 2 Feb 2021 at 09:08, hey...@gmail.com <> > 
>>> wrote:
>>> Hi,
>>> 
>>> I have an io.Reader whose content is encoded in base64 with encoding type 
>>> unknown. 

Re: [go-nuts] Contributing to golang based project on github - collaboration best practice

[Shulhan]

> On 3 Feb 2021, at 10.17, Nicholas Yue  wrote:
> 
> Hi,
> 
>   I am relatively new to golang coming from a C++/Python world.
> 
>   When contributing to C++/Python projects on GitHub, I usually fork the 
> project, make changes to hit and submit a pull request to the 
> author/maintainer.
> 
>   I found that the above didn't work for me.
> 
>   I am hoping to contribute to the following project
> 
> https://github.com/scipipe/scipipe
>   
>   I am trying to ascertain if my challenges is because Golang has specific 
> ways for contribution or the above project is setup differently. I can't tell 
> as I am new to Golang.
> 
>   I clone the forked project to $HOME/go/src/github/scipipe and testing it in 
> another directory. I found that whenever I run my small test program, it 
> pulls from some archive rather than my cloned repository and hence is not 
> able to see the changes I made.
> 
>   Is there some special way to import the module so that it picked my clone 
> directory rather than pulling down a zip archive versioned copy ?
> 


Inside the go.mod file in your test directory, add the "replace" directive. 
Something like this may works,


module mytestmodule

replace github.com/scipipe/scipipe => ../path/to/your/clone


More on Go modules,

* https://blog.golang.org/using-go-modules
* https://golang.org/ref/mod

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/D4B065B8-12C3-4DDF-8190-13A8D7EA0FAA%40gmail.com.

[go-nuts] Contributing to golang based project on github - collaboration best practice

[Nicholas Yue]

Hi,

  I am relatively new to golang coming from a C++/Python world.

  When contributing to C++/Python projects on GitHub, I usually fork the 
project, make changes to hit and submit a pull request to the 
author/maintainer.

  I found that the above didn't work for me.

  I am hoping to contribute to the following project

https://github.com/scipipe/scipipe
  
  I am trying to ascertain if my challenges is because Golang has specific 
ways for contribution or the above project is setup differently. I can't 
tell as I am new to Golang.

  I clone the forked project to $HOME/go/src/github/scipipe and testing it 
in another directory. I found that whenever I run my small test program, it 
pulls from some archive rather than my cloned repository and hence is not 
able to see the changes I made.

  Is there some special way to import the module so that it picked my clone 
directory rather than pulling down a zip archive versioned copy ?

```
package main

import (
// Import the SciPipe package, aliased to 'sp'
sp "github.com/scipipe/scipipe"
)

func main() {
// Init workflow with a name, and max concurrent tasks
wf := sp.NewWorkflow("hello_world", 4)

// Initialize processes and set output file paths
hello := wf.NewProc("hello", "echo 'Hello ' > {o:out}")
world := wf.NewProc("world", "echo $(cat {i:in}) World >> {o:out}")

// Connect network
world.In("in").From(hello.Out("out"))

wf.Nicholas() // Failed to find this new exported function I added
}
```

Cheers

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/d312ffe3-24b5-4aad-a3e1-4e50a35e318an%40googlegroups.com.

Re: [go-nuts] Possible to make base64 pick the decode encoding automatically?

[hey...@gmail.com]

Your translate reader works really well, thanks for sharing it.

I have seen code that tried to decode base64 four times in the wild, which 
led me to posting this, hope something like this could be incorporated into 
the standard library. 

On Tuesday, February 2, 2021 at 8:50:08 PM UTC+8 rog wrote:

> In case you find it helpful, here's a clone of the base64 command that I 
> wrote in Go. I did it precisely because I wanted to be able to decode any 
> encoding scheme interchangeably.
>
> https://github.com/rogpeppe/misc/blob/master/cmd/base64/base64.go
>
> I agree that it might be useful to have some of this functionality 
> available in the standard library.
>
>   cheers,
> rog.
>
> On Tue, 2 Feb 2021 at 09:08, hey...@gmail.com  wrote:
>
>> Hi,
>>
>> I have an io.Reader whose content is encoded in base64 with encoding type 
>> unknown. Since there shouldn't be any ambiguity between the two, is it 
>> possible to make the base64 automatically pick the right one to decode?
>>
>> Currently I have to read everything out to pin down the encoding, which 
>> defeats the purpose of using an io.Reader.
>>
>> Is there a solution to this problem?
>>
>> Thanks in advance.
>>
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/0ccee37d-319e-41b3-9bfd-3dc46e0fad78n%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/15c0911e-cf08-48e2-b979-054ee7525d97n%40googlegroups.com.

Re: [go-nuts] Re: C function return value of type float changes when used with COG

[Alex]

You need to provide a complete runnable example, the sniplet you provided 
is missing all the interesting parts.

On Tuesday, 2 February 2021 at 3:58:39 pm UTC+8 Robert M. Münch wrote:

>
> Here we go: https://pastebin.com/6c6rq92K
>
> The code to access the C lib was generated with c-for-go, so there is a 
> lot of house-keeping stuff going on. However, I don't think this has any 
> influence on the C code data.
>
> I implemented getCHeight (see commented lines 155ff, to access the C data 
> as direct as possible. The functions prints the output as in my previous 
> post.
> Ian Lance Taylor schrieb am Dienstag, 2. Februar 2021 um 00:58:01 UTC+1:
>
>> On Mon, Feb 1, 2021 at 1:52 PM Robert M. Münch 
>>  wrote: 
>> > 
>> > I know all about the problems of FP comparison, and yes I know that I 
>> need to provide more information. What makes me suspicious is, that I have 
>> a bunch of equivalent tests before the failing one, and all pass. I still 
>> believe in the determinism of programs. Hence, this is all really strange. 
>> > 
>> > I run the C test code (without any Go) in GDB and the values are 
>> correctly set. This is how the data looks before the call, setting the 
>> correct values: 
>> > 
>> > 804 YGNodeCalculateLayout(root, YGUndefined, YGUndefined, 
>> YGDirectionLTR); 
>> > (gdb) p root.layout_.dimensions 
>> > $9 = { 
>> > _M_elems = {nan(0x40), nan(0x40)} 
>> > } 
>> > 
>> > and this is after the call: 
>> > 
>> > (gdb) p root.layout_.dimensions 
>> > $13 = { 
>> > _M_elems = {320, 10} 
>> > } 
>> > 
>> > I write a C wrapper function, that outputs the return value just before 
>> it's returned and makes its way through the Go FFI. Here is the output 
>> before the call: 
>> > 
>> > YGNodeLayoutGetWidth: 1.#QNAN0 7FC0 
>> > YGNodeLayoutGetHeight: 1.#QNAN0 7FC0 
>> > 
>> > and after the call 
>> > 
>> > YGNodeLayoutGetWidth: 320.00 43A0 
>> > YGNodeLayoutGetHeight: 10002004087730.00 6258D727 
>> > 
>> > So, why are the values before the call are different in C and the C 
>> side within the Go program? I mean why once 0x40 and 0x7FC0? I 
>> would have expected that the C code behaves the same. 
>> > And as you can see the 320 is set in both cases, but the expected 10 is 
>> a totally screwed up floating-point value. 
>> > 
>> > Could this be an alignment problem? Does CGO add some wrapping code, 
>> which does some strange things? 
>> > 
>> > Unfortunately, I didn't find a way, how I can enter the C code from a 
>> debugger when using the Go binary. Looks like there is no C debug 
>> information available. 
>>
>> cgo adds multiple wrapping functions. 
>>
>> I don't know what the problem is here. I can imagine several 
>> different possibilities. Show us the code, and we won't have to 
>> guess. Thanks. 
>>
>> Ian 
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/d0c2a315-66b3-479d-b537-f5bdbe624cefn%40googlegroups.com.

Re: [go-nuts] go.mod: 1: module: not found

[saurav deshpande]

Okay, will try with a different approach
Thank you

On Wed, 3 Feb, 2021, 1:43 am Ian Lance Taylor,  wrote:

> On Tue, Feb 2, 2021 at 11:16 AM saurav deshpande
>  wrote:
> >
> > I tried by giving the module name as pathname by pushing the code to git
> and created go.mod using that path, but still gives the same error.
>
> I'm sorry that didn't help, but I still expect that the problem is
> that "go build -n" is printing an error message, and your Makefile is
> piping that error message directly to the shell.  You're going to have
> to figure out what "go build -n" is printing.
>
> I'll repeat my earlier comment that this build procedure is not
> supported.  "go build" is supported.  "go build -n | sed | sh" is not
> supported.
>
> Ian
>
>
> > On Tuesday, February 2, 2021 at 5:33:27 AM UTC+5:30 Ian Lance Taylor
> wrote:
> >>
> >> On Mon, Feb 1, 2021 at 1:42 PM saurav deshpande
> >>  wrote:
> >> >
> >> > it just contains:
> >> >
> >> > module babyLotus
> >> >
> >> > go 1.15
> >>
> >> The module name should probably be a path name.
> >>
> >> I bet that "go build -n" is printing an error, and you are piping that
> >> error to sh.
> >>
> >> Ian
> >>
> >>
> >> > On Tuesday, February 2, 2021 at 3:04:39 AM UTC+5:30 Ian Lance Taylor
> wrote:
> >> >>
> >> >> On Mon, Feb 1, 2021 at 11:24 AM saurav deshpande
> >> >>  wrote:
> >> >> >
> >> >> > We have a go.mod file.
> >> >> > The same make file worked with go 1.9.7 version, it did not
> support mod.
> >> >> > I am trying to make this work with go 1.15, can you please suggest
> changes that would make this work.
> >> >>
> >> >> What does your go.mod file look like?
> >> >>
> >> >> Ian
> >> >>
> >> >>
> >> >> > On Tuesday, February 2, 2021 at 12:48:51 AM UTC+5:30 Ian Lance
> Taylor wrote:
> >> >> >>
> >> >> >> On Mon, Feb 1, 2021 at 10:35 AM saurav deshpande
> >> >> >>  wrote:
> >> >> >> >
> >> >> >> > Hello, I am trying to run a make file but it gives me the
> following error:
> >> >> >> > go.mod: 1: module: not found
> >> >> >> > go 1.15: unknown command
> >> >> >>
> >> >> >> ...
> >> >> >>
> >> >> >> > go.o:
> >> >> >> > @mkdir -p $(BUILD_DIR)
> >> >> >> > @echo "[go] compiling go sources into a standalone .o file"
> >> >> >> > echo $(GOPATH)
> >> >> >> > @GOARCH=386 GOOS=linux GOPATH=$(GOPATH) COG_ENABLED=1 go build
> -n 2>&1 | sed \
> >> >> >> > -e "1s|^|set -e\n|" \
> >> >> >> > -e "1s|^|export GOOS=linux\n|" \
> >> >> >> > -e "1s|^|export GOARCH=386\n|" \
> >> >> >> > -e "1s|^|export CGO_ENABLED=1\n|" \
> >> >> >> > -e "1s|^|WORK='$(BUILD_ABS_DIR)'\n|" \
> >> >> >> > -e "1s|^|alias pack='go tool pack'\n|" \
> >> >> >> > -e "/^mv/d" \
> >> >> >> > -e "s|-extld|-tmpdir='$(BUILD_ABS_DIR)' -linkmode=external
> -extldflags='-nostdlib' -extld|g" \
> >> >> >> > | sh 2>&1 | sed -e "s/^/ | /g"
> >> >> >>
> >> >> >> This procedure--using "go build -n" and editing the output--is not
> >> >> >> supported and is going to be highly version-specific. Has this
> ever
> >> >> >> worked? Have you changed the version of Go that you are using?
> >> >> >>
> >> >> >> Do your sources have a go.mod file?
> >> >> >>
> >> >> >> Ian
> >> >> >
> >> >> > --
> >> >> > You received this message because you are subscribed to the Google
> Groups "golang-nuts" group.
> >> >> > To unsubscribe from this group and stop receiving emails from it,
> send an email to golang-nuts...@googlegroups.com.
> >> >> > To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/4cfebff8-f9ad-40b1-afab-253b8ba25e48n%40googlegroups.com
> .
> >> >
> >> > --
> >> > You received this message because you are subscribed to the Google
> Groups "golang-nuts" group.
> >> > To unsubscribe from this group and stop receiving emails from it,
> send an email to golang-nuts...@googlegroups.com.
> >> > To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/487b32a6--4025-b754-790d5f3820can%40googlegroups.com
> .
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "golang-nuts" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to golang-nuts+unsubscr...@googlegroups.com.
> > To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/bef3aea9-2ecb-4e9c-a395-992d8e0ed3edn%40googlegroups.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAHXDL5ryfOifPaXNQNJWTkz4E9rSr9-X4juquUEALOVnr7uDvg%40mail.gmail.com.

Re: [go-nuts] go.mod: 1: module: not found

[Ian Lance Taylor]

On Tue, Feb 2, 2021 at 11:16 AM saurav deshpande
 wrote:
>
> I tried by giving the module name as pathname by pushing the code to git and 
> created go.mod using that path, but still gives the same error.

I'm sorry that didn't help, but I still expect that the problem is
that "go build -n" is printing an error message, and your Makefile is
piping that error message directly to the shell.  You're going to have
to figure out what "go build -n" is printing.

I'll repeat my earlier comment that this build procedure is not
supported.  "go build" is supported.  "go build -n | sed | sh" is not
supported.

Ian


> On Tuesday, February 2, 2021 at 5:33:27 AM UTC+5:30 Ian Lance Taylor wrote:
>>
>> On Mon, Feb 1, 2021 at 1:42 PM saurav deshpande
>>  wrote:
>> >
>> > it just contains:
>> >
>> > module babyLotus
>> >
>> > go 1.15
>>
>> The module name should probably be a path name.
>>
>> I bet that "go build -n" is printing an error, and you are piping that
>> error to sh.
>>
>> Ian
>>
>>
>> > On Tuesday, February 2, 2021 at 3:04:39 AM UTC+5:30 Ian Lance Taylor wrote:
>> >>
>> >> On Mon, Feb 1, 2021 at 11:24 AM saurav deshpande
>> >>  wrote:
>> >> >
>> >> > We have a go.mod file.
>> >> > The same make file worked with go 1.9.7 version, it did not support mod.
>> >> > I am trying to make this work with go 1.15, can you please suggest 
>> >> > changes that would make this work.
>> >>
>> >> What does your go.mod file look like?
>> >>
>> >> Ian
>> >>
>> >>
>> >> > On Tuesday, February 2, 2021 at 12:48:51 AM UTC+5:30 Ian Lance Taylor 
>> >> > wrote:
>> >> >>
>> >> >> On Mon, Feb 1, 2021 at 10:35 AM saurav deshpande
>> >> >>  wrote:
>> >> >> >
>> >> >> > Hello, I am trying to run a make file but it gives me the following 
>> >> >> > error:
>> >> >> > go.mod: 1: module: not found
>> >> >> > go 1.15: unknown command
>> >> >>
>> >> >> ...
>> >> >>
>> >> >> > go.o:
>> >> >> > @mkdir -p $(BUILD_DIR)
>> >> >> > @echo "[go] compiling go sources into a standalone .o file"
>> >> >> > echo $(GOPATH)
>> >> >> > @GOARCH=386 GOOS=linux GOPATH=$(GOPATH) COG_ENABLED=1 go build -n 
>> >> >> > 2>&1 | sed \
>> >> >> > -e "1s|^|set -e\n|" \
>> >> >> > -e "1s|^|export GOOS=linux\n|" \
>> >> >> > -e "1s|^|export GOARCH=386\n|" \
>> >> >> > -e "1s|^|export CGO_ENABLED=1\n|" \
>> >> >> > -e "1s|^|WORK='$(BUILD_ABS_DIR)'\n|" \
>> >> >> > -e "1s|^|alias pack='go tool pack'\n|" \
>> >> >> > -e "/^mv/d" \
>> >> >> > -e "s|-extld|-tmpdir='$(BUILD_ABS_DIR)' -linkmode=external 
>> >> >> > -extldflags='-nostdlib' -extld|g" \
>> >> >> > | sh 2>&1 | sed -e "s/^/ | /g"
>> >> >>
>> >> >> This procedure--using "go build -n" and editing the output--is not
>> >> >> supported and is going to be highly version-specific. Has this ever
>> >> >> worked? Have you changed the version of Go that you are using?
>> >> >>
>> >> >> Do your sources have a go.mod file?
>> >> >>
>> >> >> Ian
>> >> >
>> >> > --
>> >> > You received this message because you are subscribed to the Google 
>> >> > Groups "golang-nuts" group.
>> >> > To unsubscribe from this group and stop receiving emails from it, send 
>> >> > an email to golang-nuts...@googlegroups.com.
>> >> > To view this discussion on the web visit 
>> >> > https://groups.google.com/d/msgid/golang-nuts/4cfebff8-f9ad-40b1-afab-253b8ba25e48n%40googlegroups.com.
>> >
>> > --
>> > You received this message because you are subscribed to the Google Groups 
>> > "golang-nuts" group.
>> > To unsubscribe from this group and stop receiving emails from it, send an 
>> > email to golang-nuts...@googlegroups.com.
>> > To view this discussion on the web visit 
>> > https://groups.google.com/d/msgid/golang-nuts/487b32a6--4025-b754-790d5f3820can%40googlegroups.com.
>
> --
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/bef3aea9-2ecb-4e9c-a395-992d8e0ed3edn%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAOyqgcWSV9-PqSqj8n7c9hmHUJX2ATrPfKxjLpBOYC1XAiQ_5w%40mail.gmail.com.

Re: [go-nuts] go.mod: 1: module: not found

[saurav deshpande]

I tried by giving the module name as pathname by pushing the code to git 
and created go.mod using that path, but still gives the same error.


Thank you
 

On Tuesday, February 2, 2021 at 5:33:27 AM UTC+5:30 Ian Lance Taylor wrote:

> On Mon, Feb 1, 2021 at 1:42 PM saurav deshpande
>  wrote:
> >
> > it just contains:
> >
> > module babyLotus
> >
> > go 1.15
>
> The module name should probably be a path name.
>
> I bet that "go build -n" is printing an error, and you are piping that
> error to sh.
>
> Ian
>
>
> > On Tuesday, February 2, 2021 at 3:04:39 AM UTC+5:30 Ian Lance Taylor 
> wrote:
> >>
> >> On Mon, Feb 1, 2021 at 11:24 AM saurav deshpande
> >>  wrote:
> >> >
> >> > We have a go.mod file.
> >> > The same make file worked with go 1.9.7 version, it did not support 
> mod.
> >> > I am trying to make this work with go 1.15, can you please suggest 
> changes that would make this work.
> >>
> >> What does your go.mod file look like?
> >>
> >> Ian
> >>
> >>
> >> > On Tuesday, February 2, 2021 at 12:48:51 AM UTC+5:30 Ian Lance Taylor 
> wrote:
> >> >>
> >> >> On Mon, Feb 1, 2021 at 10:35 AM saurav deshpande
> >> >>  wrote:
> >> >> >
> >> >> > Hello, I am trying to run a make file but it gives me the 
> following error:
> >> >> > go.mod: 1: module: not found
> >> >> > go 1.15: unknown command
> >> >>
> >> >> ...
> >> >>
> >> >> > go.o:
> >> >> > @mkdir -p $(BUILD_DIR)
> >> >> > @echo "[go] compiling go sources into a standalone .o file"
> >> >> > echo $(GOPATH)
> >> >> > @GOARCH=386 GOOS=linux GOPATH=$(GOPATH) COG_ENABLED=1 go build -n 
> 2>&1 | sed \
> >> >> > -e "1s|^|set -e\n|" \
> >> >> > -e "1s|^|export GOOS=linux\n|" \
> >> >> > -e "1s|^|export GOARCH=386\n|" \
> >> >> > -e "1s|^|export CGO_ENABLED=1\n|" \
> >> >> > -e "1s|^|WORK='$(BUILD_ABS_DIR)'\n|" \
> >> >> > -e "1s|^|alias pack='go tool pack'\n|" \
> >> >> > -e "/^mv/d" \
> >> >> > -e "s|-extld|-tmpdir='$(BUILD_ABS_DIR)' -linkmode=external 
> -extldflags='-nostdlib' -extld|g" \
> >> >> > | sh 2>&1 | sed -e "s/^/ | /g"
> >> >>
> >> >> This procedure--using "go build -n" and editing the output--is not
> >> >> supported and is going to be highly version-specific. Has this ever
> >> >> worked? Have you changed the version of Go that you are using?
> >> >>
> >> >> Do your sources have a go.mod file?
> >> >>
> >> >> Ian
> >> >
> >> > --
> >> > You received this message because you are subscribed to the Google 
> Groups "golang-nuts" group.
> >> > To unsubscribe from this group and stop receiving emails from it, 
> send an email to golang-nuts...@googlegroups.com.
> >> > To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/4cfebff8-f9ad-40b1-afab-253b8ba25e48n%40googlegroups.com
> .
> >
> > --
> > You received this message because you are subscribed to the Google 
> Groups "golang-nuts" group.
> > To unsubscribe from this group and stop receiving emails from it, send 
> an email to golang-nuts...@googlegroups.com.
> > To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/487b32a6--4025-b754-790d5f3820can%40googlegroups.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/bef3aea9-2ecb-4e9c-a395-992d8e0ed3edn%40googlegroups.com.

Re: [go-nuts] Go Create

[Kevin Chadwick]

> 
>> Note, that this is essentially one of the workflows suggested here (which I
>> would consider the canonical documentation for the "eng org wants to onboard
>> Go" case):
>> https://golang.org/ref/mod#private-modules
> 
> I have realised a goproxy may be needed rather than a http server as I expect 
> a
> domain with https, will be required for security despite being LOCALLY secure.
> 

>> Jim, I used replace this way just yesterday. It works fine, but is not easily
>> discoverable as a process and it took me a minute or two to figure it out.

It works well mostly but seems buggy at times. Saying functions aren't defined
with illogical fixes like connecting the internet and "updating transitive
dependencies. I'm starting to reluctantly come to the conclusion that I will
simply have to spend the time setting up a GoProxy, to be in a more tested 
place.

Having to occasionally delete empty lines and put them back seemingly randomly
after a build tag is enough to deal with already.

Maybe I should dos2unix all my files or something?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/ce6d8693-c722-165d-6eee-5ac2e559a6a6%40gmail.com.

Re: [go-nuts] Possible to make base64 pick the decode encoding automatically?

['Axel Wagner' via golang-nuts]

On Tue, Feb 2, 2021 at 6:43 PM Robert Engels  wrote:

> What “padding” are you referring to? Each must be 2 characters. And there
> is a standard that covers this https://tools.ietf.org/html/rfc4648
>

Yes, there indeed is. Section 5 describes a second encoding scheme, used
for URLs and the like. Section 3.2 also talks about the padding I'm
referring to (it's defined elsewhere in the standard) and mentions that, in
certain situations, it can be omitted. In particular, you can omit padding
and, in the decoder, implicitly pad to a multiple of 4 bytes.

I don't really understand what's the argument is here. The question was if
it is possible to handle all four encoding schemas
 supported by the Go
base64 package in one swoop, because as-is, the API requires you to pick
one schema and just see if it returns an error. Roger provided, IMO, a
pretty good answer to that: You can wrap the io.Reader in one that
transparently rewrites any of the four into one well-known one, which can
then be handled by the corresponding decoder. His link provides the code
for an implementation of such a reader.


>
> On Feb 2, 2021, at 10:57 AM, 'Axel Wagner' via golang-nuts <
> golang-nuts@googlegroups.com> wrote:
>
> 
> Rogers approach seems like the best one to me - wrap the input in a custom
> `io.Reader` that transparently replaces `-_` with `+/` respectively (and
> drop trailing `=`). The bufio approach doesn't work, because there is no
> guarantee that one of the distinguishing characters is early in the stream
> and the "send it to multiple decoders" approach duplicates effort and
> wastes resources.
>
> On Tue, Feb 2, 2021 at 5:43 PM Amnon  wrote:
>
>> Reading through a bufio.Reader is often useful for these situations.
>> You can peek the beginning of the input in order to decide which decoder
>> to use.
>>
>> Another option is to use the io.TeeReader to duplicate the reader,
>> and then send one copy to each decoder.
>> One will succeed, and give you the output.
>> But you will need to drain the one that fails to prevent the TeeReader
>> form stalling.
>>
>>
>> On Tuesday, 2 February 2021 at 14:17:19 UTC axel.wa...@googlemail.com
>> wrote:
>>
>>> This question isn't about the decoded data, but about *which* base64
>>> format is used - i.e. if it uses padding or not and what 2 characters are
>>> used outside of a-zA-Z0-9. The most common ones use +/ and -_, so it's easy
>>> to tell which is used and just accept either (and padding can be viewed as
>>> optional during decoding anyway).
>>>
>>> On Tue, Feb 2, 2021 at 2:37 PM Robert Engels 
>>> wrote:
>>>
 Base64 is always ASCII. The encoded data may be in an arbitrary format.
 You need to pass additional metadata or try and detect its encoding.

 On Feb 2, 2021, at 6:50 AM, roger peppe  wrote:

 
 In case you find it helpful, here's a clone of the base64 command that
 I wrote in Go. I did it precisely because I wanted to be able to decode any
 encoding scheme interchangeably.

 https://github.com/rogpeppe/misc/blob/master/cmd/base64/base64.go

 I agree that it might be useful to have some of this functionality
 available in the standard library.

   cheers,
 rog.

 On Tue, 2 Feb 2021 at 09:08, hey...@gmail.com  wrote:

> Hi,
>
> I have an io.Reader whose content is encoded in base64 with encoding
> type unknown. Since there shouldn't be any ambiguity between the two, is 
> it
> possible to make the base64 automatically pick the right one to decode?
>
> Currently I have to read everything out to pin down the encoding,
> which defeats the purpose of using an io.Reader.
>
> Is there a solution to this problem?
>
> Thanks in advance.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to golang-nuts...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/0ccee37d-319e-41b3-9bfd-3dc46e0fad78n%40googlegroups.com
> 
> .
>
 --
 You received this message because you are subscribed to the Google
 Groups "golang-nuts" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to golang-nuts...@googlegroups.com.
 To view this discussion on the web visit
 https://groups.google.com/d/msgid/golang-nuts/CAJhgacjkUUSr-dOPFU-W4vG_AXZRY_dYYe2ti-iPuu_XUL%2BNVw%40mail.gmail.com
 
 .

 --
 You received th

Re: [go-nuts] Re: Virtual time for testing

[Christian Worm Mortensen]

Hi Roger,

Thank you for sharing how you have solved the problem in the past and the
problems you have had. As I see it, my proposal would solve your
problem perfectly in many cases without the need to keep track of anything.
If you like it, it may be helpful to express your support:
https://github.com/golang/go/issues/44026

Thanks,

Christian

On Mon, Feb 1, 2021 at 11:54 AM roger peppe  wrote:

>
> On Sat, 30 Jan 2021 at 20:12, Christian Worm Mortensen 
> wrote:
>
>> Hi Mike,
>>
>> Thank you for your consideration. I think you exactly got the essence of
>> my question: How do I wait on all go routines to finish (or be blocked on
>> one or more channels) before advancing time.
>>
>
> This is an interesting problem that's not currently solved.
>
> In the past, I've made a lot of use of this package:
> https://pkg.go.dev/github.com/rogpeppe/clock
> with its associated implementation for tests:
> https://pkg.go.dev/github.com/rogpeppe/clock@v0.0.0-20190514195947-2896927a307a/testclock
> It was originally developed as part of Canonical's Juju project.
>
> To wait for goroutines to finish, you can use the WaitAdvance
> 
> method, which waits for at least n goroutines to block
> on the clock before advancing time. This relies, of course, on all the
> code under test using the Clock interface,
> but that's not usually that hard to arrange.
>
> There are a couple of deeper problems with this particular approach though:
>
>  - in order to use WaitAdvance, you need to know the total number of
> goroutines involved, but this is implementation-dependent, so someone
> can break tests by making an apparently innocuous change that happens to
> change goroutine count.
>
> - it's still easy to make mistakes. It's easy to assume that when the
> goroutines are blocked, the state that you're trying to observe
> is static, but there may well be other goroutines still running that have
> previously been triggered. This means that one
> can end up polling state anyway if you're trying to test behaviour of an
> independent "agent" goroutine.
>
> In the end, I've largely given up on this fake clock approach in favour of
> testing with real time (on the order of 10s of milliseconds not seconds)
> and polling to wait for externally visible changes. This approach isn't
> ideal either - if you make the time intervals too short, your
> tests will be flaky; too long and you're waiting too long for tests to
> run. But at least the tests aren't relying on details of
> the internal implementation.
>
> I'd love to see a way of fixing this in the standard Go runtime, but it's
> not easy. Goroutines can be blocked in system calls (e.g. making an HTTP
> call),
> while still making progress, so just "wait for everything to be blocked
> before advancing the clock" isn't a sufficient heuristic.
> Also I'm not sure that a single clock is good enough because you might
> well want to be able to time out your tests even as you're faking out
> the clock for the code being tested.
>
>   cheers,
> rog.
>
>
> A key thing I would like from such a solution is that it does not require
>> too heavy modifications to the code to be tested or put restrictions on how
>> it can do things.
>>
>> I think it may be possible to solve it with some explicit check in /
>> check out as I think you also suggest. I guess in essence you will check
>> out before you call select and check in again after select is done waiting.
>> I think this will still not work if buffered channels are used. But maybe
>> if buffered channels are mocked, it may be doable.
>>
>> I think I may want to make a feature request on this. I see several
>> options:
>>
>> * Make a version of runtime.gosched that only returns when no other go
>> routines can run
>> * Make it possible to read the number of go routines that are ready to
>> run. You could then make a loop where you call runtime.gosched until that
>> value is 0.
>> * Make it possible to start a special go routine when the system is
>> deadlocked.
>>
>> One problem is what to do if the program is waiting on external IO such
>> as the completion of an HTTP request. I guess in an ideal solution it would
>> be possible for the program to decide if it will advance time in that
>> situation or not.
>>
>> Please let me know if you have any ideas of other things to put into the
>> feature request.
>>
>> Thanks,
>>
>> Christian
>>
>> On Fri, Jan 29, 2021 at 9:25 PM mspr...@us.ibm.com 
>> wrote:
>>
>>> Volker: injecting sleep is a nice idea, in the general vein that Jesper
>>> said of injecting time.  However, as soon as we zoom out a step and need to
>>> test both that generator and the goroutine(s) consuming and acting upon
>>> that channel activity, we get back to the essence of the original question:
>>> how to test when we have a bunch of goroutines doing stuff and the test
>>> needs to wait for them all to finish before advancing t

Re: [go-nuts] Possible to make base64 pick the decode encoding automatically?

[Robert Engels]

What “padding” are you referring to? Each must be 2 characters. And there is a 
standard that covers this https://tools.ietf.org/html/rfc4648

> On Feb 2, 2021, at 10:57 AM, 'Axel Wagner' via golang-nuts 
>  wrote:
> 
> 
> Rogers approach seems like the best one to me - wrap the input in a custom 
> `io.Reader` that transparently replaces `-_` with `+/` respectively (and drop 
> trailing `=`). The bufio approach doesn't work, because there is no guarantee 
> that one of the distinguishing characters is early in the stream and the 
> "send it to multiple decoders" approach duplicates effort and wastes 
> resources.
> 
>> On Tue, Feb 2, 2021 at 5:43 PM Amnon  wrote:
>> Reading through a bufio.Reader is often useful for these situations.
>> You can peek the beginning of the input in order to decide which decoder to 
>> use.
>> 
>> Another option is to use the io.TeeReader to duplicate the reader,
>> and then send one copy to each decoder.
>> One will succeed, and give you the output.
>> But you will need to drain the one that fails to prevent the TeeReader form 
>> stalling.
>> 
>> 
>>> On Tuesday, 2 February 2021 at 14:17:19 UTC axel.wa...@googlemail.com wrote:
>>> This question isn't about the decoded data, but about *which* base64 format 
>>> is used - i.e. if it uses padding or not and what 2 characters are used 
>>> outside of a-zA-Z0-9. The most common ones use +/ and -_, so it's easy to 
>>> tell which is used and just accept either (and padding can be viewed as 
>>> optional during decoding anyway).
>>> 
 On Tue, Feb 2, 2021 at 2:37 PM Robert Engels  wrote:
 Base64 is always ASCII. The encoded data may be in an arbitrary format. 
 You need to pass additional metadata or try and detect its encoding. 
 
>> On Feb 2, 2021, at 6:50 AM, roger peppe  wrote:
>> 
> 
> In case you find it helpful, here's a clone of the base64 command that I 
> wrote in Go. I did it precisely because I wanted to be able to decode any 
> encoding scheme interchangeably.
> 
> https://github.com/rogpeppe/misc/blob/master/cmd/base64/base64.go
> 
> I agree that it might be useful to have some of this functionality 
> available in the standard library.
> 
>   cheers,
> rog.
> 
>> On Tue, 2 Feb 2021 at 09:08, hey...@gmail.com  wrote:
>> Hi,
>> 
>> I have an io.Reader whose content is encoded in base64 with encoding 
>> type unknown. Since there shouldn't be any ambiguity between the two, is 
>> it possible to make the base64 automatically pick the right one to 
>> decode?
>> 
>> Currently I have to read everything out to pin down the encoding, which 
>> defeats the purpose of using an io.Reader.
>> 
>> Is there a solution to this problem?
>> 
>> Thanks in advance.
>> 
>> 
>> -- 
>> You received this message because you are subscribed to the Google 
>> Groups "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send 
>> an email to golang-nuts...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/0ccee37d-319e-41b3-9bfd-3dc46e0fad78n%40googlegroups.com.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAJhgacjkUUSr-dOPFU-W4vG_AXZRY_dYYe2ti-iPuu_XUL%2BNVw%40mail.gmail.com.
 
 -- 
 You received this message because you are subscribed to the Google Groups 
 "golang-nuts" group.
 To unsubscribe from this group and stop receiving emails from it, send an 
 email to golang-nuts...@googlegroups.com.
>>> 
 To view this discussion on the web visit 
 https://groups.google.com/d/msgid/golang-nuts/CF76EF37-6F82-42FF-B4D6-4B9FC02F25FC%40ix.netcom.com.
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/44621cb2-fae7-4f40-9e50-35b157f4e838n%40googlegroups.com.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAEkBMfGGkOX9HzF6e178Cu6BD9Hg4x5LT7AutHkUDPvyK%2BFmYg%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe fr

Re: [go-nuts] Possible to make base64 pick the decode encoding automatically?

['Axel Wagner' via golang-nuts]

Rogers approach seems like the best one to me - wrap the input in a custom
`io.Reader` that transparently replaces `-_` with `+/` respectively (and
drop trailing `=`). The bufio approach doesn't work, because there is no
guarantee that one of the distinguishing characters is early in the stream
and the "send it to multiple decoders" approach duplicates effort and
wastes resources.

On Tue, Feb 2, 2021 at 5:43 PM Amnon  wrote:

> Reading through a bufio.Reader is often useful for these situations.
> You can peek the beginning of the input in order to decide which decoder
> to use.
>
> Another option is to use the io.TeeReader to duplicate the reader,
> and then send one copy to each decoder.
> One will succeed, and give you the output.
> But you will need to drain the one that fails to prevent the TeeReader
> form stalling.
>
>
> On Tuesday, 2 February 2021 at 14:17:19 UTC axel.wa...@googlemail.com
> wrote:
>
>> This question isn't about the decoded data, but about *which* base64
>> format is used - i.e. if it uses padding or not and what 2 characters are
>> used outside of a-zA-Z0-9. The most common ones use +/ and -_, so it's easy
>> to tell which is used and just accept either (and padding can be viewed as
>> optional during decoding anyway).
>>
>> On Tue, Feb 2, 2021 at 2:37 PM Robert Engels 
>> wrote:
>>
>>> Base64 is always ASCII. The encoded data may be in an arbitrary format.
>>> You need to pass additional metadata or try and detect its encoding.
>>>
>>> On Feb 2, 2021, at 6:50 AM, roger peppe  wrote:
>>>
>>> 
>>> In case you find it helpful, here's a clone of the base64 command that I
>>> wrote in Go. I did it precisely because I wanted to be able to decode any
>>> encoding scheme interchangeably.
>>>
>>> https://github.com/rogpeppe/misc/blob/master/cmd/base64/base64.go
>>>
>>> I agree that it might be useful to have some of this functionality
>>> available in the standard library.
>>>
>>>   cheers,
>>> rog.
>>>
>>> On Tue, 2 Feb 2021 at 09:08, hey...@gmail.com  wrote:
>>>
 Hi,

 I have an io.Reader whose content is encoded in base64 with encoding
 type unknown. Since there shouldn't be any ambiguity between the two, is it
 possible to make the base64 automatically pick the right one to decode?

 Currently I have to read everything out to pin down the encoding, which
 defeats the purpose of using an io.Reader.

 Is there a solution to this problem?

 Thanks in advance.


 --
 You received this message because you are subscribed to the Google
 Groups "golang-nuts" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to golang-nuts...@googlegroups.com.
 To view this discussion on the web visit
 https://groups.google.com/d/msgid/golang-nuts/0ccee37d-319e-41b3-9bfd-3dc46e0fad78n%40googlegroups.com
 
 .

>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to golang-nuts...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/golang-nuts/CAJhgacjkUUSr-dOPFU-W4vG_AXZRY_dYYe2ti-iPuu_XUL%2BNVw%40mail.gmail.com
>>> 
>>> .
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to golang-nuts...@googlegroups.com.
>>>
>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/golang-nuts/CF76EF37-6F82-42FF-B4D6-4B9FC02F25FC%40ix.netcom.com
>>> 
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/44621cb2-fae7-4f40-9e50-35b157f4e838n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAEkBMfGGkOX9HzF6

Re: [go-nuts] Possible to make base64 pick the decode encoding automatically?

[Amnon]

Reading through a bufio.Reader is often useful for these situations.
You can peek the beginning of the input in order to decide which decoder to 
use.

Another option is to use the io.TeeReader to duplicate the reader,
and then send one copy to each decoder.
One will succeed, and give you the output.
But you will need to drain the one that fails to prevent the TeeReader form 
stalling.


On Tuesday, 2 February 2021 at 14:17:19 UTC axel.wa...@googlemail.com wrote:

> This question isn't about the decoded data, but about *which* base64 
> format is used - i.e. if it uses padding or not and what 2 characters are 
> used outside of a-zA-Z0-9. The most common ones use +/ and -_, so it's easy 
> to tell which is used and just accept either (and padding can be viewed as 
> optional during decoding anyway).
>
> On Tue, Feb 2, 2021 at 2:37 PM Robert Engels  wrote:
>
>> Base64 is always ASCII. The encoded data may be in an arbitrary format. 
>> You need to pass additional metadata or try and detect its encoding. 
>>
>> On Feb 2, 2021, at 6:50 AM, roger peppe  wrote:
>>
>> 
>> In case you find it helpful, here's a clone of the base64 command that I 
>> wrote in Go. I did it precisely because I wanted to be able to decode any 
>> encoding scheme interchangeably.
>>
>> https://github.com/rogpeppe/misc/blob/master/cmd/base64/base64.go
>>
>> I agree that it might be useful to have some of this functionality 
>> available in the standard library.
>>
>>   cheers,
>> rog.
>>
>> On Tue, 2 Feb 2021 at 09:08, hey...@gmail.com  wrote:
>>
>>> Hi,
>>>
>>> I have an io.Reader whose content is encoded in base64 with encoding 
>>> type unknown. Since there shouldn't be any ambiguity between the two, is it 
>>> possible to make the base64 automatically pick the right one to decode?
>>>
>>> Currently I have to read everything out to pin down the encoding, which 
>>> defeats the purpose of using an io.Reader.
>>>
>>> Is there a solution to this problem?
>>>
>>> Thanks in advance.
>>>
>>>
>>> -- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to golang-nuts...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/golang-nuts/0ccee37d-319e-41b3-9bfd-3dc46e0fad78n%40googlegroups.com
>>>  
>>> 
>>> .
>>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/CAJhgacjkUUSr-dOPFU-W4vG_AXZRY_dYYe2ti-iPuu_XUL%2BNVw%40mail.gmail.com
>>  
>> 
>> .
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts...@googlegroups.com.
>>
> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/CF76EF37-6F82-42FF-B4D6-4B9FC02F25FC%40ix.netcom.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/44621cb2-fae7-4f40-9e50-35b157f4e838n%40googlegroups.com.

Re: [go-nuts] Possible to make base64 pick the decode encoding automatically?

['Axel Wagner' via golang-nuts]

This question isn't about the decoded data, but about *which* base64 format
is used - i.e. if it uses padding or not and what 2 characters are used
outside of a-zA-Z0-9. The most common ones use +/ and -_, so it's easy to
tell which is used and just accept either (and padding can be viewed as
optional during decoding anyway).

On Tue, Feb 2, 2021 at 2:37 PM Robert Engels  wrote:

> Base64 is always ASCII. The encoded data may be in an arbitrary format.
> You need to pass additional metadata or try and detect its encoding.
>
> On Feb 2, 2021, at 6:50 AM, roger peppe  wrote:
>
> 
> In case you find it helpful, here's a clone of the base64 command that I
> wrote in Go. I did it precisely because I wanted to be able to decode any
> encoding scheme interchangeably.
>
> https://github.com/rogpeppe/misc/blob/master/cmd/base64/base64.go
>
> I agree that it might be useful to have some of this functionality
> available in the standard library.
>
>   cheers,
> rog.
>
> On Tue, 2 Feb 2021 at 09:08, hey...@gmail.com  wrote:
>
>> Hi,
>>
>> I have an io.Reader whose content is encoded in base64 with encoding type
>> unknown. Since there shouldn't be any ambiguity between the two, is it
>> possible to make the base64 automatically pick the right one to decode?
>>
>> Currently I have to read everything out to pin down the encoding, which
>> defeats the purpose of using an io.Reader.
>>
>> Is there a solution to this problem?
>>
>> Thanks in advance.
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/golang-nuts/0ccee37d-319e-41b3-9bfd-3dc46e0fad78n%40googlegroups.com
>> 
>> .
>>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/CAJhgacjkUUSr-dOPFU-W4vG_AXZRY_dYYe2ti-iPuu_XUL%2BNVw%40mail.gmail.com
> 
> .
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/CF76EF37-6F82-42FF-B4D6-4B9FC02F25FC%40ix.netcom.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAEkBMfGFqX_aBoecB-BAMeFaDVQJnMh%2BbxTMig4wyBuM5RLWOw%40mail.gmail.com.

Re: [go-nuts] Possible to make base64 pick the decode encoding automatically?

[Robert Engels]

Base64 is always ASCII. The encoded data may be in an arbitrary format. You 
need to pass additional metadata or try and detect its encoding. 

> On Feb 2, 2021, at 6:50 AM, roger peppe  wrote:
> 
> 
> In case you find it helpful, here's a clone of the base64 command that I 
> wrote in Go. I did it precisely because I wanted to be able to decode any 
> encoding scheme interchangeably.
> 
> https://github.com/rogpeppe/misc/blob/master/cmd/base64/base64.go
> 
> I agree that it might be useful to have some of this functionality available 
> in the standard library.
> 
>   cheers,
> rog.
> 
>> On Tue, 2 Feb 2021 at 09:08, hey...@gmail.com  wrote:
>> Hi,
>> 
>> I have an io.Reader whose content is encoded in base64 with encoding type 
>> unknown. Since there shouldn't be any ambiguity between the two, is it 
>> possible to make the base64 automatically pick the right one to decode?
>> 
>> Currently I have to read everything out to pin down the encoding, which 
>> defeats the purpose of using an io.Reader.
>> 
>> Is there a solution to this problem?
>> 
>> Thanks in advance.
>> 
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/0ccee37d-319e-41b3-9bfd-3dc46e0fad78n%40googlegroups.com.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CAJhgacjkUUSr-dOPFU-W4vG_AXZRY_dYYe2ti-iPuu_XUL%2BNVw%40mail.gmail.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CF76EF37-6F82-42FF-B4D6-4B9FC02F25FC%40ix.netcom.com.

Re: [go-nuts] Possible to make base64 pick the decode encoding automatically?

[roger peppe]

In case you find it helpful, here's a clone of the base64 command that I
wrote in Go. I did it precisely because I wanted to be able to decode any
encoding scheme interchangeably.

https://github.com/rogpeppe/misc/blob/master/cmd/base64/base64.go

I agree that it might be useful to have some of this functionality
available in the standard library.

  cheers,
rog.

On Tue, 2 Feb 2021 at 09:08, hey...@gmail.com  wrote:

> Hi,
>
> I have an io.Reader whose content is encoded in base64 with encoding type
> unknown. Since there shouldn't be any ambiguity between the two, is it
> possible to make the base64 automatically pick the right one to decode?
>
> Currently I have to read everything out to pin down the encoding, which
> defeats the purpose of using an io.Reader.
>
> Is there a solution to this problem?
>
> Thanks in advance.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/0ccee37d-319e-41b3-9bfd-3dc46e0fad78n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAJhgacjkUUSr-dOPFU-W4vG_AXZRY_dYYe2ti-iPuu_XUL%2BNVw%40mail.gmail.com.

[go-nuts] Possible to make base64 pick the decode encoding automatically?

[hey...@gmail.com]

Hi,

I have an io.Reader whose content is encoded in base64 with encoding type 
unknown. Since there shouldn't be any ambiguity between the two, is it 
possible to make the base64 automatically pick the right one to decode?

Currently I have to read everything out to pin down the encoding, which 
defeats the purpose of using an io.Reader.

Is there a solution to this problem?

Thanks in advance.


-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/0ccee37d-319e-41b3-9bfd-3dc46e0fad78n%40googlegroups.com.

Re: [go-nuts] How to improve the parallelism of go routine？

[颜文泽]

ok
在2021年2月2日星期二 UTC+8 下午4:24:12 写道：

> On Mon, 2021-02-01 at 23:48 -0800, 颜文泽 wrote:
> > This is my code:
> > 2021-02-02 15-45-01 的屏幕截图.png
>
> Please don't post code as images.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/2c6cc907-28b1-4999-ba98-a4abb20373f0n%40googlegroups.com.

Re: [go-nuts] How to improve the parallelism of go routine？

['Dan Kortschak' via golang-nuts]

On Mon, 2021-02-01 at 23:48 -0800, 颜文泽 wrote:
> This is my code:
> 2021-02-02 15-45-01 的屏幕截图.png

Please don't post code as images.


-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/47e0791d802d01e56ac1526b226420a09047970a.camel%40kortschak.io.

Re: [go-nuts] How to improve the parallelism of go routine？

[颜文泽]

thank you。

在2021年2月2日星期二 UTC+8 下午4:17:46 写道：

> On Tue, Feb 2, 2021 at 8:48 AM 颜文泽  wrote:
>
>> And then this is the result, it's amazing.I think I know why my program 
>> is slow, the number of routines is too high
>
>
> 13 goroutines is certainly not "too high".
>  
>
>> but I found that the GOMAXPROCS function doesn't work, it's a really 
>> confusing phenomenon for me.
>>
> My example did not do anything, my understanding of the number of runtines 
>> should be 1 only Ah.
>>
>
> GOMAXPROCS specifies the maximum number of goroutines executing go code at 
> the same time - it does not limit the number of threads or the number of 
> goroutines you can start. The extra goroutines are probably both from 
> third-party dependencies and from the runtime itself (e.g. for the GC).
>  
>
>> [image: 2021-02-02 15-45-49 的屏幕截图.png]
>> 在2021年2月2日星期二 UTC+8 下午3:27:45 写道：
>>
>>> Vtune is very useful for squeezing the ultimate performance out of Go 
>>> programs, once you have done
>>> the usual optimisation, mimized allocations, io etc. 
>>>
>>> pprof is more than adequate for the average programmer. But when you 
>>> need to super-optimise 
>>> functions which implement math kernels, crypto functions, video codecs 
>>> etc, then without a HW perfomance
>>> counter based profiler such as vtune or linux perf, (
>>> https://perf.wiki.kernel.org/index.php/Main_Page)  you are shooting in 
>>> the dark.
>>> vtune not only tells you which functions are taking the most time, but 
>>> WHY these are taking a long time,
>>> how long the code is spending waiting for cache misses, and the 
>>> different kind of stall cycles which 
>>> kill performance on a modern CPU.
>>>
>>> Vtune or perf is also a great tool for teaching us about processors, and 
>>> helping us understand what influences
>>> the rate at which instructions are executed by them.
>>>
>>> The problem with vtune is that it is quite unfriendly and expensive (> 
>>> $3000 for a single floating license)!
>>> It also does not work on ARM processors (such as Apple M1).
>>>
>>> There has been a proposal to add performance counters to pprof.
>>>
>>> https://go.googlesource.com/proposal/+/refs/changes/08/219508/2/design/36821-perf-counter-pprof.md
>>> If accepted, this would give the power of vtune to the masses for free..
>>>
>>> On Tuesday, 2 February 2021 at 06:37:37 UTC nnsm...@gmail.com wrote:
>>>
 One more question, is it effective to use vtune to tune golang. I am 
 afraid that vtune is not suitable, although intel claims to be effective.
 在2021年2月2日星期二 UTC+8 下午2:32:40<颜文泽> 写道：

> Thanks, it's not memory db, but my current test is not involving io. 
> I'll take time to look at your information, thanks a lot. Also I found 
> that 
> many of the functions with high cpi rate are runtime functions, is the 
> overhead of these functions unavoidable?The following diagram is for a 
> single routine:
> [image: 2021-02-02 14-25-33 的屏幕截图.png]
> The following chart is for the 8 routines:
> [image: 2021-02-02 14-25-56 的屏幕截图.png]
> 在2021年2月2日星期二 UTC+8 下午2:27:39 写道：
>
>> Unless it is an in memory database, I would expect the IO costs to 
>> dwarf the cpu costs, but I guess a lot depends on how you define 
>> ‘analytical processing’.
>>
>> In my experience, “out of the box” performance of Go routines in IO 
>> processing is outstanding.
>>
>> For the cpu bound case, I think with threads, cpu assignments 
>> (cpuset), etc. you can probably create a higher performing system in 
>> some 
>> cases - but it’s a lot of work.
>>
>> Even without that, I think the scheduler in most Linux systems is 
>> more mature than the Go scheduler, and makes better choices for cache 
>> affinity, etc. It’s very hard to design a high performance cpu bound 
>> system 
>> that runs on a general purpose OS or language/platform. Without 
>> knowledge 
>> of the olap db design it is very hard to make a recommendation.
>>
>> This is some suggested reading to help you in your journey 
>> https://dave.cheney.net/high-performance-go-workshop/dotgo-paris.html
>>
>> On Feb 2, 2021, at 12:07 AM, 颜文泽  wrote:
>>
>> I don't know much about the internal implementation of golang, sorry. 
>> I was a c programmer and I tried to implement the original logic (olap 
>> database) by using routine as a thread replacement. But I found that I 
>> would encounter bottlenecks, and I don't know how to solve them. Maybe I 
>> should study the implementation of routine before I can write the right 
>> code.
>>
>> 在2021年2月2日星期二 UTC+8 下午12:21:44 写道：
>>
>>> You wrote “I found that cache misses from routines switching is also 
>>> a headache”. 
>>>
>>> They would not be switching if they are cpu bound and there are less 
>>> of than number of cpus. Remember too that you need some % of the cpus 
>>> to 
>>> execute the

Re: [go-nuts] How to improve the parallelism of go routine？

['Axel Wagner' via golang-nuts]

On Tue, Feb 2, 2021 at 8:48 AM 颜文泽  wrote:

> And then this is the result, it's amazing.I think I know why my program is
> slow, the number of routines is too high


13 goroutines is certainly not "too high".


> but I found that the GOMAXPROCS function doesn't work, it's a really
> confusing phenomenon for me.
>
My example did not do anything, my understanding of the number of runtines
> should be 1 only Ah.
>

GOMAXPROCS specifies the maximum number of goroutines executing go code at
the same time - it does not limit the number of threads or the number of
goroutines you can start. The extra goroutines are probably both from
third-party dependencies and from the runtime itself (e.g. for the GC).


> [image: 2021-02-02 15-45-49 的屏幕截图.png]
> 在2021年2月2日星期二 UTC+8 下午3:27:45 写道：
>
>> Vtune is very useful for squeezing the ultimate performance out of Go
>> programs, once you have done
>> the usual optimisation, mimized allocations, io etc.
>>
>> pprof is more than adequate for the average programmer. But when you need
>> to super-optimise
>> functions which implement math kernels, crypto functions, video codecs
>> etc, then without a HW perfomance
>> counter based profiler such as vtune or linux perf, (
>> https://perf.wiki.kernel.org/index.php/Main_Page)  you are shooting in
>> the dark.
>> vtune not only tells you which functions are taking the most time, but
>> WHY these are taking a long time,
>> how long the code is spending waiting for cache misses, and the different
>> kind of stall cycles which
>> kill performance on a modern CPU.
>>
>> Vtune or perf is also a great tool for teaching us about processors, and
>> helping us understand what influences
>> the rate at which instructions are executed by them.
>>
>> The problem with vtune is that it is quite unfriendly and expensive (>
>> $3000 for a single floating license)!
>> It also does not work on ARM processors (such as Apple M1).
>>
>> There has been a proposal to add performance counters to pprof.
>>
>> https://go.googlesource.com/proposal/+/refs/changes/08/219508/2/design/36821-perf-counter-pprof.md
>> If accepted, this would give the power of vtune to the masses for free..
>>
>> On Tuesday, 2 February 2021 at 06:37:37 UTC nnsm...@gmail.com wrote:
>>
>>> One more question, is it effective to use vtune to tune golang. I am
>>> afraid that vtune is not suitable, although intel claims to be effective.
>>> 在2021年2月2日星期二 UTC+8 下午2:32:40<颜文泽> 写道：
>>>
 Thanks, it's not memory db, but my current test is not involving io.
 I'll take time to look at your information, thanks a lot. Also I found that
 many of the functions with high cpi rate are runtime functions, is the
 overhead of these functions unavoidable?The following diagram is for a
 single routine:
 [image: 2021-02-02 14-25-33 的屏幕截图.png]
 The following chart is for the 8 routines:
 [image: 2021-02-02 14-25-56 的屏幕截图.png]
 在2021年2月2日星期二 UTC+8 下午2:27:39 写道：

> Unless it is an in memory database, I would expect the IO costs to
> dwarf the cpu costs, but I guess a lot depends on how you define
> ‘analytical processing’.
>
> In my experience, “out of the box” performance of Go routines in IO
> processing is outstanding.
>
> For the cpu bound case, I think with threads, cpu assignments
> (cpuset), etc. you can probably create a higher performing system in some
> cases - but it’s a lot of work.
>
> Even without that, I think the scheduler in most Linux systems is more
> mature than the Go scheduler, and makes better choices for cache affinity,
> etc. It’s very hard to design a high performance cpu bound system that 
> runs
> on a general purpose OS or language/platform. Without knowledge of the 
> olap
> db design it is very hard to make a recommendation.
>
> This is some suggested reading to help you in your journey
> https://dave.cheney.net/high-performance-go-workshop/dotgo-paris.html
>
> On Feb 2, 2021, at 12:07 AM, 颜文泽  wrote:
>
> I don't know much about the internal implementation of golang, sorry.
> I was a c programmer and I tried to implement the original logic (olap
> database) by using routine as a thread replacement. But I found that I
> would encounter bottlenecks, and I don't know how to solve them. Maybe I
> should study the implementation of routine before I can write the right
> code.
>
> 在2021年2月2日星期二 UTC+8 下午12:21:44 写道：
>
>> You wrote “I found that cache misses from routines switching is also
>> a headache”.
>>
>> They would not be switching if they are cpu bound and there are less
>> of than number of cpus. Remember too that you need some % of the cpus to
>> execute the runtime GC code and other housekeeping.
>>
>> > On Feb 1, 2021, at 10:04 PM, 颜文泽  wrote:
>> >
>> > I found that cache misses from routines switching is also a
>> headache
>>
>>
> --
> You received

Re: [go-nuts] How to improve the parallelism of go routine？

[颜文泽]

Probably introduced by a third-party package. I'll troubleshoot.

在2021年2月2日星期二 UTC+8 下午3:50:32<颜文泽> 写道：

> Note: I don't use the init function
> 在2021年2月2日星期二 UTC+8 下午3:48:26<颜文泽> 写道：
>
>> If it works, it's fine, I'll just keep using vtune. I only work on x86 
>> anyway. That said, I found another miracle, my program has 13 routines as 
>> soon as it starts. It's so peculiar. I simply can't understand why this is.
>>
>> This is my code:
>>
>> [image: 2021-02-02 15-45-01 的屏幕截图.png]
>> And then this is the result, it's amazing.I think I know why my program 
>> is slow, the number of routines is too high, but I found that the 
>> GOMAXPROCS function doesn't work, it's a really confusing phenomenon for me.
>> My example did not do anything, my understanding of the number of 
>> runtines should be 1 only Ah.
>> [image: 2021-02-02 15-45-49 的屏幕截图.png]
>> 在2021年2月2日星期二 UTC+8 下午3:27:45 写道：
>>
>>> Vtune is very useful for squeezing the ultimate performance out of Go 
>>> programs, once you have done
>>> the usual optimisation, mimized allocations, io etc. 
>>>
>>> pprof is more than adequate for the average programmer. But when you 
>>> need to super-optimise 
>>> functions which implement math kernels, crypto functions, video codecs 
>>> etc, then without a HW perfomance
>>> counter based profiler such as vtune or linux perf, (
>>> https://perf.wiki.kernel.org/index.php/Main_Page)  you are shooting in 
>>> the dark.
>>> vtune not only tells you which functions are taking the most time, but 
>>> WHY these are taking a long time,
>>> how long the code is spending waiting for cache misses, and the 
>>> different kind of stall cycles which 
>>> kill performance on a modern CPU.
>>>
>>> Vtune or perf is also a great tool for teaching us about processors, and 
>>> helping us understand what influences
>>> the rate at which instructions are executed by them.
>>>
>>> The problem with vtune is that it is quite unfriendly and expensive (> 
>>> $3000 for a single floating license)!
>>> It also does not work on ARM processors (such as Apple M1).
>>>
>>> There has been a proposal to add performance counters to pprof.
>>>
>>> https://go.googlesource.com/proposal/+/refs/changes/08/219508/2/design/36821-perf-counter-pprof.md
>>> If accepted, this would give the power of vtune to the masses for free..
>>>
>>> On Tuesday, 2 February 2021 at 06:37:37 UTC nnsm...@gmail.com wrote:
>>>
 One more question, is it effective to use vtune to tune golang. I am 
 afraid that vtune is not suitable, although intel claims to be effective.
 在2021年2月2日星期二 UTC+8 下午2:32:40<颜文泽> 写道：

> Thanks, it's not memory db, but my current test is not involving io. 
> I'll take time to look at your information, thanks a lot. Also I found 
> that 
> many of the functions with high cpi rate are runtime functions, is the 
> overhead of these functions unavoidable?The following diagram is for a 
> single routine:
> [image: 2021-02-02 14-25-33 的屏幕截图.png]
> The following chart is for the 8 routines:
> [image: 2021-02-02 14-25-56 的屏幕截图.png]
> 在2021年2月2日星期二 UTC+8 下午2:27:39 写道：
>
>> Unless it is an in memory database, I would expect the IO costs to 
>> dwarf the cpu costs, but I guess a lot depends on how you define 
>> ‘analytical processing’.
>>
>> In my experience, “out of the box” performance of Go routines in IO 
>> processing is outstanding.
>>
>> For the cpu bound case, I think with threads, cpu assignments 
>> (cpuset), etc. you can probably create a higher performing system in 
>> some 
>> cases - but it’s a lot of work.
>>
>> Even without that, I think the scheduler in most Linux systems is 
>> more mature than the Go scheduler, and makes better choices for cache 
>> affinity, etc. It’s very hard to design a high performance cpu bound 
>> system 
>> that runs on a general purpose OS or language/platform. Without 
>> knowledge 
>> of the olap db design it is very hard to make a recommendation.
>>
>> This is some suggested reading to help you in your journey 
>> https://dave.cheney.net/high-performance-go-workshop/dotgo-paris.html
>>
>> On Feb 2, 2021, at 12:07 AM, 颜文泽  wrote:
>>
>> I don't know much about the internal implementation of golang, sorry. 
>> I was a c programmer and I tried to implement the original logic (olap 
>> database) by using routine as a thread replacement. But I found that I 
>> would encounter bottlenecks, and I don't know how to solve them. Maybe I 
>> should study the implementation of routine before I can write the right 
>> code.
>>
>> 在2021年2月2日星期二 UTC+8 下午12:21:44 写道：
>>
>>> You wrote “I found that cache misses from routines switching is also 
>>> a headache”. 
>>>
>>> They would not be switching if they are cpu bound and there are less 
>>> of than number of cpus. Remember too that you need some % of the cpus 
>>> t