Re: [go-nuts] Does not match go tool version "go1.11.2"

[Ian Lance Taylor]

On Mon, Nov 26, 2018 at 8:08 PM Freddy Martinez  wrote:
>
> Hey Guys.. I'm really new with go (only 4 days) and I'm trying to install 
> graphql-go and I have this error
>
>
> freddy@Freddys-MacBook-Pro:~/go$ go get github.com/graphql-go/graphql
>
> # internal/race
>
> compile: version "go1.11" does not match go tool version "go1.11.2"
>
> # unicode/utf8
>
> compile: version "go1.11" does not match go tool version "go1.11.2"
>
> # errors
>
> compile: version "go1.11" does not match go tool version "go1.11.2"
>
> # runtime/internal/sys
>
> compile: version "go1.11" does not match go tool version "go1.11.2"
>
> # math/bits
>
> compile: version "go1.11" does not match go tool version "go1.11.2"
>
> # encoding
>
> compile: version "go1.11" does not match go tool version "go1.11.2"
>
> # unicode/utf16
>
> compile: version "go1.11" does not match go tool version "go1.11.2"
>
> # github.com/graphql-go/graphql/language/kinds
>
> compile: version "go1.11" does not match go tool version "go1.11.2"
>
> # github.com/graphql-go/graphql/language/source
>
> compile: version "go1.11" does not match go tool version "go1.11.2"
>
> # unicode
>
> compile: version "go1.11" does not match go tool version "go1.11.2"
>
> # internal/cpu
>
> compile: version "go1.11" does not match go tool version "go1.11.2"
>
> # runtime/internal/atomic
>
> compile: version "go1.11" does not match go tool version "go1.11.2"
>
> # sync/atomic
>
> compile: version "go1.11" does not match go tool version "go1.11.2"
>
>
>
> What does this error means ?

It means that you are using compiler version 1.11 with pre-built
standard library packages built by version 1.11.2.  I'm not sure how
that could happen, unless perhaps you installed 1.11, copied the go
tools somewhere, then installed 1.11.2, then copied them back.  I
suggest that you remove your current installation entirely and start
afresh.

Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] contracts thought experiment: dimension checks

[Tamás Gulácsi]

Just a silly idea: what if you make the dimensions (m,n) part of the type, not 
just T, somehow?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: contracts thought experiment: dimension checks

[Ian Lance Taylor]

On Mon, Nov 26, 2018 at 5:14 PM Dan Kortschak
 wrote:
>
> Background:
>
> As part of the Gonum dev team, I have an interest in using Go for
> scientific computing. This stems from long experience with unsafe
> computing environments often used in, particularly, bioinformatics
> (perl, python and R being standouts in the field, with packages that
> often use dubious design and implementation practices that allow silent
> failures giving plausible but incorrect results rather than following
> than following the rule of repair. In Gonum packages we (perhaps non-
> idiomatically in the context of the broader Go ecosystem) insist on
> failing noisily when we can rather than allowing silent progression to
> an incorrect result; we use panics instead of errors a fair bit because
> they cannot be implicitly ignored.
>
> Question:
>
> One of the areas where we do this is in the mat package for matrix
> arithmetic and linear algebra. We use panics in lieue of a compile-time
> type check for appropriate input matrix shape (e.g. the check needed to
> ensure [mxw] <- [mxn] x [nxw] is a valid operation). I don't see any
> way to encode this kind of information into a contract as it stands in
> the current contracts draft. I'm not entirely sure that any kind of
> reasonably simple contract system *would* allow this. If we assume that
> there is some type parameterised function that operates on say [n][m]T
> to perform a matrix mult, can anyone see a way of encoding the required
> constraint?

There is no way to express that constraint.  In general contracts are
only intended to express things that can be described in the type
system.  What you are asking for--a relationship between the lengths
of different slice types--can not be described in the type system.

This of it this way: the purpose of a contract is to decide when it is
OK to instantiate a generic function with a given type.  When making
that decision at compile time, we don't know the length of a slice.
So we can't use the length of a slice when deciding whether to
instantiate a generic function.

(That said, one could imagine extending contracts to permit them to
describe non-type arguments, and then one could use an integer
argument to set the size of an array, and then one could write a
contract describing the relationship between arrays.  Of course the
caller would have to pass arrays rather than slices, which would not
work well with sub-matrixes.)

Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] contracts thought experiment: dimension checks

[Burak Serdar]

On Mon, Nov 26, 2018 at 6:15 PM Dan Kortschak
 wrote:
>
> Background:
>
> As part of the Gonum dev team, I have an interest in using Go for
> scientific computing. This stems from long experience with unsafe
> computing environments often used in, particularly, bioinformatics
> (perl, python and R being standouts in the field, with packages that
> often use dubious design and implementation practices that allow silent
> failures giving plausible but incorrect results rather than following
> than following the rule of repair. In Gonum packages we (perhaps non-
> idiomatically in the context of the broader Go ecosystem) insist on
> failing noisily when we can rather than allowing silent progression to
> an incorrect result; we use panics instead of errors a fair bit because
> they cannot be implicitly ignored.
>
> Question:
>
> One of the areas where we do this is in the mat package for matrix
> arithmetic and linear algebra. We use panics in lieue of a compile-time
> type check for appropriate input matrix shape (e.g. the check needed to
> ensure [mxw] <- [mxn] x [nxw] is a valid operation). I don't see any
> way to encode this kind of information into a contract as it stands in
> the current contracts draft. I'm not entirely sure that any kind of
> reasonably simple contract system *would* allow this. If we assume that
> there is some type parameterised function that operates on say [n][m]T
> to perform a matrix mult, can anyone see a way of encoding the required
> constraint?

Is this hypothetical matrix mult function working on arrays, or
slices? If it is to work on arrays, is that not very restrictive,
since all matrices passed to it must have constant dimensions? If it
is to work on slices, then there is no compile-time size information.


>
> thanks
> Dan
>
> --
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] How to write unit test for init()?

[Mhd Shulhan]

On Tue, 27 Nov 2018, 11:08 Vast Peng 
>
>   when importing *component*, Golang will call *init() *automatically.
>

One way to do it is by moving the init into another function and call it
from init.  That way you can create test for the function.

>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] How to write unit test for init()?

[Vast Peng]

The code below:

package component

import "os"

type BaseicConfig struct{
}

func init() {
 configDir := os.Getenv(constant.ProjectRootEnv)
 if configDir == "" {
 // run here.
 log.Panic(constant.InvalidEnvVariables)
 }


 configPath := filepath.Join(configDir, constant.ConfigName)
 if _, err := toml.DecodeFile(configPath, ); err != nil {
 log.Panic(err.Error())
 }
}

I want to test it.

Problem:

  when importing *component*, Golang will call *init() *automatically.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Does not match go tool version "go1.11.2"

[Freddy Martinez]

Hey Guys.. I'm really new with go (only 4 days) and I'm trying to install 
graphql-go and I have this error


freddy@Freddys-MacBook-Pro:~/go$ go get github.com/graphql-go/graphql

# internal/race

compile: version "go1.11" does not match go tool version "go1.11.2"

# unicode/utf8

compile: version "go1.11" does not match go tool version "go1.11.2"

# errors

compile: version "go1.11" does not match go tool version "go1.11.2"

# runtime/internal/sys

compile: version "go1.11" does not match go tool version "go1.11.2"

# math/bits

compile: version "go1.11" does not match go tool version "go1.11.2"

# encoding

compile: version "go1.11" does not match go tool version "go1.11.2"

# unicode/utf16

compile: version "go1.11" does not match go tool version "go1.11.2"

# github.com/graphql-go/graphql/language/kinds

compile: version "go1.11" does not match go tool version "go1.11.2"

# github.com/graphql-go/graphql/language/source

compile: version "go1.11" does not match go tool version "go1.11.2"

# unicode

compile: version "go1.11" does not match go tool version "go1.11.2"

# internal/cpu

compile: version "go1.11" does not match go tool version "go1.11.2"

# runtime/internal/atomic

compile: version "go1.11" does not match go tool version "go1.11.2"

# sync/atomic

compile: version "go1.11" does not match go tool version "go1.11.2"


What does this error means ?

Regards

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] The most performant deque of all time?

[robert engels]

No problem, but just one last word on this… 

You have 748 lines of internal unit tests and only 295 lines of actual code.. 
This IMO does not lend itself to maintainable & flexible code. If your 295 
lines needs 748 to verify its correctness something is wrong. You have an 
additional 400 lines of integration tests - if those are full coverage they 
should be more than enough IMO, but again, just something to think about and to 
each his own.



> On Nov 26, 2018, at 9:56 PM, Christian Petrin  
> wrote:
> 
> Moved the non-unit tests to the "deque_test" package. The tests 
> , package-wise, look as they should now. 
> Thanks Robert for the suggestion. :-)
> 
> On Mon, Nov 26, 2018 at 2:01 PM robert engels  > wrote:
> It’s funny though, if you look at the container/ring tests, they test the 
> internal structure, but there are no tests of the behavior, so if the data 
> structure design is not correct, the tests will pass but the operations may 
> not work as expected (especially after a refactor). A case of I know what 
> should happen…
> 
> And then some public methods like Do()/Move() are not tested at all.
> 
> 
>> On Nov 26, 2018, at 3:43 PM, robert engels > > wrote:
>> 
>> My “none” looks to have been a little strong… Clearly whoever wrote the 
>> container package believes in testing as the OP. A more in-depth review 
>> shows other packages with similar “private tests”, but typically they seem 
>> to be testing the public API only. Then there are packages like sync that 
>> have no private tests…
>> 
>> It appears there is no standard in the stdlib as far as this goes :)
>> 
>> Like I said, it is my opinion, and I’m sure others feel differently. It’s 
>> served me well - mileage may vary.
>> 
>>> On Nov 26, 2018, at 3:32 PM, Dan Kortschak >> > wrote:
>>> 
>>> container/ring (a arguably non-idiomatic stdlib package does indeed
>>> contain this kind of test). It also does not have in code asserts,
>>> which are something that I've found to be very rare in Go code.
>>> 
>>> On Mon, 2018-11-26 at 12:09 -0600, robert engels wrote:
 I am just offering, and many would disagree, that unit tests that
 like are brittle and not worthwhile. I don’t see them in the Go
 stdlib for major packages, but I could be wrong.
 
 My thought on this is that if you are testing the nitty details of
 the implementation, you need to get that “correct” twice - and if it
 is failing, you don’t really know which one is wrong… This is
 especially problematic as things are refactored - all of those types
 of tests break.
 
 I’m of the opinion - test the behavior comprehensively, and expose
 the expected usage (and corresponding design) there.
 
> 
> On Nov 26, 2018, at 11:58 AM, Christian Petrin  il.com > wrote:
> 
> Hi Robert,
> 
> the deque has unit, integration and API tests. The unit tests, I
> agree, are hard to follow as they have to check all internal
> properties to make sure the code is doing what it is supposed to do
> (keep a well formed ring linked list during all steps). Given their
> nature (unit tests), the only way to access the internal deque
> variables is to have them in the same package.
> 
> Regarding the other, API and integration tests, I agree 100%. They
> should be in the dequeue_test package as they are not supposed to
> access (or know of) any of the deque internal variables. I'll move
> these tests to a different file in the dequeue_test package.
> 
> Really appreciate the suggestion (correction, really).
> 
> Thanks,
> Christian
> 
> 
> 
> 
> On Mon, Nov 26, 2018 at 9:42 AM robert engels  
> m >> wrote:
> Just an FYI, IMO your testing is incorrect.
> 
> Your tests should be in a package dequeue_test so that they cannot
> access the internal details. This makes them brittle and hard to
> follow.
> 
> For reference, look at the Go stdlib sync/map.go and
> sync/map_test.go. All of the tests are in sync_test for all of the
> structs/methods.
> 
> 
>> 
>> On Nov 26, 2018, at 10:59 AM, Christian Petrin > mail.com  > >> wrote:
>> 
>> Hello,
>> 
>> I'm working on a logging system called CloudLogger > 
>> b.com/cloud-logger/docs > and to cut to 
>> the chase, CloudLogger
>> needs an unbounded in-memory queue. The idea is to use the queue
>> as a sequential data store. As there's no telling beforehand how
>> much data will need to be stored, the queue has to be able to

Re: [go-nuts] The most performant deque of all time?

[Christian Petrin]

Moved the non-unit tests to the "deque_test" package. The tests
, package-wise, look as they should now.
Thanks Robert for the suggestion. :-)

On Mon, Nov 26, 2018 at 2:01 PM robert engels  wrote:

> It’s funny though, if you look at the container/ring tests, they test the
> internal structure, but there are no tests of the behavior, so if the data
> structure design is not correct, the tests will pass but the operations may
> not work as expected (especially after a refactor). A case of I know what
> should happen…
>
> And then some public methods like Do()/Move() are not tested at all.
>
>
> On Nov 26, 2018, at 3:43 PM, robert engels  wrote:
>
> My “none” looks to have been a little strong… Clearly whoever wrote the
> container package believes in testing as the OP. A more in-depth review
> shows other packages with similar “private tests”, but typically they seem
> to be testing the public API only. Then there are packages like sync that
> have no private tests…
>
> It appears there is no standard in the stdlib as far as this goes :)
>
> Like I said, it is my opinion, and I’m sure others feel differently. It’s
> served me well - mileage may vary.
>
> On Nov 26, 2018, at 3:32 PM, Dan Kortschak 
> wrote:
>
> container/ring (a arguably non-idiomatic stdlib package does indeed
> contain this kind of test). It also does not have in code asserts,
> which are something that I've found to be very rare in Go code.
>
> On Mon, 2018-11-26 at 12:09 -0600, robert engels wrote:
>
> I am just offering, and many would disagree, that unit tests that
> like are brittle and not worthwhile. I don’t see them in the Go
> stdlib for major packages, but I could be wrong.
>
> My thought on this is that if you are testing the nitty details of
> the implementation, you need to get that “correct” twice - and if it
> is failing, you don’t really know which one is wrong… This is
> especially problematic as things are refactored - all of those types
> of tests break.
>
> I’m of the opinion - test the behavior comprehensively, and expose
> the expected usage (and corresponding design) there.
>
>
> On Nov 26, 2018, at 11:58 AM, Christian Petrin  il.com> wrote:
>
> Hi Robert,
>
> the deque has unit, integration and API tests. The unit tests, I
> agree, are hard to follow as they have to check all internal
> properties to make sure the code is doing what it is supposed to do
> (keep a well formed ring linked list during all steps). Given their
> nature (unit tests), the only way to access the internal deque
> variables is to have them in the same package.
>
> Regarding the other, API and integration tests, I agree 100%. They
> should be in the dequeue_test package as they are not supposed to
> access (or know of) any of the deque internal variables. I'll move
> these tests to a different file in the dequeue_test package.
>
> Really appreciate the suggestion (correction, really).
>
> Thanks,
> Christian
>
>
>
>
> On Mon, Nov 26, 2018 at 9:42 AM robert engels  m > wrote:
> Just an FYI, IMO your testing is incorrect.
>
> Your tests should be in a package dequeue_test so that they cannot
> access the internal details. This makes them brittle and hard to
> follow.
>
> For reference, look at the Go stdlib sync/map.go and
> sync/map_test.go. All of the tests are in sync_test for all of the
> structs/methods.
>
>
>
> On Nov 26, 2018, at 10:59 AM, Christian Petrin  mail.com >
> wrote:
>
> Hello,
>
> I'm working on a logging system called CloudLogger  b.com/cloud-logger/docs> and to cut to the chase, CloudLogger
> needs an unbounded in-memory queue. The idea is to use the queue
> as a sequential data store. As there's no telling beforehand how
> much data will need to be stored, the queue has to be able to
> scale to support large amounts of data, so CloudLogger needs an
> unbounded queue, a very fast and efficient one.
>
> Noticing Go doesn't offer an unbounded queue implementation, I
> came up with an idea to build a queue based on linked slices.
>
> I was so impressed with the results that I decided to spend some
> time researching about other queue designs and ways to improve
> it. I'm finally done with the research and I feel like the new
> queue is worth being considered to be part of the standard
> library.
>
> So please take a look at the issue. All suggestions, questions,
> comments, etc are most welcome!
>
> Due to many suggestions, I have deployed the deque as a proper
> external package. In order to help validate the deque and get it
> into the standard library, I need to somehow get people to start
> using it. If you are using a queue/stack/deque, please consider
> switching to the new deque. Also if you know someone who is using
> a queue/stack/deque, please let them know about the deque.
>
> I'm extremely interested in improving the queue and I love
> challenges. If you suspect there's a possibly better design or a
> 

[go-nuts] contracts thought experiment: dimension checks

[Dan Kortschak]

Background:

As part of the Gonum dev team, I have an interest in using Go for
scientific computing. This stems from long experience with unsafe
computing environments often used in, particularly, bioinformatics
(perl, python and R being standouts in the field, with packages that
often use dubious design and implementation practices that allow silent
failures giving plausible but incorrect results rather than following
than following the rule of repair. In Gonum packages we (perhaps non-
idiomatically in the context of the broader Go ecosystem) insist on
failing noisily when we can rather than allowing silent progression to
an incorrect result; we use panics instead of errors a fair bit because
they cannot be implicitly ignored.

Question:

One of the areas where we do this is in the mat package for matrix
arithmetic and linear algebra. We use panics in lieue of a compile-time
type check for appropriate input matrix shape (e.g. the check needed to
ensure [mxw] <- [mxn] x [nxw] is a valid operation). I don't see any
way to encode this kind of information into a contract as it stands in
the current contracts draft. I'm not entirely sure that any kind of
reasonably simple contract system *would* allow this. If we assume that
there is some type parameterised function that operates on say [n][m]T
to perform a matrix mult, can anyone see a way of encoding the required
constraint?

thanks
Dan

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Important design missing from Errors overview

[Ian Lance Taylor]

On Mon, Nov 26, 2018 at 1:31 PM  wrote:
>
> While reading 
> https://go.googlesource.com/proposal/+/master/design/go2draft-error-values-overview.md#other-go-designs
>  I noticed that an important design is missing from the links below. I am 
> talking about Ben's https://middlemost.com/failure-is-your-domain/ which I 
> think offers some very good insights that are missing from the current 
> proposals. I think it is important to be added and be considered.

Thanks, it's a good article, but I'm not sure what feedback to take
from it regarding the error design drafts that were made.  In any case
a good place to respond is the wiki feedback pages mentioned at
https://go.googlesource.com/proposal/+/master/design/go2draft.md .

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] The most performant deque of all time?

[robert engels]

It’s funny though, if you look at the container/ring tests, they test the 
internal structure, but there are no tests of the behavior, so if the data 
structure design is not correct, the tests will pass but the operations may not 
work as expected (especially after a refactor). A case of I know what should 
happen…

And then some public methods like Do()/Move() are not tested at all.


> On Nov 26, 2018, at 3:43 PM, robert engels  wrote:
> 
> My “none” looks to have been a little strong… Clearly whoever wrote the 
> container package believes in testing as the OP. A more in-depth review shows 
> other packages with similar “private tests”, but typically they seem to be 
> testing the public API only. Then there are packages like sync that have no 
> private tests…
> 
> It appears there is no standard in the stdlib as far as this goes :)
> 
> Like I said, it is my opinion, and I’m sure others feel differently. It’s 
> served me well - mileage may vary.
> 
>> On Nov 26, 2018, at 3:32 PM, Dan Kortschak > > wrote:
>> 
>> container/ring (a arguably non-idiomatic stdlib package does indeed
>> contain this kind of test). It also does not have in code asserts,
>> which are something that I've found to be very rare in Go code.
>> 
>> On Mon, 2018-11-26 at 12:09 -0600, robert engels wrote:
>>> I am just offering, and many would disagree, that unit tests that
>>> like are brittle and not worthwhile. I don’t see them in the Go
>>> stdlib for major packages, but I could be wrong.
>>> 
>>> My thought on this is that if you are testing the nitty details of
>>> the implementation, you need to get that “correct” twice - and if it
>>> is failing, you don’t really know which one is wrong… This is
>>> especially problematic as things are refactored - all of those types
>>> of tests break.
>>> 
>>> I’m of the opinion - test the behavior comprehensively, and expose
>>> the expected usage (and corresponding design) there.
>>> 
 
 On Nov 26, 2018, at 11:58 AM, Christian Petrin >>> il.com > wrote:
 
 Hi Robert,
 
 the deque has unit, integration and API tests. The unit tests, I
 agree, are hard to follow as they have to check all internal
 properties to make sure the code is doing what it is supposed to do
 (keep a well formed ring linked list during all steps). Given their
 nature (unit tests), the only way to access the internal deque
 variables is to have them in the same package.
 
 Regarding the other, API and integration tests, I agree 100%. They
 should be in the dequeue_test package as they are not supposed to
 access (or know of) any of the deque internal variables. I'll move
 these tests to a different file in the dequeue_test package.
 
 Really appreciate the suggestion (correction, really).
 
 Thanks,
 Christian
 
 
 
 
 On Mon, Nov 26, 2018 at 9:42 AM robert engels >>> 
 m >> wrote:
 Just an FYI, IMO your testing is incorrect.
 
 Your tests should be in a package dequeue_test so that they cannot
 access the internal details. This makes them brittle and hard to
 follow.
 
 For reference, look at the Go stdlib sync/map.go and
 sync/map_test.go. All of the tests are in sync_test for all of the
 structs/methods.
 
 
> 
> On Nov 26, 2018, at 10:59 AM, Christian Petrin  mail.com   >> wrote:
> 
> Hello,
> 
> I'm working on a logging system called CloudLogger  
> b.com/cloud-logger/docs > and to cut to 
> the chase, CloudLogger
> needs an unbounded in-memory queue. The idea is to use the queue
> as a sequential data store. As there's no telling beforehand how
> much data will need to be stored, the queue has to be able to
> scale to support large amounts of data, so CloudLogger needs an
> unbounded queue, a very fast and efficient one.
> 
> Noticing Go doesn't offer an unbounded queue implementation, I
> came up with an idea to build a queue based on linked slices.
> 
> I was so impressed with the results that I decided to spend some
> time researching about other queue designs and ways to improve
> it. I'm finally done with the research and I feel like the new
> queue is worth being considered to be part of the standard
> library.
> 
> So please take a look at the issue. All suggestions, questions,
> comments, etc are most welcome!
> 
> Due to many suggestions, I have deployed the deque as a proper
> external package. In order to help validate the deque and get it
> into the standard library, I need to somehow get people to start
> using it. If you are 

Re: [go-nuts] The most performant deque of all time?

[robert engels]

My “none” looks to have been a little strong… Clearly whoever wrote the 
container package believes in testing as the OP. A more in-depth review shows 
other packages with similar “private tests”, but typically they seem to be 
testing the public API only. Then there are packages like sync that have no 
private tests…

It appears there is no standard in the stdlib as far as this goes :)

Like I said, it is my opinion, and I’m sure others feel differently. It’s 
served me well - mileage may vary.

> On Nov 26, 2018, at 3:32 PM, Dan Kortschak  
> wrote:
> 
> container/ring (a arguably non-idiomatic stdlib package does indeed
> contain this kind of test). It also does not have in code asserts,
> which are something that I've found to be very rare in Go code.
> 
> On Mon, 2018-11-26 at 12:09 -0600, robert engels wrote:
>> I am just offering, and many would disagree, that unit tests that
>> like are brittle and not worthwhile. I don’t see them in the Go
>> stdlib for major packages, but I could be wrong.
>> 
>> My thought on this is that if you are testing the nitty details of
>> the implementation, you need to get that “correct” twice - and if it
>> is failing, you don’t really know which one is wrong… This is
>> especially problematic as things are refactored - all of those types
>> of tests break.
>> 
>> I’m of the opinion - test the behavior comprehensively, and expose
>> the expected usage (and corresponding design) there.
>> 
>>> 
>>> On Nov 26, 2018, at 11:58 AM, Christian Petrin >> il.com> wrote:
>>> 
>>> Hi Robert,
>>> 
>>> the deque has unit, integration and API tests. The unit tests, I
>>> agree, are hard to follow as they have to check all internal
>>> properties to make sure the code is doing what it is supposed to do
>>> (keep a well formed ring linked list during all steps). Given their
>>> nature (unit tests), the only way to access the internal deque
>>> variables is to have them in the same package.
>>> 
>>> Regarding the other, API and integration tests, I agree 100%. They
>>> should be in the dequeue_test package as they are not supposed to
>>> access (or know of) any of the deque internal variables. I'll move
>>> these tests to a different file in the dequeue_test package.
>>> 
>>> Really appreciate the suggestion (correction, really).
>>> 
>>> Thanks,
>>> Christian
>>> 
>>> 
>>> 
>>> 
>>> On Mon, Nov 26, 2018 at 9:42 AM robert engels >> m >> wrote:
>>> Just an FYI, IMO your testing is incorrect.
>>> 
>>> Your tests should be in a package dequeue_test so that they cannot
>>> access the internal details. This makes them brittle and hard to
>>> follow.
>>> 
>>> For reference, look at the Go stdlib sync/map.go and
>>> sync/map_test.go. All of the tests are in sync_test for all of the
>>> structs/methods.
>>> 
>>> 
 
 On Nov 26, 2018, at 10:59 AM, Christian Petrin >>> mail.com  >> wrote:
 
 Hello,
 
 I'm working on a logging system called CloudLogger 
 b.com/cloud-logger/docs > and to cut to 
 the chase, CloudLogger
 needs an unbounded in-memory queue. The idea is to use the queue
 as a sequential data store. As there's no telling beforehand how
 much data will need to be stored, the queue has to be able to
 scale to support large amounts of data, so CloudLogger needs an
 unbounded queue, a very fast and efficient one.
 
 Noticing Go doesn't offer an unbounded queue implementation, I
 came up with an idea to build a queue based on linked slices.
 
 I was so impressed with the results that I decided to spend some
 time researching about other queue designs and ways to improve
 it. I'm finally done with the research and I feel like the new
 queue is worth being considered to be part of the standard
 library.
 
 So please take a look at the issue. All suggestions, questions,
 comments, etc are most welcome!
 
 Due to many suggestions, I have deployed the deque as a proper
 external package. In order to help validate the deque and get it
 into the standard library, I need to somehow get people to start
 using it. If you are using a queue/stack/deque, please consider
 switching to the new deque. Also if you know someone who is using
 a queue/stack/deque, please let them know about the deque.
 
 I'm extremely interested in improving the queue and I love
 challenges. If you suspect there's a possibly better design or a
 better, more performant and balanced deque implementation out
 there, please let me know and I'm very glad to benchmark it
 against deque.
 
 Proposal: https://github.com/golang/go/issues/27935 
 
 

Re: [go-nuts] The most performant deque of all time?

[Dan Kortschak]

container/ring (a arguably non-idiomatic stdlib package does indeed
contain this kind of test). It also does not have in code asserts,
which are something that I've found to be very rare in Go code.

On Mon, 2018-11-26 at 12:09 -0600, robert engels wrote:
> I am just offering, and many would disagree, that unit tests that
> like are brittle and not worthwhile. I don’t see them in the Go
> stdlib for major packages, but I could be wrong.
>
> My thought on this is that if you are testing the nitty details of
> the implementation, you need to get that “correct” twice - and if it
> is failing, you don’t really know which one is wrong… This is
> especially problematic as things are refactored - all of those types
> of tests break.
>
> I’m of the opinion - test the behavior comprehensively, and expose
> the expected usage (and corresponding design) there.
>
> >
> > On Nov 26, 2018, at 11:58 AM, Christian Petrin  > il.com> wrote:
> >
> > Hi Robert,
> >
> > the deque has unit, integration and API tests. The unit tests, I
> > agree, are hard to follow as they have to check all internal
> > properties to make sure the code is doing what it is supposed to do
> > (keep a well formed ring linked list during all steps). Given their
> > nature (unit tests), the only way to access the internal deque
> > variables is to have them in the same package.
> >
> > Regarding the other, API and integration tests, I agree 100%. They
> > should be in the dequeue_test package as they are not supposed to
> > access (or know of) any of the deque internal variables. I'll move
> > these tests to a different file in the dequeue_test package.
> >
> > Really appreciate the suggestion (correction, really).
> >
> > Thanks,
> > Christian
> >
> >
> >
> >
> > On Mon, Nov 26, 2018 at 9:42 AM robert engels  > m > wrote:
> > Just an FYI, IMO your testing is incorrect.
> >
> > Your tests should be in a package dequeue_test so that they cannot
> > access the internal details. This makes them brittle and hard to
> > follow.
> >
> > For reference, look at the Go stdlib sync/map.go and
> > sync/map_test.go. All of the tests are in sync_test for all of the
> > structs/methods.
> >
> >
> > >
> > > On Nov 26, 2018, at 10:59 AM, Christian Petrin  > > mail.com > wrote:
> > >
> > > Hello,
> > >
> > > I'm working on a logging system called CloudLogger  > > b.com/cloud-logger/docs> and to cut to the chase, CloudLogger
> > > needs an unbounded in-memory queue. The idea is to use the queue
> > > as a sequential data store. As there's no telling beforehand how
> > > much data will need to be stored, the queue has to be able to
> > > scale to support large amounts of data, so CloudLogger needs an
> > > unbounded queue, a very fast and efficient one.
> > >
> > > Noticing Go doesn't offer an unbounded queue implementation, I
> > > came up with an idea to build a queue based on linked slices.
> > >
> > > I was so impressed with the results that I decided to spend some
> > > time researching about other queue designs and ways to improve
> > > it. I'm finally done with the research and I feel like the new
> > > queue is worth being considered to be part of the standard
> > > library.
> > >
> > > So please take a look at the issue. All suggestions, questions,
> > > comments, etc are most welcome!
> > >
> > > Due to many suggestions, I have deployed the deque as a proper
> > > external package. In order to help validate the deque and get it
> > > into the standard library, I need to somehow get people to start
> > > using it. If you are using a queue/stack/deque, please consider
> > > switching to the new deque. Also if you know someone who is using
> > > a queue/stack/deque, please let them know about the deque.
> > >
> > > I'm extremely interested in improving the queue and I love
> > > challenges. If you suspect there's a possibly better design or a
> > > better, more performant and balanced deque implementation out
> > > there, please let me know and I'm very glad to benchmark it
> > > against deque.
> > >
> > > Proposal: https://github.com/golang/go/issues/27935
> > > 
> > > Deque package: https://github.com/ef-ds/deque
> > > 
> > >
> > > Thanks,
> > > Christian
> > >
> > > --
> > > You received this message because you are subscribed to the
> > > Google Groups "golang-nuts" group.
> > > To unsubscribe from this group and stop receiving emails from it,
> > > send an email to golang-nuts+unsubscr...@googlegroups.com
> > > .
> > > For more options, visit https://groups.google.com/d/optout
> > > .
> >
> >
> > --
> > Thanks,
> > Christian
--
CRICOS provider code 00123M

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, 

[go-nuts] Important design missing from Errors overview

[gopher . fortress]

Greetings,

While reading 
https://go.googlesource.com/proposal/+/master/design/go2draft-error-values-overview.md#other-go-designs
 
I noticed that an important design is missing from the links below. I am 
talking about Ben's https://middlemost.com/failure-is-your-domain/ which I 
think offers some very good insights that are missing from the current 
proposals. I think it is important to be added and be considered.

Thanks

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] The most performant deque of all time?

[Christian Petrin]

I like your thinking. Replacing UTs with asserts sounds pretty interesting
to me. Let me give a good thought about it. There's clearly enormous
potential, but there's also potentially many unforeseen implications. But
we should continue this discussion offline or in a separate thread. I have
duly noted your suggestion. :-)

On Mon, Nov 26, 2018 at 12:48 PM robert engels 
wrote:

> If you re-read what you wrote you will see the problem - you weren’t
> setting a prev/next link correctly - the proper way test test these
> invariants IMO is to use asserts within the implementation itself - that
> way the implementation change change without breaking all of the tests.
> Otherwise code is impossible to refactor - as you need to “determine the
> design” by reading very low-level unit tests - when I review your unit
> tests they tell me me nothing explicitly about the design - I have to
> “figure that out”… not good.
>
> In many ways, your unit tests present more complex/obscure logic to
> understand than the code itself…
>
> I’ll repeat, I can’t find any “unit tests” of that type in the Go stdlib,
> and reviewing the ones I do find (not in a different _test package) it
> seems it was a mistake, as they still only exercise the public API  -
> that’s a hint to me that you are doing something wrong.
>
>
>
> On Nov 26, 2018, at 2:34 PM, Christian Petrin 
> wrote:
>
> It's funny because I'm not a big fan of unit testing for all the same
> reasons you mentioned. Impl7
> ,
> for instance, I didn't implement any unit tests as the implementation was
> very simple and integration tests is sufficient.
>
> With deque, I initially implemented just integration tests. They were all
> passing and deque was behaving, according to the tests, correctly, always
> returning the expected results. When I deemed to have finalized the
> implementation (checking its correctness via integration tests), the deque
> was not performing as expected in a number of test scenarios. That caused
> me to scratch my head pretty hard for a good deal of time until I realized
> there was nothing wrong with the design. There should have some issue with
> the implementation. I tried my best to find out where the problem was, but
> failed. So I realized I should implement unit tests to make sure the
> implementation was not only correct, but follows the design perfectly.
>
> With the UTs, I was able to find the bug. A very sneaky one where I was
> not setting a link (prev/next) correctly, and that caused the deque to not
> use the spare links when the design dictate the deque should. Even with the
> bug, the queue was performing correctly, but not efficiently.
>
> In any case, my point is a well validated implementation should have tests
> at all levels (unit, integration, API, e2e) as some bugs can only be
> discovered with some types of tests.
>
> On Mon, Nov 26, 2018 at 10:09 AM robert engels 
> wrote:
>
>> I am just offering, and many would disagree, that unit tests that like
>> are brittle and not worthwhile. I don’t see them in the Go stdlib for major
>> packages, but I could be wrong.
>>
>> My thought on this is that if you are testing the nitty details of the
>> implementation, you need to get that “correct” twice - and if it is
>> failing, you don’t really know which one is wrong… This is especially
>> problematic as things are refactored - all of those types of tests break.
>>
>> I’m of the opinion - test the behavior comprehensively, and expose the
>> expected usage (and corresponding design) there.
>>
>> On Nov 26, 2018, at 11:58 AM, Christian Petrin 
>> wrote:
>>
>> Hi Robert,
>>
>> the deque has unit, integration and API tests. The unit tests, I agree,
>> are hard to follow as they have to check all internal properties to make
>> sure the code is doing what it is supposed to do (keep a well formed ring
>> linked list during all steps). Given their nature (unit tests), the only
>> way to access the internal deque variables is to have them in the same
>> package.
>>
>> Regarding the other, API and integration tests, I agree 100%. They should
>> be in the dequeue_test package as they are not supposed to access (or know
>> of) any of the deque internal variables. I'll move these tests to a
>> different file in the dequeue_test package.
>>
>> Really appreciate the suggestion (correction, really).
>>
>> Thanks,
>> Christian
>>
>>
>>
>>
>> On Mon, Nov 26, 2018 at 9:42 AM robert engels 
>> wrote:
>>
>>> Just an FYI, IMO your testing is incorrect.
>>>
>>> Your tests should be in a package dequeue_test so that they cannot
>>> access the internal details. This makes them brittle and hard to follow.
>>>
>>> For reference, look at the Go stdlib sync/map.go and sync/map_test.go.
>>> All of the tests are in sync_test for all of the structs/methods.
>>>
>>>
>>> On Nov 26, 2018, at 10:59 AM, Christian Petrin <
>>> christianpet...@gmail.com> wrote:
>>>
>>> Hello,
>>>

Re: [go-nuts] The most performant deque of all time?

[robert engels]

If you re-read what you wrote you will see the problem - you weren’t setting a 
prev/next link correctly - the proper way test test these invariants IMO is to 
use asserts within the implementation itself - that way the implementation 
change change without breaking all of the tests. Otherwise code is impossible 
to refactor - as you need to “determine the design” by reading very low-level 
unit tests - when I review your unit tests they tell me me nothing explicitly 
about the design - I have to “figure that out”… not good.

In many ways, your unit tests present more complex/obscure logic to understand 
than the code itself…

I’ll repeat, I can’t find any “unit tests” of that type in the Go stdlib, and 
reviewing the ones I do find (not in a different _test package) it seems it was 
a mistake, as they still only exercise the public API  - that’s a hint to me 
that you are doing something wrong.



> On Nov 26, 2018, at 2:34 PM, Christian Petrin  
> wrote:
> 
> It's funny because I'm not a big fan of unit testing for all the same reasons 
> you mentioned. Impl7 
> ,
>  for instance, I didn't implement any unit tests as the implementation was 
> very simple and integration tests is sufficient.
> 
> With deque, I initially implemented just integration tests. They were all 
> passing and deque was behaving, according to the tests, correctly, always 
> returning the expected results. When I deemed to have finalized the 
> implementation (checking its correctness via integration tests), the deque 
> was not performing as expected in a number of test scenarios. That caused me 
> to scratch my head pretty hard for a good deal of time until I realized there 
> was nothing wrong with the design. There should have some issue with the 
> implementation. I tried my best to find out where the problem was, but 
> failed. So I realized I should implement unit tests to make sure the 
> implementation was not only correct, but follows the design perfectly.
> 
> With the UTs, I was able to find the bug. A very sneaky one where I was not 
> setting a link (prev/next) correctly, and that caused the deque to not use 
> the spare links when the design dictate the deque should. Even with the bug, 
> the queue was performing correctly, but not efficiently.
> 
> In any case, my point is a well validated implementation should have tests at 
> all levels (unit, integration, API, e2e) as some bugs can only be discovered 
> with some types of tests.
> 
> On Mon, Nov 26, 2018 at 10:09 AM robert engels  > wrote:
> I am just offering, and many would disagree, that unit tests that like are 
> brittle and not worthwhile. I don’t see them in the Go stdlib for major 
> packages, but I could be wrong.
> 
> My thought on this is that if you are testing the nitty details of the 
> implementation, you need to get that “correct” twice - and if it is failing, 
> you don’t really know which one is wrong… This is especially problematic as 
> things are refactored - all of those types of tests break.
> 
> I’m of the opinion - test the behavior comprehensively, and expose the 
> expected usage (and corresponding design) there.
> 
>> On Nov 26, 2018, at 11:58 AM, Christian Petrin > > wrote:
>> 
>> Hi Robert,
>> 
>> the deque has unit, integration and API tests. The unit tests, I agree, are 
>> hard to follow as they have to check all internal properties to make sure 
>> the code is doing what it is supposed to do (keep a well formed ring linked 
>> list during all steps). Given their nature (unit tests), the only way to 
>> access the internal deque variables is to have them in the same package.
>> 
>> Regarding the other, API and integration tests, I agree 100%. They should be 
>> in the dequeue_test package as they are not supposed to access (or know of) 
>> any of the deque internal variables. I'll move these tests to a different 
>> file in the dequeue_test package.
>> 
>> Really appreciate the suggestion (correction, really).
>> 
>> Thanks,
>> Christian
>> 
>> 
>> 
>> 
>> On Mon, Nov 26, 2018 at 9:42 AM robert engels > > wrote:
>> Just an FYI, IMO your testing is incorrect.
>> 
>> Your tests should be in a package dequeue_test so that they cannot access 
>> the internal details. This makes them brittle and hard to follow.
>> 
>> For reference, look at the Go stdlib sync/map.go and sync/map_test.go. All 
>> of the tests are in sync_test for all of the structs/methods.
>> 
>> 
>>> On Nov 26, 2018, at 10:59 AM, Christian Petrin >> > wrote:
>>> 
>>> Hello,
>>> 
>>> I'm working on a logging system called CloudLogger 
>>>  and to cut to the chase, CloudLogger 
>>> needs an unbounded in-memory queue. The idea is to use the queue as a 
>>> sequential data store. As there's no telling beforehand how 

Re: [go-nuts] The most performant deque of all time?

[Christian Petrin]

It's funny because I'm not a big fan of unit testing for all the same
reasons you mentioned. Impl7
,
for instance, I didn't implement any unit tests as the implementation was
very simple and integration tests is sufficient.

With deque, I initially implemented just integration tests. They were all
passing and deque was behaving, according to the tests, correctly, always
returning the expected results. When I deemed to have finalized the
implementation (checking its correctness via integration tests), the deque
was not performing as expected in a number of test scenarios. That caused
me to scratch my head pretty hard for a good deal of time until I realized
there was nothing wrong with the design. There should have some issue with
the implementation. I tried my best to find out where the problem was, but
failed. So I realized I should implement unit tests to make sure the
implementation was not only correct, but follows the design perfectly.

With the UTs, I was able to find the bug. A very sneaky one where I was not
setting a link (prev/next) correctly, and that caused the deque to not use
the spare links when the design dictate the deque should. Even with the
bug, the queue was performing correctly, but not efficiently.

In any case, my point is a well validated implementation should have tests
at all levels (unit, integration, API, e2e) as some bugs can only be
discovered with some types of tests.

On Mon, Nov 26, 2018 at 10:09 AM robert engels 
wrote:

> I am just offering, and many would disagree, that unit tests that like are
> brittle and not worthwhile. I don’t see them in the Go stdlib for major
> packages, but I could be wrong.
>
> My thought on this is that if you are testing the nitty details of the
> implementation, you need to get that “correct” twice - and if it is
> failing, you don’t really know which one is wrong… This is especially
> problematic as things are refactored - all of those types of tests break.
>
> I’m of the opinion - test the behavior comprehensively, and expose the
> expected usage (and corresponding design) there.
>
> On Nov 26, 2018, at 11:58 AM, Christian Petrin 
> wrote:
>
> Hi Robert,
>
> the deque has unit, integration and API tests. The unit tests, I agree,
> are hard to follow as they have to check all internal properties to make
> sure the code is doing what it is supposed to do (keep a well formed ring
> linked list during all steps). Given their nature (unit tests), the only
> way to access the internal deque variables is to have them in the same
> package.
>
> Regarding the other, API and integration tests, I agree 100%. They should
> be in the dequeue_test package as they are not supposed to access (or know
> of) any of the deque internal variables. I'll move these tests to a
> different file in the dequeue_test package.
>
> Really appreciate the suggestion (correction, really).
>
> Thanks,
> Christian
>
>
>
>
> On Mon, Nov 26, 2018 at 9:42 AM robert engels 
> wrote:
>
>> Just an FYI, IMO your testing is incorrect.
>>
>> Your tests should be in a package dequeue_test so that they cannot access
>> the internal details. This makes them brittle and hard to follow.
>>
>> For reference, look at the Go stdlib sync/map.go and sync/map_test.go.
>> All of the tests are in sync_test for all of the structs/methods.
>>
>>
>> On Nov 26, 2018, at 10:59 AM, Christian Petrin 
>> wrote:
>>
>> Hello,
>>
>> I'm working on a logging system called CloudLogger
>>  and to cut to the chase,
>> CloudLogger needs an unbounded in-memory queue. The idea is to use the
>> queue as a sequential data store. As there's no telling beforehand how much
>> data will need to be stored, the queue has to be able to scale to support
>> large amounts of data, so CloudLogger needs an unbounded queue, a very fast
>> and efficient one.
>>
>> Noticing Go doesn't offer an unbounded queue implementation, I came up
>> with an idea to build a queue based on linked slices.
>>
>> I was so impressed with the results that I decided to spend some time
>> researching about other queue designs and ways to improve it. I'm finally
>> done with the research and I feel like the new queue is worth being
>> considered to be part of the standard library.
>>
>> So please take a look at the issue. All suggestions, questions, comments,
>> etc are most welcome!
>>
>> Due to many suggestions, I have deployed the deque as a proper external
>> package. In order to help validate the deque and get it into the standard
>> library, I need to somehow get people to start using it. If you are using a
>> queue/stack/deque, please consider switching to the new deque. Also if you
>> know someone who is using a queue/stack/deque, please let them know about
>> the deque.
>>
>> I'm extremely interested in improving the queue and I love challenges. If
>> you suspect there's a possibly better design or a better, more 

Re: [go-nuts] Re: GoAWK: an AWK interpreter written in Go

[Scott Pakin]

On Friday, November 23, 2018 at 1:58:37 PM UTC-7, Ben Hoyt wrote:
>
> So my main suggestion (for spakin/awk) would be able to wrap os.Stdout in 
> a bufio.NewWriter (and be sure to call Flush before Run finishes). If the 
> user wants to pass an unbuffered version, they still can, but at least the 
> default is performant.
>
> I also added CPU profiling to the spakin/awk script, and it looks like 
> it's doing a bunch more garbage collection than GoAWK, as well as some 
> regexp stuff. I suspect NewValue() is probably quite slow as it takes an 
> interface{} and does type checking. Also, strings are converted to numbers 
> using a regex, which is probably slower than a dedicated conversion/check 
> function (see parseFloatPrefix in goawk/interp/value.go).
>
> See more optimization ideas in my post at 
> https://benhoyt.com/writings/goawk/
>

Thanks for doing the performance analysis of spakin/awk 
!  Once I find the time to work on that 
project some more, I'll certainly look into implementing some of your 
suggestions.

— Scott

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Any NULL CIPHER available in TLS package?

[Localhost shell]

Hello All,

Is there a TLS package in go that supports NULL CIPHER in TLS communication 
in go? 

I couldn't find any null cipher from the listed cipher suite IDs 
https://golang.org/pkg/crypto/tls/#pkg-constants
for ex: SSL_RSA_WITH_NULL_MD5

I am working on a project that involves a go client to communicate with a 
Kafka cluster. We plan to use Null Cipher based data exchange to just get 
authentication(without encryption) to prevent the significant performance 
impact we have observed in our testing when we use encryption.

--Unilocal

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] At runtime, tell if a package path belongs to stdlib ?

[Ian Lance Taylor]

On Mon, Nov 26, 2018 at 7:07 AM  wrote:
>
> I m looking for a reliable way to determine if a package path string belongs 
> to the stdlib or not.
>
> It happens at runtime, the program can not access its sources.
>
> I m thinking to write package that stores those information into package 
> variables, into a map[go version][]packagesPaths.
>
> Do you have any suggestions to achieve that ?
>
> I digged into go/types, loader etc but i did find the desired api.

On tip, take a look at the internal/goroot package.  It's not
externally accessible, but it may show the kind of thing you are
looking for.

Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: is it possible to speed up type assertion?

[larytet]

This logger https://github.com/ScottMansfield/nanolog is roughly 2x faster 
than ZAP and will work on all Go platforms.

I want my logs to be collected and stored. I do not want to think where and 
when I can or can not log.  300ns and up (and not deterministic in my 
tests) blocking call is something I would like to avoid. I can run a 
separate log in every thread and reorder, merge the logs offline. When I 
need to actually read the log I can spend quite lot of time to decode the 
data.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: is it possible to speed up type assertion?

[robert engels]

Why are you putting the ResetTimer() and StopTimer() calls in there ? 
Unnecessary AFAIK.

Also, all this test is benchmarking is the time it takes to construct a Field 
struct. What is the point ?

> On Nov 26, 2018, at 1:47 PM, lary...@gmail.com wrote:
> 
> Talking about loggers and ZAP (interesting idea to accommodate API to the 
> JSON)
> The following code gets 40ns/op 
> type FieldType uint8
> 
> type Field struct {
> Key   string
> Type  FieldType
> Integer   int64
> Stringstring
> Interface interface{}
> }
> 
> const (
> // UnknownType is the default field type. Attempting to add it to an 
> encoder will panic.
> UnknownType FieldType = iota
> // Int64Type indicates that the field carries an int64.
> Uint64Type
> )
> 
> func Uint64(key string, val uint64) Field {
> return Field{Key: key, Type: Uint64Type, Integer: int64(val)}
> }
> 
> func handleFields(s string, fields ...Field) {
> 
> }
> 
> func BenchmarkZapApi(b *testing.B) {
> b.ResetTimer()
> for i := 0; i < b.N; i++ {
> handleFields("Hello ",
> Uint64("world", 0),
> Uint64("world", 1),
> Uint64("world", 2),
> )
> }
> b.StopTimer()
> }
> 
> 
> 
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com 
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: is it possible to speed up type assertion?

[larytet]

Talking about loggers and ZAP (interesting idea to accommodate API to the 
JSON)
The following code gets 40ns/op 
type FieldType uint8

type Field struct {
Key   string
Type  FieldType
Integer   int64
Stringstring
Interface interface{}
}

const (
// UnknownType is the default field type. Attempting to add it to an 
encoder will panic.
UnknownType FieldType = iota
// Int64Type indicates that the field carries an int64.
Uint64Type
)

func Uint64(key string, val uint64) Field {
return Field{Key: key, Type: Uint64Type, Integer: int64(val)}
}

func handleFields(s string, fields ...Field) {

}

func BenchmarkZapApi(b *testing.B) {
b.ResetTimer()
for i := 0; i < b.N; i++ {
handleFields("Hello ",
Uint64("world", 0),
Uint64("world", 1),
Uint64("world", 2),
)
}
b.StopTimer()
}




-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] The most performant deque of all time?

[robert engels]

I am just offering, and many would disagree, that unit tests that like are 
brittle and not worthwhile. I don’t see them in the Go stdlib for major 
packages, but I could be wrong.

My thought on this is that if you are testing the nitty details of the 
implementation, you need to get that “correct” twice - and if it is failing, 
you don’t really know which one is wrong… This is especially problematic as 
things are refactored - all of those types of tests break.

I’m of the opinion - test the behavior comprehensively, and expose the expected 
usage (and corresponding design) there.

> On Nov 26, 2018, at 11:58 AM, Christian Petrin  
> wrote:
> 
> Hi Robert,
> 
> the deque has unit, integration and API tests. The unit tests, I agree, are 
> hard to follow as they have to check all internal properties to make sure the 
> code is doing what it is supposed to do (keep a well formed ring linked list 
> during all steps). Given their nature (unit tests), the only way to access 
> the internal deque variables is to have them in the same package.
> 
> Regarding the other, API and integration tests, I agree 100%. They should be 
> in the dequeue_test package as they are not supposed to access (or know of) 
> any of the deque internal variables. I'll move these tests to a different 
> file in the dequeue_test package.
> 
> Really appreciate the suggestion (correction, really).
> 
> Thanks,
> Christian
> 
> 
> 
> 
> On Mon, Nov 26, 2018 at 9:42 AM robert engels  > wrote:
> Just an FYI, IMO your testing is incorrect.
> 
> Your tests should be in a package dequeue_test so that they cannot access the 
> internal details. This makes them brittle and hard to follow.
> 
> For reference, look at the Go stdlib sync/map.go and sync/map_test.go. All of 
> the tests are in sync_test for all of the structs/methods.
> 
> 
>> On Nov 26, 2018, at 10:59 AM, Christian Petrin > > wrote:
>> 
>> Hello,
>> 
>> I'm working on a logging system called CloudLogger 
>>  and to cut to the chase, CloudLogger 
>> needs an unbounded in-memory queue. The idea is to use the queue as a 
>> sequential data store. As there's no telling beforehand how much data will 
>> need to be stored, the queue has to be able to scale to support large 
>> amounts of data, so CloudLogger needs an unbounded queue, a very fast and 
>> efficient one.
>> 
>> Noticing Go doesn't offer an unbounded queue implementation, I came up with 
>> an idea to build a queue based on linked slices.
>> 
>> I was so impressed with the results that I decided to spend some time 
>> researching about other queue designs and ways to improve it. I'm finally 
>> done with the research and I feel like the new queue is worth being 
>> considered to be part of the standard library.
>> 
>> So please take a look at the issue. All suggestions, questions, comments, 
>> etc are most welcome!
>> 
>> Due to many suggestions, I have deployed the deque as a proper external 
>> package. In order to help validate the deque and get it into the standard 
>> library, I need to somehow get people to start using it. If you are using a 
>> queue/stack/deque, please consider switching to the new deque. Also if you 
>> know someone who is using a queue/stack/deque, please let them know about 
>> the deque.
>> 
>> I'm extremely interested in improving the queue and I love challenges. If 
>> you suspect there's a possibly better design or a better, more performant 
>> and balanced deque implementation out there, please let me know and I'm very 
>> glad to benchmark it against deque.
>> 
>> Proposal: https://github.com/golang/go/issues/27935 
>> 
>> Deque package: https://github.com/ef-ds/deque 
>> 
>> 
>> Thanks,
>> Christian
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts+unsubscr...@googlegroups.com 
>> .
>> For more options, visit https://groups.google.com/d/optout 
>> .
> 
> 
> 
> -- 
> Thanks,
> Christian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] The most performant deque of all time?

[Christian Petrin]

Hi Robert,

the deque has unit, integration and API tests. The unit tests, I agree, are
hard to follow as they have to check all internal properties to make sure
the code is doing what it is supposed to do (keep a well formed ring linked
list during all steps). Given their nature (unit tests), the only way to
access the internal deque variables is to have them in the same package.

Regarding the other, API and integration tests, I agree 100%. They should
be in the dequeue_test package as they are not supposed to access (or know
of) any of the deque internal variables. I'll move these tests to a
different file in the dequeue_test package.

Really appreciate the suggestion (correction, really).

Thanks,
Christian




On Mon, Nov 26, 2018 at 9:42 AM robert engels  wrote:

> Just an FYI, IMO your testing is incorrect.
>
> Your tests should be in a package dequeue_test so that they cannot access
> the internal details. This makes them brittle and hard to follow.
>
> For reference, look at the Go stdlib sync/map.go and sync/map_test.go. All
> of the tests are in sync_test for all of the structs/methods.
>
>
> On Nov 26, 2018, at 10:59 AM, Christian Petrin 
> wrote:
>
> Hello,
>
> I'm working on a logging system called CloudLogger
>  and to cut to the chase,
> CloudLogger needs an unbounded in-memory queue. The idea is to use the
> queue as a sequential data store. As there's no telling beforehand how much
> data will need to be stored, the queue has to be able to scale to support
> large amounts of data, so CloudLogger needs an unbounded queue, a very fast
> and efficient one.
>
> Noticing Go doesn't offer an unbounded queue implementation, I came up
> with an idea to build a queue based on linked slices.
>
> I was so impressed with the results that I decided to spend some time
> researching about other queue designs and ways to improve it. I'm finally
> done with the research and I feel like the new queue is worth being
> considered to be part of the standard library.
>
> So please take a look at the issue. All suggestions, questions, comments,
> etc are most welcome!
>
> Due to many suggestions, I have deployed the deque as a proper external
> package. In order to help validate the deque and get it into the standard
> library, I need to somehow get people to start using it. If you are using a
> queue/stack/deque, please consider switching to the new deque. Also if you
> know someone who is using a queue/stack/deque, please let them know about
> the deque.
>
> I'm extremely interested in improving the queue and I love challenges. If
> you suspect there's a possibly better design or a better, more performant
> and balanced deque implementation out there, please let me know and I'm
> very glad to benchmark it against deque.
>
> Proposal: https://github.com/golang/go/issues/27935
> Deque package: https://github.com/ef-ds/deque
>
> Thanks,
> Christian
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
>
>

-- 
Thanks,
Christian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] The most performant deque of all time?

[robert engels]

Just an FYI, IMO your testing is incorrect.

Your tests should be in a package dequeue_test so that they cannot access the 
internal details. This makes them brittle and hard to follow.

For reference, look at the Go stdlib sync/map.go and sync/map_test.go. All of 
the tests are in sync_test for all of the structs/methods.


> On Nov 26, 2018, at 10:59 AM, Christian Petrin  
> wrote:
> 
> Hello,
> 
> I'm working on a logging system called CloudLogger 
>  and to cut to the chase, CloudLogger 
> needs an unbounded in-memory queue. The idea is to use the queue as a 
> sequential data store. As there's no telling beforehand how much data will 
> need to be stored, the queue has to be able to scale to support large amounts 
> of data, so CloudLogger needs an unbounded queue, a very fast and efficient 
> one.
> 
> Noticing Go doesn't offer an unbounded queue implementation, I came up with 
> an idea to build a queue based on linked slices.
> 
> I was so impressed with the results that I decided to spend some time 
> researching about other queue designs and ways to improve it. I'm finally 
> done with the research and I feel like the new queue is worth being 
> considered to be part of the standard library.
> 
> So please take a look at the issue. All suggestions, questions, comments, etc 
> are most welcome!
> 
> Due to many suggestions, I have deployed the deque as a proper external 
> package. In order to help validate the deque and get it into the standard 
> library, I need to somehow get people to start using it. If you are using a 
> queue/stack/deque, please consider switching to the new deque. Also if you 
> know someone who is using a queue/stack/deque, please let them know about the 
> deque.
> 
> I'm extremely interested in improving the queue and I love challenges. If you 
> suspect there's a possibly better design or a better, more performant and 
> balanced deque implementation out there, please let me know and I'm very glad 
> to benchmark it against deque.
> 
> Proposal: https://github.com/golang/go/issues/27935
> Deque package: https://github.com/ef-ds/deque
> 
> Thanks,
> Christian
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com 
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] The most performant deque of all time?

[Christian Petrin]

Hello,

I'm working on a logging system called CloudLogger 
 and to cut to the chase, CloudLogger 
needs an unbounded in-memory queue. The idea is to use the queue as a 
sequential data store. As there's no telling beforehand how much data will 
need to be stored, the queue has to be able to scale to support large 
amounts of data, so CloudLogger needs an unbounded queue, a very fast and 
efficient one.

Noticing Go doesn't offer an unbounded queue implementation, I came up with 
an idea to build a queue based on linked slices.

I was so impressed with the results that I decided to spend some time 
researching about other queue designs and ways to improve it. I'm finally 
done with the research and I feel like the new queue is worth being 
considered to be part of the standard library.

So please take a look at the issue. All suggestions, questions, comments, 
etc are most welcome!

Due to many suggestions, I have deployed the deque as a proper external 
package. In order to help validate the deque and get it into the standard 
library, I need to somehow get people to start using it. If you are using a 
queue/stack/deque, please consider switching to the new deque. Also if you 
know someone who is using a queue/stack/deque, please let them know about 
the deque.

I'm extremely interested in improving the queue and I love challenges. If 
you suspect there's a possibly better design or a better, more performant 
and balanced deque implementation out there, please let me know and I'm 
very glad to benchmark it against deque.

Proposal: https://github.com/golang/go/issues/27935
Deque package: https://github.com/ef-ds/deque

Thanks,
Christian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] net/http/httputil.ReverseProxy Expect: 100-continue broken?

[gregoryh]

Thanks, I pushed up the change you recommended -- reading the request body 
-- and indeed that fixed the test case.

The weird TLS config thing I did was because the original problem we had 
(backend services, etc.) went away if we configured InsecureSkipVerify the 
usual straightforward way. The Go stdlib sources treat setting a 
TLSClientConfig with just InsecureSkipVerify:true is different than letting 
the net/http library set up its own config and then modifying it. e.g. in 
net/http/transport.go you can see where if TLSClientConfig != nil it will 
skip several items related to http2 configuration. In this test case I 
tried to recreate, it turns out all of this has no impact since the body 
being unread was the real issue, as you pointed out.

Anyway, back to the drawing board to figure out the original problem. We 
have a backend service (Rails) running in Google's Kubernetes Engine, 
fronted by Google's Cloud Load Balancer. Next in line is a Go ReverseProxy 
based router (also in GKE), also fronted by Google's Cloud Load Balancer, 
which clients talk to. So... client <-> GCLBa <-> ReverseProxy <-> GCLBb 
<-> Service. Of course, there are Kubernetes "things" inline as well, all 
of this making debugging much more... interesting.

If a client talks HTTPS/1.1 but our ReverseProxy talks to the backside with 
HTTPS/2, we get very strange behavior where the response the client 
receives has an explicit Content-Length but the body has obvious 
Transfer-Encoding:chunked markers, though no such Transfer-Encoding header 
since it had a Content-Length. If the client talks HTTPS/2 instead, 
everything works as it should. If we force the ReverseProxy to only talk 
HTTPS/1.1, that will also fix the issue.

Our immediate fix is to configure ReverseProxy to never use HTTP/2, since 
we can't control what the clients do. But, we'd rather use HTTP/2 if it's 
available; and, if we ReverseProxy GRPC services, we'll have no choice 
anyway.

Thanks for your help, I hope I can create a reproduction of the issue we're 
seeing, or find our bug finally. I'm still pretty sure it's triggered when 
the client uses Expect:100-continue but... So many herrings...


On Thursday, November 22, 2018 at 5:08:35 AM UTC-8, Peter Waller wrote:
>
> I suspect this has to do with the fact that you're doing a roundtrip for 
> its side effects, but not reading the response body or closing it. If I fix 
> that, everything seems to work as expected.
>
> Try configuring the TLS client with t.TLSClientConfig = 
> {InsecureSkipVerify: true}
>
> On Mon, 19 Nov 2018 at 20:30, > wrote:
>
>> Hi folks!
>>
>> Hoping somebody can help me figure out what I'm doing wrong (or what Go's 
>> doing wrong in the small chance it's that).
>>
>> It _seems_ Go's reverse proxy doesn't support 100 Continue when the 
>> backend is HTTP/2 (but I'm guessing).
>>
>> I put up the sample at https://github.com/gholt/proxrepro -- just `go 
>> run main.go` and then look at the `trace` file that's output.
>>
>> You can see where curl sends its request headers with the Expect: 
>> 100-continue but the first thing it gets back is 200 OK and then 
>> "weirdness" ensues.
>>
>> -- Greg Holt
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: is it possible to speed up type assertion?

[roger peppe]

On Mon, 26 Nov 2018 at 16:44,  wrote:

> Do you mean this
> // This is straight from the https://github.com/uber-go/zap plabook
> func (b *Binlog) LogStructured(msg string, fields ...Field) error
>
> My first goal was to allow drop in replacement for the  log package
> I have to handle 50K queries/s on a mediocre machine. My time budget is
> about ~200 micro/query
>

How many log messages are being produced per query? What are your queries
actually doing? What does the profiler tell you?
If logging is really a bottleneck, you could consider other approaches such
as sampling or just logging less often, or differently.

BTW if you're handling 50K queries per second, that gives you 20µs/query,
not 200 AFAICS.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: is it possible to speed up type assertion?

[larytet]

Do you mean this 
// This is straight from the https://github.com/uber-go/zap plabook
func (b *Binlog) LogStructured(msg string, fields ...Field) error

My first goal was to allow drop in replacement for the  log package
I have to handle 50K queries/s on a mediocre machine. My time budget is 
about ~200 micro/query

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: is it possible to speed up type assertion?

[roger peppe]

On Mon, 26 Nov 2018 at 14:16, Arkady  wrote:

> ZAP is significantly slower than what I do. The binary log has an inherent
> edge.


AFAIK there is nothing about zap which precludes it from writing binary
files.

Unlike zap, your scheme will inherently incur more memory allocations, as
it takes a memory allocation to put a non-pointer value into an interface
(that's the reason why zap is designed the way it is), so I suspect it will
end up slower in more realistic scenarios.

For comparison call to Log() is 3x faster than fmt.Fprintf()
>

At least fmt.Fprintf works predictably and reliably on all Go
architectures. Correctness is more important than speed. I'd suggest
starting with a more complete solution and making it work before trying to
optimise the heck out of it.

  cheers,
rog.

I have added BenchmarkSingleIntRogerPeppe() - 13ns more
> On Mon, Nov 26, 2018 at 4:09 PM roger peppe  wrote:
> >
> > On Sun, 25 Nov 2018 at 19:17,  wrote:
> >>
> >> These are great tips! Thank you!
> >> This is the context for the code above
> https://github.com/larytet/binlog/blob/master/binlog.go#L548
> >
> >
> > I took a look at this code. It seems that you have a deep understanding
> of how the runtime works, but to me it really seems like you're prematurely
> optimising here, and running serious risk of broken code. The code is full
> of unsafe and non-portable operations that will almost certainly break in
> the future. For example, reading /proc to determine the base offset for
> static strings is... inadvisable.
> >
> > Just because you know what's going on under the covers doesn't mean that
> you should write code that relies on that information.
> >
> > If you want to see a highly performant logging implementation that does
> not seriously rely on unsafe practices, I'd encourage you to take a look at
> the zap package (https://godoc.org/go.uber.org/zap).
> >
> > By the way, your SingleInt benchmark is misleading. You're using a
> constant argument to Log, which means that the runtime can use a single
> interface value for every call, with no allocation required. If you change
> the loop so that it passes a different number each time:
> >
> > for i := 0; i < b.N; i++ {
> > binlog.Log(fmtString, i)
> > }
> >
> > you are likely to find that the performance gap is considerably smaller.
> >
> >   cheers,
> > rog.
> >
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] At runtime, tell if a package path belongs to stdlib ?

[Jan Mercl]

On Mon, Nov 26, 2018 at 4:55 PM  wrote:

> I was not referring to the file system path, i m referring to the import
package path.

FTR: The later is canonically called just "the import path".

-- 

-j

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] At runtime, tell if a package path belongs to stdlib ?

[clementauger888]

thanks! It will be useful, indeed.

Le lundi 26 novembre 2018 16:15:51 UTC+1, Shulhan a écrit :
>
> On Mon, 26 Nov 2018 07:07:34 -0800 (PST) 
> clement...@gmail.com  wrote: 
>
> > HI, 
> > 
> > I m looking for a reliable way to determine if a package path string 
> > belongs to the stdlib or not. 
> > 
> > It happens at runtime, the program can not access its sources. 
> > 
> > I m thinking to write package that stores those information into 
> > package variables, into a map[go version][]packagesPaths. 
> > 
> > Do you have any suggestions to achieve that ? 
> > 
> > I digged into go/types, loader etc but i did find the desired api. 
> > 
> > Thanks! 
> > 
>
> Here is a function to get list of packages under $GOROOT/src. [1] 
>
> [1] https://github.com/shuLhan/beku/blob/master/env.go#L398 
>
> -- 
> { "github":"github.com/shuLhan", "site":"kilabit.info" } 
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] At runtime, tell if a package path belongs to stdlib ?

[clementauger888]

Hi,

I was not referring to the file system path, i m referring to the import 
package path.

Given the string bytes, reliably tells this is a package that belongs to 
the stdlib of version Y.

I m using the reflect.Type.PkgPath value, to be more specific.

Le lundi 26 novembre 2018 16:11:40 UTC+1, Jan Mercl a écrit :
>
> On Mon, Nov 26, 2018 at 4:07 PM > 
> wrote:
>
> > I m looking for a reliable way to determine if a package path string 
> belongs to the stdlib or not.
>
> Check that the package path is bellow runtime.GOROOT().
> -- 
>
> -j
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] At runtime, tell if a package path belongs to stdlib ?

[Shulhan]

On Mon, 26 Nov 2018 07:07:34 -0800 (PST)
clementauger...@gmail.com wrote:

> HI,
> 
> I m looking for a reliable way to determine if a package path string 
> belongs to the stdlib or not.
> 
> It happens at runtime, the program can not access its sources.
> 
> I m thinking to write package that stores those information into
> package variables, into a map[go version][]packagesPaths.
> 
> Do you have any suggestions to achieve that ?
> 
> I digged into go/types, loader etc but i did find the desired api.
> 
> Thanks!
> 

Here is a function to get list of packages under $GOROOT/src. [1]

[1] https://github.com/shuLhan/beku/blob/master/env.go#L398

-- 
{ "github":"github.com/shuLhan", "site":"kilabit.info" }

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] At runtime, tell if a package path belongs to stdlib ?

[Jan Mercl]

On Mon, Nov 26, 2018 at 4:07 PM  wrote:

> I m looking for a reliable way to determine if a package path string
belongs to the stdlib or not.

Check that the package path is bellow runtime.GOROOT().
-- 

-j

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] At runtime, tell if a package path belongs to stdlib ?

[clementauger888]

HI,

I m looking for a reliable way to determine if a package path string 
belongs to the stdlib or not.

It happens at runtime, the program can not access its sources.

I m thinking to write package that stores those information into package 
variables, into a map[go version][]packagesPaths.

Do you have any suggestions to achieve that ?

I digged into go/types, loader etc but i did find the desired api.

Thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: is it possible to speed up type assertion?

[Robert Johnstone]

Hello,

Separate question, why are you passing an unsafe pointer to writer?  You 
are (probably) forcing that int to escape to the heap.  If you want to 
write an uint64, pass in a uint64.

1) I suspect that you are using pointer arithmetic inside writer, don't.  
The code will not be portable.  Instead, you should use shift and mask to 
extract the bytes (e.x, uint8(i>>16).  I expect the resulting code will be 
faster.

2)  Even if you keep the pointer arithmetic, you should defer taking the 
address until necessary.


Good luck.

Robert


On Sunday, 25 November 2018 11:10:09 UTC-5, Arkady M wrote:
>
> The code below consumes ~40% of the total execution time. According to 
> the profiler i := uint64(arg.(uint32)) is a major contributor
>
> // Cast the integer argument to uint64 and call a "writer"
> // The "writer" knows how many bytes to add to the binary stream
> // Type casts from interface{} to integer consume 40% of the overall
> // time. Can I do better? What is interface{} in Golang?
> func (b *Binlog) writeArgumentToOutput(writer writer, arg interface{}, 
> argKind reflect.Kind) error {
> // unsafe pointer to the data depends on the data type
> var err error
> switch argKind {
> case reflect.Int8:
> i := uint64(arg.(int8))
> err = writer.write(b.ioWriter, unsafe.Pointer())
> case reflect.Int16:
> i := uint64(arg.(int16))
> err = writer.write(b.ioWriter, unsafe.Pointer())
> case reflect.Int32:
> i := uint64(arg.(int32))
> err = writer.write(b.ioWriter, unsafe.Pointer())
> case reflect.Int64:
> i := uint64(arg.(int64))
> err = writer.write(b.ioWriter, unsafe.Pointer())
> case reflect.Uint8:
> i := uint64(arg.(uint8))
> err = writer.write(b.ioWriter, unsafe.Pointer())
> case reflect.Uint16:
> i := uint64(arg.(uint16))
> err = writer.write(b.ioWriter, unsafe.Pointer())
> case reflect.Uint32:
> i := uint64(arg.(uint32))
> err = writer.write(b.ioWriter, unsafe.Pointer())
> case reflect.Uint64:
> i := uint64(arg.(uint64))
> err = writer.write(b.ioWriter, unsafe.Pointer())
> case reflect.Int:
> i := uint64(arg.(int))
> err = writer.write(b.ioWriter, unsafe.Pointer())
> case reflect.Uint:
> i := uint64(arg.(uint))
> err = writer.write(b.ioWriter, unsafe.Pointer())
> default:
> return fmt.Errorf("Unsupported type: %T\n", reflect.TypeOf(arg))
> }
> return err
> }
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: is it possible to speed up type assertion?

[roger peppe]

On Sun, 25 Nov 2018 at 19:17,  wrote:

> These are great tips! Thank you!
> This is the context for the code above
> https://github.com/larytet/binlog/blob/master/binlog.go#L548
>

I took a look at this code. It seems that you have a deep understanding of
how the runtime works, but to me it really seems like you're prematurely
optimising here, and running serious risk of broken code. The code is full
of unsafe and non-portable operations that will almost certainly break in
the future. For example, reading /proc to determine the base offset for
static strings is... inadvisable.

Just because you know what's going on under the covers doesn't mean that
you should write code that relies on that information.

If you want to see a highly performant logging implementation that does not
seriously rely on unsafe practices, I'd encourage you to take a look at the
zap package (https://godoc.org/go.uber.org/zap).

By the way, your SingleInt benchmark is misleading. You're using a constant
argument to Log, which means that the runtime can use a single interface
value for every call, with no allocation required. If you change the loop
so that it passes a different number each time:

for i := 0; i < b.N; i++ {
binlog.Log(fmtString, i)
}

you are likely to find that the performance gap is considerably smaller.

  cheers,
rog.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] gosnip: run small snippets of Go code from the command line

[Ben Hoyt]

>
> I've actually been using gomacro for this (testing a quick bit of syntax).
> I have to import the std libs that I need (auto import would be nice if
> it's not ambiguous), but it has nice output evaluation and state.
>
> https://github.com/cosmos72/gomacro
>

Ha, nice! You know, I think I remember finding that tool when I had the
idea for gosnip, but somehow forgot about it in the meantime.

"gomacro -e 'statements'" is similar, and of course much faster because it
avoids build/run. The gomacro guy is even open to adding the auto-import
feature, and there's an open issue ("good first issue") to add this:
https://github.com/cosmos72/gomacro/issues/22

-Ben

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: is it possible to speed up type assertion?

[Arkady]

Jan,

I have just measured overall performance of my code - call to Log(string, 
args []interface{}). 
This code makes the difference 

type iface struct {
tab  *unsafe.Pointer
data *unsafe.Pointer
}

func getInterfaceData(arg interface{}) unsafe.Pointer {
return unsafe.Pointer*iface)(unsafe.Pointer())).data))
}

func (b *Binlog) writeArgumentToOutput(writer writer, arg interface{}) 
error {
// unsafe pointer to the data depends on the data type
var err error
err = writer.write(b.ioWriter, getInterfaceData(arg))
return err
}



-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: is it possible to speed up type assertion?

[Jan Mercl]

> The code below consumes ~40% of the total execution time. According to
the profiler i := uint64(arg.(uint32)) is a major contributor
>
> // Cast the integer argument to uint64 and call a "writer"
> // The "writer" knows how many bytes to add to the binary stream
> // Type casts from interface{} to integer consume 40% of the overall
> // time. Can I do better? What is interface{} in Golang?
> func (b *Binlog) writeArgumentToOutput(writer writer, arg interface{},
argKind reflect.Kind) error {
> // unsafe pointer to the data depends on the data type
> var err error
> switch argKind {
> case reflect.Int8:
> i := uint64(arg.(int8))
> err = writer.write(b.ioWriter, unsafe.Pointer())
> case reflect.Int16:
> i := uint64(arg.(int16))
> err = writer.write(b.ioWriter, unsafe.Pointer())
> case reflect.Int32:
> i := uint64(arg.(int32))
> err = writer.write(b.ioWriter, unsafe.Pointer())
> case reflect.Int64:
> i := uint64(arg.(int64))
> err = writer.write(b.ioWriter, unsafe.Pointer())
> case reflect.Uint8:
> i := uint64(arg.(uint8))
> err = writer.write(b.ioWriter, unsafe.Pointer())
> case reflect.Uint16:
> i := uint64(arg.(uint16))
> err = writer.write(b.ioWriter, unsafe.Pointer())
> case reflect.Uint32:
> i := uint64(arg.(uint32))
> err = writer.write(b.ioWriter, unsafe.Pointer())
> case reflect.Uint64:
> i := uint64(arg.(uint64))
> err = writer.write(b.ioWriter, unsafe.Pointer())
> case reflect.Int:
> i := uint64(arg.(int))
> err = writer.write(b.ioWriter, unsafe.Pointer())
> case reflect.Uint:
> i := uint64(arg.(uint))
> err = writer.write(b.ioWriter, unsafe.Pointer())
> default:
> return fmt.Errorf("Unsupported type: %T\n",
reflect.TypeOf(arg))
> }
> return err
> }

What roger peppe said.

- Performing an unchecked type assertion is incorrect in the general case.

Example: https://play.golang.org/p/YmZyw0MnOK8

- Unsafe seems to be used for no good reason.

- Reflect ditto.

This code https://play.golang.org/p/4sd1yayjoNY prints

500035.3 ns/op

While this one https://play.golang.org/p/tSfWbrqv4PN prints

1   18.7 ns/op

Using Intel® Xeon(R) CPU E5-1650 v2 @ 3.50GHz × 12.

PS: The last two links point to code that cannot run in the playground.

-- 

-j

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: is it possible to speed up type assertion?

[Arkady]

P.S. shaves 20% from the total execution time.
On Mon, Nov 26, 2018 at 10:22 AM  wrote:
>
> The ugly hack below shaves 20% from the switch-case
> type iface struct {
> tab  *unsafe.Pointer
> data *unsafe.Pointer
> }
>
> func getInterfaceData(arg interface{}) unsafe.Pointer {
> return unsafe.Pointer*iface)(unsafe.Pointer())).data))
> }
>
> // Cast the integer argument to uint64 and call a "writer"
> // The "writer" knows how many bytes to add to the binary stream
> //
> // Type casts from interface{} to integer consume 40% of the overall
> // time. Can I do better? What is interface{} in Golang?
> // Switching to args *[]interface makes the performance 2x worse
> // Before you jump to conlusions see
> // https://groups.google.com/forum/#!topic/golang-nuts/Og8s9Y-Kif4
> func (b *Binlog) writeArgumentToOutput(writer writer, arg interface{}) error {
> // unsafe pointer to the data depends on the data type
> var err error
> err = writer.write(b.ioWriter, getInterfaceData(arg))
> return err
> }
>
>
> --
> You received this message because you are subscribed to a topic in the Google 
> Groups "golang-nuts" group.
> To unsubscribe from this topic, visit 
> https://groups.google.com/d/topic/golang-nuts/Og8s9Y-Kif4/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to 
> golang-nuts+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: is it possible to speed up type assertion?

[larytet]

The ugly hack below shaves 20% from the switch-case
type iface struct {
tab  *unsafe.Pointer
data *unsafe.Pointer
}

func getInterfaceData(arg interface{}) unsafe.Pointer {
return unsafe.Pointer*iface)(unsafe.Pointer())).data))
}

// Cast the integer argument to uint64 and call a "writer"
// The "writer" knows how many bytes to add to the binary stream
//
// Type casts from interface{} to integer consume 40% of the overall
// time. Can I do better? What is interface{} in Golang?
// Switching to args *[]interface makes the performance 2x worse
// Before you jump to conlusions see
// https://groups.google.com/forum/#!topic/golang-nuts/Og8s9Y-Kif4
func (b *Binlog) writeArgumentToOutput(writer writer, arg interface{}) 
error {
// unsafe pointer to the data depends on the data type
var err error
err = writer.write(b.ioWriter, getInterfaceData(arg))
return err
}


-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.