Re: [go-nuts] Crash using gccheckmark=1, possible GC bug?

[Ian Lance Taylor]

On Thu, Aug 17, 2023 at 5:41 PM Tibor Halter  wrote:
>
> I get this crash when using the GODEBUG option gccheckmark=1:
>
> runtime: marking free object 0xc0016c5668 found at *(0xc0046ca8e0+0x8)
> base=0xc0046ca8e0 s.base()=0xc0046ca000 s.limit=0xc0046cc000 s.spanclass=8 
> s.elemsize=32 s.state=mSpanInUse
>  *(base+0) = 0x15e11f0
>  *(base+8) = 0xc0016c5668 <==
>  *(base+16) = 0x28
>  *(base+24) = 0xc004642030
> obj=0xc0016c5668 s.base()=0xc0016c4000 s.limit=0xc0016c5ff8 s.spanclass=6 
> s.elemsize=24 s.state=mSpanInUse
>  *(obj+0) = 0x15e0e80
>  *(obj+8) = 0xc00039c080
>  *(obj+16) = 0xc0016c5650
> fatal error: marking free object
>
> runtime stack:
> runtime.throw({0x132362a?, 0x3?})
> /usr/local/go-1.21.0/src/runtime/panic.go:1077 +0x5c fp=0x7f9636366d48 
> sp=0x7f9636366d18 pc=0x43aa9c
> runtime.greyobject(0xc0016c5668, 0x1?, 0x7f9636366df8?, 0x6590?, 
> 0x7f9636366df8?, 0x7f9636366de0?)
> /usr/local/go-1.21.0/src/runtime/mgcmark.go:1476 +0x285 fp=0x7f9636366d98 
> sp=0x7f9636366d48 pc=0x424d85
> runtime.scanobject(0xc0046ca8e0, 0xc4d240)
> /usr/local/go-1.21.0/src/runtime/mgcmark.go:1336 +0x171 fp=0x7f9636366e20 
> sp=0x7f9636366d98 pc=0x424711
> runtime.gcDrain(0xc4d240, 0x7)
> /usr/local/go-1.21.0/src/runtime/mgcmark.go:1103 +0x1ba fp=0x7f9636366e80 
> sp=0x7f9636366e20 pc=0x423fba
> runtime.gcBgMarkWorker.func2()
> /usr/local/go-1.21.0/src/runtime/mgc.go:1385 +0x6f fp=0x7f9636366ed0 
> sp=0x7f9636366e80 pc=0x4208af
> runtime.systemstack()
> /usr/local/go-1.21.0/src/runtime/asm_amd64.s:509 +0x4a fp=0x7f9636366ee0 
> sp=0x7f9636366ed0 pc=0x46b1aa
>
>
> From the docs, I understand this 2nd pass stop-the-world verification
> guarantees the concurrent mark step did not follow the pointer, aka this is a 
> GC bug?
>
> A confirmation would be greatly appreciated to help me follow the right trail.
>
> Context
> I'm building a type-system on top of Go for user-defined data structures. For 
> efficient implementation, I'm using unsafe.Pointer-s heavily. I believe I 
> understand pointer conversion rules, span classes, that there is metadata on 
> the side to differentiate ordinary values from pointers and how it is used to 
> follow pointer chains.
> Go vet detects no issues. I've been debugging this for 4 days.

That error looks like the GC found a pointer to an object that was
already freed.  This can happen, for example, if you convert a uintptr
to a pointer type, but the object to which the pointer points has been
freed.  While it could be a GC bug, that would not be my first guess.
The contents of the block shown above are intended to help you
identify where the pointer came from.

Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAOyqgcX%3DD6WOoLhsjWTnQqJyAAzYTW03qV56nzKLijSJZOSuwA%40mail.gmail.com.

[go-nuts] Crash using gccheckmark=1, possible GC bug?

[Tibor Halter]

Hi!

I get this crash when using the GODEBUG option gccheckmark=1:

runtime: marking free object 0xc0016c5668 found at *(0xc0046ca8e0+0x8)
base=0xc0046ca8e0 s.base()=0xc0046ca000 s.limit=0xc0046cc000 s.spanclass=8 
s.elemsize=32 s.state=mSpanInUse
 *(base+0) = 0x15e11f0
 *(base+8) = 0xc0016c5668 <==
 *(base+16) = 0x28
 *(base+24) = 0xc004642030
obj=0xc0016c5668 s.base()=0xc0016c4000 s.limit=0xc0016c5ff8 s.spanclass=6 
s.elemsize=24 s.state=mSpanInUse
 *(obj+0) = 0x15e0e80
 *(obj+8) = 0xc00039c080
 *(obj+16) = 0xc0016c5650
fatal error: marking free object

runtime stack:
runtime.throw({0x132362a?, 0x3?})
/usr/local/go-1.21.0/src/runtime/panic.go:1077 +0x5c fp=0x7f9636366d48 
sp=0x7f9636366d18 pc=0x43aa9c
runtime.greyobject(0xc0016c5668, 0x1?, 0x7f9636366df8?, 0x6590?, 
0x7f9636366df8?, 0x7f9636366de0?)
/usr/local/go-1.21.0/src/runtime/mgcmark.go:1476 +0x285 fp=0x7f9636366d98 
sp=0x7f9636366d48 pc=0x424d85
runtime.scanobject(0xc0046ca8e0, 0xc4d240)
/usr/local/go-1.21.0/src/runtime/mgcmark.go:1336 +0x171 fp=0x7f9636366e20 
sp=0x7f9636366d98 pc=0x424711
runtime.gcDrain(0xc4d240, 0x7)
/usr/local/go-1.21.0/src/runtime/mgcmark.go:1103 +0x1ba fp=0x7f9636366e80 
sp=0x7f9636366e20 pc=0x423fba
runtime.gcBgMarkWorker.func2()
/usr/local/go-1.21.0/src/runtime/mgc.go:1385 +0x6f fp=0x7f9636366ed0 
sp=0x7f9636366e80 pc=0x4208af
runtime.systemstack()
/usr/local/go-1.21.0/src/runtime/asm_amd64.s:509 +0x4a fp=0x7f9636366ee0 
sp=0x7f9636366ed0 pc=0x46b1aa


>From the docs, I understand this 2nd pass stop-the-world verification
guarantees the concurrent mark step did not follow the pointer, aka this is 
a GC bug?

A confirmation would be greatly appreciated to help me follow the right 
trail.

*Context*
I'm building a type-system on top of Go for user-defined data structures. 
For efficient implementation, I'm using unsafe.Pointer-s heavily. I believe 
I understand pointer conversion rules, span classes, that there is metadata 
on the side to differentiate ordinary values from pointers and how it is 
used to follow pointer chains.
Go vet detects no issues. I've been debugging this for 4 days.

Many thanks!

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/59b6e020-a8e6-4310-b28a-ffd228b30787n%40googlegroups.com.

Re: [go-nuts] Why is reflect.Type defined as an interface, but reflect.Value is defined as a struct?

[Ian Lance Taylor]

On Thu, Aug 17, 2023 at 10:02 AM Hu Jian  wrote:
>
> I'm curious why reflect.Type is defined as an interface, but reflect.Value is 
> defined as a struct? I don't see any other implementation of the reflect.Type 
> interface.

reflect.Value is an ordinary struct.  The reflect.Type interface is
really implemented by several different types, each of which begins
with the same struct but is then followed by type-specific
information.  On tip this is abi.PointerType, abi.StructType, and so
forth.  That data structure can in turn be followed by
abi.UncommonType.  So we have several possible implementations for
reflect.Type, although they all exist in the reflect package.

Also reflect.Value is small enough to pass around as a value, rather
than taking a pointer and sometimes forcing the Value into the heap
There's no reason to do that for reflect.Type, as we are normally
using pointers to data structures created by the compiler, not data
structures in the heap.

I think it's true that we could today push everything into a single
struct, and do all the pointer conversions behind the scenes, but it
wouldn't help much, and we can't make that change today anyhow.

Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAOyqgcWUvj5kkw6wUeMneGyEqh8MRZXO3CLd5Oz6rSY_z%2BhYuQ%40mail.gmail.com.

[go-nuts] Why is reflect.Type defined as an interface, but reflect.Value is defined as a struct?

[Hu Jian]

hi all,
I'm curious why reflect.Type is defined as an interface, but reflect.Value 
is defined as a struct? I don't see any other implementation of the 
reflect.Type interface.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/d2163138-8bc2-40d8-84d1-2a0874273a6fn%40googlegroups.com.

[go-nuts] Re: how to constrain a type

[Brian Candler]

Then make a dummy interface, with a method with a unique name that you 
never call.

On Thursday, 17 August 2023 at 16:48:28 UTC+1 Mark wrote:

> Hi Brian,
> Sorry for not being clearer. Ideally I don't want an interface: each Shape 
> is essentially a store of different kinds of values so at some point I'm 
> going to iterate over them and within that iterating use a type switch. 
> This is because the shapes _don't_ know how to draw themselves but an 
> external function that reads them will know.
>
> On Thursday, August 17, 2023 at 4:35:10 PM UTC+1 Brian Candler wrote:
>
>> I'm not sure without knowing *how* you want to use []*Shape{}.  Perhaps 
>> you want Shape to be an interface?
>>
>> type Shape interface {
>> DrawMe()
>> }
>>
>> type BoxShape struct ...
>> func (*BoxShape) DrawMe() { }
>>
>> type LineShape struct ...
>> func (*LineShape) DrawMe() { }
>>
>> shapes := []Shape{}
>>
>> (Note that interfaces internally carry pointers, so you don't generally 
>> want to have pointers to interface types)
>>
>> On Thursday, 17 August 2023 at 13:44:10 UTC+1 Mark wrote:
>>
>>> Here's something that works but uses `any` which is too broad in scope:
>>> ```go
>>> type Shape any // Want to constrain to BoxShape and LineShape only
>>>
>>> type BaseShape struct {
>>> X int
>>> Y int
>>> PenColor  color
>>> PenWidth  int
>>> FillColor color
>>> }
>>>
>>> type BoxShape struct {
>>> BaseShape
>>> Width  int
>>> Height int
>>> }
>>>
>>> type LineShape struct {
>>> BaseShape
>>> X2 int
>>> Y2 int
>>> }
>>>
>>> shapes := []*Shape{}
>>> ```
>>> Is there a nice way to create a `Shape` type which will allow me to do 
>>> `shapes := []*Shape{}` and that _also_ constrains the shapes to `BoxShape` 
>>> and `LineShape`?
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/09798318-1a13-4570-a8a2-af6ccb3f0155n%40googlegroups.com.

[go-nuts] Re: how to constrain a type

[Brian Candler]

I'm not sure without knowing *how* you want to use []*Shape{}.  Perhaps you 
want Shape to be an interface?

type Shape interface {
DrawMe()
}

type BoxShape struct ...
func (*BoxShape) DrawMe() { }

type LineShape struct ...
func (*LineShape) DrawMe() { }

shapes := []Shape{}

(Note that interfaces internally carry pointers, so you don't generally 
want to have pointers to interface types)

On Thursday, 17 August 2023 at 13:44:10 UTC+1 Mark wrote:

> Here's something that works but uses `any` which is too broad in scope:
> ```go
> type Shape any // Want to constrain to BoxShape and LineShape only
>
> type BaseShape struct {
> X int
> Y int
> PenColor  color
> PenWidth  int
> FillColor color
> }
>
> type BoxShape struct {
> BaseShape
> Width  int
> Height int
> }
>
> type LineShape struct {
> BaseShape
> X2 int
> Y2 int
> }
>
> shapes := []*Shape{}
> ```
> Is there a nice way to create a `Shape` type which will allow me to do 
> `shapes := []*Shape{}` and that _also_ constrains the shapes to `BoxShape` 
> and `LineShape`?
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/b2d706b5-71f8-461f-ab5f-7da5b238ec17n%40googlegroups.com.

Re: [go-nuts] keyword= support for function calls

[Jason Phillips]

Even if we assume that the addition of keyword arguments would be a 
positive for the Go language and ecosystem, it wouldn't be a backward 
compatible change to add a keyword argument unless we make additional 
changes to the language to support implicit conversions between 
function-with-keyword types. For example, a completely valid way to use 
os.Open today would be to pass it into a function as an argument:

  func DoThing(open func(string) (*os.File, error)) {
f, err := open("foo.json")
// etc
  }
  ...
  DoThing(os.Open)

How does the above code work after adding a new keyword argument to 
os.Open? If a new non-keyword argument was added, this would clearly be a 
backward incompatible change today. How would I implement DoThing such that 
I _can_ use new keyword arguments?

Any real language change needs to precisely define what happens in the 
above cases, and more. So, if you're serious about such a proposal, you 
need to state the exact changes you're proposing to the language 
specification, the expected impact on the language implementation(s), and 
have some evidence that the proposed changes are worth the effort and are 
better than the alternatives.
On Thursday, August 17, 2023 at 6:13:55 AM UTC-4 Dan Kortschak wrote:

> On Thu, 2023-08-17 at 11:34 +0200, Peter Herth wrote:
> > I think the omission of keyword parameters in Go is a weakness. In
> > many cases, keyword parameters are a simple way of creating APIs,
> > which depend on a lot of possible parameters, of which most are not
> > necessarily specified. Their omission is especially ironic, as there
> > is a strong push to initialize structs only via using the keyword
> > parameter syntax. And the reasons for that are very good. So when we
> > have a nice system - which essentially is just syntactic sugar at the
> > call site - for structs, why can't we have the same system for
> > functions? Like with structs, it should be mostly a syntax for
> > calling functions. They should probably be able to specify which
> > parameters are positional and which can be specified by keyword
> > parameters. 
> > Of course you can do the common "trick" via generating a struct which
> > allows the keyword syntax for creation, but I think it would be a
> > good addition to not need that indirection.
>
> A significant problem with additions like this is that people would use
> it. kwargs-rich functions enable overly complex APIs. I'm very glad Go
> doesn't have them.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/90a6abfc-d878-4935-a718-193acfa9e853n%40googlegroups.com.

Re: [go-nuts] keyword= support for function calls

['Dan Kortschak' via golang-nuts]

On Thu, 2023-08-17 at 11:34 +0200, Peter Herth wrote:
> I think the omission of keyword parameters in Go is a weakness. In
> many cases, keyword parameters are a simple way of creating APIs,
> which depend on a lot of possible parameters, of which most are not
> necessarily specified. Their omission is especially ironic, as there
> is a strong push to initialize structs only via using the keyword
> parameter syntax. And the reasons for that are very good. So when we
> have a nice system - which essentially is just syntactic sugar at the
> call site - for structs, why can't we have the same system for
> functions? Like with structs, it should be mostly a syntax for
> calling functions. They should probably be able to specify which
> parameters are positional and which can be specified by keyword
> parameters. 
> Of course you can do the common "trick" via generating a struct which
> allows the keyword syntax for creation, but I think it would be a
> good addition to not need that indirection.

A significant problem with additions like this is that people would use
it. kwargs-rich functions enable overly complex APIs. I'm very glad Go
doesn't have them.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/3a3163c2d6827bb2c40a850c2d24b39dced1285c.camel%40kortschak.io.

Re: [go-nuts] keyword= support for function calls

[Peter Herth]

I think the omission of keyword parameters in Go is a weakness. In many
cases, keyword parameters are a simple way of creating APIs, which depend
on a lot of possible parameters, of which most are not necessarily
specified. Their omission is especially ironic, as there is a strong push
to initialize structs only via using the keyword parameter syntax. And the
reasons for that are very good. So when we have a nice system - which
essentially is just syntactic sugar at the call site - for structs, why
can't we have the same system for functions? Like with structs, it should
be mostly a syntax for calling functions. They should probably be able to
specify which parameters are positional and which can be specified by
keyword parameters.
Of course you can do the common "trick" via generating a struct which
allows the keyword syntax for creation, but I think it would be a good
addition to not need that indirection.



On Thu, Aug 17, 2023 at 3:03 AM Jon Perryman  wrote:

> Do you feel your GOLANG API implementations allows your API's
> and functions to evolve and grow? Personally, positional args has pissed me
> off for years when a proven simple solution has never been adopted by most
> Unix programming languages (except Javascript). Most notably, it's missing
> from GOLANG.
>
> Everyone here understands and uses os.open from the os package.  For
> argument's sake, let's say os.open() is your responsibility to integrate
> nosql, memory files, compression, encryption, btree, key indexed files and
> more. How do you modify os.open( ) to integrate these file related
> features? This was solved years ago in languages not used in Unix
> using keyword arguments that also included positional. For example:
>
> func os.open(name string, memoryfile= bool, compress= bool, btree{
> btree_optons string }, encrypt{ key= string }, nosql{ nosql_options string
> } ) (*File, error)
> os.open('myfile.txt', encrypt{ key='abc' }, compress=true)
>
> The os.open args 1, 3 and 5 would be set from the specified arguments.
>
>  Is this something that others need and I should pursue or should I just
> forget it? Proposal https://github.com/golang/go/issues/62078 was closed
> with "I don't see anything concrete being proposed here.". If I should
> pursue this, should I reword it or reduce it to this Email?
>
> Thanks, Jon.
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/CAByJhJnkuai27VNiE6PraU9-5hoO85Hm__0UQJrT75a7KqD8uw%40mail.gmail.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CANVuwyFU-O4hfJckFtZnNGen_jZ-Dmg_fKXR90ChLogTBiFOKA%40mail.gmail.com.

[go-nuts] Re: Test coverage: joint unit test and integration test coverage

[Jan]

I tried to use the `-test.gocoverdir` (*) and while the test(s) being 
executed use the given directory, unfortunately it still sets `GOCOVERDIR` 
to some newly created temporary directory. 

Since my integration tests are executed from a *_test.go  
test 
(it's a test after all), in the end the `-test.gocoverdir` flag contents 
are discarded (or not passed along in GOCOVERDIR). If the flag is set, 
shouldn't it be used for `GOCOVERDIR` as well ?

cheers

(*) Btw, I can't find documentation on -test.gocoverdir in Google 
,
 
except the insides of code. Also `go help testflag | grep gocoverdir` 
doesn't return anything -- in go1.21.0.

On Thursday, August 17, 2023 at 8:08:51 AM UTC+2 Jan wrote:

> Oh yes, it's true that the new mechanism is more general, and allows 
> integration tests, it's a huge improvement for cases like this! Thanks for 
> the design btw!
>
> Now the ergonomics of the case of unit tests + integration tests could be 
> made easier/more ergonomic -- I would assume it is the common case for 
> projects with integration tests (because everyone has also unit tests). And 
> a suggestion of maybe a mention / an example in the integration-test-coverage 
> blog  ?
>
> On Wednesday, August 16, 2023 at 10:03:55 PM UTC+2 TheDiveO wrote:
>
>> Hmm, my previous response got deleted for no reason. So here we go: the 
>> new mechanism is more general, and as you can see in my example I actually 
>> run the "same" unit tests twice. The reason is because some code paths 
>> might be only exercised when not being run as root, especially error 
>> handling. So there is some value in the new mechanism even "just" for unit 
>> tests. Admittedly, not all will need that in their unit tests.
>>
>> On Wednesday, August 16, 2023 at 8:53:56 PM UTC+2 Jan wrote:
>>
>>> Thanks, that's very helpful! 
>>>
>>> I was doing something similar 
>>> ,
>>>  
>>> but I was extracting the temporary coverage directory created by Go because 
>>> I didn't know about the --test.gocoverdir flag. 
>>>
>>> Still it feels wrong one has to do the trick of creating a temporary 
>>> directory (or extract it from the go tool) , where for unit tests only it's 
>>> not needed ... I wonder if this is done deliberately, or just a bug ?
>>>
>>> cheers
>>>
>>> On Wednesday, August 16, 2023 at 8:33:23 PM UTC+2 TheDiveO wrote:
>>>
 Maybe similar to this? 
 https://github.com/thediveo/lxkns/blob/cef5a31d7517cb126378f81628f51672cb793527/scripts/cov.sh#L28

 On Wednesday, August 16, 2023 at 1:54:48 PM UTC+2 Jan wrote:

> hi all, 
>
> After reading the documentation 
> , I managed to set up 
> a process in which I compile and run my integration tests, and get 
> coverage 
> from them in the `$GOCOVERDIR` subdirectory.
>
> Now I would like to have a combined unit tests + integration tests 
> report, all in one go, and I'm not sure how to get that.
>
> I was expecting that, if I go to my modules root directory, and I do:
>
> ```
> go test --cover --coverpkg=./... --coverprofile=/tmp/cover_profile.out 
> ./...
> go tool cover -func /tmp/cover_profile.out > /tmp/cover_func.out
> ```
>
> I  would get all the results, including integration tests (since they 
> are called with GOCOVERDIR set). But instead I only see the coverage of 
> the 
> unit tests, and the information from the integration tests seems to be 
> ignored, even though it is generated.
>
> I'm sure it is generated because if I run the command above with 
> `--work` (preserve the temporary files), and log the value of $GOCOVERDIR 
> I 
> can see where the test stores the coverage data files. And if I manually 
> do:
>
> ```
> go tool covdata func -i /tmp/go-build287472875/b001/gocoverdir
> ```
> (where /tmp/go-build/gocoverdir is the temporary directory 
> reported for GOCOVERDIR)
>
> I see the results I expected (some of my functions that I want to make 
> sure are covered) are there. But they are not reported in 
> `/tmp/cover_func.out` above.
>
> Any ideas why ? 
>
> Or any other suggestions on how to merge the report from unit tests 
> (`/tmp/cover_profile.out` in my example) and integration tests ? 
>
> thanks!
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To 

Re: [go-nuts] keyword= support for function calls

['Axel Wagner' via golang-nuts]

The reason `os.Open` has not changed is because it doesn't have to change.
None of the features you mention really make sense for what it does. At
least that would be my interpretation. If you want to support your
interpretation, I would suggest digging up proposals to add those features
which got rejected because we didn't want to add more functions. That would
show that a lack of keyword arguments is to blame for a lack of `os.Open`
evolution, instead of a lack of need for evolution.

On the other hand, `net.Dial` empirically shows that it *is* possible to
use this approach to not add more functions, while expanding functionality.
Pre Go 1.0 there was only net.Dial. For Go 1.0, we determined that we would
want to be able to specify timeouts, so we added net.DialTimeout
. Then, for Go 1.1, we also wanted to specify a
local address to use. However, it became clear that it is not sustainable
to add more and more versions of a `Dial` function, so net.Dialer was added
in Go 1.1 . Since then, we have added
many more options to it . As adding fields
to a struct is backwards-compatible, we can add as many options as we like,
while still maintaining compatibility and not adding new package-scoped API
surface or functions.

The actual history of `net.Dial` strongly implies that if we *wanted* to
evolve `os.Open`, then what we would do is add a new struct type
`os.Opener`, with fields for all these optional features, which would then
have an `func (*Opener) Open(name string) (*File, error)` method. It's not
a perfect mechanism, but so far, it seems to have sufficed.

To be clear, if you feel strongly that we should support keyword arguments,
you can write and file a proposal to that effect. All I can say is that in
the past, this has come up repeatedly, but obviously has not been accepted.
So your case should better be strong - and in particular, it should
acknowledge the evidence and history we've built in over a decade. And you
might ultimately save yourself some frustration by instead adopting the
imperfect solutions we do have.

On Thu, Aug 17, 2023 at 7:32 AM Jon Perryman  wrote:

> The net.dialer solution you mention uses a struct which does not simplify.
> How does this get programmers out of the add another function mentality?
> GOLANG and C/C++ programmers have the same solution because it's not
> convenient to use a struct. Compare charts in C++ versus JavaScript. C++
> solution chose to implement 1,000 functions instead of using the struct as
> you suggest. JavaScript looked to the future and implemented it using the
> method that I'm proposing for GOLANG. The method I'm suggesting is so easy
> and convenient that people use it as seen with chartjs.
>
> Ask yourself why os.open() has remained unchanged since the beginning
> instead of expanding to include new features. Adding new features is
> extremely inconvenient. Using a struct is inconvenient.  Let's just add
> more functions.
>
> On Wed, Aug 16, 2023 at 9:14 PM Axel Wagner 
> wrote:
>
>> Have a look at net.Dialer . It is
>> essentially a struct that adds optional, keyword arguments to net.Dial
>> .
>>
>> On Thu, Aug 17, 2023 at 3:44 AM Jon Perryman 
>> wrote:
>>
>>> Struct does not solve the basic problem. In what way are you suggesting
>>> to solve the described os.open() changes? Should each API be seperate from
>>> os.open where it must be used with every read/write?
>>>
>>> On Wed, Aug 16, 2023 at 6:14 PM Kurtis Rader 
>>> wrote:
>>>
 Personally, and I say this as a happy user of Python, I dislike keyword
 parameters. Note that in Python keyword parameters are (or should be)
 seldom used for anything other than optional arguments. And if your API has
 so many parameters that they need names the API probably needs refactoring.
 Alternatively, pass a struct whose contents can be initialized using the
 struct member names.

 On Wed, Aug 16, 2023 at 6:04 PM Jon Perryman 
 wrote:

> Do you feel your GOLANG API implementations allows your API's
> and functions to evolve and grow? Personally, positional args has pissed 
> me
> off for years when a proven simple solution has never been adopted by most
> Unix programming languages (except Javascript). Most notably, it's missing
> from GOLANG.
>
> Everyone here understands and uses os.open from the os package.  For
> argument's sake, let's say os.open() is your responsibility to integrate
> nosql, memory files, compression, encryption, btree, key indexed files and
> more. How do you modify os.open( ) to integrate these file related
> features? This was solved years ago in languages not used in Unix
> using keyword arguments that also included positional. For example:
>
> func os.open(name string, memoryfile= bool, compress= bool, btree{
> 

Re: [go-nuts] weird "index out of range" in strconv.formatBits

['Dan Kortschak' via golang-nuts]

On Wed, 2023-08-16 at 23:43 -0700, metronome wrote:
> Thanks for commenting, a few supplements.
> 
> # 1. Version of Go?
> We observed the issue with both go1.20.5 and go1.18.10 on linux/amd64
> (centos)
> 
> # 2. Context?
> All panics we observed so far are from either 
>                  strconv.FormatInt -> strconv.formatBits chain, or
>                  strconv.FormatUint -> strconv.formatBits chain
> where the base is always 10.
> 
> // typical call site, toId is an "*int64".
> if com_count > 1 {
> com_string = anchor + "," + strconv.FormatInt(*toId, 10)
> }
> 
> # 3. If your program using pure Go (statically linked) or Cgo?
> Binary was built with CGO_ENABLED=1 and -buildmode=exe.
> All panic call sites we observed so far are "pure go", that is, no C
> calling go path.
> 
> # 4. panic stack trace
> panic: runtime error: index out of range [18446744073708732603] with
> length 200
> 
> goroutine 1 [running]:
> strconv.formatBits({0x0?, 0x0?, 0x0?}, 0xc09e00b750?, 0x1?, 0x1?,
> 0x0?)
>   /usr/lib/go-1.20/src/strconv/itoa.go:140 +0x4b9
> strconv.FormatInt(0x0?, 0xc07393df80?)
>   /usr/lib/go-1.20/src/strconv/itoa.go:29 +0xa5
>                   ...


Have you eliminated the possibility of races? Have you built with
CGO_ENABLED=0?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/79322d9278fab127f2a75f51986cc784a8c8666c.camel%40kortschak.io.

Re: [go-nuts] weird "index out of range" in strconv.formatBits

[metronome]

Thanks for commenting, a few supplements.

*# 1. Version of Go?*
We observed the issue with both go1.20.5 and go1.18.10 on linux/amd64 
(centos)

*# 2. Context?*
All panics we observed so far are from either 
 *strconv.FormatInt -> strconv.formatBits* chain, or
 *strconv.FormatUint -> strconv.formatBits* chain
where the base is always 10.

// typical call site, toId is an "*int64".

if com_count > 1 {

com_string = anchor + "," + strconv.FormatInt(*toId, 10)

}


*# 3. If your program using pure Go (statically linked) or Cgo?*

Binary was built with CGO_ENABLED=1 and -buildmode=exe.

All panic call sites we observed so far are "pure go", that is, no C 
calling go path.


*# 4. panic stack trace*
panic: runtime error: index out of range [18446744073708732603] with length 
200

goroutine 1 [running]:
strconv.formatBits({0x0?, 0x0?, 0x0?}, 0xc09e00b750?, 0x1?, 0x1?, 0x0?)
/usr/lib/go-1.20/src/strconv/itoa.go:140 +0x4b9
strconv.FormatInt(0x0?, 0xc07393df80?)
/usr/lib/go-1.20/src/strconv/itoa.go:29 +0xa5

  ...


On Thursday, August 17, 2023 at 12:56:17 PM UTC+8 Kurtis Rader wrote:

> Insufficient information. Version of Go? Since formatBits is private we 
> need to see the actual code of a call to a public API that resulted in the 
> call to formatBits that failed. Also, show us the literal panic stack. 
> Showing us the assembly code with no context is not useful. If your program 
> using pure Go (statically linked) or Cgo? Wild guesses, what I used to call 
> SWAGS (silly wild ass guesses) as a Unix support engineer, are seldom 
> useful. If your guess is scientifically informed that is a different matter 
> but you should be able to articulate why you think your guess is more 
> likely to be true than a random coin flip.
>
> On Wed, Aug 16, 2023 at 8:31 PM metronome  wrote:
>
>> Hi,
>>
>> We ran into a weird *out of range* issue of *strconv.formatBits*, hope 
>> someone can shed a light on what could be the root cause, any comment is 
>> highly appreciated.
>>
>> problem description:
>> *  random out of range at code 
>> ,
>>  
>> most of the time the indexing is a huge int but we observed at least one 
>> exception (#2).*
>>
>> *  #1: runtime error: index out of range [18446744073709449339] with 
>> length 200*
>> * #2: runtime error: index out of range [102511] with length 200*
>>
>> Wild guesses:
>> 1. The machine code seems to suggest it's unlikely a data race or memory 
>> corruption? But perhaps 
>> relevant registers, like R10, had been saved and restored, then it might 
>> be due to stack corruption?
>> Given that R12 is scratch reg, is it possible that R12 is clobbered 
>> somehow, say, by signal handling?
>>
>> ===
>>0x00495b0a<+810>:mov%rdi,%r10
>>
>>0x00495b0d<+813>:shr%rdi
>>
>>0x00495b10<+816>:mov%rax,%rsi
>>
>>0x00495b13<+819>:movabs $0xa3d70a3d70a3d70b,%rax
>>
>>0x00495b1d<+829>:mov%rdx,%r11
>>
>>0x00495b20<+832>:mul%rdi
>>
>>0x00495b23<+835>:shr$0x5,%rdx
>>
>> *   0x00495b27<+839>:imul   $0x64,%rdx,%r12*
>>
>> *   0x00495b2b<+843>:sub%r12,%r10*
>>
>> *   0x00495b2e<+846>:lea(%r10,%r10,1),%rax*
>>
>>0x00495b32<+850>:lea0x1(%rax),%rax
>>
>>0x00495b36<+854>:nopw   0x0(%rax,%rax,1)
>>
>>0x00495b3f<+863>:nop
>>
>>0x00495b40<+864>:cmp$0xc8,%rax
>>
>>0x00495b46<+870>:jae0x495c8f 
>> 
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/ff804d4c-24ee-480d-8ed1-219f9b8d7cbcn%40googlegroups.com
>>  
>> 
>> .
>>
>
>
> -- 
> Kurtis Rader
> Caretaker of the exceptional canines Junior and Hank
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/003ec4ce-443e-48ec-9ac8-22aa57b0c006n%40googlegroups.com.

[go-nuts] Re: Test coverage: joint unit test and integration test coverage

[Jan]

Oh yes, it's true that the new mechanism is more general, and allows 
integration tests, it's a huge improvement for cases like this! Thanks for 
the design btw!

Now the ergonomics of the case of unit tests + integration tests could be 
made easier/more ergonomic -- I would assume it is the common case for 
projects with integration tests (because everyone has also unit tests). And 
a suggestion of maybe a mention / an example in the integration-test-coverage 
blog  ?

On Wednesday, August 16, 2023 at 10:03:55 PM UTC+2 TheDiveO wrote:

> Hmm, my previous response got deleted for no reason. So here we go: the 
> new mechanism is more general, and as you can see in my example I actually 
> run the "same" unit tests twice. The reason is because some code paths 
> might be only exercised when not being run as root, especially error 
> handling. So there is some value in the new mechanism even "just" for unit 
> tests. Admittedly, not all will need that in their unit tests.
>
> On Wednesday, August 16, 2023 at 8:53:56 PM UTC+2 Jan wrote:
>
>> Thanks, that's very helpful! 
>>
>> I was doing something similar 
>> ,
>>  
>> but I was extracting the temporary coverage directory created by Go because 
>> I didn't know about the --test.gocoverdir flag. 
>>
>> Still it feels wrong one has to do the trick of creating a temporary 
>> directory (or extract it from the go tool) , where for unit tests only it's 
>> not needed ... I wonder if this is done deliberately, or just a bug ?
>>
>> cheers
>>
>> On Wednesday, August 16, 2023 at 8:33:23 PM UTC+2 TheDiveO wrote:
>>
>>> Maybe similar to this? 
>>> https://github.com/thediveo/lxkns/blob/cef5a31d7517cb126378f81628f51672cb793527/scripts/cov.sh#L28
>>>
>>> On Wednesday, August 16, 2023 at 1:54:48 PM UTC+2 Jan wrote:
>>>
 hi all, 

 After reading the documentation 
 , I managed to set up a 
 process in which I compile and run my integration tests, and get coverage 
 from them in the `$GOCOVERDIR` subdirectory.

 Now I would like to have a combined unit tests + integration tests 
 report, all in one go, and I'm not sure how to get that.

 I was expecting that, if I go to my modules root directory, and I do:

 ```
 go test --cover --coverpkg=./... --coverprofile=/tmp/cover_profile.out 
 ./...
 go tool cover -func /tmp/cover_profile.out > /tmp/cover_func.out
 ```

 I  would get all the results, including integration tests (since they 
 are called with GOCOVERDIR set). But instead I only see the coverage of 
 the 
 unit tests, and the information from the integration tests seems to be 
 ignored, even though it is generated.

 I'm sure it is generated because if I run the command above with 
 `--work` (preserve the temporary files), and log the value of $GOCOVERDIR 
 I 
 can see where the test stores the coverage data files. And if I manually 
 do:

 ```
 go tool covdata func -i /tmp/go-build287472875/b001/gocoverdir
 ```
 (where /tmp/go-build/gocoverdir is the temporary directory reported 
 for GOCOVERDIR)

 I see the results I expected (some of my functions that I want to make 
 sure are covered) are there. But they are not reported in 
 `/tmp/cover_func.out` above.

 Any ideas why ? 

 Or any other suggestions on how to merge the report from unit tests 
 (`/tmp/cover_profile.out` in my example) and integration tests ? 

 thanks!





-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/38d1228d-26d3-4fc5-ad89-4eddc335cfe4n%40googlegroups.com.