Re: [go-nuts] possible bug in slogtest or ReplaceAttr

['Sean Liao' via golang-nuts]

please file it as a bug for jsonhandler

- sean

On Fri, Aug 18, 2023, 21:17 jal...@gmail.com  wrote:

> Hi all,
>
> In slog, I am experimenting with the the ReplaceAttr func in the
> HandlerOptions with json handler. My understanding is that if i want to
> remove an attribute then return an empty slog.Attr. However, logtest test
> fails with a json unmarshal error. Note I am literally using the code
> example from the official docs.
>
> Is this expected or a  bug? Not sure what my user error is here.
>
> https://go.dev/play/p/i7VtMtNK4eV
>
> Thanks,
> Joe
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/89810de6-6a02-440f-988b-9503845a8c90n%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAGabyPpww3p8%2BpU%3DWM4QTOYmOE2eqd-VCh4Rpnvq9aHyfnKxVw%40mail.gmail.com.

Re: [go-nuts] Crash using gccheckmark=1, possible GC bug?

[Tibor Halter]

Ah reflect.StructOf! I totally missed that, that was the missing piece.

Thanks for the detailed response Ian, much appreciated.

Tibor

On Friday, August 18, 2023 at 11:06:27 PM UTC+2 Ian Lance Taylor wrote:

> On Fri, Aug 18, 2023 at 8:12 AM Tibor Halter  wrote:
> >
> > Hi Elias, thanks for chiming in!
> >
> > > The Go garbage collector is precise in that it only considers pointers 
> when determining memory that can be freed.
> >
> > Agreed, but it is not as precise on _what_ is a pointer. Specifically, 
> which matters?
> > - the type used upon allocating the memory, or
> > - the type used upon writing to it.
> >
> > Because *T or unsafe.Pointer is required upon writing to a memory 
> location for it to be marked as a pointer, I came to the conclusion that 
> the type upon writing is what matters.
> >
> > So then both types need to be pointers?
> >
> > Does that mean there is no way to define a struct-like memory layout 
> dynamically, that contains a mix of pointer and non-pointer types, in a way 
> that is supported by the GC?
> > (This is what I'm trying to do.)
>
> The rule is pretty simple: a pointer must always have a pointer type.
> That includes when it is in a variable and when it is stored in
> memory. In your terms, what matters is the type used upon allocating
> the memory.
>
> Your code is not following any of the patterns permitted for
> unsafe.Pointer (any patterns that are not explicitly permitted are
> forbidden). The code is effectively converting from *byte to *Num.
> As it says at https://pkg.go.dev/unsafe#Pointer that kind of
> conversion is only permitted if the types in question "share an
> equivalent memory layout". byte and Num have different memory
> layouts.
>
> You can define a struct-like memory layout dynamically by using
> reflect.StructOf.
>
> Ian
>
>
> > On Friday, August 18, 2023 at 3:49:03 PM UTC+2 ma...@eliasnaur.com 
> wrote:
> >>
> >> On Friday, 18 August 2023 at 05:57:38 UTC-6 Tibor Halter wrote:
> >>
> >> Thanks Ian!
> >>
> >> I have managed to strip it down to its essence, see code below.
> >>
> >> Summary of what I'm doing
> >> - reserve memory with make([]byte)
> >> - get pointer to underlying array
> >> - cast it to type Num
> >> - write pointer #2 into it
> >> - force GC()
> >> - GC does not follow pointer #2, memory is freed
> >>
> >>
> >> The Go garbage collector is precise in that it only considers pointers 
> when determining memory that can be freed. In your scenario and 
> demonstration program you explicitly store a pointer in a byte slice, 
> effectively making it invisible to the collector.
> >>
> >>
> >> --
> >>
> >>
> >> package debug
> >>
> >> import (
> >> "testing"
> >> "runtime"
> >> "unsafe"
> >> )
> >>
> >> type Num struct {
> >> numData *uint64
> >> }
> >>
> >> func Test(t *testing.T) {
> >> n := uint64(123456789)
> >> numMemSize := 8
> >> b := make([]byte, numMemSize, numMemSize)
> >> sliceDataPtr := unsafe.Pointer(unsafe.SliceData(b))
> >> (*Num)(sliceDataPtr).numData = &n
> >>
> >>
> >> The pointer to &n stored in b here is effectively hidden from the 
> collector.
> >>
> >>
> >>
> >> runtime.GC()
> >>
> >>
> >> With no more visible references to (the supposedly heap-allocated) n, 
> it can be freed.
> >>
> >>
> >>
> >> act := (*Num)(sliceDataPtr).numData
> >> if *act != 123456789 {
> >> t.Fatalf("v[%X]", *act)
> >> }
> >> }
> >>
> >>
> >> Elias
> >
> > --
> > You received this message because you are subscribed to the Google 
> Groups "golang-nuts" group.
> > To unsubscribe from this group and stop receiving emails from it, send 
> an email to golang-nuts...@googlegroups.com.
> > To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/5051da1c-36de-4699-906c-ff8f50754404n%40googlegroups.com
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/1d6e5233-cef4-4466-8fee-9359ca69938dn%40googlegroups.com.

Re: [go-nuts] Crash using gccheckmark=1, possible GC bug?

[Ian Lance Taylor]

On Fri, Aug 18, 2023 at 8:12 AM Tibor Halter  wrote:
>
> Hi Elias, thanks for chiming in!
>
> > The Go garbage collector is precise in that it only considers pointers when 
> > determining memory that can be freed.
>
> Agreed, but it is not as precise on _what_ is a pointer. Specifically, which 
> matters?
> - the type used upon allocating the memory, or
> - the type used upon writing to it.
>
> Because *T or unsafe.Pointer is required upon writing to a memory location 
> for it to be marked as a pointer, I came to the conclusion that the type upon 
> writing is what matters.
>
> So then both types need to be pointers?
>
> Does that mean there is no way to define a struct-like memory layout 
> dynamically, that contains a mix of pointer and non-pointer types, in a way 
> that is supported by the GC?
> (This is what I'm trying to do.)

The rule is pretty simple: a pointer must always have a pointer type.
That includes when it is in a variable and when it is stored in
memory.  In your terms, what matters is the type used upon allocating
the memory.

Your code is not following any of the patterns permitted for
unsafe.Pointer (any patterns that are not explicitly permitted are
forbidden).  The code is effectively converting from *byte to *Num.
As it says at https://pkg.go.dev/unsafe#Pointer that kind of
conversion is only permitted if the types in question "share an
equivalent memory layout".  byte and Num have different memory
layouts.

You can define a struct-like memory layout dynamically by using
reflect.StructOf.

Ian


> On Friday, August 18, 2023 at 3:49:03 PM UTC+2 ma...@eliasnaur.com wrote:
>>
>> On Friday, 18 August 2023 at 05:57:38 UTC-6 Tibor Halter wrote:
>>
>> Thanks Ian!
>>
>> I have managed to strip it down to its essence, see code below.
>>
>> Summary of what I'm doing
>> - reserve memory with make([]byte)
>> - get pointer to underlying array
>> - cast it to type Num
>> - write pointer #2 into it
>> - force GC()
>> - GC does not follow pointer #2, memory is freed
>>
>>
>> The Go garbage collector is precise in that it only considers pointers when 
>> determining memory that can be freed. In your scenario and demonstration 
>> program you explicitly store a pointer in a byte slice, effectively making 
>> it invisible to the collector.
>>
>>
>> --
>>
>>
>> package debug
>>
>> import (
>> "testing"
>> "runtime"
>> "unsafe"
>> )
>>
>> type Num struct {
>> numData *uint64
>> }
>>
>> func Test(t *testing.T) {
>> n := uint64(123456789)
>> numMemSize := 8
>> b := make([]byte, numMemSize, numMemSize)
>> sliceDataPtr := unsafe.Pointer(unsafe.SliceData(b))
>> (*Num)(sliceDataPtr).numData = &n
>>
>>
>> The pointer to &n stored in b here is effectively hidden from the collector.
>>
>>
>>
>> runtime.GC()
>>
>>
>> With no more visible references to (the supposedly heap-allocated) n, it can 
>> be freed.
>>
>>
>>
>> act := (*Num)(sliceDataPtr).numData
>> if *act != 123456789 {
>> t.Fatalf("v[%X]", *act)
>> }
>> }
>>
>>
>> Elias
>
> --
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/5051da1c-36de-4699-906c-ff8f50754404n%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAOyqgcXKkZdzVebd3U%3DBQ25scsyZC5g%2BKWNS2DFgFhmgHQkb-g%40mail.gmail.com.

[go-nuts] How to try out the new range func CL in a module

[Hein Meling]

Hi all,

I wanted to play around with the new range func CL 
.

Doing simple stuff works just fine, but if I need to import packages to 
construct my custom iterator func, the go/gotip command insists on a go.mod 
file, which effectively resets the go version to 1.21.0 (due to go.mod), 
instead of the "(w/ rangefunc)" CL.

Anyone know any workarounds for this?

Thanks,
:) Hein

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/f01ff376-b789-4d8a-89f5-165a6527325fn%40googlegroups.com.

[go-nuts] possible bug in slogtest or ReplaceAttr

[jal...@gmail.com]

Hi all,

In slog, I am experimenting with the the ReplaceAttr func in the 
HandlerOptions with json handler. My understanding is that if i want to 
remove an attribute then return an empty slog.Attr. However, logtest test 
fails with a json unmarshal error. Note I am literally using the code 
example from the official docs.

Is this expected or a  bug? Not sure what my user error is here.

https://go.dev/play/p/i7VtMtNK4eV

Thanks,
Joe

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/89810de6-6a02-440f-988b-9503845a8c90n%40googlegroups.com.

Re: [go-nuts] Crash using gccheckmark=1, possible GC bug?

[Tibor Halter]

For the record, allocating the memory as []unsafe.Pointer fixes the issues. 
(Working example below.)

That said, I'm not sure this is a future proof idea. It seems the GC 
handles it correctly now, but may change in the future.
I'd greatly prefer to find a spec-compliant solution...


package debug

import (
"testing"
"runtime"
"unsafe"
)

type Num struct {
val uint64
ptr *uint64
}
func Test(t *testing.T) {
var sliceDataPtr unsafe.Pointer
func() {
n := uint64(123456789)
numMemSize := 2
b := make([]unsafe.Pointer, numMemSize, numMemSize)

sliceDataPtr = unsafe.Pointer(unsafe.SliceData(b))
numVal := (*Num)(sliceDataPtr)
numVal.val = 123
numVal.ptr = &n

escape(sliceDataPtr) // force on heap
}()


runtime.GC()


numVal := (*Num)(sliceDataPtr)
if numVal.val != 123 {
t.Fatalf("v[%X]", numVal.val)
}
if *numVal.ptr != 123456789 {
t.Fatalf("v[%X]", *numVal.ptr)
}
}

var sink any
func escape(x any) {
sink = x
sink = nil
}


On Friday, August 18, 2023 at 5:12:18 PM UTC+2 Tibor Halter wrote:

> Hi Elias, thanks for chiming in!
>
> > The Go garbage collector is precise in that it only considers pointers 
> when determining memory that can be freed.
>
> Agreed, but it is not as precise on _what_ is a pointer. Specifically, 
> which matters?
> - the type used upon allocating the memory, or
> - the type used upon writing to it.
>
> Because *T or unsafe.Pointer is required upon writing to a memory location 
> for it to be marked as a pointer, I came to the conclusion that the type 
> upon writing is what matters.
>
> So then both types need to be pointers?
>
> Does that mean there is no way to define a struct-like memory layout 
> dynamically, that contains a mix of pointer and non-pointer types, in a way 
> that is supported by the GC?
> (This is what I'm trying to do.)
>
> On Friday, August 18, 2023 at 3:49:03 PM UTC+2 ma...@eliasnaur.com wrote:
>
>> On Friday, 18 August 2023 at 05:57:38 UTC-6 Tibor Halter wrote:
>>
>> Thanks Ian!
>>
>> I have managed to strip it down to its essence, see code below.
>>
>> *Summary of what I'm doing*
>> - reserve memory with make([]byte)
>> - get pointer to underlying array
>> - cast it to type Num
>> - write pointer #2 into it
>> - force GC()
>> - GC does not follow pointer #2, memory is freed
>>
>>
>> The Go garbage collector is precise in that it only considers pointers 
>> when determining memory that can be freed. In your scenario and 
>> demonstration program you explicitly store a pointer in a byte slice, 
>> effectively making it invisible to the collector.
>>  
>>
>> --
>>
>>
>> package debug
>>
>> import (
>> "testing"
>> "runtime"
>> "unsafe"
>> )
>>
>> type Num struct {
>> numData *uint64
>> }
>>
>> func Test(t *testing.T) {
>> n := uint64(123456789)
>> numMemSize := 8
>> b := make([]byte, numMemSize, numMemSize)
>> sliceDataPtr := unsafe.Pointer(unsafe.SliceData(b))
>> (*Num)(sliceDataPtr).numData = &n
>>
>>
>> The pointer to &n stored in b here is effectively hidden from the 
>> collector.
>>  
>>
>>
>> runtime.GC()
>>
>>
>> With no more visible references to (the supposedly heap-allocated) n, it 
>> can be freed.
>>  
>>
>>
>> act := (*Num)(sliceDataPtr).numData
>> if *act != 123456789 {
>> t.Fatalf("v[%X]", *act)
>> }
>> }
>>
>>
>> Elias 
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/11c27ff5-0e1b-42a7-9f00-6bda4f860f90n%40googlegroups.com.

Re: [go-nuts] Crash using gccheckmark=1, possible GC bug?

[Tibor Halter]

Hi Elias, thanks for chiming in!

> The Go garbage collector is precise in that it only considers pointers 
when determining memory that can be freed.

Agreed, but it is not as precise on _what_ is a pointer. Specifically, 
which matters?
- the type used upon allocating the memory, or
- the type used upon writing to it.

Because *T or unsafe.Pointer is required upon writing to a memory location 
for it to be marked as a pointer, I came to the conclusion that the type 
upon writing is what matters.

So then both types need to be pointers?

Does that mean there is no way to define a struct-like memory layout 
dynamically, that contains a mix of pointer and non-pointer types, in a way 
that is supported by the GC?
(This is what I'm trying to do.)

On Friday, August 18, 2023 at 3:49:03 PM UTC+2 ma...@eliasnaur.com wrote:

> On Friday, 18 August 2023 at 05:57:38 UTC-6 Tibor Halter wrote:
>
> Thanks Ian!
>
> I have managed to strip it down to its essence, see code below.
>
> *Summary of what I'm doing*
> - reserve memory with make([]byte)
> - get pointer to underlying array
> - cast it to type Num
> - write pointer #2 into it
> - force GC()
> - GC does not follow pointer #2, memory is freed
>
>
> The Go garbage collector is precise in that it only considers pointers 
> when determining memory that can be freed. In your scenario and 
> demonstration program you explicitly store a pointer in a byte slice, 
> effectively making it invisible to the collector.
>  
>
> --
>
>
> package debug
>
> import (
> "testing"
> "runtime"
> "unsafe"
> )
>
> type Num struct {
> numData *uint64
> }
>
> func Test(t *testing.T) {
> n := uint64(123456789)
> numMemSize := 8
> b := make([]byte, numMemSize, numMemSize)
> sliceDataPtr := unsafe.Pointer(unsafe.SliceData(b))
> (*Num)(sliceDataPtr).numData = &n
>
>
> The pointer to &n stored in b here is effectively hidden from the 
> collector.
>  
>
>
> runtime.GC()
>
>
> With no more visible references to (the supposedly heap-allocated) n, it 
> can be freed.
>  
>
>
> act := (*Num)(sliceDataPtr).numData
> if *act != 123456789 {
> t.Fatalf("v[%X]", *act)
> }
> }
>
>
> Elias 
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/5051da1c-36de-4699-906c-ff8f50754404n%40googlegroups.com.

Re: [go-nuts] Crash using gccheckmark=1, possible GC bug?

[ma...@eliasnaur.com]

On Friday, 18 August 2023 at 05:57:38 UTC-6 Tibor Halter wrote:

Thanks Ian!

I have managed to strip it down to its essence, see code below.

*Summary of what I'm doing*
- reserve memory with make([]byte)
- get pointer to underlying array
- cast it to type Num
- write pointer #2 into it
- force GC()
- GC does not follow pointer #2, memory is freed


The Go garbage collector is precise in that it only considers pointers when 
determining memory that can be freed. In your scenario and demonstration 
program you explicitly store a pointer in a byte slice, effectively making 
it invisible to the collector.
 

--


package debug

import (
"testing"
"runtime"
"unsafe"
)

type Num struct {
numData *uint64
}

func Test(t *testing.T) {
n := uint64(123456789)
numMemSize := 8
b := make([]byte, numMemSize, numMemSize)
sliceDataPtr := unsafe.Pointer(unsafe.SliceData(b))
(*Num)(sliceDataPtr).numData = &n


The pointer to &n stored in b here is effectively hidden from the collector.
 


runtime.GC()


With no more visible references to (the supposedly heap-allocated) n, it 
can be freed.
 


act := (*Num)(sliceDataPtr).numData
if *act != 123456789 {
t.Fatalf("v[%X]", *act)
}
}


Elias 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/f299fa55-9046-45bd-ba5c-0904bd6c3e0an%40googlegroups.com.

Re: [go-nuts] Crash using gccheckmark=1, possible GC bug?

[Tibor Halter]

I am testing on Go 1.21.0, Linux, amd64


On Friday, August 18, 2023 at 1:57:38 PM UTC+2 Tibor Halter wrote:

> Thanks Ian!
>
> I have managed to strip it down to its essence, see code below.
>
> *Summary of what I'm doing*
> - reserve memory with make([]byte)
> - get pointer to underlying array
> - cast it to type Num
> - write pointer #2 into it
> - force GC()
> - GC does not follow pointer #2, memory is freed
>
> *Notes*
> - If I run it with `go test` it passes.
> - If I run it with `GODEBUG=clobberfree=1 go test` it fails with:
>   --- FAIL: Test (0.00s)
>   debug_test.go:26: v[DEADBEEFDEADBEEF]
> - If I inline the variable `numMemSize` it passes.
>
>
> As per the rules in unsafe.Pointer docs I would understand the conversions 
> I'm doing are valid.
> What do I miss?
>
> Thanks a lot,
> Tibor
>
>
> --
>
>
> package debug
>
> import (
> "testing"
> "runtime"
> "unsafe"
> )
>
> type Num struct {
> numData *uint64
> }
>
> func Test(t *testing.T) {
> n := uint64(123456789)
> numMemSize := 8
> b := make([]byte, numMemSize, numMemSize)
> sliceDataPtr := unsafe.Pointer(unsafe.SliceData(b))
> (*Num)(sliceDataPtr).numData = &n
>
>
> runtime.GC()
>
>
> act := (*Num)(sliceDataPtr).numData
> if *act != 123456789 {
> t.Fatalf("v[%X]", *act)
> }
> }
>
>
> On Friday, August 18, 2023 at 2:59:46 AM UTC+2 Ian Lance Taylor wrote:
>
>> On Thu, Aug 17, 2023 at 5:41 PM Tibor Halter  wrote: 
>> > 
>> > I get this crash when using the GODEBUG option gccheckmark=1: 
>> > 
>> > runtime: marking free object 0xc0016c5668 found at *(0xc0046ca8e0+0x8) 
>> > base=0xc0046ca8e0 s.base()=0xc0046ca000 s.limit=0xc0046cc000 
>> s.spanclass=8 s.elemsize=32 s.state=mSpanInUse 
>> > *(base+0) = 0x15e11f0 
>> > *(base+8) = 0xc0016c5668 <== 
>> > *(base+16) = 0x28 
>> > *(base+24) = 0xc004642030 
>> > obj=0xc0016c5668 s.base()=0xc0016c4000 s.limit=0xc0016c5ff8 
>> s.spanclass=6 s.elemsize=24 s.state=mSpanInUse 
>> > *(obj+0) = 0x15e0e80 
>> > *(obj+8) = 0xc00039c080 
>> > *(obj+16) = 0xc0016c5650 
>> > fatal error: marking free object 
>> > 
>> > runtime stack: 
>> > runtime.throw({0x132362a?, 0x3?}) 
>> > /usr/local/go-1.21.0/src/runtime/panic.go:1077 +0x5c fp=0x7f9636366d48 
>> sp=0x7f9636366d18 pc=0x43aa9c 
>> > runtime.greyobject(0xc0016c5668, 0x1?, 0x7f9636366df8?, 0x6590?, 
>> 0x7f9636366df8?, 0x7f9636366de0?) 
>> > /usr/local/go-1.21.0/src/runtime/mgcmark.go:1476 +0x285 
>> fp=0x7f9636366d98 sp=0x7f9636366d48 pc=0x424d85 
>> > runtime.scanobject(0xc0046ca8e0, 0xc4d240) 
>> > /usr/local/go-1.21.0/src/runtime/mgcmark.go:1336 +0x171 
>> fp=0x7f9636366e20 sp=0x7f9636366d98 pc=0x424711 
>> > runtime.gcDrain(0xc4d240, 0x7) 
>> > /usr/local/go-1.21.0/src/runtime/mgcmark.go:1103 +0x1ba 
>> fp=0x7f9636366e80 sp=0x7f9636366e20 pc=0x423fba 
>> > runtime.gcBgMarkWorker.func2() 
>> > /usr/local/go-1.21.0/src/runtime/mgc.go:1385 +0x6f fp=0x7f9636366ed0 
>> sp=0x7f9636366e80 pc=0x4208af 
>> > runtime.systemstack() 
>> > /usr/local/go-1.21.0/src/runtime/asm_amd64.s:509 +0x4a 
>> fp=0x7f9636366ee0 sp=0x7f9636366ed0 pc=0x46b1aa 
>> > 
>> > 
>> > From the docs, I understand this 2nd pass stop-the-world verification 
>> > guarantees the concurrent mark step did not follow the pointer, aka 
>> this is a GC bug? 
>> > 
>> > A confirmation would be greatly appreciated to help me follow the right 
>> trail. 
>> > 
>> > Context 
>> > I'm building a type-system on top of Go for user-defined data 
>> structures. For efficient implementation, I'm using unsafe.Pointer-s 
>> heavily. I believe I understand pointer conversion rules, span classes, 
>> that there is metadata on the side to differentiate ordinary values from 
>> pointers and how it is used to follow pointer chains. 
>> > Go vet detects no issues. I've been debugging this for 4 days. 
>>
>> That error looks like the GC found a pointer to an object that was 
>> already freed. This can happen, for example, if you convert a uintptr 
>> to a pointer type, but the object to which the pointer points has been 
>> freed. While it could be a GC bug, that would not be my first guess. 
>> The contents of the block shown above are intended to help you 
>> identify where the pointer came from. 
>>
>> Ian 
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/5d0be612-39ec-48dd-a567-a8eb0172cfb1n%40googlegroups.com.

Re: [go-nuts] Crash using gccheckmark=1, possible GC bug?

[Tibor Halter]

Thanks Ian!

I have managed to strip it down to its essence, see code below.

*Summary of what I'm doing*
- reserve memory with make([]byte)
- get pointer to underlying array
- cast it to type Num
- write pointer #2 into it
- force GC()
- GC does not follow pointer #2, memory is freed

*Notes*
- If I run it with `go test` it passes.
- If I run it with `GODEBUG=clobberfree=1 go test` it fails with:
  --- FAIL: Test (0.00s)
  debug_test.go:26: v[DEADBEEFDEADBEEF]
- If I inline the variable `numMemSize` it passes.


As per the rules in unsafe.Pointer docs I would understand the conversions 
I'm doing are valid.
What do I miss?

Thanks a lot,
Tibor


--


package debug

import (
"testing"
"runtime"
"unsafe"
)

type Num struct {
numData *uint64
}

func Test(t *testing.T) {
n := uint64(123456789)
numMemSize := 8
b := make([]byte, numMemSize, numMemSize)
sliceDataPtr := unsafe.Pointer(unsafe.SliceData(b))
(*Num)(sliceDataPtr).numData = &n


runtime.GC()


act := (*Num)(sliceDataPtr).numData
if *act != 123456789 {
t.Fatalf("v[%X]", *act)
}
}


On Friday, August 18, 2023 at 2:59:46 AM UTC+2 Ian Lance Taylor wrote:

> On Thu, Aug 17, 2023 at 5:41 PM Tibor Halter  wrote:
> >
> > I get this crash when using the GODEBUG option gccheckmark=1:
> >
> > runtime: marking free object 0xc0016c5668 found at *(0xc0046ca8e0+0x8)
> > base=0xc0046ca8e0 s.base()=0xc0046ca000 s.limit=0xc0046cc000 
> s.spanclass=8 s.elemsize=32 s.state=mSpanInUse
> > *(base+0) = 0x15e11f0
> > *(base+8) = 0xc0016c5668 <==
> > *(base+16) = 0x28
> > *(base+24) = 0xc004642030
> > obj=0xc0016c5668 s.base()=0xc0016c4000 s.limit=0xc0016c5ff8 
> s.spanclass=6 s.elemsize=24 s.state=mSpanInUse
> > *(obj+0) = 0x15e0e80
> > *(obj+8) = 0xc00039c080
> > *(obj+16) = 0xc0016c5650
> > fatal error: marking free object
> >
> > runtime stack:
> > runtime.throw({0x132362a?, 0x3?})
> > /usr/local/go-1.21.0/src/runtime/panic.go:1077 +0x5c fp=0x7f9636366d48 
> sp=0x7f9636366d18 pc=0x43aa9c
> > runtime.greyobject(0xc0016c5668, 0x1?, 0x7f9636366df8?, 0x6590?, 
> 0x7f9636366df8?, 0x7f9636366de0?)
> > /usr/local/go-1.21.0/src/runtime/mgcmark.go:1476 +0x285 
> fp=0x7f9636366d98 sp=0x7f9636366d48 pc=0x424d85
> > runtime.scanobject(0xc0046ca8e0, 0xc4d240)
> > /usr/local/go-1.21.0/src/runtime/mgcmark.go:1336 +0x171 
> fp=0x7f9636366e20 sp=0x7f9636366d98 pc=0x424711
> > runtime.gcDrain(0xc4d240, 0x7)
> > /usr/local/go-1.21.0/src/runtime/mgcmark.go:1103 +0x1ba 
> fp=0x7f9636366e80 sp=0x7f9636366e20 pc=0x423fba
> > runtime.gcBgMarkWorker.func2()
> > /usr/local/go-1.21.0/src/runtime/mgc.go:1385 +0x6f fp=0x7f9636366ed0 
> sp=0x7f9636366e80 pc=0x4208af
> > runtime.systemstack()
> > /usr/local/go-1.21.0/src/runtime/asm_amd64.s:509 +0x4a fp=0x7f9636366ee0 
> sp=0x7f9636366ed0 pc=0x46b1aa
> >
> >
> > From the docs, I understand this 2nd pass stop-the-world verification
> > guarantees the concurrent mark step did not follow the pointer, aka this 
> is a GC bug?
> >
> > A confirmation would be greatly appreciated to help me follow the right 
> trail.
> >
> > Context
> > I'm building a type-system on top of Go for user-defined data 
> structures. For efficient implementation, I'm using unsafe.Pointer-s 
> heavily. I believe I understand pointer conversion rules, span classes, 
> that there is metadata on the side to differentiate ordinary values from 
> pointers and how it is used to follow pointer chains.
> > Go vet detects no issues. I've been debugging this for 4 days.
>
> That error looks like the GC found a pointer to an object that was
> already freed. This can happen, for example, if you convert a uintptr
> to a pointer type, but the object to which the pointer points has been
> freed. While it could be a GC bug, that would not be my first guess.
> The contents of the block shown above are intended to help you
> identify where the pointer came from.
>
> Ian
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/90e4be23-0f6b-4a34-8424-89535d74bb22n%40googlegroups.com.

Re: [go-nuts] Why is foo in my go.sum file?

[TheDiveO]

test case?

On Friday, August 18, 2023 at 9:18:05 AM UTC+2 Dan Kortschak wrote:

> On Fri, 2023-08-18 at 16:49 +1000, Nigel Tao wrote:
> > The go.sum file in the golang.org/x/image repo has a line that is not
> > another golang.org.x/* module:
> > 
> > github.com/yuin/goldmark v1.4.13/go.mod
> > h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
> > 
> > 
> https://github.com/golang/image/blob/2b687b56714d59d061135c735913a64fe2b70778/go.sum#L1
> > 
> > That line was added in a go...@golang.org authored commit last
> > October:
> > 
> https://github.com/golang/image/commit/ffcb3fe7d1bf4ed2e01a95a552bb3b7f5dab24d1
> > 
> > I'm just curious why goldmark is considered a dependency. "go mod
> > why"
> > and plain old "grep" doesn't give me any leads.
> > 
> > ---
> > $ git checkout ffcb3fe7d1bf4ed2e01a95a552bb3b7f5dab24d1
> > HEAD is now at ffcb3fe go.mod: update golang.org/x dependencies
> > 
> > $ go mod why github.com/yuin/goldmark
> > # github.com/yuin/goldmark
> > (main module does not need package github.com/yuin/goldmark)
> > 
> > $ grep goldmark -R .
> > ./go.sum:github.com/yuin/goldmark v1.4.13/go.mod
> > h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
> > ---
> > 
> > Am I holding "go mod why" wrong? What else can I try?
> > 
>
> My guess is the go.mod in x/tools.
>
> https://github.com/golang/tools/blob/master/go.mod
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/cc9f4f90-1b39-436f-99eb-4dfcf8dad6ddn%40googlegroups.com.

[go-nuts] Re: Test coverage: joint unit test and integration test coverage

[TheDiveO]

The only "documentation" is this issue comment 
https://github.com/golang/go/issues/51430#issuecomment-1344711300 and the 
reply to my question about the "official 
status" https://github.com/golang/go/issues/51430#issuecomment-1485320781

On Thursday, August 17, 2023 at 9:51:12 AM UTC+2 Jan wrote:

> I tried to use the `-test.gocoverdir` (*) and while the test(s) being 
> executed use the given directory, unfortunately it still sets `GOCOVERDIR` 
> to some newly created temporary directory. 
>
> Since my integration tests are executed from a *_test.go  
> test
>  
> (it's a test after all), in the end the `-test.gocoverdir` flag contents 
> are discarded (or not passed along in GOCOVERDIR). If the flag is set, 
> shouldn't it be used for `GOCOVERDIR` as well ?
>
> cheers
>
> (*) Btw, I can't find documentation on -test.gocoverdir in Google 
> ,
>  
> except the insides of code. Also `go help testflag | grep gocoverdir` 
> doesn't return anything -- in go1.21.0.
>
> On Thursday, August 17, 2023 at 8:08:51 AM UTC+2 Jan wrote:
>
>> Oh yes, it's true that the new mechanism is more general, and allows 
>> integration tests, it's a huge improvement for cases like this! Thanks for 
>> the design btw!
>>
>> Now the ergonomics of the case of unit tests + integration tests could be 
>> made easier/more ergonomic -- I would assume it is the common case for 
>> projects with integration tests (because everyone has also unit tests). And 
>> a suggestion of maybe a mention / an example in the 
>> integration-test-coverage 
>> blog  ?
>>
>> On Wednesday, August 16, 2023 at 10:03:55 PM UTC+2 TheDiveO wrote:
>>
>>> Hmm, my previous response got deleted for no reason. So here we go: the 
>>> new mechanism is more general, and as you can see in my example I actually 
>>> run the "same" unit tests twice. The reason is because some code paths 
>>> might be only exercised when not being run as root, especially error 
>>> handling. So there is some value in the new mechanism even "just" for unit 
>>> tests. Admittedly, not all will need that in their unit tests.
>>>
>>> On Wednesday, August 16, 2023 at 8:53:56 PM UTC+2 Jan wrote:
>>>
 Thanks, that's very helpful! 

 I was doing something similar 
 ,
  
 but I was extracting the temporary coverage directory created by Go 
 because 
 I didn't know about the --test.gocoverdir flag. 

 Still it feels wrong one has to do the trick of creating a temporary 
 directory (or extract it from the go tool) , where for unit tests only 
 it's 
 not needed ... I wonder if this is done deliberately, or just a bug ?

 cheers

 On Wednesday, August 16, 2023 at 8:33:23 PM UTC+2 TheDiveO wrote:

> Maybe similar to this? 
> https://github.com/thediveo/lxkns/blob/cef5a31d7517cb126378f81628f51672cb793527/scripts/cov.sh#L28
>
> On Wednesday, August 16, 2023 at 1:54:48 PM UTC+2 Jan wrote:
>
>> hi all, 
>>
>> After reading the documentation 
>> , I managed to set up 
>> a process in which I compile and run my integration tests, and get 
>> coverage 
>> from them in the `$GOCOVERDIR` subdirectory.
>>
>> Now I would like to have a combined unit tests + integration tests 
>> report, all in one go, and I'm not sure how to get that.
>>
>> I was expecting that, if I go to my modules root directory, and I do:
>>
>> ```
>> go test --cover --coverpkg=./... 
>> --coverprofile=/tmp/cover_profile.out ./...
>> go tool cover -func /tmp/cover_profile.out > /tmp/cover_func.out
>> ```
>>
>> I  would get all the results, including integration tests (since they 
>> are called with GOCOVERDIR set). But instead I only see the coverage of 
>> the 
>> unit tests, and the information from the integration tests seems to be 
>> ignored, even though it is generated.
>>
>> I'm sure it is generated because if I run the command above with 
>> `--work` (preserve the temporary files), and log the value of 
>> $GOCOVERDIR I 
>> can see where the test stores the coverage data files. And if I manually 
>> do:
>>
>> ```
>> go tool covdata func -i /tmp/go-build287472875/b001/gocoverdir
>> ```
>> (where /tmp/go-build/gocoverdir is the temporary directory 
>> reported for GOCOVERDIR)
>>
>> I see the results I expected (some of my functions that I want to 
>> make sure are covered) are there. But they are not reported in 
>> `/tmp/cover_func.out` above.
>>

Re: [go-nuts] weird "index out of range" in strconv.formatBits

['Dan Kortschak' via golang-nuts]

On Thu, 2023-08-17 at 23:32 -0700, metronome wrote:
> > > Have you built with CGO_ENABLED=0?
> Building with CGO_ENABLED=0 succeeded, does that mean the binary's
> runtime behavior has nothing to do with CGO, deploying
> a CGO_ENABLED=0 binary online is not an option as well, for now (We
> are trying, but not sure if we can make it happen).

Do you get the same behaviour with CGO_ENABLED=0?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/35aff4ec8fd97582dcfec804f1d8714c4e55c26c.camel%40kortschak.io.

Re: [go-nuts] Why is foo in my go.sum file?

['Dan Kortschak' via golang-nuts]

On Fri, 2023-08-18 at 16:49 +1000, Nigel Tao wrote:
> The go.sum file in the golang.org/x/image repo has a line that is not
> another golang.org.x/* module:
> 
> github.com/yuin/goldmark v1.4.13/go.mod
> h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
> 
> https://github.com/golang/image/blob/2b687b56714d59d061135c735913a64fe2b70778/go.sum#L1
> 
> That line was added in a go...@golang.org authored commit last
> October:
> https://github.com/golang/image/commit/ffcb3fe7d1bf4ed2e01a95a552bb3b7f5dab24d1
> 
> I'm just curious why goldmark is considered a dependency. "go mod
> why"
> and plain old "grep" doesn't give me any leads.
> 
> ---
> $ git checkout ffcb3fe7d1bf4ed2e01a95a552bb3b7f5dab24d1
> HEAD is now at ffcb3fe go.mod: update golang.org/x dependencies
> 
> $ go mod why github.com/yuin/goldmark
> # github.com/yuin/goldmark
> (main module does not need package github.com/yuin/goldmark)
> 
> $ grep goldmark -R .
> ./go.sum:github.com/yuin/goldmark v1.4.13/go.mod
> h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
> ---
> 
> Am I holding "go mod why" wrong? What else can I try?
> 

My guess is the go.mod in x/tools.

https://github.com/golang/tools/blob/master/go.mod

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/0b42133f457f308f976203fcdcf9b4042c513d49.camel%40kortschak.io.