[go-nuts] IOPS profiling

[Ethan Reesor]

I'm seeing some pretty strong indications that my application (which is 
running on AWS EC2) is bottlenecked by IOPS (I/O operations per second).

Is there anything like `go test -profile=...` and `go tool pprof ...` for 
IOPS? We're looking at optimizing the database we're using, but I'd much 
prefer to capture real world data so we're not optimizing blindly.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/eaf9b1ff-e553-4544-9da5-e7969d2fee98n%40googlegroups.com.

Re: [go-nuts] Nil pointer panic that should be impossible

[Ethan Reesor]

I hadn't used the race detector before. I do see a race warning for
(*URL).String() among an embarrassing number of other results. I'm going to
update (*URL).String() to use atomic.Pointer to remove the race.

Thanks,
Ethan

On Thu, Mar 21, 2024 at 8:59 AM 'Axel Wagner' via golang-nuts <
golang-nuts@googlegroups.com> wrote:

> On Thu, Mar 21, 2024 at 2:48 PM 王李荣  wrote:
>
>> hi Axel,
>>
>> is not modifying `u.memoize.str` thread-safe?  the len and the data
>> point should become visible at same time?
>>
>
> What makes you think that? To be clear, there are no benign data races.
> Even a data-race on a variable smaller than a word is still a data-race,
> unless you do it holding a lock or using atomic instructions. But strings
> are *larger* than single words.
>
> To demonstrate that the effect I am talking about is real, look at this
> code: https://go.dev/play/p/LzRq9-OH-Xb
>
>
>>
>> 在2024年3月16日星期六 UTC+8 06:29:06 写道：
>>
>>> Have you tried running the code with the race detector enabled? I
>>> suspect that you are concurrently modifying `u.memoize.str` by calling
>>> `u.String()` from multiple goroutines. And the non-zero length of the
>>> string header written by one goroutine becomes visible to the other one,
>>> before the modification to the data pointer.
>>>
>>> On Fri, Mar 15, 2024 at 11:15 PM Ethan Reesor 
>>> wrote:
>>>
>>>> From this CI job
>>>> <https://gitlab.com/accumulatenetwork/accumulate/-/jobs/6398114923>:
>>>>
>>>> panic: runtime error: invalid memory address or nil pointer dereference
>>>> [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x51d8b7]
>>>> goroutine 1589381 [running]:
>>>> strings.EqualFold({0xc000beec20?, 0x0?}, {0x0?, 0xacace7?})
>>>>  /usr/local/go/src/strings/strings.go: +0x37
>>>>
>>>> gitlab.com/accumulatenetwork/accumulate/pkg/url.(*URL).Equal(0xc000a74e40?,
>>>> 0xc00094c540)
>>>>  /builds/accumulatenetwork/accumulate/pkg/url/url.go:472 +0x10c
>>>>
>>>> This is in a docker container based on the go:1.22 image, so the panic
>>>> appears to be happening here:
>>>>
>>>> func EqualFold(s, t string) bool {
>>>> // ASCII fast path
>>>> i := 0
>>>> for ; i < len(s) && i < len(t); i++ {
>>>> sr := s[i]
>>>> tr := t[i] // <-- line 
>>>>
>>>> (*URL).Equal
>>>> <https://gitlab.com/accumulatenetwork/accumulate/-/blob/5b1cb612d76d4163a101303e51a6fd352224cdab/pkg/url/url.go#L465>
>>>> :
>>>>
>>>> func (u *URL) Equal(v *URL) bool {
>>>> if u == v {
>>>> return true
>>>> }
>>>> if u == nil || v == nil {
>>>> return false
>>>> }
>>>> return strings.EqualFold(u.String(), v.String())
>>>> }
>>>>
>>>> (*URL).String
>>>> <https://gitlab.com/accumulatenetwork/accumulate/-/blob/5b1cb612d76d4163a101303e51a6fd352224cdab/pkg/url/url.go#L240>
>>>> :
>>>>
>>>> func (u *URL) String() string {
>>>> if u.memoize.str != "" {
>>>> return u.memoize.str
>>>> }
>>>>
>>>> u.memoize.str = u.format(nil, true)
>>>> return u.memoize.str
>>>> }
>>>>
>>>> (*URL).format
>>>> <https://gitlab.com/accumulatenetwork/accumulate/-/blob/5b1cb612d76d4163a101303e51a6fd352224cdab/pkg/url/url.go#L189>
>>>> :
>>>>
>>>> func (u *URL) format(txid []byte, encode bool) string {
>>>> var buf strings.Builder
>>>> // ... write to the builder
>>>> return buf.String()
>>>> }
>>>>
>>>> How is this possible? Based on `addr=0x0` in the panic I think this is
>>>> a nil pointer panic, as opposed to some other kind of segfault. The only
>>>> way I can reproduce panic-on-string-index is with 
>>>> `(*reflect.StringHeader)(unsafe.Pointer(&s)).Data
>>>> = 0`, but I don't see how that can be happening here. I'm saving the string
>>>> but I'm not doing anything weird with it. And the string header is a value
>>>> type so code that manipulates the returned string shouldn't modify the
>>>> original. And I'm definitely not doing any kind of unsafe string
>>>> manipulation like that in my code, anywhere. The only reference to unsafe
>

[go-nuts] Nil pointer panic that should be impossible

[Ethan Reesor]

>From this CI job 
:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x51d8b7]
goroutine 1589381 [running]:
strings.EqualFold({0xc000beec20?, 0x0?}, {0x0?, 0xacace7?})
 /usr/local/go/src/strings/strings.go: +0x37
gitlab.com/accumulatenetwork/accumulate/pkg/url.(*URL).Equal(0xc000a74e40?, 
0xc00094c540)
 /builds/accumulatenetwork/accumulate/pkg/url/url.go:472 +0x10c

This is in a docker container based on the go:1.22 image, so the panic 
appears to be happening here:

func EqualFold(s, t string) bool {
// ASCII fast path
i := 0
for ; i < len(s) && i < len(t); i++ {
sr := s[i]
tr := t[i] // <-- line 

(*URL).Equal 

:

func (u *URL) Equal(v *URL) bool {
if u == v {
return true
}
if u == nil || v == nil {
return false
}
return strings.EqualFold(u.String(), v.String())
}

(*URL).String 

:

func (u *URL) String() string {
if u.memoize.str != "" {
return u.memoize.str
}

u.memoize.str = u.format(nil, true)
return u.memoize.str
}

(*URL).format 

:

func (u *URL) format(txid []byte, encode bool) string {
var buf strings.Builder
// ... write to the builder
return buf.String()
}

How is this possible? Based on `addr=0x0` in the panic I think this is a 
nil pointer panic, as opposed to some other kind of segfault. The only way 
I can reproduce panic-on-string-index is with 
`(*reflect.StringHeader)(unsafe.Pointer(&s)).Data 
= 0`, but I don't see how that can be happening here. I'm saving the string 
but I'm not doing anything weird with it. And the string header is a value 
type so code that manipulates the returned string shouldn't modify the 
original. And I'm definitely not doing any kind of unsafe string 
manipulation like that in my code, anywhere. The only reference to unsafe 
anywhere in my code is for parameters for calling GetDiskFreeSpaceExW 
(Windows kernel32.dll call).

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/d6f6bb75-45e9-4a38-9bbd-d332e7f3e57cn%40googlegroups.com.