[go-nuts] Re: How to Optimize A Golang Application Which Has More Than 100 Million Objects at Peak?

[Sen Han]

Thanks a lot. You are so kind and nice :-D

I tried golang's heap profile and it is very very helpful. By the way, what 
do you think about this article:

https://dgraph.io/blog/post/manual-memory-management-golang-jemalloc/



在2021年7月18日星期日 UTC+8 下午11:27:38 写道：

> Two other replies have mentioned Sync.Pool. I agree that Sync.Pool is a 
> valuable tool. 
>
> However, for the benefit of any beginning gophers who may read this 
> thread, I wanted to point out that, in a situation like yours, I would want 
> to try to reduce heap allocation in other ways first. Not that Sync.pool is 
> a last resort exactly, but it does have non-trivial overhead, and is not a 
> substitute for thinking about other ways to clean up heap allocations. For 
> example, pulling allocations out of loops, or manual object re-use where it 
> fits naturally into the code.
>
> On Friday, July 16, 2021 at 8:27:03 AM UTC-4 rmfr wrote:
>
>> I run it at an 8 cores 16GB machine and it occupies all cpu cores it 
>> could.
>>
>> 1. It is ~95% cpu intensive and with ~5% network communications.
>>
>> 2. The codebase is huge and has more than 300 thousands of lines of code 
>> (even didn't count the third party library yet).
>>
>> 3. The tool pprof tells nearly 50% percent of the time is spending on the 
>> runtime, something related to gc, mallocgc, memclrNoHeapPointers, and so on.
>>
>> 4. It has ~100 million dynamic objects.
>>
>> Do you guys have some good advice to optimize the performance?
>>
>> One idea that occurs to me is to do something like sync.Pool to buffer 
>> some most frequently allocated and freed objects. But the problem is I 
>> didn't manage to find a golang tool to find such objects. The runtime 
>> provides api to get the amount of objects but it didn't provide api to get 
>> the detailed statistics of all objects. Please correct me if I'm wrong. 
>> Thanks a lot :-)
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/e8c0bec6-54aa-4b4a-9ea9-280f9e8ecc59n%40googlegroups.com.

Re: [go-nuts] Re: `ltrace` yields "Couldn't find .dynsym or .dynstr in "/proc/*/exe" with binaries generated by "go build *.go"

[Sen Han]

Which in this case, the `strace` cannot trace a single syscall that
involves the time.

On Mon, Sep 18, 2017 at 7:08 AM, Sen Han <yunthana...@gmail.com> wrote:

> In `go1.9 @ linux 3.10.0-327.el7.x86_64`, the call __vdso_clock_gettime_sym
> is
> completely in userspace, and does not involve any syscall in the kernel,
> and that is
> the reason __vdso_clock_gettime_sym with CLOCK_MONOTONIC is so fast. ( In
> C,
> the perf of __vdso_clock_gettime_sym is about 14,000,000 op/s, it drops
> to nearly
> 7,000,000 op/s in golang's implementation. )
>
> On Mon, Sep 18, 2017 at 5:28 AM, David Collier-Brown <davecb...@gmail.com>
> wrote:
>
>> You may need to look and see what __vdso_clock_gettime_sym calls at the
>> system-call level, and compare that with the system calls reported by
>> strace from your executable.
>>
>> I assume you want some particular thing that __vdso_clock_gettime_sym
>> does: try asking the folks about *that*, as they may already know.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to golang-nuts+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: `ltrace` yields "Couldn't find .dynsym or .dynstr in "/proc/*/exe" with binaries generated by "go build *.go"

[Sen Han]

In `go1.9 @ linux 3.10.0-327.el7.x86_64`, the call __vdso_clock_gettime_sym
is
completely in userspace, and does not involve any syscall in the kernel,
and that is
the reason __vdso_clock_gettime_sym with CLOCK_MONOTONIC is so fast. ( In
C,
the perf of __vdso_clock_gettime_sym is about 14,000,000 op/s, it drops to
nearly
7,000,000 op/s in golang's implementation. )

On Mon, Sep 18, 2017 at 5:28 AM, David Collier-Brown 
wrote:

> You may need to look and see what __vdso_clock_gettime_sym calls at the
> system-call level, and compare that with the system calls reported by
> strace from your executable.
>
> I assume you want some particular thing that __vdso_clock_gettime_sym
> does: try asking the folks about *that*, as they may already know.
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Latency profiling

[Sen Han]

(Sorry for the noise.)

Hi Juliusz, this tool go-tool-trace-greediest-goroutines
would be
helpful.(
may also refer the issue 20792 )

All best

-- Sen

On Sun, Sep 17, 2017 at 9:12 PM, Juliusz Chroboczek  wrote:

> Hi,
>
> I've got a CPU-bound function that is not called very frequently, and
> hence doesn't appear prominently in the CPU profile, but I'm guessing
> that it takes a significant time to run (on the order of tens of
> milliseconds), and hence increases the latency of the calling goroutine.
>
> I don't see anything in pprof to give the average and maximum time spent
> in a given function, let alone to get the list of functions that have
> the highest maximum CPU time per invocation.  Not a big deal, I'm going
> to instrument the function manually, but I'm wondering if I'm missing
> something.
>
> Thanks,
>
> -- Juliusz
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] `ltrace` yields "Couldn't find .dynsym or .dynstr in "/proc/*/exe" with binaries generated by "go build *.go"

[Sen Han]

Thanks for your information. I want to use `ltrace` to find out if the
golang binary is using
the __vdso_clock_gettime_sym with CLOCK_MONOTONIC option for measuring time
interval.

But it finally turns out currently the `ltrace` cannot works very well with
go built binaries on
vdso tracing.

Is there some exist tools could trace the vdso call in golang? Or is there
any recommend methods
could achieve it on the runtime?

Thanks.

PS:

With your information then I tried again:

$ cat hi.go
package main

import (
"fmt"
"time"
)

func main() {
fmt.Println("hi!")
start := *time.Now()*
t := time.Now()
fmt.Println(t.Sub(start))
}
$ CGO_ENABLED=0 go build -a -x -*linkshared* hi.go
...

$ *ldd hi*
linux-vdso.so.1 =>  (0x7ffe96de7000)
libc.so.6 => /lib64/libc.so.6 (0x7f721e492000)
/lib64/ld-linux-x86-64.so.2 (0x7f721e85f000)

$ ltrace -s 1000 -tttTf ./hi
[pid 2305] 1505506857.418373 __libc_start_main(0x4bbcb0, 1, 0x7fff8d799738,
0x51a270hi!
540ns
 
[pid 2306] 1505506857.440419 +++ exited (status 0) +++
[pid 2307] 1505506857.440451 +++ exited (status 0) +++
[pid 2308] 1505506857.440946 +++ exited (status 0) +++
[pid 2305] 1505506857.440972 +++ exited (status 0) +++

$ *readelf -r hi*

Relocation section '.rela.dyn' at offset 0x65650 contains 2 entries:
  Offset  Info   Type   Sym. ValueSym. Name +
Addend
007e1fe8  00010006 R_X86_64_GLOB_DAT 
__gmon_start__ + 0
007e1ff0  115e0012 R_X86_64_TPOFF64   runtime.tlsg
+ 0

Relocation section '.rela.plt' at offset 0x65680 contains 2 entries:
  Offset  Info   Type   Sym. ValueSym. Name +
Addend
007e1fd8  00010007 R_X86_64_JUMP_SLO 
__gmon_start__ + 0
007e1fe0  00030007 R_X86_64_JUMP_SLO 
*__libc_start_main
+ 0*




On Fri, Sep 15, 2017 at 9:48 PM, Konstantin Khomoutov 
wrote:

> On Fri, Sep 15, 2017 at 04:09:24AM -0700, nehs...@gmail.com wrote:
>
> > Hi, I just found `ltrace` would yield "Couldn't find .dynsym or .dynstr
> in
> > "/proc/*/exe" with executable binary generated by "go build *.go".
> >
> > Is there any options in `go build` could solve this problem?
> >
> > Thanks a lot.
> >
> > PS:
> > $go version
> > go version go1.9 linux/amd64
> > $ cat hi.go
> > package main
> >
> >
> > import "fmt"
> >
> >
> > func main() {
> >  fmt.Println("hi!")
> > }
> > $ go build hi.go
> > $ ltrace ./hi
> > Couldn't find .dynsym or .dynstr in "/proc/`*`/exe"
>
> From the description of the ltrace's Debian package, I gather that
>
> | ltrace is a debugging program which runs a specified command until it
> | exits.  While the command is executing, ltrace intercepts and records
> | the dynamic library calls which are called by
> | the executed process and the signals received by that process.
> | It can also intercept and print the system calls executed by the program.
>
> The Go compiler (of the "gc" suite you seem to use) generates statically
> compiled executable which link with no dynamic libraries (at least on
> Linux/amd64, with the reasonably current Go versions).
>
> Supposedly this explains why you don't see the those ".dyn*" sections in
> the generated ELF file.
>
> So what is your goal, exactly?  What do you want to achieve?
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Go's scheduler always “stuck” after certain time when executing the code below

[Sen Han]

> just add a function call occasionally depending on your required latency
bound

Thanks Jesper for your all detailed explanation. I quite
agree with your point of view and conclusion.

One thing left here should to be discussed is about the
go scheduler's unwanted behaviour.

Unwanted behaviour of the scheduler:

After go run the codes at the topmost of this thread, we got:

$ time go run na.go
> heartbeat...
> 1000
> 2000
> 3000
> 4000
> 5000
> 6000
> 7000
> 8000
> 9000
> 1
> 11000
> heartbeat...
> 12000
> 13000
> heartbeat...
> 14000
> 15000
> 16000
> 17000
> 18000
> 19000
> heartbeat...
> ^Csignal: interrupt
> real0m0.779s
> user0m0.720s
> sys 0m0.152s


(*The process stuck here forever and no new outputs anymore*)

As the runtime.GOMAXPROCS is 5 now, why can't the scheduler just
let the other several goroutines which is already pending all switch
to the 4 left idly goroutine 'slots' and run on instead of suspend forever?
I think there is probably some design flaw about it.

On Wed, Jun 21, 2017 at 10:18 PM, Sen Han <yunthana...@gmail.com> wrote:

> > just add a function call occasionally depending on your required latency
> bound
>
> Thanks Jesper for your all detailed explanation. I quite
> agree with your point of view and conclusion.
>
> One thing left here should to be discussed is about the
> go scheduler's unwanted behaviour.
>
> Unwanted behaviour of the scheduler:
>
> After go run the codes at the topmost of this thread, we got:
>
> $ time go run na.go
>> heartbeat...
>> 1000
>> 2000
>> 3000
>> 4000
>> 5000
>> 6000
>> 7000
>> 8000
>> 9000
>> 1
>> 11000
>> heartbeat...
>> 12000
>> 13000
>> heartbeat...
>> 14000
>> 15000
>> 16000
>> 17000
>> 18000
>> 19000
>> heartbeat...
>> ^Csignal: interrupt
>> real0m0.779s
>> user0m0.720s
>> sys 0m0.152s
>
>
> (*The process stuck here forever and no new outputs anymore*)
>
> As the runtime.GOMAXPROCS is 5 now, why can't the scheduler just
> let the other several goroutines which is already pending all switch
> to the 4 left idly goroutine 'slots' and run on instead of suspend forever?
> I think there is probably some design flaw about it.
>
> On Wed, Jun 21, 2017 at 9:40 PM, Jesper Louis Andersen <
> jesper.louis.ander...@gmail.com> wrote:
>
>> On Wed, Jun 21, 2017 at 5:40 AM Ronald <yunthana...@gmail.com> wrote:
>>
>>> Hi, I found when running this code below (play
>>> <https://play.golang.org/p/XyxV_1f6Ri>):
>>>
>>>
>>> You are looking at a property of a specific Go implementation, not a
>> property of the language.
>>
>> Any system must define a set of "preemption points" at which you can
>> switch out one process for another. Historically go only preempted on
>> channel communication and (blocking) system calls. Later it also added
>> preemption on function calls (it piggybacks on stack checks). Adding more
>> preemption requires you to add checks inside loops so a loop eventually
>> gets preempted, but the caveat is that this might slow you program down
>> when you have a tight loop for which the compiler can prove no eventual
>> bound.
>>
>> The current best workaround is to make sure that lengthy work
>> periodically hits a preemption point some way or the other. Later versions
>> of Go might solve this problem by adding those checks itself automatically
>> to your programs to improve their latencies.
>>
>> Other concurrent languages use different methods:
>>
>> Erlang preempts like Go does currently, but since loops are implemented
>> with recursion, you are sure to hit a preemption point every time you go
>> round the loop.
>>
>> Haskell preempts on memory allocation and Haskell programs tend to
>> allocate memory quite frequently. The exception is tight (fusioned) loop
>> code on arrays.
>>
>> Go is doing a *far* better job here than almost every other programming
>> language out there when it comes to fair scheduling. And there is a
>> somewhat simple workaround: just add a function call occasionally depending
>> on your required latency bound.
>>
>>
>>
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [go-nuts] Re: Go's scheduler always “stuck” after certain time when executing the code below

[Sen Han]

>
> This piece of code prohibits cooperative scheduling in 599 999 999 out of

600 000 000 times. It's a nice example how to not code in Go.


Thanks Jan for your reply :)


> Particularly, delays should be implemented using the functions in the time

package, not by burning CPU cycles at the expense of other useful work

that could have been done instead.


Sure this kinda of dead loop is just an simulation about some really CPU
intensive jobs.
And I think for those kind of job, the delay about range [20 ms- inf ms) is
quite ordinary.
Thus I think that kind of writing is acceptable :D

On Wed, Jun 21, 2017 at 8:08 PM, Jan Mercl <0xj...@gmail.com> wrote:

> On Wed, Jun 21, 2017 at 1:50 PM Ronald  wrote:
>
> > for {
> > ct++
> > if ct%6 == 0 {
> > t2 := time.Now().UnixNano()
> > fmt.Println("hooray", t2, t1, float64(t2-t1)/1e6)
> > t1 = t2
> > }
> > }
>
> > This piece code is not "dead spin", it is just cpu intensive.
>
> This piece of code prohibits cooperative scheduling in 599 999 999 out of
> 600 000 000 times. It's a nice example how to not code in Go. Particularly,
> delays should be implemented using the functions in the time package, not
> by burning CPU cycles at the expense of other useful work that could have
> been done instead.
>
> --
>
> -j
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.