Re: [go-nuts] Why traversing Go hashmap so slower then C++ hashmap

[Ian Lance Taylor]

On Tue, Jun 28, 2022 at 9:56 PM Yan Li  wrote:
>
> Question:
> Go 161.385148ms vs C++ ms:31
> Go 851.517707ms vs C++ ms:104
> My question is why traversing go map Significantly slowly?
> Can it be improved?

I don't know exactly why there is a speed difference.

That said, there are significant differences between Go map and C++
unordered_map.  For example:

- Go map iteration is in random order.  C++ unordered_map iteration is
unpredictable but consistent.

- If you add an element to a Go map during the iteration, the behavior
is specified.  If you add an element to a C++ unordered_map all
iterators may be invalidated, breaking the loop.

Also, there is an implementation difference: the C++ code is compiled
inline with the exact types being used.  This takes more compile time
but produces faster run time.  The Go code uses a single shared
implementation for maps of all types.  This takes less compile time
but more run time.

Perhaps a combination of all these things explains the difference, or
perhaps there is more going on that I don't know about.

Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAOyqgcVnYnGacowVGgzoUPT6Q-o0SoA_ne%2B0LABJ80Lbs%3DsHjA%40mail.gmail.com.

[go-nuts] Why traversing Go hashmap so slower then C++ hashmap

[Yan Li]

*System env:*
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:Ubuntu 18.04.6 LTS
Release:18.04
Codename:   bionic

$ go version
go version go1.18.3 linux/amd64

$ g++ -v
gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)

*Test traversing Go hashmap:*
package main

import (
"fmt"
"time"
)

var size = 999;

func test_map() {
m1 := make(map[int]int)
for i := 0; i < size; i++ {
m1[i] = i
}

sum := 0
s := time.Now()
for _, v := range(m1) {
sum += v
}
fmt.Printf("m1  range sum:%v %v\n", sum, time.Since(s))

sum = 0
s = time.Now()
for i := 0; i < size; i++ {
sum += m1[i]
}
fmt.Printf("m1 direct sum:%v %v\n", sum, time.Since(s))
}

func main() {
test_map()
}

*Test traversing C++ hashmap:*
#include 

#include 
#include 
#include 

int size = 999;

static uint64_t
gettime() {
uint64_t t;
struct timespec ti;
clock_gettime(CLOCK_MONOTONIC, );
t = (uint64_t)ti.tv_sec * 1000;
t += ti.tv_nsec / 100;
return t;
}

void test_hashmap() {
std::unordered_map m1;
for (int i = 0; i < size; i++) {
m1[i] = i;
}

uint64_t sum = 0;
uint64_t s = gettime();
s = gettime();
for (auto& v : m1) {
sum += v.second;
}
printf("v1 foreach sum:%lu ms:%lu\n", sum, gettime() - s);

sum = 0;
s = gettime();
for (int i = 0; i < size; i++) {
sum += m1[i];
}
printf("v1  direct sum:%lu ms:%lu\n", sum, gettime() - s);
}

int main() {
test_hashmap();
return 0;
}

*Compare running result:*
*running Go*
$ go build test.go
$ ./test
m1  range sum:498501 161.385148ms
m1 direct sum:498501 851.517707ms
*running C++*
$ g++ -g -Wall -O2 test.cc
$ ./a.out
v1 foreach sum:498501 ms:31
v1  direct sum:498501 ms:104

*Question:*
Go 161.385148ms vs C++ ms:31 
Go 851.517707ms vs C++ ms:104 
My question is why traversing go map Significantly slowly?
Can it be improved? 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/67f2994b-5f05-401a-b207-cc730a6f822en%40googlegroups.com.

Re: [go-nuts] Temporary prevent preemption?

[Kurtis Rader]

The "asyncpreemptoff" debug option may be useful. See
https://pkg.go.dev/runtime. Specifically, try running your program after
doing "export GODEBUG=asyncpreemptoff=1". I'll leave it to others to
explain why this can be risky; especially in a context like you described.

On Tue, Jun 28, 2022 at 6:45 PM TH  wrote:

> Hey,
>
> I'm writing some latency & speed sensitive routines that are processing
> large amount of signal data in batches. When I run this in a standalone
> program the speed is fine. However when I combine this to the rest of
> software (>1000 goroutines, mainly network code) the speed gets reduced by
> 2x.
>
> Are there any low-level ways to temporary prevent preemption of a function
> or a goroutine?
>
> Alternatively I tried to run two piece of software in parallel, via
> shm+IPC synchronization methods, but futex, posix mutex, pipes, all
> produced latency jitter from 20us - 200us. If there's lower latency methods
> or more stable ones, please do recommend.
>
> Thanks
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/60220814-5763-4be4-a811-a9a84f64f12dn%40googlegroups.com
> 
> .
>


-- 
Kurtis Rader
Caretaker of the exceptional canines Junior and Hank

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CABx2%3DD97wBz36mF%3Dx5Kht3cGmx0Sz%3DoTgLxsPYUyvxpq3Z7rZQ%40mail.gmail.com.

[go-nuts] Re: Temporary prevent preemption?

[TH]

Btw.
I forgot to mention that CPU load is typically <2-5% on a 32 core server.
- Running in stand-alone as testing purposes speed is let say 100%
- Running two pieces code separately and connected via IPC methods, speed 
is also 100%, but IPC latency jitter is a problem
- But running two pieces code together (data processing part as 5 
additional goroutines) speed decreases to <50% (No lock contention between 
the two pieces of code, by analyzing more carefully it seems that 
processing routine gets interrupted in midway)

Thanks

On Wednesday, 29 June 2022 at 04:45:10 UTC+3 TH wrote:

> Hey,
>
> I'm writing some latency & speed sensitive routines that are processing 
> large amount of signal data in batches. When I run this in a standalone 
> program the speed is fine. However when I combine this to the rest of 
> software (>1000 goroutines, mainly network code) the speed gets reduced by 
> 2x.
>
> Are there any low-level ways to temporary prevent preemption of a function 
> or a goroutine?
>
> Alternatively I tried to run two piece of software in parallel, via 
> shm+IPC synchronization methods, but futex, posix mutex, pipes, all 
> produced latency jitter from 20us - 200us. If there's lower latency methods 
> or more stable ones, please do recommend.
>
> Thanks
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/9d036cc6-9ace-4169-b586-23a0d93915d5n%40googlegroups.com.

[go-nuts] Temporary prevent preemption?

[TH]

Hey,

I'm writing some latency & speed sensitive routines that are processing 
large amount of signal data in batches. When I run this in a standalone 
program the speed is fine. However when I combine this to the rest of 
software (>1000 goroutines, mainly network code) the speed gets reduced by 
2x.

Are there any low-level ways to temporary prevent preemption of a function 
or a goroutine?

Alternatively I tried to run two piece of software in parallel, via shm+IPC 
synchronization methods, but futex, posix mutex, pipes, all produced 
latency jitter from 20us - 200us. If there's lower latency methods or more 
stable ones, please do recommend.

Thanks

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/60220814-5763-4be4-a811-a9a84f64f12dn%40googlegroups.com.

Re: [go-nuts] ListenAndServeTLS() (pem and key files for private network)

[Martin Schnabel]

the package only helps to setup a special tls listener to be used with
the http server, as shown in the example. the http handler code and
routing is not affected in any way and is completely independent.

the special listener uses a ca cert to generate new server certs every
time you reach under a new name. client only need to trust the ca cert
at one time. if they connect to the server the listener returns the
existing server cert or creates a new cert (with the self signed ca
cert) that includes the name or ip it was called with.

hope that helps

On 6/28/22 16:03, Hugh Myrie wrote:

Hi Martin,

How does localca handle CORS?

Originally I defined an handler and pass it as a parameter to 
ListenAndServeTLS


c := cors.New(cors.Options{
         AllowedOrigins:   []string{"*"},
         AllowedMethods:   []string{"GET", "PUT", "POST", "DELETE"},
         AllowedHeaders:   []string{"*"},
         AllowCredentials: true,
     })
handler := c.Handler(r)

On Tue, Jun 28, 2022 at 7:03 AM Hugh Myrie > wrote:


Thanks Martin for your help.

On Tue, Jun 28, 2022, 06:26 Martin Schnabel mailto:m...@mb0.org>> wrote:

hi hugh,

i played around with the same idea six years ago, resulting in
https://github.com/mb0/localca 

it will use a self signed root cerificate and creates new child
certificates specific to the requested hostname, be it an ip or
a name
like my_local.box

i haven't tested it for awhile, but the idea does apply.

have fun

On 6/28/22 02:35, Hugh Myrie wrote:
 > I wish to create a secure private network using a self-signed
 > certificate with a Go web server: See the following code block:
 >
 > // Code
 >      err := http.ListenAndServeTLS(":"+port, "auto.org.pem",
 > "auto.org-key.pem", handler)
 >      if err != nil {
 >
 >          logError((err.Error()))
 >          log.Fatal("ListenAndServe: ", err)
 >      }
 > // End of Code
 >
 > Could I auto  generate (and register) the .pem and .key files
using GO?
 > I wish to create a trust certificate if there files do not exist.
 >
 > I came across the following website:
 >
 >
"https://gist.github.com/shaneutt/5e1995295cff6721c89a71d13a71c251
"
 >
 > I am not sure how to implement this. Your help is appreciated.
 >
 > --
 > You received this message because you are subscribed to the
Google
 > Groups "golang-nuts" group.
 > To unsubscribe from this group and stop receiving emails from
it, send
 > an email to golang-nuts+unsubscr...@googlegroups.com

 > >.
 > To view this discussion on the web visit
 >

https://groups.google.com/d/msgid/golang-nuts/6ce2a407-600d-44b7-97d1-53e90e91e7c8n%40googlegroups.com



 >

>.

-- 
You received this message because you are subscribed to a topic

in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/golang-nuts/0XKZ3AkAA9w/unsubscribe
.
To unsubscribe from this group and all its topics, send an email
to golang-nuts+unsubscr...@googlegroups.com
.
To view this discussion on the web visit

https://groups.google.com/d/msgid/golang-nuts/63b7a250-5afc-6171-6526-00796498ea36%40mb0.org

.



--
http://www.jaxtr.com/blessed_hope 


--
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/2c4546e6-22b7-302b-311b-d6f4e198c60a%40mb0.org.

Re: [go-nuts] Is this a compilation optimization bug ?

[Ian Lance Taylor]

On Tue, Jun 28, 2022 at 5:47 AM 'Dan Kortschak' via golang-nuts
 wrote:
>
> On Mon, 2022-06-27 at 23:32 -0700, iori yamada wrote:
> > Running the code in the link below times out, is this a bug?
> > Looking at the assembly, it seems that the instructions corresponding
> > to the if statement have been removed due to optimization.
> >
> > https://go.dev/play/p/CZX4mbyrp37
> >
> > The following is a description of how I thought it was a bug.
> >
> > In the first loop, the value of i is 9223372036854775807 (the maximum
> > value of type int), which is incremented and overflows. Then it
> > becomes -9223372036854775808 (minimum value of type int) and I
> > thought the condition in the if statement exits from the true
> > neighbor loop.
> >
> > I couldn't figure out where the bug was happening, so I decided to
> > look at an assembly of the same code as above, built it locally, and
> > checked it with objdump.
> > Perhaps, the CMP instruction that compares the conditions of the if
> > statement has been removed due to optimization.
> >
> > ```
> > ...
> > TEXT main.main(SB) Path/main.go
> >   main.go:3 0x100051550 92f0 MOVD $9223372036854775807, R0
> >   main.go:4 0x100051554 1402 JMP 2(PC)
> >   main.go:4 0x100051558 91000400 ADD $1, R0, R0
> >   main.go:4 0x10005155c 92f0001b MOVD $9223372036854775807, R27
> >   main.go:4 0x100051560 eb1b001f CMP R27, R0
> >   main.go:4 0x100051564 54ad BLE -3(PC)
> >   main.go:9 0x100051568 d65f03c0 RET
> >   main.go:9 0x10005156c  ?
> > ```
> >
> > So I compiled it without optimization (go build -gcflags '-N -l') and
> > the program terminated immediately as intended.
> >
> > I may have misunderstood something, may I ask anyone to check once?
>
> Yes, I see the same behaviour when I run this locally (also tried with
> an old version go1.14.9).
>
> Also, the assembly generated at godbolt shows that no code is generated
> for the terminating branch  https://godbolt.org/z/9jW1GaKfv unless
> optimisation is turned off https://godbolt.org/z/o7o6x3bzd.

Interesting.  Looks like Go 1.10 got this right, but 1.11 and higher
go into an endless loop.  gccgo gets it right.

Please open an issue.  Thanks.

Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAOyqgcUPy6VdKbmc%2BOCaMDAw3c6FdtJVcLiJBGji1BK-DPzKaQ%40mail.gmail.com.

Re: [go-nuts] Is this a compilation optimization bug ?

['Dan Kortschak' via golang-nuts]

On Mon, 2022-06-27 at 23:32 -0700, iori yamada wrote:
> Running the code in the link below times out, is this a bug?
> Looking at the assembly, it seems that the instructions corresponding
> to the if statement have been removed due to optimization.
>
> https://go.dev/play/p/CZX4mbyrp37
>
> The following is a description of how I thought it was a bug.
>
> In the first loop, the value of i is 9223372036854775807 (the maximum
> value of type int), which is incremented and overflows. Then it
> becomes -9223372036854775808 (minimum value of type int) and I
> thought the condition in the if statement exits from the true
> neighbor loop.
>
> I couldn't figure out where the bug was happening, so I decided to
> look at an assembly of the same code as above, built it locally, and
> checked it with objdump.
> Perhaps, the CMP instruction that compares the conditions of the if
> statement has been removed due to optimization.
>
> ```
> ...
> TEXT main.main(SB) Path/main.go
>   main.go:3 0x100051550 92f0 MOVD $9223372036854775807, R0
>   main.go:4 0x100051554 1402 JMP 2(PC)
>   main.go:4 0x100051558 91000400 ADD $1, R0, R0
>   main.go:4 0x10005155c 92f0001b MOVD $9223372036854775807, R27
>   main.go:4 0x100051560 eb1b001f CMP R27, R0
>   main.go:4 0x100051564 54ad BLE -3(PC)
>   main.go:9 0x100051568 d65f03c0 RET
>   main.go:9 0x10005156c  ?
> ```
>
> So I compiled it without optimization (go build -gcflags '-N -l') and
> the program terminated immediately as intended.
>
> I may have misunderstood something, may I ask anyone to check once?

Yes, I see the same behaviour when I run this locally (also tried with
an old version go1.14.9).

Also, the assembly generated at godbolt shows that no code is generated
for the terminating branch  https://godbolt.org/z/9jW1GaKfv unless
optimisation is turned off https://godbolt.org/z/o7o6x3bzd.

Dan

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/ac0efc0fc8bb57ec2e3fb12b9f69774f28c16e84.camel%40kortschak.io.

[go-nuts] Is this a compilation optimization bug ?

[iori yamada]

Running the code in the link below times out, is this a bug?
Looking at the assembly, it seems that the instructions corresponding to 
the if statement have been removed due to optimization.

https://go.dev/play/p/CZX4mbyrp37

The following is a description of how I thought it was a bug.

In the first loop, the value of i is 9223372036854775807 (the maximum value 
of type int), which is incremented and overflows. Then it becomes 
-9223372036854775808 (minimum value of type int) and I thought the 
condition in the if statement exits from the true neighbor loop.

I couldn't figure out where the bug was happening, so I decided to look at 
an assembly of the same code as above, built it locally, and checked it 
with objdump.
Perhaps, the CMP instruction that compares the conditions of the if 
statement has been removed due to optimization.

```
...
TEXT main.main(SB) Path/main.go
  main.go:3 0x100051550 92f0 MOVD $9223372036854775807, R0
  main.go:4 0x100051554 1402 JMP 2(PC)
  main.go:4 0x100051558 91000400 ADD $1, R0, R0
  main.go:4 0x10005155c 92f0001b MOVD $9223372036854775807, R27
  main.go:4 0x100051560 eb1b001f CMP R27, R0
  main.go:4 0x100051564 54ad BLE -3(PC)
  main.go:9 0x100051568 d65f03c0 RET
  main.go:9 0x10005156c  ? 
```

So I compiled it without optimization (go build -gcflags '-N -l') and the 
program terminated immediately as intended.

I may have misunderstood something, may I ask anyone to check once?

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/c4f21dc1-d450-42c8-996f-4f8f1461a9c2n%40googlegroups.com.

Re: [go-nuts] ListenAndServeTLS() (pem and key files for private network)

[Martin Schnabel]

hi hugh,

i played around with the same idea six years ago, resulting in 
https://github.com/mb0/localca


it will use a self signed root cerificate and creates new child 
certificates specific to the requested hostname, be it an ip or a name 
like my_local.box


i haven't tested it for awhile, but the idea does apply.

have fun

On 6/28/22 02:35, Hugh Myrie wrote:
I wish to create a secure private network using a self-signed 
certificate with a Go web server: See the following code block:


// Code
     err := http.ListenAndServeTLS(":"+port, "auto.org.pem", 
"auto.org-key.pem", handler)

     if err != nil {

         logError((err.Error()))
         log.Fatal("ListenAndServe: ", err)
     }
// End of Code

Could I auto  generate (and register) the .pem and .key files using GO?  
I wish to create a trust certificate if there files do not exist.


I came across the following website:

"https://gist.github.com/shaneutt/5e1995295cff6721c89a71d13a71c251;

I am not sure how to implement this. Your help is appreciated.

--
You received this message because you are subscribed to the Google 
Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send 
an email to golang-nuts+unsubscr...@googlegroups.com 
.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/6ce2a407-600d-44b7-97d1-53e90e91e7c8n%40googlegroups.com 
.


--
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/63b7a250-5afc-6171-6526-00796498ea36%40mb0.org.

[go-nuts] Re: ListenAndServeTLS() (pem and key files for private network)

[Peter Galbavy]

If it helps, I put this together for my own project - it's not a package 
but a bunch of convenience functions/methods for generating TLS certs and 
keys (there is a lot of implementation specific stuff), but the general 
shape may help you:

https://github.com/itrs-group/cordial/blob/main/tools/geneos/internal/instance/tls.go

Peter

On Tuesday, 28 June 2022 at 01:35:38 UTC+1 hugh@gmail.com wrote:

> I wish to create a secure private network using a self-signed certificate 
> with a Go web server: See the following code block:
>
> // Code 
> err := http.ListenAndServeTLS(":"+port, "auto.org.pem", 
> "auto.org-key.pem", handler)
> if err != nil {
>
> logError((err.Error()))
> log.Fatal("ListenAndServe: ", err)
> }
> // End of Code
>
> Could I auto  generate (and register) the .pem and .key files using GO?  I 
> wish to create a trust certificate if there files do not exist.
>
> I came across the following website:
>
> "https://gist.github.com/shaneutt/5e1995295cff6721c89a71d13a71c251;
>
> I am not sure how to implement this. Your help is appreciated.
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/2c4a6aee-0b1f-454b-a725-c4233759ed27n%40googlegroups.com.