Re: [go-nuts] Suggestions on optimizing Go GC

[Zhihui Jiang]

On Monday, October 30, 2023 at 10:12:08 PM UTC-7 Robert Engels wrote:

What is the average wall time of a request?

The average latency is a few hundred milliseconds.  


Based on what you wrote it appears that handling a single request generates 
a lot of garbage - high allocation rate - and for this to be significant I 
suspect the runtime is also significant - which implies to me a spawn and 
destroy request handler is your best bet. 

I actually didn't quite get your suggestion earlier. We are using gRPC and 
I think for each request we already have separate goroutines to handle it. 
Can you explain a little bit more about spawn and destroy request handler? 


On Oct 30, 2023, at 11:56 PM, Zhihui Jiang  wrote:

Hi Michael, Jason and Robert, thanks a lot for the replies and suggestions!


I did some profiling today, here are some specific findings:
1, CPUs used for GC is around 35% after we enabled soft memory limit, and 
it was 45%+ before. I don't have too much experience here on how much CPU 
we should spend on GCs ideally, but my intuition 35% is pretty high.
2, For GC, most of the CPU is on *runtime.scanObject* which I guess is 
dependent on how many object we allocate and how fast that is. 
3, With some further look at the heap profiling, it turns out most of the 
objects (70%+) allocated are due to complex protobuf messages we use for 
communications between services which can be big and might have deep-nested 
submessages.

On Monday, October 30, 2023 at 2:19:23 PM UTC-7 Michael Knyszek wrote:

I second Jason's message, and +1 to off-heap memory as a last resort. 

Yes, indeed. One of the advantage using Go is we don't need to manage 
memory by ourselves, I will try other options first and see how much we can 
improve. 

Here are a few more details:

For a starting point on how to reduce memory allocations directly, see 
https://go.dev/doc/gc-guide#Optimization_guide. Note that this may require 
restructuring your program in places. (e.g. passing byte slices to 
functions to be filled instead of returning byte slices; that sort of 
thing.)
RE: pooling memory, take a look look at sync.Pool (
https://pkg.go.dev/sync#Pool). A sync.Pool can be really effective at 
reducing the number of memory allocations that are made in the steady-state.

Object pooling is actually one of the most promising option we are trying 
to implement right now. One quick question: is sync.Pool also feasible for 
complex protobuf messages? any pitfall we should be take into 
consideration? 


On Monday, October 30, 2023 at 2:33:21 PM UTC-4 Jason E. Aten wrote:

Similar to Robert's suggestion, you could just use non-GC-ed memory within 
the process.

https://github.com/glycerine/offheap provides an example. 

The central idea is that the Go GC will never touch memory that you have 
requested
yourself from the OS. So you can make your own Arenas. 
https://en.wikipedia.org/wiki/Region-based_memory_management

But I would save these as last resorts of course. Before that:

a) can you reduce the objects allocated per request?  
b) can you allocate everything else on the stack? There are flags to see 
why things are escaping to the heap, use those in your analysis.
(This is by far the simplest and fastest thing. Since the stack is 
automatically unwound when the user request finishes, typically, there is 
no GC to do.)

Will try this out and let you know if we have interesting findings here. 

c) can you allocate a pool of objects that is just reused instead of 
allocating for each new user request?
d) Is there anything that can be effectively cached and re-used instead of 
allocated?

Good point! We actually have an in-memory cache which already haas very 
high cache hit ratio of 95%+. Seems not too much headroom here to further 
reduce the CPUs on GC. 


Use the profiler pprof to figure out what is going on.

Thanks! pprof indeed is very helpful tool and the problem we are facing 
seems to boil down to too many large/complex protobuf message passed around 
different services which allocates too many objects during the proto 
unmarshal. 

-- 

You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an 
email to golang-nuts...@googlegroups.com.

To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/2e3ac44e-923b-4b6b-88ec-743f8474c83an%40googlegroups.com
 

.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/29875b49-316d-4332-9854-35da4c043005n%40googlegroups.com.

Re: [go-nuts] Suggestions on optimizing Go GC

[Robert Engels]

What is the average wall time of a request?

Based on what you wrote it appears that handling a single request generates a 
lot of garbage - high allocation rate - and for this to be significant I 
suspect the runtime is also significant - which implies to me a spawn and 
destroy request handler is your best bet. 

> On Oct 30, 2023, at 11:56 PM, Zhihui Jiang  wrote:
> 
> Hi Michael, Jason and Robert, thanks a lot for the replies and suggestions!
> 
> I did some profiling today, here are some specific findings:
> 1, CPUs used for GC is around 35% after we enabled soft memory limit, and it 
> was 45%+ before. I don't have too much experience here on how much CPU we 
> should spend on GCs ideally, but my intuition 35% is pretty high.
> 2, For GC, most of the CPU is on runtime.scanObject which I guess is 
> dependent on how many object we allocate and how fast that is. 
> 3, With some further look at the heap profiling, it turns out most of the 
> objects (70%+) allocated are due to complex protobuf messages we use for 
> communications between services which can be big and might have deep-nested 
> submessages.
> 
> On Monday, October 30, 2023 at 2:19:23 PM UTC-7 Michael Knyszek wrote:
> I second Jason's message, and +1 to off-heap memory as a last resort.
> Yes, indeed. One of the advantage using Go is we don't need to manage memory 
> by ourselves, I will try other options first and see how much we can improve. 
> Here are a few more details:
> 
> For a starting point on how to reduce memory allocations directly, see 
> https://go.dev/doc/gc-guide#Optimization_guide. Note that this may require 
> restructuring your program in places. (e.g. passing byte slices to functions 
> to be filled instead of returning byte slices; that sort of thing.)
> RE: pooling memory, take a look look at sync.Pool 
> (https://pkg.go.dev/sync#Pool). A sync.Pool can be really effective at 
> reducing the number of memory allocations that are made in the steady-state.
> Object pooling is actually one of the most promising option we are trying to 
> implement right now. One quick question: is sync.Pool also feasible for 
> complex protobuf messages? any pitfall we should be take into consideration? 
> 
> On Monday, October 30, 2023 at 2:33:21 PM UTC-4 Jason E. Aten wrote:
> Similar to Robert's suggestion, you could just use non-GC-ed memory within 
> the process.
> 
> https://github.com/glycerine/offheap provides an example. 
> 
> The central idea is that the Go GC will never touch memory that you have 
> requested
> yourself from the OS. So you can make your own Arenas. 
> https://en.wikipedia.org/wiki/Region-based_memory_management
> 
> But I would save these as last resorts of course. Before that:
> 
> a) can you reduce the objects allocated per request?  
> b) can you allocate everything else on the stack? There are flags to see why 
> things are escaping to the heap, use those in your analysis.
> (This is by far the simplest and fastest thing. Since the stack is 
> automatically unwound when the user request finishes, typically, there is no 
> GC to do.)
> Will try this out and let you know if we have interesting findings here. 
> c) can you allocate a pool of objects that is just reused instead of 
> allocating for each new user request?
> d) Is there anything that can be effectively cached and re-used instead of 
> allocated?
> Good point! We actually have an in-memory cache which already haas very high 
> cache hit ratio of 95%+. Seems not too much headroom here to further reduce 
> the CPUs on GC. 
> 
> Use the profiler pprof to figure out what is going on.
> Thanks! pprof indeed is very helpful tool and the problem we are facing seems 
> to boil down to too many large/complex protobuf message passed around 
> different services which allocates too many objects during the proto 
> unmarshal. 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/2e3ac44e-923b-4b6b-88ec-743f8474c83an%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/0A66E87C-1BD6-4486-B152-763188B404C7%40ix.netcom.com.

Re: [go-nuts] Suggestions on optimizing Go GC

[Zhihui Jiang]

Hi Michael, Jason and Robert, thanks a lot for the replies and suggestions!

I did some profiling today, here are some specific findings:
1, CPUs used for GC is around 35% after we enabled soft memory limit, and 
it was 45%+ before. I don't have too much experience here on how much CPU 
we should spend on GCs ideally, but my intuition 35% is pretty high.
2, For GC, most of the CPU is on *runtime.scanObject* which I guess is 
dependent on how many object we allocate and how fast that is. 
3, With some further look at the heap profiling, it turns out most of the 
objects (70%+) allocated are due to complex protobuf messages we use for 
communications between services which can be big and might have deep-nested 
submessages.

On Monday, October 30, 2023 at 2:19:23 PM UTC-7 Michael Knyszek wrote:

I second Jason's message, and +1 to off-heap memory as a last resort. 

Yes, indeed. One of the advantage using Go is we don't need to manage 
memory by ourselves, I will try other options first and see how much we can 
improve. 

Here are a few more details:

For a starting point on how to reduce memory allocations directly, see 
https://go.dev/doc/gc-guide#Optimization_guide. Note that this may require 
restructuring your program in places. (e.g. passing byte slices to 
functions to be filled instead of returning byte slices; that sort of 
thing.)
RE: pooling memory, take a look look at sync.Pool (
https://pkg.go.dev/sync#Pool). A sync.Pool can be really effective at 
reducing the number of memory allocations that are made in the steady-state.

Object pooling is actually one of the most promising option we are trying 
to implement right now. One quick question: is sync.Pool also feasible for 
complex protobuf messages? any pitfall we should be take into 
consideration? 


On Monday, October 30, 2023 at 2:33:21 PM UTC-4 Jason E. Aten wrote:

Similar to Robert's suggestion, you could just use non-GC-ed memory within 
the process.

https://github.com/glycerine/offheap provides an example. 

The central idea is that the Go GC will never touch memory that you have 
requested
yourself from the OS. So you can make your own Arenas. 
https://en.wikipedia.org/wiki/Region-based_memory_management

But I would save these as last resorts of course. Before that:

a) can you reduce the objects allocated per request?  
b) can you allocate everything else on the stack? There are flags to see 
why things are escaping to the heap, use those in your analysis.
(This is by far the simplest and fastest thing. Since the stack is 
automatically unwound when the user request finishes, typically, there is 
no GC to do.)

Will try this out and let you know if we have interesting findings here. 

c) can you allocate a pool of objects that is just reused instead of 
allocating for each new user request?
d) Is there anything that can be effectively cached and re-used instead of 
allocated?

Good point! We actually have an in-memory cache which already haas very 
high cache hit ratio of 95%+. Seems not too much headroom here to further 
reduce the CPUs on GC. 


Use the profiler pprof to figure out what is going on.

Thanks! pprof indeed is very helpful tool and the problem we are facing 
seems to boil down to too many large/complex protobuf message passed around 
different services which allocates too many objects during the proto 
unmarshal. 

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/2e3ac44e-923b-4b6b-88ec-743f8474c83an%40googlegroups.com.

Re: [go-nuts] Re: Clace: Secure web application development platform using Starlark

['Dan Kortschak' via golang-nuts]

On Mon, 2023-10-30 at 18:43 -0700, Jason E. Aten wrote:
> I'm surprised by that claim. I seriously doubt, from reading the
> licenses, that you can legally use the Apache2 license, since
> it removes the MPL requirements; which the MPL forbids you from
> doing.  
> 

The Mozilla FAQ https://www.mozilla.org/en-US/MPL/2.0/FAQ/ appears to
think it's OK.

> Q13: May I combine MPL-licensed code and BSD-licensed code in the
> same executable program? What about Apache?
>
> Yes to both. Mozilla currently does this with BSD-licensed code. For
> example, libvpx, which is used in Firefox to decode WebM video, is
> under a BSD license.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/3b85733a7f1f495abf00b54ef4a33e4d65113d3d.camel%40kortschak.io.

Re: [go-nuts] Re: Clace: Secure web application development platform using Starlark

[Jason E. Aten]

I'm surprised by that claim. I seriously doubt, from reading the licenses, 
that you can legally use the Apache2 license, since
it removes the MPL requirements; which the MPL forbids you from doing.  

Moreover I don't think a court would consider relevant what the Cloud 
Native Foundation thought was "okay".

On Tuesday, October 31, 2023 at 1:13:15 AM UTC Ajay Kidave wrote:

> Clace itself is Apache-2 licensed, using a MPL licensed library in an 
> Apache-2 licensed project is fine from what I understand. I do not plan to 
> make any code changes to the go-plugin code. The go-plugin library is 
> specifically allowed by CNCF projects if that matters 
> https://github.com/cncf/foundation/blob/main/license-exceptions/cncf-exceptions-2023-06-27.json#L3
> .
>
> The intent in Clace is to allow multiple versions of a plugin to be usable 
> for backward compatibility. That, plus the security and stability benefits 
> of the out of process model make the go-plugin approach work better than 
> the .so/.dll approach.
>
> Ajay
>
> On Mon, Oct 30, 2023 at 11:55 AM Jason E. Aten  wrote:
>
>> I would just be aware that using software that is MPL licensed,
>>
>> https://github.com/hashicorp/go-plugin/blob/main/LICENSE
>>
>> means that nobody with commercial aspirations will touch your stuff. In 
>> practice, that means relatively little adoption. 
>>
>> -- 
>> You received this message because you are subscribed to a topic in the 
>> Google Groups "golang-nuts" group.
>> To unsubscribe from this topic, visit 
>> https://groups.google.com/d/topic/golang-nuts/FyaMylLPGEw/unsubscribe.
>> To unsubscribe from this group and all its topics, send an email to 
>> golang-nuts...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/d56f9aab-19f8-40d4-87a5-59d05e1adb9fn%40googlegroups.com
>>  
>> 
>> .
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/d659156e-b0d0-4c3a-b42a-9ff1a0672384n%40googlegroups.com.

Re: [go-nuts] Re: recommended statistics package? Requirements: ANOVA, Brown-Forsythe

[王富民awaw]

Hi Jason

Thanks for your suggestion.
I have renamed the function Levene to BrownForsythe, as well as added a 
comment to note that it assumes its input to be sorted ascendingly.

On Tuesday, 31 October 2023 at 02:18:51 UTC+8 Jason E. Aten wrote:

>
> On Monday, October 30, 2023 at 3:14:29 AM UTC 王富民awaw wrote:
>
> Therefore, perhaps I should not be following scipy and rename my function 
> from Levene to BrownForsythe?
>
>  
> Since you have made it available to the public Go community, yes, it would 
> be a good idea to name it BrownForsythe,
> as it is using the deviations from the group medians.
>
> Also the function makes the assumption, as does Median, that each group's 
> samples[j] is already sorted. You
> should document that assumption, or sort.Slice() each group's sample 
> before calling Median.
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/e12012dc-36c5-43e2-a165-02b0a480cce0n%40googlegroups.com.

Re: [go-nuts] Re: Clace: Secure web application development platform using Starlark

[Ajay Kidave]

Clace itself is Apache-2 licensed, using a MPL licensed library in an
Apache-2 licensed project is fine from what I understand. I do not plan to
make any code changes to the go-plugin code. The go-plugin library is
specifically allowed by CNCF projects if that matters
https://github.com/cncf/foundation/blob/main/license-exceptions/cncf-exceptions-2023-06-27.json#L3
.

The intent in Clace is to allow multiple versions of a plugin to be usable
for backward compatibility. That, plus the security and stability benefits
of the out of process model make the go-plugin approach work better than
the .so/.dll approach.

Ajay

On Mon, Oct 30, 2023 at 11:55 AM Jason E. Aten  wrote:

> I would just be aware that using software that is MPL licensed,
>
> https://github.com/hashicorp/go-plugin/blob/main/LICENSE
>
> means that nobody with commercial aspirations will touch your stuff. In
> practice, that means relatively little adoption.
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "golang-nuts" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/golang-nuts/FyaMylLPGEw/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/d56f9aab-19f8-40d4-87a5-59d05e1adb9fn%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAFHo40kZpkuAw%2B5r-UV-VjRoJBvg3OAZSZxS3gwq11%3DKxKdeOA%40mail.gmail.com.

Re: [go-nuts] Suggestions on optimizing Go GC

['Michael Knyszek' via golang-nuts]

I second Jason's message, and +1 to off-heap memory as a last resort. Here 
are a few more details:

For a starting point on how to reduce memory allocations directly, see 
https://go.dev/doc/gc-guide#Optimization_guide. Note that this may require 
restructuring your program in places. (e.g. passing byte slices to 
functions to be filled instead of returning byte slices; that sort of 
thing.)
RE: pooling memory, take a look look at sync.Pool (
https://pkg.go.dev/sync#Pool). A sync.Pool can be really effective at 
reducing the number of memory allocations that are made in the steady-state.

On Monday, October 30, 2023 at 2:33:21 PM UTC-4 Jason E. Aten wrote:

> Similar to Robert's suggestion, you could just use non-GC-ed memory within 
> the process.
>
> https://github.com/glycerine/offheap provides an example. 
>
> The central idea is that the Go GC will never touch memory that you have 
> requested
> yourself from the OS. So you can make your own Arenas. 
> https://en.wikipedia.org/wiki/Region-based_memory_management
>
> But I would save these as last resorts of course. Before that:
>
> a) can you reduce the objects allocated per request?  
> b) can you allocate everything else on the stack? There are flags to see 
> why things are escaping to the heap, use those in your analysis.
> (This is by far the simplest and fastest thing. Since the stack is 
> automatically unwound when the user request finishes, typically, there is 
> no GC to do.)
> c) can you allocate a pool of objects that is just reused instead of 
> allocating for each new user request?
> d) Is there anything that can be effectively cached and re-used instead of 
> allocated?
>
> Use the profiler pprof to figure out what is going on.
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/184e160a-d30d-43e4-a822-b7dc61a03b1bn%40googlegroups.com.

[go-nuts] Re: Clace: Secure web application development platform using Starlark

[TheDiveO]

sadly, OpenDoufu(*) is the epitaph of HashiCorp breaking bad. So that 
go-plugin package is a no-go now (pun intended).

(*) I can't get myself using the British Empire misspelling of Standard 
Chinese (putonghua). And don't call that Ma... either.

On Monday, October 30, 2023 at 7:54:50 PM UTC+1 Jason E. Aten wrote:

> I would just be aware that using software that is MPL licensed,
>
> https://github.com/hashicorp/go-plugin/blob/main/LICENSE
>
> means that nobody with commercial aspirations will touch your stuff. In 
> practice, that means relatively little adoption.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/742c67bd-53f7-4df4-abb3-0e1583553a1cn%40googlegroups.com.

Re: [go-nuts] Re: recommended statistics package? Requirements: ANOVA, Brown-Forsythe

[Jason E. Aten]

On Monday, October 30, 2023 at 3:14:29 AM UTC 王富民awaw wrote:

Therefore, perhaps I should not be following scipy and rename my function 
from Levene to BrownForsythe?

 
Since you have made it available to the public Go community, yes, it would 
be a good idea to name it BrownForsythe,
as it is using the deviations from the group medians.

Also the function makes the assumption, as does Median, that each group's 
samples[j] is already sorted. You
should document that assumption, or sort.Slice() each group's sample before 
calling Median.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/2722f546-62e3-4cb1-80bb-1104d19ced4an%40googlegroups.com.

[go-nuts] Symbolic/Concolic execution of Go compiled Binaries

['Karolina GORNA' via golang-nuts]

Hi Gophers,

Hope that your are alright!

*Does any of you know symbolic or concolic execution tools that are 
compatible with Go binaries please ? *
>From what I currently understand, it seams possible to adapt Angr 
 for instance so that it understands Go binaries, but I 
am not 100% sure.
Also, I am curious if someone worked on that before.

Thank you for your time,
Karolina


-- 

Les informations contenues dans ce message électronique ainsi que celles 
contenues dans les documents attachés sont strictement confidentielles et 
sont destinées à l'usage exclusif du (des) destinataire(s) nommé(s).
Toute 
divulgation, distribution ou reproduction, même partielle, en est 
strictement interdite sauf autorisation écrite et expresse de l’émetteur.
Si vous recevez ce message par erreur, veuillez le notifier immédiatement à 
son émetteur par retour, et le détruire ainsi que tous les documents qui y 
sont attachés.


The information contained in this email and in any 
document enclosed is strictly confidential and is intended solely for the 
use of the individual or entity to which it is addressed.
Partial or total 
disclosure, distribution or reproduction of its contents is strictly 
prohibited unless expressly approved in writing by the sender.
If you have 
received this communication in error, please notify us immediately by 
responding to this email, and then delete the message and its attached 
files from your system.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/bbe7b8ed-cf7f-45ba-b984-bb42cc716dbdn%40googlegroups.com.

Re: [go-nuts] Re: Go called by C: is the goroutine locked to a thread?

[Domenico Andreoli]

On Sun, Oct 29, 2023 at 11:32 PM Jason E. Aten  wrote:

> You could also look at the existing Go <-> Python interfaces and see how
> they handle such issues. Might give you hints.
>
> https://github.com/qur/gopy. (python 3.11)
> https://github.com/glycerine/pyg (python 3.7.1)
>
> Thanks for the pointers, I'll check what they do.

-- 
rsa4096: 3B10 0CA1 8674 ACBA B4FE  FCD2 CE5B CF17 9960 DE13
ed25519: FFB4 0CC3 7F2E 091D F7DA  356E CC79 2832 ED38 CB05

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAGH6_pqkYp_-HxhBHZnkpPdWKG4DMo%3DUSrJ%3DveNQ8Ua7h9%3D4YA%40mail.gmail.com.