On Monday, May 12, 2025 at 3:58:54 PM UTC-4 Jason E. Aten wrote:
possibly very relevant -- https://github.com/golang/go/issues/70683 On Monday, May 12, 2025 at 8:50:33 PM UTC+1 Jason E. Aten wrote: Interesting. Thanks, Robert, for the pointer to your tests in https://github.com/robaho/go-concurrency-test ; I've not tried to run them. It seems there have been a dozen or so commits to sync.Map since 2018, and there is even an experimental hash-trie version mentioned below in the history. If you revisit it and find your concerns still hold, I would be quite interested to see how my simple benchmarks were fooled. https://cs.opensource.google/go/go/+/refs/tags/go1.24.3:src/sync/map.go;l=40;bpv=1;bpt=0;drc=aecb8faa91e2b40e3651ee93c8e70635ed367677;dlc=53b2b64b649b26c7bb3397bec5d86d3b203eb015 On Monday, May 12, 2025 at 8:03:49 PM UTC+1 Robert Engels wrote: That’s interesting on the performance of sync.Map as the issues cited here are still open https://github.com/golang/go/issues/28938#issuecomment-441681208 Those issues should have been closed with https://github.com/golang/go/issues/70683. I just went ahead and did that with a final comment on each. sync.Map still isn't perfect but might be worth a second look now. On May 12, 2025, at 12:48 PM, Jason E. Aten <j.e....@gmail.com> wrote: Robert's suggestion is obviously the most direct and correct route if you can change the IDL of generated type. After all, code generation is there to simplify generating alot of code from a little definition. I was assuming that one of constraints was not being able to modify the struct received. You should take that simpler route, if at all possible. > If contention is indeed a problem I can mitigate by sharding the map You are in luck. sync.Map already shards for you. It is very well engineered, and in my benchmarks, you will be very hard pressed to do better. Less than 50ns for 10% writes, and 12ns for 100% reads. Its documentation does it a bit of a disservice by discouraging its wider use, but it is awesome; one of the reasons that net/rpc itself was so performant. There are some studies here (with benchmark code you can run yourself to reproduce it: https://github.com/glycerine/uart), the most salient part is this benchmark, going from 100% writes at the top, to 100% reads at the bottom, in steps of 10%. BenchmarkReadWriteSyncMap BenchmarkReadWriteSyncMap/frac_0 BenchmarkReadWriteSyncMap/frac_0-8 11585511 128.9 ns/op 111 B/op 5 allocs/op BenchmarkReadWriteSyncMap/frac_1 BenchmarkReadWriteSyncMap/frac_1-8 12977352 131.1 ns/op 101 B/op 4 allocs/op BenchmarkReadWriteSyncMap/frac_2 BenchmarkReadWriteSyncMap/frac_2-8 14838945 122.2 ns/op 90 B/op 4 allocs/op BenchmarkReadWriteSyncMap/frac_3 BenchmarkReadWriteSyncMap/frac_3-8 14907528 114.5 ns/op 80 B/op 3 allocs/op BenchmarkReadWriteSyncMap/frac_4 BenchmarkReadWriteSyncMap/frac_4-8 18078240 100.8 ns/op 70 B/op 3 allocs/op BenchmarkReadWriteSyncMap/frac_5 BenchmarkReadWriteSyncMap/frac_5-8 18791480 89.52 ns/op 59 B/op 3 allocs/op BenchmarkReadWriteSyncMap/frac_6 BenchmarkReadWriteSyncMap/frac_6-8 21172922 79.55 ns/op 49 B/op 2 allocs/op BenchmarkReadWriteSyncMap/frac_7 BenchmarkReadWriteSyncMap/frac_7-8 25711459 73.04 ns/op 38 B/op 2 allocs/op BenchmarkReadWriteSyncMap/frac_8 BenchmarkReadWriteSyncMap/frac_8-8 29925382 60.51 ns/op 28 B/op 1 allocs/op BenchmarkReadWriteSyncMap/frac_9 BenchmarkReadWriteSyncMap/frac_9-8 41599728 46.34 ns/op 18 B/op 1 allocs/op BenchmarkReadWriteSyncMap/frac_10 BenchmarkReadWriteSyncMap/frac_10-8 100000000 11.96 ns/op 8 B/op 1 allocs/op On Monday, May 12, 2025 at 5:34:08 PM UTC+1 Robert Engels wrote: Why not just modify the code generator to add a user data pointer to the structure? I can’t imagine something like that isn’t already there? Either the code generator should be easy to enhance or it should have a side car data field - anything else is hard to rationalize as even rudimentary design. On May 12, 2025, at 11:13 AM, Alexander Shopov <a...@kambanaria.org> wrote: Now that I think about it again - I guess your suggestion could work for my case with some caveats: 1. There can be contention on the map - but as you point out there is sync.Map as well 2. The state should be removed from the map but that can also be arranged by deferring a delete operation 3. If contention is indeed a problem I can mitigate by sharding the map 4. If contention continues to be a problem perhaps weak pointers with this recipe https://github.com/golang/go/issues/67552#issuecomment-2195479919 may help. It won't be pretty but hey - help beggars can't be choosers! Kind regards: al_shopov На пн, 12.05.2025 г. в 17:22 Jason E. Aten <j.e....@gmail.com> написа: Forgive me if this off base, as I'm still a little fuzzy on the exact constraints of you problem... but, as stated, if you want to associate additional optional behavior and state with any given response that is constrained to the generated, just use the responses's pointer (to its struct) (if local an ephemeral is all you need, or a cryptographic hash of its contents, if you need to survive reboots) as the key into a separate map/sync.Map/ database. Standard Go does not move pointers; the GC is a non-moving collector by intent to enable sane interfacing with C code, and does very well at avoiding fragmentation (the motivation for compaction/copying GC) by using size classes instead. Since you can rely on the pointer not moving, just use it as a key to a map[*generated.Response]*EnrichedResponse to lookup your additional state on either the caller or responder side. On Monday, May 12, 2025 at 1:12:25 PM UTC+1 Alexander Shopov wrote: seems a very common assumption I would have guessed it is due to compatibility with C but https://pkg.go.dev/cmd/cgo#hdr-C_references_to_Go explicitly states: *Go struct types are not supported; use a C struct type. * So it may be completely at the decision of the implementer. The caveats in https://go.dev/wiki/cgo#common-pitfalls also sound too tricky. The pointer to the entire value stays alive while any pointer to one of its fields is alive That requires some cooperation from the GC. In the case where structs are reasonably small and they contain values rather than pointers this would probably hold. Thanx for even further details (I am writing down the offset trick) but I'd prefer to not stray too far away from usual conventions. Kind regards: al_shopov На пн, 12.05.2025 г. в 12:14 Axel Wagner <axel.wa...@googlemail.com> написа: On Mon, 12 May 2025 at 11:12, Alexander Shopov <a...@kambanaria.org> wrote: Is the equality of pointer to struct and pointer to its first member actually always guaranteed by Go lang? https://go.dev/ref/spec#Package_unsafe says The effect of converting between Pointer and uintptr is implementation-defined but I guess it is not the same thing. The sections on Struct types and Alignment do not give such guarantees. Perhaps not, though it seems a very common assumption and if an implementation violates it, I would expect that to break a bunch of code. You can always make sure, by accounting for a potential offset: https://go.dev/play/p/whdFALRG20P I guess that still makes the assumption, that a pointer to the entire value stays alive while any pointer to one of its field is alive. I'm not sure *that* is guaranteed either, but at least it's a weaker assumption. In the end it is not wise to use such a technique on a large code base shared by many developers. I will not be going this way but many thanx for the example it taught me something I did not know. Axel, you may use it for a 2nd part of Advanced Generics Patterns (nice presentation! I liked it) - Advanced Type Patterns (use types wrongly so we can learn how to make the wrong way the right way). I was hoping that there is a trick that I am missing. Like - a returned function can access state in a closure but I see no way to do it for methods. I definitely do not want to go the direction - since I want to add a method I should change the code generation, types (struct vs interface-s) to suit that small change. Kind regards: al_shopov На пн, 12.05.2025 г. в 7:12 Axel Wagner <axel.wa...@googlemail.com> написа: On Mon, 12 May 2025 at 07:05, Axel Wagner <axel.wa...@googlemail.com> wrote: A less evil, but still bad way, would be to store it in a global map: https://go.dev/play/p/ZHBJVCduf25 (note: I'm not sure this use of weak pointers is actually correct, I haven't used them yet). Okay, I'm pretty sure it's not correct. It should be possible to make this work somehow, but I'm a bit too lazy to try right now, especially as you shouldn't do it anyways. Otherwise, no. The state has to live somewhere. And really, both of these tricks are still really bad and you shouldn't do them. The best way is to respect the type system and if you want extra methods, you need a new type and if that needs extra state, that should be stored in the type. ----- Why would I want to do this? What am I trying to achieve? Basically there is a lot of generated code and I want to keep compatibility with it. Similar to the way Go embeds wider interfaces into narrower ones I want to be able to add methods to the generated code without having to change it. Change the code generator, and/or add additional code in an extra file. Hacking the language will cause you more pain, in the long run. Then - whenever some code calls the *Get* method on the server - based on the whether the returned value implements the *Enriched* interface or not and the value it returns - I can dispatch behavior. Kind regards: al_shopov -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/bbe6bcd8-e33c-41bf-868a-e498561c3e72n%40googlegroups.com <https://groups.google.com/d/msgid/golang-nuts/bbe6bcd8-e33c-41bf-868a-e498561c3e72n%40googlegroups.com?utm_medium=email&utm_source=footer> . -- You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/5gx3CCMfyJg/unsubscribe. To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/e823a08d-11c3-49d0-bd24-89eef57efc0an%40googlegroups.com <https://groups.google.com/d/msgid/golang-nuts/e823a08d-11c3-49d0-bd24-89eef57efc0an%40googlegroups.com?utm_medium=email&utm_source=footer> . -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/CAP6f5MmNXt-AuUPA4XGc1ZbbYqaMibtdzcxikx04KWNUpy21nA%40mail.gmail.com <https://groups.google.com/d/msgid/golang-nuts/CAP6f5MmNXt-AuUPA4XGc1ZbbYqaMibtdzcxikx04KWNUpy21nA%40mail.gmail.com?utm_medium=email&utm_source=footer> . -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/eeb31b6d-6253-4633-9c9b-006ef2df44e1n%40googlegroups.com <https://groups.google.com/d/msgid/golang-nuts/eeb31b6d-6253-4633-9c9b-006ef2df44e1n%40googlegroups.com?utm_medium=email&utm_source=footer> . -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/golang-nuts/bd7231c5-6be9-446e-a8c8-0a871a92cc99n%40googlegroups.com.
