[go-nuts] Re: Async process state snapshotting in Golang

[jul . semaan]

As a follow-up, I managed to get everything to work as I wanted with 
forking but I got a major issue..

The golang runtime mutexes are invalidated by the forking, that means that 
anything in the runtime that needs a mutex is broken so garbage collection 
had to be disabled and additional memory allocation that required the heap 
to be enlarged was also broken.
When a runtime operation was requiring a mutex, it would lock forever and 
since the process was single threaded after the fork, then having the main 
thread locked would make the process hang forever.

Although this didn't happen in all cases (would succeed 9/10 times with 
memory pre-allocation), it isn't suitable for me and will certainly go 
beyond my knowledge of Golang when I hit those issues in production.

So, instead of fighting against Golang and most likely lose the battle, I 
decided to go with the fine grain locking using the following logic:
- RLock the map and grab a copy of its keys
- Iterate through all the keys
-- RLock the map to get the pointer of the struct associated to the current 
key and release it right after
-- Check that the pointer is not pointing to nil (keys can disapear in my 
case)
-- RLock the structure
-- Serialize the structure
-- Release the RLock on the structure
- Write the serialized structures to the disk

I'll be optimizing it as well to write to the disk while I'm serializing 
each individual struct to same some more memory but I found it to be 
effective and I already had part of the plumbing in place to make this lock 
strategy work

I would have found it nice though to be able to make the Redis strategy 
work but I do realize now that the candies that come with Golang (GC + 
heap) have some limitations when working with very low level system calls 
(at least this is my understanding of it)

Anyway, thanks to everybody for the input

- Julien

On Tuesday, 30 January 2018 07:36:26 UTC-5, Julien Semaan wrote:
>
> I believe copying the map containing structs and not pointers would make 
> it consume twice the amount of memory which isn't desirable in my case.
>
> - Julien
>
> On Tuesday, 30 January 2018 00:09:21 UTC-5, Tamás Gulácsi wrote:
>>
>> Do you really need pointers tonyour structs in the map? Because that 
>> causes the headache here: if no pointers, then simply rlock-ing the map, 
>> copying / serializing would be enough.
>>
>> Could even have two maps, each gets every modification, except when 
>> copying, when the secondary is serialized, thn updated with the accumulated 
>> changes. With a goroutine managing the secondary, executing commands 
>> (update, delete, marshal) from a buffered channel, this wouldn't uterfere 
>> with the main map.
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Async process state snapshotting in Golang

[jul . semaan]

I believe copying the map containing structs and not pointers would make it 
consume twice the amount of memory which isn't desirable in my case.

- Julien

On Tuesday, 30 January 2018 00:09:21 UTC-5, Tamás Gulácsi wrote:
>
> Do you really need pointers tonyour structs in the map? Because that 
> causes the headache here: if no pointers, then simply rlock-ing the map, 
> copying / serializing would be enough.
>
> Could even have two maps, each gets every modification, except when 
> copying, when the secondary is serialized, thn updated with the accumulated 
> changes. With a goroutine managing the secondary, executing commands 
> (update, delete, marshal) from a buffered channel, this wouldn't uterfere 
> with the main map.
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Async process state snapshotting in Golang

[jul . semaan]

My main goal is to avoid complex granular locking inside the data structure 
since writes are very common in the map (whether its adding items or 
modifying the actual item).
My idea was to implement MarshalJSON for my map and then while creating the 
JSON I'd be holding locks at the appropriate times.
That is not my preferred solution though because of the locking.

As for forking...

I believe forking on Linux is now very efficient due to copy-on-write so 
the cost should be minimal.

I got some kind of basic prototype working but I need to figure out if it 
will stay memory and performance efficient when my parent starts writing to 
the map. 
Since its a map of pointers, I think the footprint will be minimal.

When I get my prototype done, I'll likely post it here to get some input on 
whether or not it does make sense

- Julien

On Monday, 29 January 2018 10:39:35 UTC-5, matthe...@gmail.com wrote:
>
> Can you describe your performance needs in more detail?
>
> I missed the need for the struct values in the copy (obviously needed to 
> encode to JSON). Going back to locks for a moment, my thought now is that a 
> sync.RWMutex on this map would have an RLock taken by every struct 
> read/write and then the Lock would be taken for the map copy which would 
> return a map to copied structs instead of a map to pointers. Wouldn’t a 
> fork lock the entire process while a larger copy happens anyway?
>
> Where I’d start otherwise is adding additional data. Perhaps a second map 
> could help with performance here.
>
> Matt
>
> On Monday, January 29, 2018 at 6:44:17 AM UTC-6, Julien Semaan wrote:
>>
>> Hi,
>>
>> Thanks for all the input.
>>
>> Unfortunately, as Tamás and Jake pointed out, I can't simply grab a copy 
>> of the map that has the same pointers since I'll need to be locking the 
>> structure everytime I'm copying it.
>> So that means Matt's idea wouldn't really work for my use-case
>>
>> I was really hoping to have something similar to the fork behavior where 
>> I could be working with a memory copy of the process in order to be able to 
>> persist everything without holding a single lock either on the map or on 
>> the actual structure.
>>
>> - Julien
>>
>> On Friday, 26 January 2018 15:44:46 UTC-5, Julien Semaan wrote:
>>>
>>> Hi,
>>>
>>> I'm currently tackling a head scratching problem (at least for me).
>>>
>>> First, I'll start by explaining my main goal:
>>> I have an in memory 'map[string]*SomeStructPointer' that I want to JSON 
>>> encode and write to a file.
>>> Now since I want to prevent concurrent access to the map, I need to 
>>> lock-down access to it during the JSON encoding.
>>> I want to avoid that locking or have it held for the smallest amount of 
>>> time.
>>>
>>> Next, my idea:
>>> Fork the process, leveraging the copy-on-write, JSON encode the good 
>>> stuff, write it to a file, then exit the child process.
>>> During that time, I wouldn't need to lock-down access to the map since 
>>> the forked process would have its own copy of the memory
>>> I pretty much wanted to do what Redis does, but in Golang, and for a 
>>> multi-threaded process which I now learned is not really possible (please 
>>> correct me if I'm wrong I'd be happy).
>>>
>>> So, with that failed idea behind me, I'm looking at how I could 
>>> accomplish this in Golang while being memory efficient.
>>>
>>> I'm opened to a lot of ideas, but I'd consider myself intermediate in 
>>> Golang and beginner in C (if the answer involves CGO) so I might need 
>>> detailed examples if we get in deep low level stuff.
>>>
>>> I'd also like to have something that is pure Golang code since I'll be 
>>> cross compiling this on multiple architectures using the go compiler.
>>>
>>> Obviously, if more details are needed to answer the question, I'd be 
>>> happy to provide them.
>>>
>>> Thanks in advance!
>>>
>>> - Julien
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Async process state snapshotting in Golang

[matthewjuran]

Can you describe your performance needs in more detail?

I missed the need for the struct values in the copy (obviously needed to 
encode to JSON). Going back to locks for a moment, my thought now is that a 
sync.RWMutex on this map would have an RLock taken by every struct 
read/write and then the Lock would be taken for the map copy which would 
return a map to copied structs instead of a map to pointers. Wouldn’t a 
fork lock the entire process while a larger copy happens anyway?

Where I’d start otherwise is adding additional data. Perhaps a second map 
could help with performance here.

Matt

On Monday, January 29, 2018 at 6:44:17 AM UTC-6, Julien Semaan wrote:
>
> Hi,
>
> Thanks for all the input.
>
> Unfortunately, as Tamás and Jake pointed out, I can't simply grab a copy 
> of the map that has the same pointers since I'll need to be locking the 
> structure everytime I'm copying it.
> So that means Matt's idea wouldn't really work for my use-case
>
> I was really hoping to have something similar to the fork behavior where I 
> could be working with a memory copy of the process in order to be able to 
> persist everything without holding a single lock either on the map or on 
> the actual structure.
>
> - Julien
>
> On Friday, 26 January 2018 15:44:46 UTC-5, Julien Semaan wrote:
>>
>> Hi,
>>
>> I'm currently tackling a head scratching problem (at least for me).
>>
>> First, I'll start by explaining my main goal:
>> I have an in memory 'map[string]*SomeStructPointer' that I want to JSON 
>> encode and write to a file.
>> Now since I want to prevent concurrent access to the map, I need to 
>> lock-down access to it during the JSON encoding.
>> I want to avoid that locking or have it held for the smallest amount of 
>> time.
>>
>> Next, my idea:
>> Fork the process, leveraging the copy-on-write, JSON encode the good 
>> stuff, write it to a file, then exit the child process.
>> During that time, I wouldn't need to lock-down access to the map since 
>> the forked process would have its own copy of the memory
>> I pretty much wanted to do what Redis does, but in Golang, and for a 
>> multi-threaded process which I now learned is not really possible (please 
>> correct me if I'm wrong I'd be happy).
>>
>> So, with that failed idea behind me, I'm looking at how I could 
>> accomplish this in Golang while being memory efficient.
>>
>> I'm opened to a lot of ideas, but I'd consider myself intermediate in 
>> Golang and beginner in C (if the answer involves CGO) so I might need 
>> detailed examples if we get in deep low level stuff.
>>
>> I'd also like to have something that is pure Golang code since I'll be 
>> cross compiling this on multiple architectures using the go compiler.
>>
>> Obviously, if more details are needed to answer the question, I'd be 
>> happy to provide them.
>>
>> Thanks in advance!
>>
>> - Julien
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Async process state snapshotting in Golang

[jul . semaan]

Hi,

Thanks for all the input.

Unfortunately, as Tamás and Jake pointed out, I can't simply grab a copy of 
the map that has the same pointers since I'll need to be locking the 
structure everytime I'm copying it.
So that means Matt's idea wouldn't really work for my use-case

I was really hoping to have something similar to the fork behavior where I 
could be working with a memory copy of the process in order to be able to 
persist everything without holding a single lock either on the map or on 
the actual structure.

- Julien

On Friday, 26 January 2018 15:44:46 UTC-5, Julien Semaan wrote:
>
> Hi,
>
> I'm currently tackling a head scratching problem (at least for me).
>
> First, I'll start by explaining my main goal:
> I have an in memory 'map[string]*SomeStructPointer' that I want to JSON 
> encode and write to a file.
> Now since I want to prevent concurrent access to the map, I need to 
> lock-down access to it during the JSON encoding.
> I want to avoid that locking or have it held for the smallest amount of 
> time.
>
> Next, my idea:
> Fork the process, leveraging the copy-on-write, JSON encode the good 
> stuff, write it to a file, then exit the child process.
> During that time, I wouldn't need to lock-down access to the map since the 
> forked process would have its own copy of the memory
> I pretty much wanted to do what Redis does, but in Golang, and for a 
> multi-threaded process which I now learned is not really possible (please 
> correct me if I'm wrong I'd be happy).
>
> So, with that failed idea behind me, I'm looking at how I could accomplish 
> this in Golang while being memory efficient.
>
> I'm opened to a lot of ideas, but I'd consider myself intermediate in 
> Golang and beginner in C (if the answer involves CGO) so I might need 
> detailed examples if we get in deep low level stuff.
>
> I'd also like to have something that is pure Golang code since I'll be 
> cross compiling this on multiple architectures using the go compiler.
>
> Obviously, if more details are needed to answer the question, I'd be happy 
> to provide them.
>
> Thanks in advance!
>
> - Julien
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Async process state snapshotting in Golang

[jake6502]

A couple of comments. 

This is probably obvious, but I wanted to point out that matthe...'s code 
would require *all *access to the map to be synchronized using the mutex. 
It's a simple and effective solution. One downside to his code is that it 
holds the lock for the duration of the Copy(). This may be unacceptable 
depending on the size of the map and your latency requirements. If the map 
is very, very large, this could block a writer for an indeterminate amount 
of time. 

If your map met the requirements for sync.Map 
, that struct contains a Range() method 
that allows for concurrent access. Depending on the requirements for the 
temporal strictness of the "snapshot", you could then use that function to 
create a copy without blocking other goroutines which might be attempting 
to write to the map. 

There are other, more complicated way that the Copy() lock time could be 
made constant, but they are significantly complicated. 



On Friday, January 26, 2018 at 9:56:38 PM UTC-5, matthe...@gmail.com wrote:
>
> Why not this?
>
> type StructMap map[string]*SomeStruct
>
> type SyncStructMap struct {
> *sync.Mutex // maybe change to *sync.RWMutex if there are mixed 
> read/write synced operations
> StructMap
> }
>
> func (a SyncStructMap) Copy() StructMap {
> out := make(StructMap)
> a.Lock()
> for key, value := range a.StructMap {
> out[key] = value
> }
> a.Unlock()
> return out
> }
>
> // call with go keyword to make it execute concurrently
> func (a SyncStructMap) EncodeAndWrite() {
> c := a.Copy()
> // encode c then write to file
> }
>
> Matt
>
> On Friday, January 26, 2018 at 4:02:09 PM UTC-6, Tamás Gulácsi wrote:
>>
>> Do those *SomeStruct change during map serialization, or only the map 
>> (new keys / delete keys)?
>> If they do, you'll have to lock them too, not just the map access!
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] Re: Async process state snapshotting in Golang

[matthewjuran]

Why not this?

type StructMap map[string]*SomeStruct

type SyncStructMap struct {
*sync.Mutex // maybe change to *sync.RWMutex if there are mixed 
read/write synced operations
StructMap
}

func (a SyncStructMap) Copy() StructMap {
out := make(StructMap)
a.Lock()
for key, value := range a.StructMap {
out[key] = value
}
a.Unlock()
return out
}

// call with go keyword to make it execute concurrently
func (a SyncStructMap) EncodeAndWrite() {
c := a.Copy()
// encode c then write to file
}

Matt

On Friday, January 26, 2018 at 4:02:09 PM UTC-6, Tamás Gulácsi wrote:
>
> Do those *SomeStruct change during map serialization, or only the map (new 
> keys / delete keys)?
> If they do, you'll have to lock them too, not just the map access!
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.