Re: [go-nuts] Upgradable RLock

[burak serdar]

On Sun, Jan 29, 2023 at 7:34 PM Diego Augusto Molina <
diegoaugustomol...@gmail.com> wrote:

> From times to times I write a scraper or some other tool that would
> authenticate to a service and then use the auth result to do stuff
> concurrently. But when auth expires, I need to synchronize all my
> goroutines and have a single one do the re-auth process, check the status,
> etc. and then arrange for all goroutines to go back to work using the new
> auth result.
>
> To generalize the problem: multiple goroutines read a cached value that
> expires at some point. When it does, they all should block and some I/O
> operation has to be performed by a single goroutine to renew the cached
> value, then unblock all other goroutines and have them use the new value.
>
> I solved this in the past in a number of ways: having a single goroutine
> that handles the cache by asking it for the value through a channel, using
> sync.Cond (which btw every time I decide to use I need to carefully re-read
> its docs and do lots of tests because I never get it right at first). But
> what I came to do lately is to implement an upgradable lock and have every
> goroutine do:
>


So how about using sync.Once:

type CacheEntry struct {
  Auth AuthInfo
  once sync.Once
}

Cache would be a map[CacheKey]*CacheEntry. When AuthInfo expires, you
simply replace it with a new one:

func (c *Cache) Expire(key CacheKey) {
  c.Lock()
  defer c.Unlock()
  c.cache[key]={}
}

func (c *Cache) Get(key CacheKey) Auth {
   c.RLock()
   entry:=c.cache[key]
c.RUnlock()
if entry==nil {
  c.Lock()
  entry=c.cache[key]
  if entry==nil {
 c.cache[key]={}
  }
  c.UnLock()
   }
entry.Do(func() {
// Authenticate
})
return entry.Auth
}



>
> **
> func (x implem) getProtectedValue() (someType, error) {
> // acquires a read lock that can be upgraded
> lock := x.upgradableLock.UpgradableRLock()
> // the Unlock method of the returned lock does the right thing
> // even if we later upgrade the lock
> defer lock.Unlock()
>
> // here we have read access to x.protectedValue
>
> // if the cached value is no longer valid, upgrade the lock
> // and update it
> if !isValid(x.protectedValue) && lock.TryUpgrade() {
>   // within this if, we know we *do* have write access
>   // to x.protectedValue
> x.protectedValue, x.protectedValueError = newProtectedValue()
> }
>
> // here we can only say everyone has read access to x.protectedValue
> // (the goroutine that got the lock upgrade could still write
> // here but as this code is shared, we should check the result
> // of the previous lock.TryUpgrade() again)
>
> return x.protectedValue, x.protectedValueError
> }
> **
>
> The implementation is quite simple:
> **
> // Upgradable implements all methods of sync.RWMutex, plus a new
> // method to acquire a read lock that can optionally be upgraded
> // to a write lock.
> type Upgradable struct {
> readers sync.RWMutex
> writers sync.RWMutex
> }
>
> func (m *Upgradable) RLock() { m.readers.RLock() }
> func (m *Upgradable) TryRLock() bool { return m.readers.TryRLock() }
> func (m *Upgradable) RUnlock() { m.readers.RUnlock() }
> func (m *Upgradable) RLocker() sync.Locker { return m.readers.RLocker() }
>
> func (m *Upgradable) Lock() {
> m.writers.Lock()
> m.readers.Lock()
> }
>
> func (m *Upgradable) TryLock() bool {
> if m.writers.TryLock() {
> if m.readers.TryLock() {
> return true
> }
> m.writers.Unlock()
> }
> return false
> }
>
> func (m *Upgradable) Unlock() {
> m.readers.Unlock()
> m.writers.Unlock()
> }
>
> // UpgradableRLock returns a read lock that can optionally be
> // upgraded to a write lock.
> func (m *Upgradable) UpgradableRLock() *UpgradableRLock {
> m.readers.RLock()
> return {
> m:  m,
> unlockFunc: m.RUnlock,
> }
> }
>
> // UpgradableRLock is a read lock that can be upgraded to a write
> // lock. This is acquired by calling (*Upgradable).
> // UpgradableRLock().
> type UpgradableRLock struct {
> mu sync.Mutex
> m  *Upgradable
> unlockFunc func()
> }
>
> // TryUpgrade will attempt to upgrade the acquired read lock to
> // a write lock, and return whether it succeeded. If it didn't
> // succeed then it will block until the goroutine that succeeded
> // calls Unlock(). After unblocking, the read lock will still be
> // valid until calling Unblock().
> //
> // TryUpgrade panics if called more than once or if called after
> // Unlock.
> func (u *UpgradableRLock) TryUpgrade() (ok bool) {
> u.mu.Lock()
> defer u.mu.Unlock()
>
> if u.m == nil {
> panic("TryUpgrade can only be called once and not after Unlock")
> }
>
> if ok = u.m.writers.TryLock(); ok {
> u.m.readers.RUnlock()
> u.m.readers.Lock()
> u.unlockFunc = u.m.Unlock
>
> } else {
> u.m.readers.RUnlock()
> 

[go-nuts] Upgradable RLock

[Diego Augusto Molina]

>From times to times I write a scraper or some other tool that would 
authenticate to a service and then use the auth result to do stuff 
concurrently. But when auth expires, I need to synchronize all my 
goroutines and have a single one do the re-auth process, check the status, 
etc. and then arrange for all goroutines to go back to work using the new 
auth result.

To generalize the problem: multiple goroutines read a cached value that 
expires at some point. When it does, they all should block and some I/O 
operation has to be performed by a single goroutine to renew the cached 
value, then unblock all other goroutines and have them use the new value.

I solved this in the past in a number of ways: having a single goroutine 
that handles the cache by asking it for the value through a channel, using 
sync.Cond (which btw every time I decide to use I need to carefully re-read 
its docs and do lots of tests because I never get it right at first). But 
what I came to do lately is to implement an upgradable lock and have every 
goroutine do:

**
func (x implem) getProtectedValue() (someType, error) {
// acquires a read lock that can be upgraded
lock := x.upgradableLock.UpgradableRLock()
// the Unlock method of the returned lock does the right thing
// even if we later upgrade the lock
defer lock.Unlock()

// here we have read access to x.protectedValue

// if the cached value is no longer valid, upgrade the lock
// and update it
if !isValid(x.protectedValue) && lock.TryUpgrade() {
  // within this if, we know we *do* have write access
  // to x.protectedValue
x.protectedValue, x.protectedValueError = newProtectedValue()
}

// here we can only say everyone has read access to x.protectedValue
// (the goroutine that got the lock upgrade could still write
// here but as this code is shared, we should check the result
// of the previous lock.TryUpgrade() again)

return x.protectedValue, x.protectedValueError
}
**

The implementation is quite simple:
**
// Upgradable implements all methods of sync.RWMutex, plus a new
// method to acquire a read lock that can optionally be upgraded
// to a write lock.
type Upgradable struct {
readers sync.RWMutex
writers sync.RWMutex
}

func (m *Upgradable) RLock() { m.readers.RLock() }
func (m *Upgradable) TryRLock() bool { return m.readers.TryRLock() }
func (m *Upgradable) RUnlock() { m.readers.RUnlock() }
func (m *Upgradable) RLocker() sync.Locker { return m.readers.RLocker() }

func (m *Upgradable) Lock() {
m.writers.Lock()
m.readers.Lock()
}

func (m *Upgradable) TryLock() bool {
if m.writers.TryLock() {
if m.readers.TryLock() {
return true
}
m.writers.Unlock()
}
return false
}

func (m *Upgradable) Unlock() {
m.readers.Unlock()
m.writers.Unlock()
}

// UpgradableRLock returns a read lock that can optionally be
// upgraded to a write lock.
func (m *Upgradable) UpgradableRLock() *UpgradableRLock {
m.readers.RLock()
return {
m:  m,
unlockFunc: m.RUnlock,
}
}

// UpgradableRLock is a read lock that can be upgraded to a write
// lock. This is acquired by calling (*Upgradable).
// UpgradableRLock().
type UpgradableRLock struct {
mu sync.Mutex
m  *Upgradable
unlockFunc func()
}

// TryUpgrade will attempt to upgrade the acquired read lock to
// a write lock, and return whether it succeeded. If it didn't
// succeed then it will block until the goroutine that succeeded
// calls Unlock(). After unblocking, the read lock will still be
// valid until calling Unblock().
//
// TryUpgrade panics if called more than once or if called after
// Unlock.
func (u *UpgradableRLock) TryUpgrade() (ok bool) {
u.mu.Lock()
defer u.mu.Unlock()

if u.m == nil {
panic("TryUpgrade can only be called once and not after Unlock")
}

if ok = u.m.writers.TryLock(); ok {
u.m.readers.RUnlock()
u.m.readers.Lock()
u.unlockFunc = u.m.Unlock

} else {
u.m.readers.RUnlock()
u.m.writers.RLock()
u.unlockFunc = u.m.writers.RUnlock
}

u.m = nil

return
}

// Unlock releases the lock, whether it was a read lock or a write
// lock acquired by calling Upgrade.
//
// Unlock panics if called more than once.
func (u *UpgradableRLock) Unlock() {
u.mu.Lock()
defer u.mu.Unlock()

if u.unlockFunc == nil {
panic("Unlock can only be called once")
}

u.unlockFunc()
u.unlockFunc = nil
u.m = nil
}
**

I obviously try to avoid using it for other than protecting values that 
require potentially long I/O operations (like in my case re-authenticating 
to a web service) and also having a lot of interested goroutines. The good 
thing is that most of the time this pattern only requires one RLock, but 
the (*UpgradableRLock).Unlock requires an additional lock/unlock to prevent 
misusage of the upgradable lock (I could potentially get rid of it but 
preferred to keep it in the side 

Re: [go-nuts] Creating and Linking to Shared Library Version of Go Runtime?

[TheDiveO]

Are the std/runtime .so's even versioned? How do you manage that?

Every time I'm feeling like finally being in $PARADISE out of the .so 
dependency $VERYVERYHOTPLACE there comes along the demand to go back. Sigh. 
;) 

On Sunday, January 29, 2023 at 9:26:51 PM UTC+1 bobj...@gmail.com wrote:

> I'm glad to see this issue getting some discussion. I have 100+ smallish 
> utility programs written in Go, and each one consumes about 1.5 MB (precise 
> average: 1,867,844 bytes); my bin directory contains 100+ copies of the Go 
> runtime. Sadly, I mainly use Windows, and there seems to be no way to use 
> linked libraries in Go for Windows.
>
> My solution has been to rewrite many of my smallish Go programs in Python 
> (except those that really need Go's speed)  -- 10K each vs. 1.5M each disk 
> storage. For these  manually invoked utilities, the speed difference is 
> often not noticeable. And the number of source lines and overall program 
> complexity is reduced by roughly 30%. (Added benefit: the Python programs 
> are always properly indented, even without a "pyfmt" program :-)
>
> I *am* a Go fan, and I understand that Go's mission is for writing big 
> server programs, but it's too bad that the size of a small Go program 
> executable is *many* times larger than a small C program.
> On Sunday, January 29, 2023 at 9:49:17 AM UTC-8 jlfo...@berkeley.edu 
> wrote:
>
>> The discussion of SSD wear and tear is going way off my original 
>> question. I'm sorry I even mentioned it.
>> Can we drop it?
>>
>> I'm still interested in the answer to the original question, which we've 
>> made some progress in answering.
>> As of now the answer is yes, subject to some permission and performance 
>> problems. I hope the experts
>> can shed some light on these. 
>>
>> Cordially,
>> Jon Forrest
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/d8f98822-c541-4508-b24c-c4a5596b160an%40googlegroups.com.

Re: [go-nuts] Creating and Linking to Shared Library Version of Go Runtime?

[bobj...@gmail.com]

I'm glad to see this issue getting some discussion. I have 100+ smallish 
utility programs written in Go, and each one consumes about 1.5 MB (precise 
average: 1,867,844 bytes); my bin directory contains 100+ copies of the Go 
runtime. Sadly, I mainly use Windows, and there seems to be no way to use 
linked libraries in Go for Windows.

My solution has been to rewrite many of my smallish Go programs in Python 
(except those that really need Go's speed)  -- 10K each vs. 1.5M each disk 
storage. For these  manually invoked utilities, the speed difference is 
often not noticeable. And the number of source lines and overall program 
complexity is reduced by roughly 30%. (Added benefit: the Python programs 
are always properly indented, even without a "pyfmt" program :-)

I *am* a Go fan, and I understand that Go's mission is for writing big 
server programs, but it's too bad that the size of a small Go program 
executable is *many* times larger than a small C program.
On Sunday, January 29, 2023 at 9:49:17 AM UTC-8 jlfo...@berkeley.edu wrote:

> The discussion of SSD wear and tear is going way off my original question. 
> I'm sorry I even mentioned it.
> Can we drop it?
>
> I'm still interested in the answer to the original question, which we've 
> made some progress in answering.
> As of now the answer is yes, subject to some permission and performance 
> problems. I hope the experts
> can shed some light on these. 
>
> Cordially,
> Jon Forrest
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/3c691095-e0d8-4da5-8513-887af7b0e9b4n%40googlegroups.com.

Re: [go-nuts] Creating and Linking to Shared Library Version of Go Runtime?

[jlfo...@berkeley.edu]

The discussion of SSD wear and tear is going way off my original question. 
I'm sorry I even mentioned it.
Can we drop it?

I'm still interested in the answer to the original question, which we've 
made some progress in answering.
As of now the answer is yes, subject to some permission and performance 
problems. I hope the experts
can shed some light on these. 

Cordially,
Jon Forrest

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/ce90a56e-dae7-492c-b935-edf734f1dfe8n%40googlegroups.com.

Re: [go-nuts] Creating and Linking to Shared Library Version of Go Runtime?

[jake...@gmail.com]

This is pretty OT, and I am no expert, but the overwhelming consensus on 
the inter-tubes seems to be that *reading *from an SSD causes no 'wear' 
whatsoever. It is only writes and deletes that age an SSD. So this use case 
should not impact SSD life. 

But it is an interesting endeavor on its own. 
On Sunday, January 29, 2023 at 1:01:42 AM UTC-5 ren...@ix.netcom.com wrote:

> Shared dynamic libraries do reduce the “wear and tear” on an SSD.  The 
> binaries are loaded a single time and shared across processes - slightly 
> longer startup times for the dynamic linkage. 
>
> It is highly beneficial with large runtimes vs small standard library 
> usage in tiny utilities.  
>
> On Jan 28, 2023, at 10:46 PM, Kurtis Rader  wrote:
>
> 
>
> It does not surprise me that your shared run-time build takes more time 
> than the usual static build. The latter case is highly optimized while your 
> use case has probably been given very little consideration. I am also 
> confused by your argument for supporting linking against a shared Go 
> runtime library. You wrote earlier that the reason you want this is to 
> "reduce ware (sic) and tear on my SSD." I see no reason why linking against 
> a shared Go run-time library would reduce the "wear and tear" on your SSD. 
> I think your premise is flawed.
>
> On Sat, Jan 28, 2023 at 8:29 PM jlfo...@berkeley.edu  
> wrote:
>
>> Thanks for the reply. I had mixed results.
>>
>> On Fedora 37, go version go1.19.4 linux/amd64, in /usr/local/go/src as 
>> root I ran
>>
>> go install -buildmode=shared std
>>
>> That seemed to work.
>>
>> In my normal working directory, as me, not root, I ran
>>
>> go build -linkshared *.go
>>
>> (I do a regular build in this directory by just running "go build *.go")
>>
>> That resulted in a whole bunch of error messages of the style
>>
>> go build internal/goarch: copying /tmp/go-build2481653269/b006/_pkg_.a: 
>> open /usr/local/go/pkg/linux_amd64_dynlink/internal/goarch.a: permission 
>> denied
>>
>> So I became root and ran go build -linkshared *.go again.
>>
>> This time it worked!! The result was 83584 byte binary, whereas the 
>> binary produced 
>> by the standard method is 4186764 bytes. That's a great savings!!! The 
>> small binary seemed
>> to work fine.
>>
>> Just for yuks, I tried building my program as me again (not root). I got 
>> permission error messages
>> again, but now they look like
>>
>> open /usr/local/go/pkg/linux_amd64_dynlink/archive/tar.shlibname: 
>> permission denied
>>
>> There are 238 such lines.
>>
>> There's another problem. Unlike what I would expect, it takes *longer* to 
>> build the shared version
>> than the static version.
>>
>> As root
>>
>> [root@localhost]# time go build *.go
>>
>> real0m0.298s
>> user0m0.346s
>> sys 0m0.091s
>>
>> [root@localhost]# time go build -linkshared *.go
>>
>> real0m1.441s
>> user0m1.193s
>> sys 0m0.325s
>>
>> That doesn't seem right.
>>
>> Any advice?
>>
>> Cordially,
>> Jon Forrest
>>
>> On Saturday, January 28, 2023 at 7:51:25 PM UTC-8 Ian Lance Taylor wrote:
>>
>>> On Sat, Jan 28, 2023 at 11:27 AM jlfo...@berkeley.edu 
>>>  wrote: 
>>> > 
>>> > For people like me who have no intention of ever distributing a Go 
>>> executable, I wonder 
>>> > if it would be useful, and possible, to link to a shared library 
>>> version of the Go 
>>> > Runtime. This would make my binaries much smaller and would reduce 
>>> ware and 
>>> > tear on my SSD. 
>>> > 
>>> > Of course, this presumes that such a shared library could be created 
>>> in the first place. 
>>> > 
>>> > I did a quick Google and I didn't find this issue being previously 
>>> discussed. 
>>> > 
>>> > Comments? 
>>>
>>> At least on Linux systems it should work to run "go install 
>>> -buildmode=shared std" and then to build other Go programs with 
>>> "-linkshared". 
>>>
>>> Ian 
>>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golang-nuts...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/4dc94780-06e6-4aa3-a9b1-64b97dd85a5en%40googlegroups.com
>>  
>> 
>> .
>>
>
>
> -- 
> Kurtis Rader
> Caretaker of the exceptional canines Junior and Hank
>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts...@googlegroups.com.
>
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/CABx2%3DD-SBB%3DvFGYUGaXpf-8m6RH4HK4xEKjGUdMXMwjK90NMPg%40mail.gmail.com
>  
>