Re: [go-nuts] An old problem: lack of priority select cases

Robert Engels Wed, 04 Sep 2019 07:06:31 -0700

No, please review this: https://play.golang.org/p/sPWrNhzRXKz

There is no need to use the lock in Read()

Encapsulation in this case means, the user of MultiWriterIntChan does not know it uses locks behind the scenes, nor does it need to know.

Due to limitations of the playground, the code referenced above runs serially. The sleeps are ONLY in there to demonstrate that early termination works - they are not needed for correctness.

That the writer does not unlock unless the channel is read is a limitation of unbuffered channels in Go - you could get around this by using a select and default. Closing a channel with waiting writers will cause a panic. Your technique is incorrect in my mind because...

The Close() will read from the channel, causing other writes - that the Producer thought were processed to be discarded. This is incorrect behavior in most use cases (think if the Producer was generating log messages).

For the version I posted, every write that completes will be handled by a consumer.

-----Original Message-----
From: Leo Lara
Sent: Sep 4, 2019 8:24 AM
To: Robert Engels
Cc: golang-nuts
Subject: Re: [go-nuts] An old problem: lack of priority select cases

By encapsulation? Are you talking about https://play.golang.org/p/YOwuYFiqtlf or other version of the code?

As I think someone already said, there is a need in the method Read for a lock. And the read from the channel is done from that method. With your pattern, the user of the hypothetical component reading from the channel needs to hold a lock.

Perhaps you do not need to check of c.closed as the closed channel with signal it already with the "ok" result. In that case you wouldn't need a lock in the reader, and it would be transparent for the reader.

Another drawback of this implementations is that the writer does not unlock unless the channel is read. Perhaps in some situations this does not matter, but this also implies that the Close method will not exit until they are read, plus the problem of not being fair, that new writer goroutines could go forward while Close is blocked in the Write mutex, as you block writing on a channel while holding a read lock that increases the chances of unfairness trying to get the write lock. This is a smaller problem in mine as the lock is hold for much shorter time, and I do not block inside the read lock.

I have seen several alternatives here, but still not a critisism of why those implementations would be preferable over mine. The only thing I see in the alternatives is that they combine less different concurrent idioms. Mine needs a lock, a waitgroup and "closing" channel but it seems to have better external qualities: transparency, fairness, performance, writer can spawn any time...

On Wed, Sep 4, 2019 at 1:22 PM Robert Engels <reng...@ix.netcom.com> wrote:
As for the readers being aware of a lock, that is not true either, as the lock is hidden by encapsulation. (The last and best version doesn’t even need the lock on the readers)

On Sep 4, 2019, at 4:55 AM, Leo Lara <l...@leopoldolara.com> wrote:

You should read up on how a RWLock works.

I am not going to answer to that ;-)

About this code:

https://play.golang.org/p/YOwuYFiqtlf

1. I wouldn't say that there are several goroutines writing, as there are several goroutines block in a mutex. Some might find this not important, although there are some people that have looked into this and think it is less efficient https://opensource.com/article/18/7/locks-versus-channels-concurrent-go ; I think it has to do on how the scheduler changes to another goroutine when the goroutine is lock on a channel.
2. More importantly IMHO, in your example the reader has to be aware that there is a lock. Hence, it is not "transparent" in the sense that the reader has to share something with the writer

The Sleep(1) in the producer is only to add some delay to demonstrate it gets terminated before the desired number of iterations.

On Aug 29, 2019, at 12:13 AM, Leo Lara <l...@leopoldolara.com> wrote:

Hi Robert,

To put you in context, it all started when I read https://go101.org/article/channel-closing.html , that said that it is impossible or at least you shouldn't close a channel that is being written by several goroutines. Then I wrote this article with my solution https://dev.to/leolara/closing-a-go-channel-written-by-several-goroutines-52j2 also in https://medium.com/@leolara/closing-a-go-channel-written-by-several-goroutines-eba3a6c9404b I then created this issue https://github.com/go101/go101/issues/132 and from there this topic was created by T L.

Your example does not have several goruitnes writing so I think it is a different problem. Perhaps that simple lock would work with several goroutines, but I think there would be more contention with this lock.

Anyway, I think I have already an optimisation to my code, I think using a RW lock, if I put the "Add(1)" in a read lock and the wait in a Write lock it might work better. The race condition that my lock prevents is only related when an "Add" and a "Wait" run concurrently, several "Add" can run concurrently.

On Thursday, August 29, 2019 at 4:05:06 AM UTC+2, robert engels wrote:
Here is a version using RWLock https://play.golang.org/p/YOwuYFiqtlf

It won’t run correctly in the playground because it terminates when all routines are asleep - which happens during the test (not sure why it does this, as sleeping is different than a deadlock).

It is probably less efficient, and less orderly than the other example using WaitGroup but you get the idea I hope. It forcibly terminates the writers before they complete by design.

On Aug 28, 2019, at 4:09 PM, Michel Levieux <m.le...@capitaldata.fr> wrote:

One should also be careful regarding the conceptual demands he or she is making.
Having a shared resource (that is complex enough that it cannot be atomically accessed or modified) means essentially that "having multiple writers being transparent to the readers", fundamentally, is not possible.

From the moment itself when such a resource is shared, there must be some sort of mecanism (that one using resources atomically usable) that ensures the integrity of it.
Maybe what you're talking about is having it transparent in terms of code, in which case we both agree, but if you're looking for something transparent in essence, as in performance, logical construction and all the rest, I think there is a misunderstanding here: even if it was added in the language, there would be many many things going on under the hood, as it is already (and cannot really be otherwise) for channel use alone.

As for the priority using selects, I think it's more of something to be dealt with on the "user-side". There are many kinds of priority in general, and trying to implement something in the language itself would IMO either be too specific compared to the nessecary time to do so or it would probably have a huge overhead on the "classical' use case of the select construct.
+ the fact that it is apparently already possible using RWMutexes.

Le mer. 28 août 2019 à 22:37, Marcin Romaszewicz <mar...@gmail.com> a écrit :
Think of a channel as existing for the lifetime of a particular data stream, and not have it be associated with either producer or consumer. Here's an example:

https://play.golang.org/p/aEAXXtz2X1g

The channel here is closed after all producers have exited, and all consumers continue to run until the channel is drained of data.

The producers are managed by something somewhere in your code - and that is the scope at which it makes sense to create channel ownership. I've used a waitgroup to ensure that the channel is closed after all producers exit, but you can use whatever barrier construct you want.

Even if you must have a channel per producer, you can safely close the producer side, without notifying the downstream about this. The example early in the thread uses multiple channels, with one channel being used to signal that the producers should exit. Channels aren't really the right model for this, you want a thread safe flag of some sort. For example:

var exitFlag uint64
func producer(chan data int, wg *sync.WaitGroup) {
    defer wg.Done()
    for {
        shouldExit := atomic.LoadUint64(&exitFlag)
        if shouldExit == 1 {
             return
        }
        chan <- rand.Intn(100)
    }
}

Here's 10 producers and 3 consumers sharing a channel and closing it safely upon receiving an exit flag:
https://play.golang.org/p/RiKi1PGVSvF

-- Marcin

On Wed, Aug 28, 2019 at 11:29 AM Leo Lara <l...@leopoldolara.com> wrote:
I do not think priority select is *necessary*, it could be a nice addition if the performance does not change.

On Wednesday, August 28, 2019 at 8:27:36 PM UTC+2, Leo Lara wrote:
Hi Robert,

From the article: """To bound more the problem, in my case, you control the writers but not the readers"""

So what I was trying to do was to be able to close, with mutiple writers, while being transparent for the readers. The readers only need to read as usual form the channel.

For example, if you want to write a library where the user just reads from a channel, this is an approach I found where the user of the lirbary deos nto have to do anything special. Of course, there might be another solution, but if you need to modify the reader we are talking about a different problem.

Cheers!!

On Wednesday, August 28, 2019 at 7:17:24 PM UTC+2, Robert Engels wrote:
A better solution is to wrap the writes using a RWLock, grab the read lock for writing, and the Write lock for closing. Pretty simple.

Just encapsulate it all in a MultiWriterChannel struct - generics would help here :)

-----Original Message-----
From: Leo Lara
Sent: Aug 28, 2019 11:24 AM
To: golang-nuts
Subject: [go-nuts] Re: An old problem: lack of priority select cases

This is connected with my article: https://dev.to/leolara/closing-a-go-channel-written-by-several-goroutines-52j2

I think there I show it is possible to workaround that limitation using standard Go tools. Of course, the code would be simple with priority select, but also perhaps select would become less efficient.

On Wednesday, August 28, 2019 at 6:06:33 PM UTC+2, T L wrote:
The old thread: https://groups.google.com/forum/#!topic/golang-nuts/ZrVIhHCrR9o

Go channels are flexible, but in practice, I often encountered some situations in which channel are hard to use.
Given an example:

import "math/rand"

type Producer struct {
    data   chan int
    closed chan struct{}
}

func NewProducer() *Producer {
    p := &Producer {
        data:   make(chan int),
        closed: make(chan struct{}),
    }

    go p.run()

    return p
}

func (p *Produce) Stream() chan int {
    return p.data
}

func (p *Producer) run() {
    for {
        // If non-blocking cases are selected by their appearance order,
        // then the following slect block is a perfect use.
        select {
        case(0) <-p.closed: return
        case p.data <- rand.Int():
        }
    }
}

func (p *Produce) Clsoe() {
    close(p.closed)
    close(p.data)
}

func main() {
    p := NewProducer()
    for n := p.Stream() {
        // use n ...
    }
}

If the first case in the select block in the above example has a higher priority than the second one,
then coding will be much happier for the use cases like the above one.

In short, the above use case requires:
* for receivers, data streaming end is notified by the close of a channel.
* for senders, data will never be sent to closed channel.

But, as Go 1 doesn't support priority select cases, it is much tedious to implement the code
satisfying the above listed requirements. The final implementation is often very ugly and inefficient.

Does anyone else also experience the pain?

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golan...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/b284f880-034a-4721-8686-ef48d3e2c14c%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golan...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/aeb38a0a-8268-42d7-a8eb-ce5ef01c5380%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golan...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CA%2Bv29LvcUhUvrZb_8AKYWj0A%2Bqd5LKBPmbz-RVBb%3DJn_gNZE6w%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golan...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CANgi337s1Low95QvqJUAOTsqcVji7uMQ_jr%3DFftpt2uMz5_XSQ%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golan...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/e5f37679-bdfb-4da3-854e-fea35cf82cb7%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/8893737c-148e-4d64-9a0c-cd72046cd0d1%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAK9c6k42QraquPapY%2BXR_wH7cCLrE3-DrAEZJsbUa%3DVDa2-sFg%40mail.gmail.com.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/308658222.2060.1567605948054%40wamui-gaston.atl.sa.earthlink.net.

Re: [go-nuts] An old problem: lack of priority select cases

Reply via email to