Re: [go-nuts] Channels may not be the best solution in Go

[Rob Pike]

>
>
> Not all programs benefit from concurrency. Writing concurrent code for
> essentially sequential programs will not benefit from multiple cores,
> like generating prime numbers. Do not forget that concurrency includes
> overhead for context switch and memory barriers. Using channels in a
> sequential program is I think misuse of channels. There is no
> performance gain in having a sequence of goroutines, each waiting for
> the previous one the complete.
>

You are conflating concurrency and parallelism, or perhaps the problem and
the solution. Some programs (solutions) benefit from expression as
concurrent code, others do not. Whether performance improves depends on how
many cores you have as well as the inherent nature of the problem. Remember
there is no such thing as parallelism on a single core, yet the idea of
using concurrency in programs arose when multicore was but a dream.

Judging the benefit of an approach to a problem depends on your figure of
merit. Your figure of merit seems to be performance, which is perfectly
valid. Mine depends on the problem and on the solution, and performance may
not be the most important component.

In other words, some solutions work well expressed concurrently; others do
not. Whether the performance improves is only one part of deciding how
well. One programmer's misuse may be another's thing of joy.

-rob

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAOXNBZSPhMm1XWtMBX7z7g%3Dbh4gXW%2BRrrQ%2BfLCmN3aeE-UzhDQ%40mail.gmail.com.

Re: [go-nuts] Re: An old problem: lack of priority select cases

[T L]

> Please, drop it. You don't know what you're talking about. I've been 
amazed at how patient everyone has been. I think it's time to be more blunt 
in pointing out you don't understand concurrency and independent events.

Maybe. But you don't know I'm talking about for sure.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/6521f318-b608-43f6-a15b-48c459bcd376%40googlegroups.com.

Re: [go-nuts] Re: An old problem: lack of priority select cases

[Kurtis Rader]

Please, drop it. You don't know what you're talking about. I've been amazed
at how patient everyone has been. I think it's time to be more blunt in
pointing out you don't understand concurrency and independent events.

On Fri, Oct 4, 2019 at 3:15 PM T L  wrote:

>
>
> On Friday, October 4, 2019 at 5:40:08 PM UTC-4, ohir wrote:
>>
>> On Fri, 4 Oct 2019 13:52:19 -0700 (PDT)
>> T L  wrote:
>>
>> > One priority-order select block is not only cleaner than two
>> random-order
>> > select blocks, but also more efficient.
>>
>> It is neither.
>>
>> 1. It is not cleaner, because you would need to somehow denote
>> priorities.
>> If you think now about "in written order", you should think of the most
>> frequent
>> cases that you do not want to have all five, ten, twenty cases be
>> serviced
>> on order. Not to mention that down the road someone might introduce an
>> unnoticeable bug for pure aesthetic reasons.
>>
>> 2. It is not more efficient, it is less efficient because every select
>> operation
>> would need at least a "priority comparison" op, or - in general case - a
>> sort
>> op, because not only two channels can be ready, it may happen that all of
>> them will. This would have to run for every select then, unless
>> introduced by a "select_with_priorities" keyword.
>>
>>
> select(ordered)  {
> case <-ctx.Done():
> return ctx.Err()
> case out <- v:
> }
>
> is undoubtedly more efficient than
>
> select(randomized) {
> case <-ctx.Done():
> return ctx.Err()
> default:
> }
> select(randomized)  {
> case <-ctx.Done():
> return ctx.Err()
> case out <- v:
> }
>
> Not only one less try-receive operation is saved,
> but also is the step 2 described in
> https://go101.org/article/channel.html#select-implementation saved.
>
>
>
>> 3. Others tried to no avail, I will too in other words:
>>
>> The select switch is deliberately randomized because language creators
>> envisioned that otherwise someone would be "smart" and will use
>> ordering to impose a **dependency** where it should not be made.
>>
>> Channel operations serviced by select are treated as sourced
>> from **independent** events. Period.
>>
>> If your out <-v must be dependent of ctx.Done not being ready you
>> must resolve this dependency by other means. Eg. by selecting for it
>> in separate select before.
>>
>
> It is not a must. It just tries to select ctx.Done if possible.
> In fact, if the two evens are independent with each other,
> there is no way to absolutely ensure the must.
>
>
>>
>> Hope this helps,
>>
>
> You can read the first comment in this thread for another use case of 
> priority-order
> select block.
>
>
>>
>> --
>> Wojciech S. Czarnecki
>>  << ^oo^ >> OHIR-RIPE
>>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/0c9aaa9b-ea9d-49f5-a1b4-4f94ee0de424%40googlegroups.com
> 
> .
>


-- 
Kurtis Rader
Caretaker of the exceptional canines Junior and Hank

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CABx2%3DD9NnBkz0jCcXM7x%3D4Ms6rw0L%2BmrWGsknD3_kKM_9mMnzw%40mail.gmail.com.

Re: [go-nuts] Re: An old problem: lack of priority select cases

[T L]

On Friday, October 4, 2019 at 5:40:08 PM UTC-4, ohir wrote:
>
> On Fri, 4 Oct 2019 13:52:19 -0700 (PDT) 
> T L > wrote: 
>
> > One priority-order select block is not only cleaner than two 
> random-order 
> > select blocks, but also more efficient. 
>
> It is neither. 
>
> 1. It is not cleaner, because you would need to somehow denote priorities. 
> If you think now about "in written order", you should think of the most 
> frequent 
> cases that you do not want to have all five, ten, twenty cases be serviced 
> on order. Not to mention that down the road someone might introduce an 
> unnoticeable bug for pure aesthetic reasons. 
>
> 2. It is not more efficient, it is less efficient because every select 
> operation 
> would need at least a "priority comparison" op, or - in general case - a 
> sort 
> op, because not only two channels can be ready, it may happen that all of 
> them will. This would have to run for every select then, unless 
> introduced by a "select_with_priorities" keyword. 
>
>
select(ordered)  {
case <-ctx.Done():
return ctx.Err()
case out <- v:
}

is undoubtedly more efficient than 

select(randomized) {
case <-ctx.Done():
return ctx.Err()
default:
}
select(randomized)  {
case <-ctx.Done():
return ctx.Err()
case out <- v:
}

Not only one less try-receive operation is saved,
but also is the step 2 described in 
https://go101.org/article/channel.html#select-implementation saved.

 

> 3. Others tried to no avail, I will too in other words: 
>
> The select switch is deliberately randomized because language creators 
> envisioned that otherwise someone would be "smart" and will use 
> ordering to impose a **dependency** where it should not be made. 
>
> Channel operations serviced by select are treated as sourced 
> from **independent** events. Period. 
>
> If your out <-v must be dependent of ctx.Done not being ready you 
> must resolve this dependency by other means. Eg. by selecting for it 
> in separate select before. 
>

It is not a must. It just tries to select ctx.Done if possible.
In fact, if the two evens are independent with each other,
there is no way to absolutely ensure the must.
 

>
> Hope this helps, 
>

You can read the first comment in this thread for another use case of 
priority-order 
select block.
 

>
> -- 
> Wojciech S. Czarnecki 
>  << ^oo^ >> OHIR-RIPE 
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/0c9aaa9b-ea9d-49f5-a1b4-4f94ee0de424%40googlegroups.com.

Re: [go-nuts] Re: An old problem: lack of priority select cases

[Wojciech S. Czarnecki]

On Fri, 4 Oct 2019 13:52:19 -0700 (PDT)
T L  wrote:

> One priority-order select block is not only cleaner than two random-order
> select blocks, but also more efficient.

It is neither. 

1. It is not cleaner, because you would need to somehow denote priorities.
If you think now about "in written order", you should think of the most frequent
cases that you do not want to have all five, ten, twenty cases be serviced
on order. Not to mention that down the road someone might introduce an
unnoticeable bug for pure aesthetic reasons.

2. It is not more efficient, it is less efficient because every select operation
would need at least a "priority comparison" op, or - in general case - a sort
op, because not only two channels can be ready, it may happen that all of
them will. This would have to run for every select then, unless 
introduced by a "select_with_priorities" keyword.

3. Others tried to no avail, I will too in other words:

The select switch is deliberately randomized because language creators
envisioned that otherwise someone would be "smart" and will use
ordering to impose a **dependency** where it should not be made.

Channel operations serviced by select are treated as sourced
from **independent** events. Period. 

If your out <-v must be dependent of ctx.Done not being ready you
must resolve this dependency by other means. Eg. by selecting for it
in separate select before. 

Hope this helps,

-- 
Wojciech S. Czarnecki
 << ^oo^ >> OHIR-RIPE

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/20191004233941.0b5c516e%40zuzia.

[go-nuts] Re: Correct way of sending HEADERS and DATA frames in http2?

[helio . guardabaxo]

Did you solve this problem?
If so, could I see the code?
I am trying to read the stream's priority field.

Em quarta-feira, 23 de maio de 2018 07:41:34 UTC-3, Ankit Gupta escreveu:
>
> I am using *golang.org/x/net/http2 * and 
> *golang.org/x/net/http2/ 
> **hpack* packages to do low level http2 
> framing. For requests with only HEADERS frame sent, I want to send a empy 
> json '{}', so I send 1 HEADERS frame and 1 DATA frame as below 
>
> headersframe, ok := frame.(*http2.HeadersFrame)
> if ok {
> println("headers frame detected")
>
> if headersframe.StreamEnded() {
>  
> // write HEADERS frame with 2 headers
>   
> hbytes := make([]byte, 0, 2048)
> hbuffer := bytes.NewBuffer(hbytes)
> hfield1 := hpack.HeaderField{
> Name: ":status",
> Value: "200",
> Sensitive: false,}
> hfield2 := hpack.HeaderField{
> Name: "content-type",
> Value: "application/json",
> Sensitive: false,}
>
> encoder := hpack.NewEncoder(hbuffer)
> err := encoder.WriteField(hfield1)
> err = encoder.WriteField(hfield2)
> if err != nil {
> println(err.Error())
> }
> hfp := http2.HeadersFrameParam{
> StreamID: frame.Header().StreamID,
> BlockFragment: hbuffer.Bytes(),
> EndStream: false,
> EndHeaders: true,
> }
> err = framer.WriteHeaders(hfp)
> if err != nil {
> println(err.Error())
> }
>
> // write DATA frame
>
> data := []byte("{}")
> framer.WriteData(frame.Header().StreamID, true, data)
> }
> return
> }
>
> On trying with *curl -v https://127.0.0.1:8000  
> -k --http2*, I get error as - 
>
> curl: (16) Error in the HTTP2 framing layer
>
> Is anything missing in my way of framing response?
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/95fe40c6-090d-47d6-a46b-a2e79eac75f3%40googlegroups.com.

Re: [go-nuts] Re: An old problem: lack of priority select cases

[Robert Engels]

But even the two select blocks does not really make a difference. I have shared code that did this previously.For another example, think of two channels A and B. A is the high priority channel where all events should be processed before B channel events - this is especially important considering that A events can be processed in 1 ms, and B events take 60 seconds.So, as soon as you start processing a B event - you have delayed the processing of a "just after" A event by 60 seconds... probably not what you want. So you either need to be able to stop the processing of B, or you need to process them "concurrently" (this is a simplification of how to actually accomplish this).This is why if closing the channel is independent of the processing of the channel, you need external synchronization and/or ordering.Are there workloads where the A and B channels are always ready at the same time, probably... but if you ALWAYS processed A first - eventually B would run out of queue/memory space - and either deadlock or crash...I will reiterate - your thinking on this problem is incorrect, which is why the solution you propose is of limited usefulness. Still, if you really want this, it is simple to write a function 'selectUsingPriority(A,B chan). But if you don't fully understand what I am saying, you are going to have obscure reliability problems in production if you use it.-Original Message-
From: T L 
Sent: Oct 4, 2019 3:52 PM
To: golang-nuts 
Subject: Re: [go-nuts] Re: An old problem: lack of priority select cases

On Friday, October 4, 2019 at 4:38:36 PM UTC-4, Marcin Romaszewicz wrote:What he's trying to say is that it is pointless to order select cases in this example, because it is impossible to guarantee ordering in an asynchronous system.You have two asynchronous data streams, ctx.Done() and "v", in that example above.Generally, those select cases will happen one by one, as those asynchronous channels deliver data. If "v" is ready first, shortly before ctx.Done(), then v will be selected first, followed by ctx.Done(). It doesn't matter that ctx.Done() wants to be higher priority, "v" arrived first, before ctx.Done() was posted, so it's handled first. If ctx.Done() happens first, the reverse happens. The only interesting case is when both "v" and ctx.Done() are ready at the same time, which will be unlikely in practice.If ctx.Done() and "v" happen together, so that both cases of that select statement are available simultaneously, then sure, you can order them, but this will happen very infrequently, and you still MUST handle the case where "v" happens first, very close to ctx.Done(). So, if you MUST handle this case, you don't really need to bother with statement priority, since if your code is written well, this ordering won't matter. ctx.Done() might happen while you're in the middle of handling "v", for example.This is not frequently, but not rare. The desire to have priority to select statements presumes that you have several cases happen at the same time, which isn't how this will generally work. If you want to order things in a select, you have to change how it behaves, and order events within a particular time window. Say you write a poller which samples the event queue every second. If you do this, then sure, you can order things however you want, but the cost you pay is that second of buffering latency.Yes, select case priority tries to always select one case of the two caseswhen both the two cases are available. Without this feature, to achievethis goal, we often need two select blocks (sorry, I mis-typed  select block as  select case in my last comment). One priority-order select block is not only cleaner than two random-order select blocks, but also more efficient.On Fri, Oct 4, 2019 at 1:19 PM T L  wrote:On Friday, October 4, 2019 at 4:09:09 PM UTC-4, Robert Engels wrote:Because ctx.Done() and v being ready for read are independent events. You can not impose ordering on them unless there is an outer mutex that both events are subject to.As an aside, this is why I think the best 'concurrent software' developers are those that have been exposed to at least some hardware design. Many programmers think in terms of 1 and 0 and everything being ordered. This is certainly not the case in hardware, nor concurrent software. (For example, in computer hardware, the clock/sync line is what is used as the outer controlling event, but still things like propagation times, etc. make even this simple statement not fully correct).Still not understanding what you new saying. ;DAgain, select case priority enables use to deduce two select casesto one select case in coding in many scenarios.This often leads to cleaner code, avoid can avoid the harm causedby missing a try-receive select case. -Original Message-
From: T L 
Sent: Oct 4, 2019 2:44 PM
To: golang-nuts 
Subject: Re: [go-nuts] Re: An old problem: lack of priority select cases

On Friday, October 4, 2019 at 3:32:31 PM UTC-4, Ro

Re: [go-nuts] Re: An old problem: lack of priority select cases

[T L]

On Friday, October 4, 2019 at 4:38:36 PM UTC-4, Marcin Romaszewicz wrote:
>
> What he's trying to say is that it is pointless to order select cases in 
> this example, because it is impossible to guarantee ordering in an 
> asynchronous system.
>
> You have two asynchronous data streams, ctx.Done() and "v", in that 
> example above.
>
> Generally, those select cases will happen one by one, as those 
> asynchronous channels deliver data. If "v" is ready first, shortly before 
> ctx.Done(), then v will be selected first, followed by ctx.Done(). It 
> doesn't matter that ctx.Done() wants to be higher priority, "v" arrived 
> first, before ctx.Done() was posted, so it's handled first. If ctx.Done() 
> happens first, the reverse happens. The only interesting case is when both 
> "v" and ctx.Done() are ready at the same time, which will be unlikely in 
> practice.
>
> If ctx.Done() and "v" happen together, so that both cases of that select 
> statement are available simultaneously, then sure, you can order them, but 
> this will happen very infrequently, and you still MUST handle the case 
> where "v" happens first, very close to ctx.Done(). So, if you MUST handle 
> this case, you don't really need to bother with statement priority, since 
> if your code is written well, this ordering won't matter. ctx.Done() might 
> happen while you're in the middle of handling "v", for example.
>
>
This is not frequently, but not rare.
 

> The desire to have priority to select statements presumes that you have 
> several cases happen at the same time, which isn't how this will generally 
> work. If you want to order things in a select, you have to change how it 
> behaves, and order events within a particular time window. Say you write a 
> poller which samples the event queue every second. If you do this, then 
> sure, you can order things however you want, but the cost you pay is that 
> second of buffering latency.
>

Yes, select case priority tries to always select one case of the two cases
when both the two cases are available. Without this feature, to achieve
this goal, we often need two select blocks (sorry, I mis-typed select block
as select case in my last comment). 

One priority-order select block is not only cleaner than two random-order
select blocks, but also more efficient.

>
>
> On Fri, Oct 4, 2019 at 1:19 PM T L > 
> wrote:
>
>>
>>
>> On Friday, October 4, 2019 at 4:09:09 PM UTC-4, Robert Engels wrote:
>>>
>>> Because ctx.Done() and v being ready for read are independent events. 
>>> You can not impose ordering on them unless there is an outer mutex that 
>>> both events are subject to.
>>>
>>> As an aside, this is why I think the best 'concurrent software' 
>>> developers are those that have been exposed to at least some hardware 
>>> design. Many programmers think in terms of 1 and 0 and everything being 
>>> ordered. This is certainly not the case in hardware, nor concurrent 
>>> software. (For example, in computer hardware, the clock/sync line is what 
>>> is used as the outer controlling event, but still things like propagation 
>>> times, etc. make even this simple statement not fully correct).
>>>
>>>
>> Still not understanding what you new saying. ;D
>>
>> Again, select case priority enables use to deduce two select cases
>> to one select case in coding in many scenarios.
>> This often leads to cleaner code, avoid can avoid the harm caused
>> by missing a try-receive select case.
>>
>>  
>>
>>> -Original Message- 
>>> From: T L 
>>> Sent: Oct 4, 2019 2:44 PM 
>>> To: golang-nuts 
>>> Subject: Re: [go-nuts] Re: An old problem: lack of priority select cases 
>>>
>>>
>>>
>>> On Friday, October 4, 2019 at 3:32:31 PM UTC-4, Robert Engels wrote:

 You still are not understanding  proper concurrent design. Priority 
 select cases do not matter in the case of asynchronous external events. 

>>>
>>> It at least avoids code verbosity and improves code quantity..
>>>
>>> BTW, I don't understand what you said. Could you elaborate more?
>>>
>>>
 On Oct 4, 2019, at 1:46 PM, T L  wrote:

 
 I just found an example in the "context" package docs:

 //  // Stream generates values with DoSomething and sends them to out
 //  // until DoSomething returns an error or ctx.Done is closed.
 //  func Stream(ctx context.Context, out chan<- Value) error {
 // for {
 // v, err := DoSomething(ctx)
 // if err != nil {
 // return err
 // }
 // select {
 // case <-ctx.Done():
 // return ctx.Err()
 // case out <- v:
 // }
 // }
 //  }

 It looks the send "out <- v" still has a possibility to be executed,
 even if "ctx.Done()" is closed.
 But if Go supports select case priority, then this will never happen.



>

Re: [go-nuts] Re: An old problem: lack of priority select cases

[Marcin Romaszewicz]

What he's trying to say is that it is pointless to order select cases in
this example, because it is impossible to guarantee ordering in an
asynchronous system.

You have two asynchronous data streams, ctx.Done() and "v", in that example
above.

Generally, those select cases will happen one by one, as those asynchronous
channels deliver data. If "v" is ready first, shortly before ctx.Done(),
then v will be selected first, followed by ctx.Done(). It doesn't matter
that ctx.Done() wants to be higher priority, "v" arrived first, before
ctx.Done() was posted, so it's handled first. If ctx.Done() happens first,
the reverse happens. The only interesting case is when both "v" and
ctx.Done() are ready at the same time, which will be unlikely in practice.

If ctx.Done() and "v" happen together, so that both cases of that select
statement are available simultaneously, then sure, you can order them, but
this will happen very infrequently, and you still MUST handle the case
where "v" happens first, very close to ctx.Done(). So, if you MUST handle
this case, you don't really need to bother with statement priority, since
if your code is written well, this ordering won't matter. ctx.Done() might
happen while you're in the middle of handling "v", for example.

The desire to have priority to select statements presumes that you have
several cases happen at the same time, which isn't how this will generally
work. If you want to order things in a select, you have to change how it
behaves, and order events within a particular time window. Say you write a
poller which samples the event queue every second. If you do this, then
sure, you can order things however you want, but the cost you pay is that
second of buffering latency.


On Fri, Oct 4, 2019 at 1:19 PM T L  wrote:

>
>
> On Friday, October 4, 2019 at 4:09:09 PM UTC-4, Robert Engels wrote:
>>
>> Because ctx.Done() and v being ready for read are independent events. You
>> can not impose ordering on them unless there is an outer mutex that both
>> events are subject to.
>>
>> As an aside, this is why I think the best 'concurrent software'
>> developers are those that have been exposed to at least some hardware
>> design. Many programmers think in terms of 1 and 0 and everything being
>> ordered. This is certainly not the case in hardware, nor concurrent
>> software. (For example, in computer hardware, the clock/sync line is what
>> is used as the outer controlling event, but still things like propagation
>> times, etc. make even this simple statement not fully correct).
>>
>>
> Still not understanding what you new saying. ;D
>
> Again, select case priority enables use to deduce two select cases
> to one select case in coding in many scenarios.
> This often leads to cleaner code, avoid can avoid the harm caused
> by missing a try-receive select case.
>
>
>
>> -Original Message-
>> From: T L
>> Sent: Oct 4, 2019 2:44 PM
>> To: golang-nuts
>> Subject: Re: [go-nuts] Re: An old problem: lack of priority select cases
>>
>>
>>
>> On Friday, October 4, 2019 at 3:32:31 PM UTC-4, Robert Engels wrote:
>>>
>>> You still are not understanding  proper concurrent design. Priority
>>> select cases do not matter in the case of asynchronous external events.
>>>
>>
>> It at least avoids code verbosity and improves code quantity..
>>
>> BTW, I don't understand what you said. Could you elaborate more?
>>
>>
>>> On Oct 4, 2019, at 1:46 PM, T L  wrote:
>>>
>>> 
>>> I just found an example in the "context" package docs:
>>>
>>> //  // Stream generates values with DoSomething and sends them to out
>>> //  // until DoSomething returns an error or ctx.Done is closed.
>>> //  func Stream(ctx context.Context, out chan<- Value) error {
>>> //  for {
>>> //  v, err := DoSomething(ctx)
>>> //  if err != nil {
>>> //  return err
>>> //  }
>>> //  select {
>>> //  case <-ctx.Done():
>>> //  return ctx.Err()
>>> //  case out <- v:
>>> //  }
>>> //  }
>>> //  }
>>>
>>> It looks the send "out <- v" still has a possibility to be executed,
>>> even if "ctx.Done()" is closed.
>>> But if Go supports select case priority, then this will never happen.
>>>
>>>
>>>
>>> On Wednesday, August 28, 2019 at 12:06:33 PM UTC-4, T L wrote:

 The old thread:
 https://groups.google.com/forum/#!topic/golang-nuts/ZrVIhHCrR9o

 Go channels are flexible, but in practice, I often encountered some
 situations in which channel are hard to use.
 Given an example:

 import "math/rand"

 type Producer struct {
 data   chan int
 closed chan struct{}
 }

 func NewProducer() *Producer {
 p := &Producer {
 data:   make(chan int),
 closed: make(chan struct{}),
 }

 go p.run()

 return p
 }

 func (p *

Re: [go-nuts] Re: An old problem: lack of priority select cases

[T L]

On Friday, October 4, 2019 at 4:09:09 PM UTC-4, Robert Engels wrote:
>
> Because ctx.Done() and v being ready for read are independent events. You 
> can not impose ordering on them unless there is an outer mutex that both 
> events are subject to.
>
> As an aside, this is why I think the best 'concurrent software' developers 
> are those that have been exposed to at least some hardware design. Many 
> programmers think in terms of 1 and 0 and everything being ordered. This is 
> certainly not the case in hardware, nor concurrent software. (For example, 
> in computer hardware, the clock/sync line is what is used as the outer 
> controlling event, but still things like propagation times, etc. make even 
> this simple statement not fully correct).
>
>
Still not understand what you new saying means. ;D

Again, select case priority enables use to deduce two select cases
to one select case in coding in many scenarios.
This often leads to cleaner code, and sometimes can avoid the harm 
caused by missing an essential try-receive select case.

 

> -Original Message- 
> From: T L 
> Sent: Oct 4, 2019 2:44 PM 
> To: golang-nuts 
> Subject: Re: [go-nuts] Re: An old problem: lack of priority select cases 
>
>
>
> On Friday, October 4, 2019 at 3:32:31 PM UTC-4, Robert Engels wrote:
>>
>> You still are not understanding  proper concurrent design. Priority 
>> select cases do not matter in the case of asynchronous external events. 
>>
>
> It at least avoids code verbosity and improves code quantity..
>
> BTW, I don't understand what you said. Could you elaborate more?
>
>
>> On Oct 4, 2019, at 1:46 PM, T L  wrote:
>>
>> 
>> I just found an example in the "context" package docs:
>>
>> //  // Stream generates values with DoSomething and sends them to out
>> //  // until DoSomething returns an error or ctx.Done is closed.
>> //  func Stream(ctx context.Context, out chan<- Value) error {
>> //   for {
>> //   v, err := DoSomething(ctx)
>> //   if err != nil {
>> //   return err
>> //   }
>> //   select {
>> //   case <-ctx.Done():
>> //   return ctx.Err()
>> //   case out <- v:
>> //   }
>> //   }
>> //  }
>>
>> It looks the send "out <- v" still has a possibility to be executed,
>> even if "ctx.Done()" is closed.
>> But if Go supports select case priority, then this will never happen.
>>
>>
>>
>> On Wednesday, August 28, 2019 at 12:06:33 PM UTC-4, T L wrote:
>>>
>>> The old thread: 
>>> https://groups.google.com/forum/#!topic/golang-nuts/ZrVIhHCrR9o
>>>
>>> Go channels are flexible, but in practice, I often encountered some 
>>> situations in which channel are hard to use.
>>> Given an example:
>>>
>>> import "math/rand"
>>>
>>> type Producer struct {
>>> data   chan int
>>> closed chan struct{}
>>> }
>>>
>>> func NewProducer() *Producer {
>>> p := &Producer {
>>> data:   make(chan int),
>>> closed: make(chan struct{}),
>>> }
>>> 
>>> go p.run()
>>> 
>>> return p
>>> }
>>>
>>> func (p *Produce) Stream() chan int {
>>> return p.data
>>> }
>>>
>>> func (p *Producer) run() {
>>> for {
>>> // If non-blocking cases are selected by their appearance order,
>>> // then the following slect block is a perfect use.
>>> select {
>>> case(0) <-p.closed: return
>>> case p.data <- rand.Int():
>>> }
>>> }
>>> }
>>>
>>> func (p *Produce) Clsoe() {
>>> close(p.closed)
>>> close(p.data)
>>> }
>>>
>>> func main() {
>>> p := NewProducer()
>>> for n := p.Stream() {
>>> // use n ...
>>> }
>>> }
>>>
>>>
>>> If the first case in the select block in the above example has a higher 
>>> priority than the second one,
>>> then coding will be much happier for the use cases like the above one.
>>>
>>> In short, the above use case requires:
>>> * for receivers, data streaming end is notified by the close of a 
>>> channel.
>>> * for senders, data will never be sent to closed channel.
>>>
>>> But, as Go 1 doesn't support priority select cases, it is much tedious 
>>> to implement the code
>>> satisfying the above listed requirements. The final implementation is 
>>> often very ugly and inefficient.
>>>
>>> Does anyone else also experience the pain?
>>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golan...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/b7a14153-253e-43c4-bda5-96079601465f%40googlegroups.com
>>  
>> 
>> .
>>
>> -- 
> You received this message because you are

Re: [go-nuts] Re: An old problem: lack of priority select cases

[T L]

On Friday, October 4, 2019 at 4:09:09 PM UTC-4, Robert Engels wrote:
>
> Because ctx.Done() and v being ready for read are independent events. You 
> can not impose ordering on them unless there is an outer mutex that both 
> events are subject to.
>
> As an aside, this is why I think the best 'concurrent software' developers 
> are those that have been exposed to at least some hardware design. Many 
> programmers think in terms of 1 and 0 and everything being ordered. This is 
> certainly not the case in hardware, nor concurrent software. (For example, 
> in computer hardware, the clock/sync line is what is used as the outer 
> controlling event, but still things like propagation times, etc. make even 
> this simple statement not fully correct).
>
>
Still not understanding what you new saying. ;D

Again, select case priority enables use to deduce two select cases
to one select case in coding in many scenarios.
This often leads to cleaner code, avoid can avoid the harm caused
by missing a try-receive select case.

 

> -Original Message- 
> From: T L 
> Sent: Oct 4, 2019 2:44 PM 
> To: golang-nuts 
> Subject: Re: [go-nuts] Re: An old problem: lack of priority select cases 
>
>
>
> On Friday, October 4, 2019 at 3:32:31 PM UTC-4, Robert Engels wrote:
>>
>> You still are not understanding  proper concurrent design. Priority 
>> select cases do not matter in the case of asynchronous external events. 
>>
>
> It at least avoids code verbosity and improves code quantity..
>
> BTW, I don't understand what you said. Could you elaborate more?
>
>
>> On Oct 4, 2019, at 1:46 PM, T L  wrote:
>>
>> 
>> I just found an example in the "context" package docs:
>>
>> //  // Stream generates values with DoSomething and sends them to out
>> //  // until DoSomething returns an error or ctx.Done is closed.
>> //  func Stream(ctx context.Context, out chan<- Value) error {
>> //   for {
>> //   v, err := DoSomething(ctx)
>> //   if err != nil {
>> //   return err
>> //   }
>> //   select {
>> //   case <-ctx.Done():
>> //   return ctx.Err()
>> //   case out <- v:
>> //   }
>> //   }
>> //  }
>>
>> It looks the send "out <- v" still has a possibility to be executed,
>> even if "ctx.Done()" is closed.
>> But if Go supports select case priority, then this will never happen.
>>
>>
>>
>> On Wednesday, August 28, 2019 at 12:06:33 PM UTC-4, T L wrote:
>>>
>>> The old thread: 
>>> https://groups.google.com/forum/#!topic/golang-nuts/ZrVIhHCrR9o
>>>
>>> Go channels are flexible, but in practice, I often encountered some 
>>> situations in which channel are hard to use.
>>> Given an example:
>>>
>>> import "math/rand"
>>>
>>> type Producer struct {
>>> data   chan int
>>> closed chan struct{}
>>> }
>>>
>>> func NewProducer() *Producer {
>>> p := &Producer {
>>> data:   make(chan int),
>>> closed: make(chan struct{}),
>>> }
>>> 
>>> go p.run()
>>> 
>>> return p
>>> }
>>>
>>> func (p *Produce) Stream() chan int {
>>> return p.data
>>> }
>>>
>>> func (p *Producer) run() {
>>> for {
>>> // If non-blocking cases are selected by their appearance order,
>>> // then the following slect block is a perfect use.
>>> select {
>>> case(0) <-p.closed: return
>>> case p.data <- rand.Int():
>>> }
>>> }
>>> }
>>>
>>> func (p *Produce) Clsoe() {
>>> close(p.closed)
>>> close(p.data)
>>> }
>>>
>>> func main() {
>>> p := NewProducer()
>>> for n := p.Stream() {
>>> // use n ...
>>> }
>>> }
>>>
>>>
>>> If the first case in the select block in the above example has a higher 
>>> priority than the second one,
>>> then coding will be much happier for the use cases like the above one.
>>>
>>> In short, the above use case requires:
>>> * for receivers, data streaming end is notified by the close of a 
>>> channel.
>>> * for senders, data will never be sent to closed channel.
>>>
>>> But, as Go 1 doesn't support priority select cases, it is much tedious 
>>> to implement the code
>>> satisfying the above listed requirements. The final implementation is 
>>> often very ugly and inefficient.
>>>
>>> Does anyone else also experience the pain?
>>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "golang-nuts" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to golan...@googlegroups.com.
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/golang-nuts/b7a14153-253e-43c4-bda5-96079601465f%40googlegroups.com
>>  
>> 
>> .
>>
>> -- 
> You received this message because you are subscribed to the Goog

Re: [go-nuts] Re: An old problem: lack of priority select cases

[Robert Engels]

Because ctx.Done() and v being ready for read are independent events. You can not impose ordering on them unless there is an outer mutex that both events are subject to.As an aside, this is why I think the best 'concurrent software' developers are those that have been exposed to at least some hardware design. Many programmers think in terms of 1 and 0 and everything being ordered. This is certainly not the case in hardware, nor concurrent software. (For example, in computer hardware, the clock/sync line is what is used as the outer controlling event, but still things like propagation times, etc. make even this simple statement not fully correct).-Original Message-
From: T L 
Sent: Oct 4, 2019 2:44 PM
To: golang-nuts 
Subject: Re: [go-nuts] Re: An old problem: lack of priority select cases

On Friday, October 4, 2019 at 3:32:31 PM UTC-4, Robert Engels wrote:You still are not understanding  proper concurrent design. Priority select cases do not matter in the case of asynchronous external events. It at least avoids code verbosity and improves code quantity..BTW, I don't understand what you said. Could you elaborate more?On Oct 4, 2019, at 1:46 PM, T L  wrote:I just found an example in the "context" package docs://  // Stream generates values with DoSomething and sends them to out
//  // until DoSomething returns an error or ctx.Done is closed.
//  func Stream(ctx context.Context, out chan<- Value) error {
//  	for {
//  		v, err := DoSomething(ctx)
//  		if err != nil {
//  			return err
//  		}
//  		select {
//  		case <-ctx.Done():
//  			return ctx.Err()
//  		case out <- v:
//  		}
//  	}
//  }It looks the send "out <- v" still has a possibility to be executed,even if "ctx.Done()" is closed.But if Go supports select case priority, then this will never happen.On Wednesday, August 28, 2019 at 12:06:33 PM UTC-4, T L wrote:The old thread: https://groups.google.com/forum/#!topic/golang-nuts/ZrVIhHCrR9oGo channels are flexible, but in practice, I often encountered some situations in which channel are hard to use.Given an example:import "math/rand"type Producer struct {    data   chan int    closed chan struct{}}func NewProducer() *Producer {    p := &Producer {        data:   make(chan int),        closed: make(chan struct{}),    }        go p.run()        return p}func (p *Produce) Stream() chan int {    return p.data}func (p *Producer) run() {    for {        // If non-blocking cases are selected by their appearance order,        // then the following slect block is a perfect use.        select {        case(0) <-p.closed: return        case p.data <- rand.Int():        }    }}func (p *Produce) Clsoe() {    close(p.closed)    close(p.data)}func main() {    p := NewProducer()    for n := p.Stream() {        // use n ...    }}If the first case in the select block in the above example has a higher priority than the second one,then coding will be much happier for the use cases like the above one.In short, the above use case requires:* for receivers, data streaming end is notified by the close of a channel.* for senders, data will never be sent to closed channel.But, as Go 1 doesn't support priority select cases, it is much tedious to implement the codesatisfying the above listed requirements. The final implementation is often very ugly and inefficient.Does anyone else also experience the pain?



-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golan...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/b7a14153-253e-43c4-bda5-96079601465f%40googlegroups.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/77fca2e9-120d-45e0-8ae9-5d24b63827bd%40googlegroups.com.




-- 
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/975777479.10069.1570219711061%40wamui-aurora.atl.sa.earthlink.net.

[go-nuts] Re: PKCs8 generated using go has errors when doing a openssl check

[rajesh nataraja]

Sorry, I speed read your email, but you were right Piers. "PRIVATE KEY" as 
header makes the difference. 

Rajesh.

On Friday, October 4, 2019 at 9:39:52 AM UTC-7, rajesh nataraja wrote:
>
> Hello Piers, 
>
> I have tried your playground snippet and the snippet I gave here. Both 
> dont work, what I meant is saving the Marshalled key into a file and then 
> using that to be processed by other applications (java, python, openssl 
> command). 
> This is with go 1.11.5, do you think there is some compatibility issue 
> with go package here?
>
> Thanks
> Rajesh
>
>
> On Friday, October 4, 2019 at 9:03:54 AM UTC-7, helloPiers wrote:
>>
>> For PKCS8 (rather than PKCS1), use PEM type "PRIVATE KEY" (rather than 
>> "RSA PRIVATE KEY").
>>
>> You may be constructing the ASN1 by hand deliberately, but just in case 
>> you didn't see it, there's also a standard library function 
>> x509.MarshalPKCS8PrivateKey() 
>> https://godoc.org/crypto/x509#MarshalPKCS8PrivateKey 
>>
>> This can take the output of rsa.GenerateKey() directly, for example like: 
>> https://play.golang.org/p/UzWACWh2TCo  (key size reduced so it runs in 
>> the playground without timing out).
>>
>> On Friday, October 4, 2019 at 1:14:15 AM UTC+1, rajesh nataraja wrote:
>>>
>>> Hi All, 
>>>
>>> I have the following piece of code to generate a private key in PKCS8 
>>> form and save it in a file. It does generate a file, but when I try to 
>>> check using the openssl command 
>>>
>>> openssl rsa -in rsapk.key -check 
>>> I get the following errors 
>>>
>>> 140092967139232:error:0D0680A8:asn1 encoding 
>>> routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1199:
>>> 140092967139232:error:0D06C03A:asn1 encoding 
>>> routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:767:
>>> 140092967139232:error:0D08303A:asn1 encoding 
>>> routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:699:Field=n, 
>>> Type=RSA
>>> 140092967139232:error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA 
>>> lib:rsa_ameth.c:121:
>>>
>>>
>>> Anyone knows what is wrong with my method?
>>>
>>> package main
>>>
>>> import (
>>> "crypto/x509"
>>> "crypto/rsa"
>>> "encoding/pem"
>>> "io/ioutil"
>>> "crypto/rand"
>>> "encoding/asn1"
>>> )
>>>
>>> type privateKeyInfo struct {
>>> Version int
>>> PrivateKeyAlgorithm []asn1.ObjectIdentifier
>>> PrivateKey  []byte
>>> }
>>>
>>>
>>> func NewPKCS8PrivateKey() {
>>>
>>> var pkey privateKeyInfo
>>> var bKey []byte
>>> oidPublicKeyRSA  := asn1.ObjectIdentifier{1, 2, 840, 113549, 1, 1, 1}
>>>
>>>
>>> key, err := rsa.GenerateKey(rand.Reader, 2048)
>>> if err != nil {
>>> return
>>> }
>>>
>>> pkey.Version = 0
>>> pkey.PrivateKeyAlgorithm = make([]asn1.ObjectIdentifier, 1)
>>> pkey.PrivateKeyAlgorithm[0] = oidPublicKeyRSA
>>> pkey.PrivateKey = x509.MarshalPKCS1PrivateKey(key)
>>>
>>> bKey , _ = asn1.Marshal(pkey)
>>>
>>> block := pem.Block{Type: "RSA PRIVATE KEY", Bytes: bKey}
>>>
>>> ioutil.WriteFile("./rsapk.key",  pem.EncodeToMemory(&block), 0600)
>>>
>>> }
>>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/bf492116-aaf3-4b49-a817-0154e3b59227%40googlegroups.com.

[go-nuts] xerrors and stack trace

[wilk]

Hi,

I was using xerrors, and now with go1.13 i wanted to remove this
dependency since I read on github.com/golang/xerrors
This repository holds the transition packages for the new Go 1.13
error values.

But unfortunately there are no more stack trace in go1.13

So xerrors is not more a transition package for Go 1.13 ?

Will the stack trace of xerrors become part of go1.14 ? Will xerrors
going to be maintained if stack trace are not going in go1.14 ?

Thanks

-- 
Wilk

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/qn88c4%2475lo%241%40blaine.gmane.org.

Re: [go-nuts] Re: An old problem: lack of priority select cases

[T L]

On Friday, October 4, 2019 at 3:32:31 PM UTC-4, Robert Engels wrote:
>
> You still are not understanding  proper concurrent design. Priority select 
> cases do not matter in the case of asynchronous external events. 
>

It at least avoids code verbosity and improves code quantity..

BTW, I don't understand what you said. Could you elaborate more?


> On Oct 4, 2019, at 1:46 PM, T L > wrote:
>
> 
> I just found an example in the "context" package docs:
>
> //  // Stream generates values with DoSomething and sends them to out
> //  // until DoSomething returns an error or ctx.Done is closed.
> //  func Stream(ctx context.Context, out chan<- Value) error {
> //for {
> //v, err := DoSomething(ctx)
> //if err != nil {
> //return err
> //}
> //select {
> //case <-ctx.Done():
> //return ctx.Err()
> //case out <- v:
> //}
> //}
> //  }
>
> It looks the send "out <- v" still has a possibility to be executed,
> even if "ctx.Done()" is closed.
> But if Go supports select case priority, then this will never happen.
>
>
>
> On Wednesday, August 28, 2019 at 12:06:33 PM UTC-4, T L wrote:
>>
>> The old thread: 
>> https://groups.google.com/forum/#!topic/golang-nuts/ZrVIhHCrR9o
>>
>> Go channels are flexible, but in practice, I often encountered some 
>> situations in which channel are hard to use.
>> Given an example:
>>
>> import "math/rand"
>>
>> type Producer struct {
>> data   chan int
>> closed chan struct{}
>> }
>>
>> func NewProducer() *Producer {
>> p := &Producer {
>> data:   make(chan int),
>> closed: make(chan struct{}),
>> }
>> 
>> go p.run()
>> 
>> return p
>> }
>>
>> func (p *Produce) Stream() chan int {
>> return p.data
>> }
>>
>> func (p *Producer) run() {
>> for {
>> // If non-blocking cases are selected by their appearance order,
>> // then the following slect block is a perfect use.
>> select {
>> case(0) <-p.closed: return
>> case p.data <- rand.Int():
>> }
>> }
>> }
>>
>> func (p *Produce) Clsoe() {
>> close(p.closed)
>> close(p.data)
>> }
>>
>> func main() {
>> p := NewProducer()
>> for n := p.Stream() {
>> // use n ...
>> }
>> }
>>
>>
>> If the first case in the select block in the above example has a higher 
>> priority than the second one,
>> then coding will be much happier for the use cases like the above one.
>>
>> In short, the above use case requires:
>> * for receivers, data streaming end is notified by the close of a channel.
>> * for senders, data will never be sent to closed channel.
>>
>> But, as Go 1 doesn't support priority select cases, it is much tedious to 
>> implement the code
>> satisfying the above listed requirements. The final implementation is 
>> often very ugly and inefficient.
>>
>> Does anyone else also experience the pain?
>>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golan...@googlegroups.com .
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/b7a14153-253e-43c4-bda5-96079601465f%40googlegroups.com
>  
> 
> .
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/77fca2e9-120d-45e0-8ae9-5d24b63827bd%40googlegroups.com.

Re: [go-nuts] Re: An old problem: lack of priority select cases

[Robert Engels]

You still are not understanding  proper concurrent design. Priority select 
cases do not matter in the case of asynchronous external events. 

> On Oct 4, 2019, at 1:46 PM, T L  wrote:
> 
> 
> I just found an example in the "context" package docs:
> 
> //  // Stream generates values with DoSomething and sends them to out
> //  // until DoSomething returns an error or ctx.Done is closed.
> //  func Stream(ctx context.Context, out chan<- Value) error {
> //for {
> //v, err := DoSomething(ctx)
> //if err != nil {
> //return err
> //}
> //select {
> //case <-ctx.Done():
> //return ctx.Err()
> //case out <- v:
> //}
> //}
> //  }
> 
> It looks the send "out <- v" still has a possibility to be executed,
> even if "ctx.Done()" is closed.
> But if Go supports select case priority, then this will never happen.
> 
> 
>> On Wednesday, August 28, 2019 at 12:06:33 PM UTC-4, T L wrote:
>> The old thread: 
>> https://groups.google.com/forum/#!topic/golang-nuts/ZrVIhHCrR9o
>> 
>> Go channels are flexible, but in practice, I often encountered some 
>> situations in which channel are hard to use.
>> Given an example:
>> 
>> import "math/rand"
>> 
>> type Producer struct {
>> data   chan int
>> closed chan struct{}
>> }
>> 
>> func NewProducer() *Producer {
>> p := &Producer {
>> data:   make(chan int),
>> closed: make(chan struct{}),
>> }
>> 
>> go p.run()
>> 
>> return p
>> }
>> 
>> func (p *Produce) Stream() chan int {
>> return p.data
>> }
>> 
>> func (p *Producer) run() {
>> for {
>> // If non-blocking cases are selected by their appearance order,
>> // then the following slect block is a perfect use.
>> select {
>> case(0) <-p.closed: return
>> case p.data <- rand.Int():
>> }
>> }
>> }
>> 
>> func (p *Produce) Clsoe() {
>> close(p.closed)
>> close(p.data)
>> }
>> 
>> func main() {
>> p := NewProducer()
>> for n := p.Stream() {
>> // use n ...
>> }
>> }
>> 
>> 
>> If the first case in the select block in the above example has a higher 
>> priority than the second one,
>> then coding will be much happier for the use cases like the above one.
>> 
>> In short, the above use case requires:
>> * for receivers, data streaming end is notified by the close of a channel.
>> * for senders, data will never be sent to closed channel.
>> 
>> But, as Go 1 doesn't support priority select cases, it is much tedious to 
>> implement the code
>> satisfying the above listed requirements. The final implementation is often 
>> very ugly and inefficient.
>> 
>> Does anyone else also experience the pain?
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit 
> https://groups.google.com/d/msgid/golang-nuts/b7a14153-253e-43c4-bda5-96079601465f%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/5DB82954-56D2-455F-ADF4-02B845D52783%40ix.netcom.com.

[go-nuts] Re: An old problem: lack of priority select cases

[T L]

On Friday, October 4, 2019 at 2:46:36 PM UTC-4, T L wrote:
>
> I just found an example in the "context" package docs:
>
> //  // Stream generates values with DoSomething and sends them to out
> //  // until DoSomething returns an error or ctx.Done is closed.
> //  func Stream(ctx context.Context, out chan<- Value) error {
> //for {
> //v, err := DoSomething(ctx)
> //if err != nil {
> //return err
> //}
> //select {
> //case <-ctx.Done():
> //return ctx.Err()
> //case out <- v:
> //}
> //}
> //  }
>
> It looks the send "out <- v" still has a possibility to be executed,
> even if "ctx.Done()" is closed.
> But if Go supports select case priority, then this will never happen.
>
>
This might be some over absolutely.
There is still a samll chance "ctx.Done()" is being closed when "out <- v" 
is being executed.

Should the docs be changed to:

//  func Stream(ctx context.Context, out chan<- Value) error {
//  for {
//  v, err := DoSomething(ctx)
//  if err != nil {
//  return err
//  }
//  select {
//  case <-ctx.Done():
//  return ctx.Err()
//  case default:
//  }
//  select {
//  case <-ctx.Done():
//  return ctx.Err()
//  case out <- v:
//  }
//  }
//  }


 

>
>
> On Wednesday, August 28, 2019 at 12:06:33 PM UTC-4, T L wrote:
>>
>> The old thread: 
>> https://groups.google.com/forum/#!topic/golang-nuts/ZrVIhHCrR9o
>>
>> Go channels are flexible, but in practice, I often encountered some 
>> situations in which channel are hard to use.
>> Given an example:
>>
>> import "math/rand"
>>
>> type Producer struct {
>> data   chan int
>> closed chan struct{}
>> }
>>
>> func NewProducer() *Producer {
>> p := &Producer {
>> data:   make(chan int),
>> closed: make(chan struct{}),
>> }
>> 
>> go p.run()
>> 
>> return p
>> }
>>
>> func (p *Produce) Stream() chan int {
>> return p.data
>> }
>>
>> func (p *Producer) run() {
>> for {
>> // If non-blocking cases are selected by their appearance order,
>> // then the following slect block is a perfect use.
>> select {
>> case(0) <-p.closed: return
>> case p.data <- rand.Int():
>> }
>> }
>> }
>>
>> func (p *Produce) Clsoe() {
>> close(p.closed)
>> close(p.data)
>> }
>>
>> func main() {
>> p := NewProducer()
>> for n := p.Stream() {
>> // use n ...
>> }
>> }
>>
>>
>> If the first case in the select block in the above example has a higher 
>> priority than the second one,
>> then coding will be much happier for the use cases like the above one.
>>
>> In short, the above use case requires:
>> * for receivers, data streaming end is notified by the close of a channel.
>> * for senders, data will never be sent to closed channel.
>>
>> But, as Go 1 doesn't support priority select cases, it is much tedious to 
>> implement the code
>> satisfying the above listed requirements. The final implementation is 
>> often very ugly and inefficient.
>>
>> Does anyone else also experience the pain?
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/c8dca21d-57c5-4942-8f05-062a78f64dbc%40googlegroups.com.

[go-nuts] Re: An old problem: lack of priority select cases

[T L]

I just found an example in the "context" package docs:

//  // Stream generates values with DoSomething and sends them to out
//  // until DoSomething returns an error or ctx.Done is closed.
//  func Stream(ctx context.Context, out chan<- Value) error {
//  for {
//  v, err := DoSomething(ctx)
//  if err != nil {
//  return err
//  }
//  select {
//  case <-ctx.Done():
//  return ctx.Err()
//  case out <- v:
//  }
//  }
//  }

It looks the send "out <- v" still has a possibility to be executed,
even if "ctx.Done()" is closed.
But if Go supports select case priority, then this will never happen.



On Wednesday, August 28, 2019 at 12:06:33 PM UTC-4, T L wrote:
>
> The old thread: 
> https://groups.google.com/forum/#!topic/golang-nuts/ZrVIhHCrR9o
>
> Go channels are flexible, but in practice, I often encountered some 
> situations in which channel are hard to use.
> Given an example:
>
> import "math/rand"
>
> type Producer struct {
> data   chan int
> closed chan struct{}
> }
>
> func NewProducer() *Producer {
> p := &Producer {
> data:   make(chan int),
> closed: make(chan struct{}),
> }
> 
> go p.run()
> 
> return p
> }
>
> func (p *Produce) Stream() chan int {
> return p.data
> }
>
> func (p *Producer) run() {
> for {
> // If non-blocking cases are selected by their appearance order,
> // then the following slect block is a perfect use.
> select {
> case(0) <-p.closed: return
> case p.data <- rand.Int():
> }
> }
> }
>
> func (p *Produce) Clsoe() {
> close(p.closed)
> close(p.data)
> }
>
> func main() {
> p := NewProducer()
> for n := p.Stream() {
> // use n ...
> }
> }
>
>
> If the first case in the select block in the above example has a higher 
> priority than the second one,
> then coding will be much happier for the use cases like the above one.
>
> In short, the above use case requires:
> * for receivers, data streaming end is notified by the close of a channel.
> * for senders, data will never be sent to closed channel.
>
> But, as Go 1 doesn't support priority select cases, it is much tedious to 
> implement the code
> satisfying the above listed requirements. The final implementation is 
> often very ugly and inefficient.
>
> Does anyone else also experience the pain?
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/b7a14153-253e-43c4-bda5-96079601465f%40googlegroups.com.

[go-nuts] Re: PKCs8 generated using go has errors when doing a openssl check

[rajesh nataraja]

Hello Piers, 

I have tried your playground snippet and the snippet I gave here. Both dont 
work, what I meant is saving the Marshalled key into a file and then using 
that to be processed by other applications (java, python, openssl command). 
This is with go 1.11.5, do you think there is some compatibility issue with 
go package here?

Thanks
Rajesh


On Friday, October 4, 2019 at 9:03:54 AM UTC-7, helloPiers wrote:
>
> For PKCS8 (rather than PKCS1), use PEM type "PRIVATE KEY" (rather than 
> "RSA PRIVATE KEY").
>
> You may be constructing the ASN1 by hand deliberately, but just in case 
> you didn't see it, there's also a standard library function 
> x509.MarshalPKCS8PrivateKey() 
> https://godoc.org/crypto/x509#MarshalPKCS8PrivateKey 
>
> This can take the output of rsa.GenerateKey() directly, for example like: 
> https://play.golang.org/p/UzWACWh2TCo  (key size reduced so it runs in 
> the playground without timing out).
>
> On Friday, October 4, 2019 at 1:14:15 AM UTC+1, rajesh nataraja wrote:
>>
>> Hi All, 
>>
>> I have the following piece of code to generate a private key in PKCS8 
>> form and save it in a file. It does generate a file, but when I try to 
>> check using the openssl command 
>>
>> openssl rsa -in rsapk.key -check 
>> I get the following errors 
>>
>> 140092967139232:error:0D0680A8:asn1 encoding 
>> routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1199:
>> 140092967139232:error:0D06C03A:asn1 encoding 
>> routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:767:
>> 140092967139232:error:0D08303A:asn1 encoding 
>> routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:699:Field=n, 
>> Type=RSA
>> 140092967139232:error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA 
>> lib:rsa_ameth.c:121:
>>
>>
>> Anyone knows what is wrong with my method?
>>
>> package main
>>
>> import (
>> "crypto/x509"
>> "crypto/rsa"
>> "encoding/pem"
>> "io/ioutil"
>> "crypto/rand"
>> "encoding/asn1"
>> )
>>
>> type privateKeyInfo struct {
>> Version int
>> PrivateKeyAlgorithm []asn1.ObjectIdentifier
>> PrivateKey  []byte
>> }
>>
>>
>> func NewPKCS8PrivateKey() {
>>
>> var pkey privateKeyInfo
>> var bKey []byte
>> oidPublicKeyRSA  := asn1.ObjectIdentifier{1, 2, 840, 113549, 1, 1, 1}
>>
>>
>> key, err := rsa.GenerateKey(rand.Reader, 2048)
>> if err != nil {
>> return
>> }
>>
>> pkey.Version = 0
>> pkey.PrivateKeyAlgorithm = make([]asn1.ObjectIdentifier, 1)
>> pkey.PrivateKeyAlgorithm[0] = oidPublicKeyRSA
>> pkey.PrivateKey = x509.MarshalPKCS1PrivateKey(key)
>>
>> bKey , _ = asn1.Marshal(pkey)
>>
>> block := pem.Block{Type: "RSA PRIVATE KEY", Bytes: bKey}
>>
>> ioutil.WriteFile("./rsapk.key",  pem.EncodeToMemory(&block), 0600)
>>
>> }
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/d60dc133-43b1-470a-8df0-12cc64c2c016%40googlegroups.com.

[go-nuts] Re: PKCs8 generated using go has errors when doing a openssl check

[helloPiers]

For PKCS8 (rather than PKCS1), use PEM type "PRIVATE KEY" (rather than "RSA 
PRIVATE KEY").

You may be constructing the ASN1 by hand deliberately, but just in case you 
didn't see it, there's also a standard library function 
x509.MarshalPKCS8PrivateKey() 
https://godoc.org/crypto/x509#MarshalPKCS8PrivateKey 

This can take the output of rsa.GenerateKey() directly, for example like: 
https://play.golang.org/p/UzWACWh2TCo  (key size reduced so it runs in the 
playground without timing out).

On Friday, October 4, 2019 at 1:14:15 AM UTC+1, rajesh nataraja wrote:
>
> Hi All, 
>
> I have the following piece of code to generate a private key in PKCS8 form 
> and save it in a file. It does generate a file, but when I try to check 
> using the openssl command 
>
> openssl rsa -in rsapk.key -check 
> I get the following errors 
>
> 140092967139232:error:0D0680A8:asn1 encoding 
> routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1199:
> 140092967139232:error:0D06C03A:asn1 encoding 
> routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:767:
> 140092967139232:error:0D08303A:asn1 encoding 
> routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:699:Field=n, 
> Type=RSA
> 140092967139232:error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA 
> lib:rsa_ameth.c:121:
>
>
> Anyone knows what is wrong with my method?
>
> package main
>
> import (
> "crypto/x509"
> "crypto/rsa"
> "encoding/pem"
> "io/ioutil"
> "crypto/rand"
> "encoding/asn1"
> )
>
> type privateKeyInfo struct {
> Version int
> PrivateKeyAlgorithm []asn1.ObjectIdentifier
> PrivateKey  []byte
> }
>
>
> func NewPKCS8PrivateKey() {
>
> var pkey privateKeyInfo
> var bKey []byte
> oidPublicKeyRSA  := asn1.ObjectIdentifier{1, 2, 840, 113549, 1, 1, 1}
>
>
> key, err := rsa.GenerateKey(rand.Reader, 2048)
> if err != nil {
> return
> }
>
> pkey.Version = 0
> pkey.PrivateKeyAlgorithm = make([]asn1.ObjectIdentifier, 1)
> pkey.PrivateKeyAlgorithm[0] = oidPublicKeyRSA
> pkey.PrivateKey = x509.MarshalPKCS1PrivateKey(key)
>
> bKey , _ = asn1.Marshal(pkey)
>
> block := pem.Block{Type: "RSA PRIVATE KEY", Bytes: bKey}
>
> ioutil.WriteFile("./rsapk.key",  pem.EncodeToMemory(&block), 0600)
>
> }
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/e5000151-f5d0-4eff-92e2-b4687619%40googlegroups.com.

Re: [go-nuts] Channels may not be the best solution in Go

[Jake Montgomery]

JuciÊ, I'm not going to endorse or dispute you assertions in the general 
case. 

But, I do want to point out that the OP started with "I've been working on 
a math library". It then provides a link to the library on github. It may 
be a matter of personal choice and style when writing an application, or 
code for private use. However, when one is writing a library intended for 
public use, a good programmer needs to consider things a more deeply. In 
the specific case outlined in the OP, I believe providing simple blocking 
function is the best approach. If the library returns a channel, it is 
imposing the channel overhead, as well as adding unnecessary goroutines, 
and potential complexity, to the code of the library user. The library user 
may not want or need concurrency for their specific use case, but they are 
stuck with it. On the other hand, If only a blocking function is provided, 
it is few trivial lines of code to funnel that into a channel if the user 
wants to. 

IMHO, in the OP's case it sounds like a blocking function is the correct 
API for their library. Of course, if they want to provide a channel version 
in addition, that's fine. 


On Thursday, October 3, 2019 at 1:58:57 PM UTC-4, JuciÊ Andrade wrote:

> Burak, feel free to correct me if I am wrong, but now I think I understood 
> the heart of the matter:
>
> Your approach to software development is different from mine. Nothing 
> wrong with that.
>
> . you normally write sequential code, and uses concurrent code where it 
> fits best. That is fine.
> . I use to write concurrent code, and use sequential code where it fits 
> best. That is fine as well.
>
> Concurrency mechanisms in Go are so easy to use that it allows me to take 
> that approach.
> With a little bit of caution to not create a big ball of mud[1], you can 
> write clean concurrent code.
> You said there is a synchronization overhead when a program uses channels. 
> That is true.
> On the other hand, when we divide the load among several cores we end up 
> with a net gain.
> Depending on the task at hand the difference can be staggering! I mean 15x 
> faster or more!
>
> If we consider that nearly all CPU is multicore these days [2], we will 
> soon conclude that writing concurrent code even for simple tasks makes 
> sense, in order to leverage that processing power.
>
> Your concurrent code will run keep running well in newer CPUs. Your single 
> threaded code won't.
>
> Again, nothing wrong with your approach, ok? To each its own.
>
> [1] http://laputan.org/mud/
> [2] https://www.amd.com/en/products/cpu/amd-epyc-7742
>
>
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/f5fc3aec-1f0d-4130-a757-82cded8cca29%40googlegroups.com.

Re: [go-nuts] micro review and hello gio: portable GUI code, all in Go

[Elias Naur]

> FYI, Elias gave a talk about this project at Gophercon: 
> https://www.youtube.com/watch?v=9D6eWP4peYM . 
>
> Ian 
>

And another at Gophercon UK:
https://www.youtube.com/watch?v=PxnL3-Sex3o

And there will be another (+workshop) at GoLab later this month :)

Thank you for the glowing (micro) review, Jason! It
warms my heart to know my work is appreciated.

You're also spot on: I want to see Go turn into a great
(or even preferred) choice for GUI programs and games.

-- elias.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/0854d8dc-1756-4bce-83ab-5477343cdbf4%40googlegroups.com.

Re: [go-nuts] micro review and hello gio: portable GUI code, all in Go

[Ian Lance Taylor]

On Thu, Oct 3, 2019 at 10:08 PM Jason E. Aten  wrote:
>
> Somehow I missed Gio ( https://gioui.org/ ) on its first announcement earlier 
> this year.
>
> Gio -- as a pure Go GUI framework -- really ought to be better known.
>
> Hence I offer here a short review, and a getting started example project.
>
>
> A) The micro review:
>
> I am impressed. Gio provides an immediate mode (read sane, no callback soup) 
> graphics framework. Animation and rendering were smooth.
>
> Supporting mobile, wasm, macOS desktop, iOS, android, windows desktop, and 
> wayland linux -- with x11 linux in the works, I'm told, is a bold and 
> stunning prospect.
>
>  I've verified myself that macOS and Windows work well.  The same smooth 
> experience across platforms that we've come to expect from Go command line 
> tools. But now in GUI form.  A game engine based on Gio cannot be far behind 
> (anyone want a fun project?!)
>
> One caveat on Windows -- you may need to ship along the ANGLE DLLs from 
> Chrome that are included in hello_gio below.  Eventually, the author of Gio, 
> Elias Naur, posted that he plans to also have Vulkan underneath, which might 
> prove more native to Windows. But relying on the same battle hardened OpenGL 
> translation layer DLLs that Chrome and Firefox utilize on Windows is a minor 
> packaging addendum at worst.
>
> While still young and evolving quickly, I appreciate the ambition of Gio. 
> What if Go were the preferred language for writing (portable) Apps and Games 
> of all kinds?
>
>
> B) The example, starter code:
>
> I think it is worth posting my hello world experience with Gio. Both as a 
> demonstration of its prowess, and a guide to help
> others getting started. Should you be so inclined, hello_gio will get you 
> drawing graphics in no time.
>
> https://github.com/glycerine/hello_gio
>
> Any questions, raise an issue on the github issues tracker, or see the Gio 
> mailing list https://lists.sr.ht/~eliasnaur/gio


FYI, Elias gave a talk about this project at Gophercon:
https://www.youtube.com/watch?v=9D6eWP4peYM .

Ian

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAOyqgcV9bGNpz4L6b%2B%2Beo5m5ojZSYP-Fi5bv89XDmzRtAbWXcQ%40mail.gmail.com.

Re: [go-nuts] Channels may not be the best solution in Go

[Michael Jones]

Travis, glad to hear that you are exploring Harshad Numbers. It is an area
where I have done more than a decade of work and I did not know that anyone
else even cared about them. If you ever want to know how many thousand
digit (or whatever) base 10 (or whatever) numbers have the Harshad
property, and of those, how many of their digits are 3 or 11 (or whatever),
let me know. I have a Go-language program that uses channels that can
answer the question. ;-)

Michael

On Thu, Oct 3, 2019 at 12:45 PM burak serdar  wrote:

> On Thu, Oct 3, 2019 at 11:59 AM JuciÊ Andrade  wrote:
> >
> > Burak, feel free to correct me if I am wrong, but now I think I
> understood the heart of the matter:
> >
> > Your approach to software development is different from mine. Nothing
> wrong with that.
> >
> > . you normally write sequential code, and uses concurrent code where it
> fits best. That is fine.
> > . I use to write concurrent code, and use sequential code where it fits
> best. That is fine as well.
> >
> > Concurrency mechanisms in Go are so easy to use that it allows me to
> take that approach.
> > With a little bit of caution to not create a big ball of mud[1], you can
> write clean concurrent code.
> > You said there is a synchronization overhead when a program uses
> channels. That is true.
> > On the other hand, when we divide the load among several cores we end up
> with a net gain.
> > Depending on the task at hand the difference can be staggering! I mean
> 15x faster or more!
> >
> > If we consider that nearly all CPU is multicore these days [2], we will
> soon conclude that writing concurrent code even for simple tasks makes
> sense, in order to leverage that processing power.
> >
> > Your concurrent code will run keep running well in newer CPUs. Your
> single threaded code won't.
>
> Not all programs benefit from concurrency. Writing concurrent code for
> essentially sequential programs will not benefit from multiple cores,
> like generating prime numbers. Do not forget that concurrency includes
> overhead for context switch and memory barriers. Using channels in a
> sequential program is I think misuse of channels. There is no
> performance gain in having a sequence of goroutines, each waiting for
> the previous one the complete.
>
>
> >
> > Again, nothing wrong with your approach, ok? To each its own.
> >
> > [1] http://laputan.org/mud/
> > [2] https://www.amd.com/en/products/cpu/amd-epyc-7742
> >
> >
> >
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "golang-nuts" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an email to golang-nuts+unsubscr...@googlegroups.com.
> > To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/41a107de-e426-456a-abab-d0b058e39373%40googlegroups.com
> .
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/CAMV2RqqGSQd61EE_UfdMYKYAWbZ1mXK8mTPDLv18igE%2BA2R0OQ%40mail.gmail.com
> .
>


-- 

*Michael T. jonesmichael.jo...@gmail.com *

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CALoEmQxhWppWEJ4EjdzNEZE2MH7pvuE2573LLn2X3i-0wFGH%2BQ%40mail.gmail.com.

[go-nuts] Re: Clarification on unsafe conversion between string <-> []byte

[francis]

I wrote a self contained implementation of the solution described by Keith 
Randall.

It seemed at the end of this long thread it would be nice to have something 
concrete to look at.

https://github.com/fmstephe/unsafeutil

On Friday, October 4, 2019 at 11:38:22 AM UTC+2, Francis Stephens wrote:
>
> Serhat,
>
> That implementation looks very tidy. But it still uses uintptr. So it 
> doesn't solve the GC problems discussed above.
>
> On Thursday, September 26, 2019 at 10:59:08 PM UTC+2, Serhat Şevki Dinçer 
> wrote:
>>
>> Hi,
>>
>> I wrote a string utility library  with 
>> many tests to ensure the conversion assumptions are correct. It could be 
>> helpful.
>>
>> Cheers..
>>
>> On Wednesday, September 18, 2019 at 12:42:31 PM UTC+3, Francis wrote:
>>>
>>> I am looking at the correct way to convert from a byte slice to a string 
>>> and back with no allocations. All very unsafe.
>>>
>>> I think these two cases are fairly symmetrical. So to simplify the 
>>> discussion below I will only talk about converting from a string to []byte.
>>>
>>> func StringToBytes(s string) (b []byte)
>>>
>>> From what I have read it is currently not clear how to perform this 
>>> correctly.
>>>
>>> When I say correctly I mean that the function returns a `[]byte` which 
>>> contains all of and only the bytes in the string and never confuses the 
>>> garbage collector. We fully expect that the `[]byte` returned will contain 
>>> the same underlying memory as the string and modifying its contents will 
>>> modify the string, making the string dangerously mutable. We are 
>>> comfortable with the dangerously mutable string.
>>>
>>> Following the directions in unsafe you _might_ think that this would be 
>>> a good solution.
>>>
>>> func StringToBytes(s string) []byte {
>>> return *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{
>>> Data: 
>>> (*(*reflect.StringHeader)(unsafe.Pointer(&s))).Data,
>>> Len:  len(s),
>>> Cap:  len(s),
>>> }))
>>> }
>>>
>>> The line
>>>
>>> Data: (*(*reflect.StringHeader)(unsafe.Pointer(&s))).Data,
>>>
>>> here is a really carefully commented example of this approach from github
>>>
>>> seems to satisfy unsafe  rule 5 about 
>>> converting uintptr to and from unsafe.Pointer in a singe expression.
>>>
>>> However, it clearly violates rule 6 which states `...SliceHeader and 
>>> StringHeader are only valid when interpreting the content of an actual 
>>> slice or string value.`. The `[]byte` we are returning here is built from a 
>>> `&reflect.SliceHeader` and not based on an existing `[]byte`.
>>>
>>> So we can switch to
>>>
>>> func StringToBytes(s string) (b []byte) {
>>> stringHeader := (*reflect.StringHeader)(unsafe.Pointer(&s))
>>> sliceHeader := (*reflect.SliceHeader)(unsafe.Pointer(&bytes))
>>> sliceHeader.Data = stringHeader.Data
>>> sliceHeader.Len = stringHeader.Len
>>> sliceHeader.Cap = stringHeader.Len
>>> return b
>>> }
>>>
>>> Now we are using an existing []byte to build `sliceHeader` which is 
>>> good. But we end up with a new problem. sliceHeader.Data and 
>>> stringHeader.Data are both uintptr. So by creating them in one expression 
>>> and then writing them in another expression we violate the rule that 
>>> `uintptr cannot be stored in variable`.
>>>
>>> There is a possible sense that we are protected because both of our 
>>> `uinptr`s are actually real pointers inside a real string and []byte. This 
>>> seems to be indicated by the line `In this usage hdr.Data is really an 
>>> alternate way to refer to the underlying pointer in the string header, not 
>>> a uintptr variable itself.`
>>>
>>> This feels very unclear to me.
>>>
>>> In particular the code example in the unsafe package
>>>
>>> var s string
>>> hdr := (*reflect.StringHeader)(unsafe.Pointer(&s)) // case 1
>>> hdr.Data = uintptr(unsafe.Pointer(p))  // case 6 (this case)
>>> hdr.Len = n
>>>
>>>
>>> is not the same as the case we are dealing with here. Specifically in 
>>> the unsafe package documentation we are writing from a uintpr stored in a 
>>> separate variable to another uinptr. They are probably very similar in 
>>> practice, but it isn't obvious and in my experience subtly incorrect code 
>>> often comes from relying on vague understandings of important documents.
>>>
>>> If we assume that our uinptrs are safe because they are backed by real 
>>> pointers then there is another issue with our string being garbage 
>>> collected.
>>>
>>> func StringToBytes(s string) (b []byte) {
>>> stringHeader := (*reflect.StringHeader)(unsafe.Pointer(&s))
>>> // Our string is no longer referenced anywhere and could 
>>> potentially be garbage collected
>>> sliceHeader := (*reflect.SliceHeader)(unsafe.Pointer(&bytes))
>>> sliceHeader.Data = stringHeader.Data
>>> sliceHeader.Len = stringHeader.Len
>>> sliceH

[go-nuts] Re: Clarification on unsafe conversion between string <-> []byte

[francis]

Serhat,

That implementation looks very tidy. But it still uses uintptr. So it 
doesn't solve the GC problems discussed above.

On Thursday, September 26, 2019 at 10:59:08 PM UTC+2, Serhat Şevki Dinçer 
wrote:
>
> Hi,
>
> I wrote a string utility library  with many 
> tests to ensure the conversion assumptions are correct. It could be helpful.
>
> Cheers..
>
> On Wednesday, September 18, 2019 at 12:42:31 PM UTC+3, Francis wrote:
>>
>> I am looking at the correct way to convert from a byte slice to a string 
>> and back with no allocations. All very unsafe.
>>
>> I think these two cases are fairly symmetrical. So to simplify the 
>> discussion below I will only talk about converting from a string to []byte.
>>
>> func StringToBytes(s string) (b []byte)
>>
>> From what I have read it is currently not clear how to perform this 
>> correctly.
>>
>> When I say correctly I mean that the function returns a `[]byte` which 
>> contains all of and only the bytes in the string and never confuses the 
>> garbage collector. We fully expect that the `[]byte` returned will contain 
>> the same underlying memory as the string and modifying its contents will 
>> modify the string, making the string dangerously mutable. We are 
>> comfortable with the dangerously mutable string.
>>
>> Following the directions in unsafe you _might_ think that this would be a 
>> good solution.
>>
>> func StringToBytes(s string) []byte {
>> return *(*[]byte)(unsafe.Pointer(&reflect.SliceHeader{
>> Data: (*(*reflect.StringHeader)(unsafe.Pointer(&s))).Data,
>> Len:  len(s),
>> Cap:  len(s),
>> }))
>> }
>>
>> The line
>>
>> Data: (*(*reflect.StringHeader)(unsafe.Pointer(&s))).Data,
>>
>> here is a really carefully commented example of this approach from github
>>
>> seems to satisfy unsafe  rule 5 about 
>> converting uintptr to and from unsafe.Pointer in a singe expression.
>>
>> However, it clearly violates rule 6 which states `...SliceHeader and 
>> StringHeader are only valid when interpreting the content of an actual 
>> slice or string value.`. The `[]byte` we are returning here is built from a 
>> `&reflect.SliceHeader` and not based on an existing `[]byte`.
>>
>> So we can switch to
>>
>> func StringToBytes(s string) (b []byte) {
>> stringHeader := (*reflect.StringHeader)(unsafe.Pointer(&s))
>> sliceHeader := (*reflect.SliceHeader)(unsafe.Pointer(&bytes))
>> sliceHeader.Data = stringHeader.Data
>> sliceHeader.Len = stringHeader.Len
>> sliceHeader.Cap = stringHeader.Len
>> return b
>> }
>>
>> Now we are using an existing []byte to build `sliceHeader` which is good. 
>> But we end up with a new problem. sliceHeader.Data and stringHeader.Data 
>> are both uintptr. So by creating them in one expression and then writing 
>> them in another expression we violate the rule that `uintptr cannot be 
>> stored in variable`.
>>
>> There is a possible sense that we are protected because both of our 
>> `uinptr`s are actually real pointers inside a real string and []byte. This 
>> seems to be indicated by the line `In this usage hdr.Data is really an 
>> alternate way to refer to the underlying pointer in the string header, not 
>> a uintptr variable itself.`
>>
>> This feels very unclear to me.
>>
>> In particular the code example in the unsafe package
>>
>> var s string
>> hdr := (*reflect.StringHeader)(unsafe.Pointer(&s)) // case 1
>> hdr.Data = uintptr(unsafe.Pointer(p))  // case 6 (this case)
>> hdr.Len = n
>>
>>
>> is not the same as the case we are dealing with here. Specifically in the 
>> unsafe package documentation we are writing from a uintpr stored in a 
>> separate variable to another uinptr. They are probably very similar in 
>> practice, but it isn't obvious and in my experience subtly incorrect code 
>> often comes from relying on vague understandings of important documents.
>>
>> If we assume that our uinptrs are safe because they are backed by real 
>> pointers then there is another issue with our string being garbage 
>> collected.
>>
>> func StringToBytes(s string) (b []byte) {
>> stringHeader := (*reflect.StringHeader)(unsafe.Pointer(&s))
>> // Our string is no longer referenced anywhere and could 
>> potentially be garbage collected
>> sliceHeader := (*reflect.SliceHeader)(unsafe.Pointer(&bytes))
>> sliceHeader.Data = stringHeader.Data
>> sliceHeader.Len = stringHeader.Len
>> sliceHeader.Cap = stringHeader.Len
>> return b
>> }
>>
>>
>> There is a discussion where this potential problem is raised
>>
>> https://github.com/golang/go/issues/25484
>>
>> we also see this issue mentioned in
>>
>> https://groups.google.com/forum/#!msg/golang-nuts/dcjzJy-bSpw/tcZYBzQqAQAJ
>>
>> The solution of 
>>
>> func StringToBytes(s string) (b []byte) {
>> stringHeader := (*reflect.StringHeader)(unsafe.P