[go-nuts] Re: How to go-imports subdirectory?

2020-06-09 Thread Tamás Gulácsi 
Yes. But then you could use go get to import it.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/8a20c3f9-9790-4e28-b6d3-3c2328b023f5o%40googlegroups.com.

Re: [go-nuts] Re: gzip.Reader.Read does not fill the given buffer

2020-06-09 Thread Amit Lavon 
Interesting points. So I guess ReadFull can be suitable for the consuming
end of those bytes.

Thank you!

On Tue, Jun 9, 2020 at 11:40 PM Brian Candler  wrote:

> On Tuesday, 9 June 2020 17:53:07 UTC+1, Amit Lavon wrote:
>>
>> Why would I ever use Reader.Read rather than ReadFull?
>>
>>
> Because it's often more efficient to process data in the "natural" chunks
> it comes in, rather than packing it out to fit a particular buffer size -
> in which case the final read may be split.
>
> For example, with HTTP chunked encoding, I'd expect each Read to give me
> one chunk - as long as it fits in the provided buffer, of course.
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "golang-nuts" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/golang-nuts/u-wNH3NyMHo/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/cbfde5a8-89eb-43f3-9f2f-1259df65cb8ao%40googlegroups.com
> 
> .
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CAPTkDQWOFsDjDbPSgprJkpfC%2BifNKHWaP4nT9jY8hYXP_jc3Zg%40mail.gmail.com.

Re: [go-nuts] Re: gzip.Reader.Read does not fill the given buffer

2020-06-09 Thread Brian Candler 
On Tuesday, 9 June 2020 17:53:07 UTC+1, Amit Lavon wrote:
>
> Why would I ever use Reader.Read rather than ReadFull?
>
>
Because it's often more efficient to process data in the "natural" chunks 
it comes in, rather than packing it out to fit a particular buffer size - 
in which case the final read may be split.

For example, with HTTP chunked encoding, I'd expect each Read to give me 
one chunk - as long as it fits in the provided buffer, of course.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/cbfde5a8-89eb-43f3-9f2f-1259df65cb8ao%40googlegroups.com.

Re: [go-nuts] Re: gzip.Reader.Read does not fill the given buffer

2020-06-09 Thread Sam Whited 
Read lets you build pipelines, it involves fewer expensive allocations
(ie. you might not want to use ReadFull in the hot path of an important
project), you could use read to read into a slice at different points,
or not read the entirety of an expensive document into memory all at
once, you can implement buffering on top of it, etc.

It's probably pretty rare that you actually want to use ReadFull, or at
least, I don't find myself reaching for it very often.

—Sam


On Tue, Jun 9, 2020, at 12:51, Amit Lavon wrote:
> Thank you!! io.ReadFull is just what I needed (and what I actually
> expected from Reader.Read). Why would I ever use Reader.Read rather
> than ReadFull?
>
> On Tue, Jun 9, 2020 at 6:11 PM Brian Candler
>  wrote:
> > There is io.ReadFull  if you
> > want to read as much as it can into the preallocated buffer; you'll
> > get io.ErrUnexpectedEOF if it's less than this.
> > https://play.golang.org/p/gIAX046vNvW
> >
> > There is ioutil.ReadAll 
> > if you want to allocate memory to read the entire stream into a
> > single buffer (assuming you have enough RAM)
> > https://play.golang.org/p/Pg17S6A74SN
>
> >  --
> >  You received this message because you are subscribed to a topic in
> >  the Google Groups "golang-nuts" group. To unsubscribe from this
> >  topic, visit
> >  https://groups.google.com/d/topic/golang-nuts/u-wNH3NyMHo/unsubscribe.
> >  To unsubscribe from this group and all its topics, send an email to
> >  golang-nuts+unsubscr...@googlegroups.com. To view this discussion
> >  on the web visit
> >  
> > https://groups.google.com/d/msgid/golang-nuts/8ff054df-218c-4d52-8984-83b1ced9o%40googlegroups.com
> >  
> > 
> >  .
>
>  --
>  You received this message because you are subscribed to the Google
>  Groups "golang-nuts" group. To unsubscribe from this group and stop
>  receiving emails from it, send an email to golang-
>  nuts+unsubscr...@googlegroups.com. To view this discussion on the web
>  visit
>  
> https://groups.google.com/d/msgid/golang-nuts/CAPTkDQVudE24DF6tWBO6yFTyF4TgZOopEfjnqZXLhVphp8SBwQ%40mail.gmail.com
>  
> 
>  .

-- 
Sam Whited

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/b203a7e4-0275-46c5-9b1e-05ed7a140e8e%40www.fastmail.com.

Re: [go-nuts] Re: what is the complexity of regexp.MustCompile()?

2020-06-09 Thread 'Thomas Bushnell, BSG' via golang-nuts 
On Tue, Jun 9, 2020 at 11:23 AM Axel Wagner 
wrote:

> If you actually read OPs second E-Mail, they did and they didn't find it
> very clear. With that in mind, your message isn't very nice.
> (Also, not to be repetitive or anything: ~80% of the messages in this
> thread aren't actually concerned with what the complexity class *is*, but
> whether it matters)
>

The OP stopped participating in this exciting thread a long time ago. I
went and read through the code, and it seems pretty clear to me that it's
linear.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/CA%2BYjuxvQgnTKMM4FF4%3D4pspM-2nBJScNfCNinDv-2cNA09N%3DaQ%40mail.gmail.com.

Re: [go-nuts] Re: what is the complexity of regexp.MustCompile()?

2020-06-09 Thread 'Axel Wagner' via golang-nuts 
If you actually read OPs second E-Mail, they did and they didn't find it
very clear. With that in mind, your message isn't very nice.
(Also, not to be repetitive or anything: ~80% of the messages in this
thread aren't actually concerned with what the complexity class *is*, but
whether it matters)

On Tue, Jun 9, 2020 at 5:12 PM 'Thomas Bushnell, BSG' via golang-nuts <
golang-nuts@googlegroups.com> wrote:

> I'm surprised none of you have taken all this time to just go read the
> code, where it is clearly linear.
>
> On Mon, Jun 8, 2020 at 9:12 PM Robert Engels 
> wrote:
>
>> The OP specifically asked about compilation - in fact it’s in the title.
>> You stated the compilation complexity is a DoS attack vector. I argued that
>> it is not. That is all.
>>
>> On Jun 8, 2020, at 6:39 PM, Michael Jones 
>> wrote:
>>
>> 
>> Ray, only the discussion is exponential.
>>
>> On Mon, Jun 8, 2020 at 4:23 PM 'Axel Wagner' via golang-nuts <
>> golang-nuts@googlegroups.com> wrote:
>>
>>> On Tue, Jun 9, 2020 at 12:48 AM Kurtis Rader 
>>> wrote:
>>>
 You're talking past each other. Robert is talking about limiting the
 length of the regex, not the string(s) evaluated by the regex.

>>>
>>> So am I. Note that e.g. a Code Search based on PCRE would break, even if
>>> you limit *both* (or rather, any limit causing it to not break would result
>>> in a completely useless piece of software).
>>>
>>> It should be possible to compile any regex of a reasonable length in a
 matter of microseconds. Regardless of whether the application of the regex
 to a given input is near linear (as in the case of the Go RE
 implementation) or exponential (as in the case of PCRE).

>>>
>>> This might be, where we talk past each other. I am using application as
>>> an example for concrete numbers on how quickly exponential growth can
>>> devolve. Of course, compilation of a regexp is fast - I am aware of that.
>>> As it turns out, so is application of a regexp, if you use RE2 (or Go's
>>> regexp package).
>>>
>>> What I take issue with are the statements that a) the question about the
>>> complexity of compiling a regexp is irrelevant, because b) limiting the
>>> algorithmic complexity of a function to counteract resource exhaustion
>>> attacks "never works". RE2 is an excellent example to show that it does
>>> work; it is carefully designed for linear complexity of the combined
>>> compilation+matching of a regular expression.
>>>
>>> I'm not saying compiling a regular expression is an
>>> exponential operation. I'm saying *if it was*, you could never reasonably
>>> build something like Code Search. And we can use the fact that
>>> *application* indeed is (in most engines) exponential in the combined input
>>> length of expression+search text as a good basis to make that case.
>>>
>>> Of course we don't even need this fantasy world to make the case - after
>>> all, saying "you can't build Code Search on PCRE, but you can on RE2" is
>>> just as effective to prove the effectiveness of lowering algorithmic
>>> complexity. It's just that OP's question was about compilation, so to talk
>>> about why OP's question *is* relevant, we need to talk about what would
>>> happen if the answer *wasn't* "it's linear".
>>>
>>> (I also genuinely don't understand the instinct to tell someone "why
>>> would you care?" instead of telling them the answer, FWIW. Especially in a
>>> case like this, where the answer is pretty simple)
>>>
>>> I'm pretty sure Robert is not arguing that the scaling problems of the
 regex implementation used by Perl, and too many others, can be mitigated
 simply by limiting the size of the string to be matched by the regex. If
 compiling a regex of reasonable length takes a non-negligible amount of
 time something is wrong.

 On Mon, Jun 8, 2020 at 3:22 PM Wojciech S. Czarnecki 
 wrote:

> Dnia 2020-06-08, o godz. 16:22:24
> Robert Engels  napisał(a):
>
> > it is trivial to limit the input size to something a user could
> input.
>
> With exponential complexity simple regex /(x+x+)+y/ blows up at input
> of 20 to 30 x-es.
> See: https://www.regular-expressions.info/catastrophic.html
>
> [Cut long explanations... Axel just posted most of what I was writing
> regarding trade-offs).
>
> Hope this helps,
>
> --
> Wojciech S. Czarnecki
>  << ^oo^ >> OHIR-RIPE
>
> --
> You received this message because you are subscribed to the Google
> Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/20200609002207.0a161adf%40xmint
> .
>


 --
 Kurtis Rader
 Caretaker of the exceptional canines Junior and Hank

 --
 You received this message because you are subscribed

Re: [go-nuts] Re: what is the complexity of regexp.MustCompile()?

2020-06-09 Thread 'Thomas Bushnell, BSG' via golang-nuts 
I'm surprised none of you have taken all this time to just go read the
code, where it is clearly linear.

On Mon, Jun 8, 2020 at 9:12 PM Robert Engels  wrote:

> The OP specifically asked about compilation - in fact it’s in the title.
> You stated the compilation complexity is a DoS attack vector. I argued that
> it is not. That is all.
>
> On Jun 8, 2020, at 6:39 PM, Michael Jones  wrote:
>
> 
> Ray, only the discussion is exponential.
>
> On Mon, Jun 8, 2020 at 4:23 PM 'Axel Wagner' via golang-nuts <
> golang-nuts@googlegroups.com> wrote:
>
>> On Tue, Jun 9, 2020 at 12:48 AM Kurtis Rader 
>> wrote:
>>
>>> You're talking past each other. Robert is talking about limiting the
>>> length of the regex, not the string(s) evaluated by the regex.
>>>
>>
>> So am I. Note that e.g. a Code Search based on PCRE would break, even if
>> you limit *both* (or rather, any limit causing it to not break would result
>> in a completely useless piece of software).
>>
>> It should be possible to compile any regex of a reasonable length in a
>>> matter of microseconds. Regardless of whether the application of the regex
>>> to a given input is near linear (as in the case of the Go RE
>>> implementation) or exponential (as in the case of PCRE).
>>>
>>
>> This might be, where we talk past each other. I am using application as
>> an example for concrete numbers on how quickly exponential growth can
>> devolve. Of course, compilation of a regexp is fast - I am aware of that.
>> As it turns out, so is application of a regexp, if you use RE2 (or Go's
>> regexp package).
>>
>> What I take issue with are the statements that a) the question about the
>> complexity of compiling a regexp is irrelevant, because b) limiting the
>> algorithmic complexity of a function to counteract resource exhaustion
>> attacks "never works". RE2 is an excellent example to show that it does
>> work; it is carefully designed for linear complexity of the combined
>> compilation+matching of a regular expression.
>>
>> I'm not saying compiling a regular expression is an
>> exponential operation. I'm saying *if it was*, you could never reasonably
>> build something like Code Search. And we can use the fact that
>> *application* indeed is (in most engines) exponential in the combined input
>> length of expression+search text as a good basis to make that case.
>>
>> Of course we don't even need this fantasy world to make the case - after
>> all, saying "you can't build Code Search on PCRE, but you can on RE2" is
>> just as effective to prove the effectiveness of lowering algorithmic
>> complexity. It's just that OP's question was about compilation, so to talk
>> about why OP's question *is* relevant, we need to talk about what would
>> happen if the answer *wasn't* "it's linear".
>>
>> (I also genuinely don't understand the instinct to tell someone "why
>> would you care?" instead of telling them the answer, FWIW. Especially in a
>> case like this, where the answer is pretty simple)
>>
>> I'm pretty sure Robert is not arguing that the scaling problems of the
>>> regex implementation used by Perl, and too many others, can be mitigated
>>> simply by limiting the size of the string to be matched by the regex. If
>>> compiling a regex of reasonable length takes a non-negligible amount of
>>> time something is wrong.
>>>
>>> On Mon, Jun 8, 2020 at 3:22 PM Wojciech S. Czarnecki 
>>> wrote:
>>>
 Dnia 2020-06-08, o godz. 16:22:24
 Robert Engels  napisał(a):

 > it is trivial to limit the input size to something a user could input.

 With exponential complexity simple regex /(x+x+)+y/ blows up at input
 of 20 to 30 x-es.
 See: https://www.regular-expressions.info/catastrophic.html

 [Cut long explanations... Axel just posted most of what I was writing
 regarding trade-offs).

 Hope this helps,

 --
 Wojciech S. Czarnecki
  << ^oo^ >> OHIR-RIPE

 --
 You received this message because you are subscribed to the Google
 Groups "golang-nuts" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to golang-nuts+unsubscr...@googlegroups.com.
 To view this discussion on the web visit
 https://groups.google.com/d/msgid/golang-nuts/20200609002207.0a161adf%40xmint
 .

>>>
>>>
>>> --
>>> Kurtis Rader
>>> Caretaker of the exceptional canines Junior and Hank
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to golang-nuts+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/golang-nuts/CABx2%3DD9sKeHnqUAqB61y5Ts-U_f%2BctVAuS4BC0ae8phhhcp1iw%40mail.gmail.com
>>> 
>>> .
>>>
>> --
>> You

[go-nuts] Re: gzip.Reader.Read does not fill the given buffer

2020-06-09 Thread Ronny Bangsund 
On Tuesday, June 9, 2020 at 4:05:38 PM UTC+2, Amit Lavon wrote:
>
> I am reading raw bytes from a gzip input. I found that it only reads up to 
> chunks of 2^15 even though there is more data to be read.
>
> Is that the intended behavior? I expected whatever internal buffering it 
> may have to be invisible to the caller. If that's intended, how could I 
> have anticipated that? The contract of Reader 
>  says "Read conventionally returns 
> what is available instead of waiting for more" but it doesn't seem to be 
> the case here.
>
We're at the mercy of what the filesystem prefers to give us, and in my 
experience it likes many small chunks on any OS. I've seen consistent 
returns as low as 4k from plain file reading, so 32k from gzip 
decompression seems good. Behind the scenes the OS may already have 
buffered your entire file if it's not too huge, so I wouldn't worry about 
it being slower to loop through small buffers repeatedly.

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/53405eec-3335-4f47-a503-594e9372a3f7o%40googlegroups.com.

[go-nuts] gzip.Reader.Read does not fill the given buffer

2020-06-09 Thread Amit Lavon 
Hi gophers,

Demonstration: https://play.golang.org/p/AY-fVWiOrFd

I am reading raw bytes from a gzip input. I found that it only reads up to 
chunks of 2^15 even though there is more data to be read.

Is that the intended behavior? I expected whatever internal buffering it 
may have to be invisible to the caller. If that's intended, how could I 
have anticipated that? The contract of Reader 
 says "Read conventionally returns what 
is available instead of waiting for more" but it doesn't seem to be the 
case here. I call Read expecting to wait for gzip to do its internal 
processing and make the data available.

Happy to get your input.

Amit

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/1d46775e-123b-43e7-ba27-bc105d03a552o%40googlegroups.com.

Re: [go-nuts] Using context package for goroutines cancellation is quite verbose

2020-06-09 Thread Yegor Roganov 
Thanks Ian, it's nice to know that we're using Go correctly.

I agree that more code doesn't really matter, and I think I'd be completely 
fine with this situation had there been a vet check that ensured that no 
selects are forgotten.
Let's see if generics change something in this area.

On Monday, June 8, 2020 at 10:46:05 PM UTC+3, Ian Lance Taylor wrote:
>
> On Mon, Jun 8, 2020 at 9:59 AM Yegor Roganov  > wrote: 
> > 
> > My team is developing a cloud-based data processing application, and a 
> large bulk of the application's job is doing network calls to other 
> services. 
> > Go's goroutines and channel-based communication are an excellent fit, 
> but in order for the application to be able to properly shut down, 
> > simple construct of sending to a channel `myChan <- value` needs to 
> actually be something like: 
> > 
> > ``` 
> > 
> > select { 
> > case myChan <- value: 
> > case <-ctx.Done(): 
> >return ctx.Err() 
> > } 
> > 
> > ``` 
> > 
> > As you see, a simple one line construct needs to be replaced with a 
> select on context.Done EVERYWHERE. 
> > My two gripes with this situation are: 
> > 
> > 1) It's more code and "clutter" which makes code less clear 
> > 2) It's easy to forget to do a select on context.Done, and then an 
> application is unable to shut down. 
> > 
> > Am I doing something wrong? Does someone relate with me? 
>
> You are not doing anything wrong, and, you're right: it's more code. 
>
> In general, Go prefers code to be explicit rather than implicit, so I 
> don't really agree that this makes the code less clear.  I think that 
> this code is clear to anybody who knows Go, and with more experience 
> it becomes fairly idiomatic. 
>
> But you're absolutely right that it's easy to forget to do a select. 
>
> Personally I think that this is an area where generics can help.  For 
> example, from the design draft published last year, see 
>
> https://go.googlesource.com/proposal/+/refs/heads/master/design/go2draft-contracts.md#channels
>  
> .  I think the right step here is going to be see how generics can 
> help with these common patterns before trying other approaches.  I 
> acknowledge that that is a slow process. 
>
> Ian 
>

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/golang-nuts/5e27f46d-44c9-41b1-a49e-c51021e0e906o%40googlegroups.com.