Re: [go-nuts] Goroutines, Nonblocking I/O, And Memory Usage

Hi! On Thu, Mar 15, 2018 at 09:09:18AM -0700, Rio wrote: > Yeah, there's no way to avoid the kernel buffer waste. You can tune kernel buffer size to minimize it. > > I did such a custom poller for a network proxy. It was a pain > > to get right (I only did the Eww!poll version). > > I tried

Re: [go-nuts] Goroutines, Nonblocking I/O, And Memory Usage

On Thu, Mar 15, 2018 at 12:04 PM, Rio wrote: > > On Friday, March 16, 2018 at 1:05:12 AM UTC+8, Ian Lance Taylor wrote: >> >> >> That's inherently racy, though. It's quite normal to have multiple >> goroutines reading from the same socket. That is awkward if we split >> up

Re: [go-nuts] Goroutines, Nonblocking I/O, And Memory Usage

On Friday, March 16, 2018 at 1:05:12 AM UTC+8, Ian Lance Taylor wrote: > > > That's inherently racy, though. It's quite normal to have multiple > goroutines reading from the same socket. That is awkward if we split > up WaitRead and Read. Better to have a single wait-then-read > operation,

Re: [go-nuts] Goroutines, Nonblocking I/O, And Memory Usage

On Wed, 14 Mar 2018 12:37:22 -0700 Ian Lance Taylor wrote: Ian Lance Taylor writes: > On Wed, Mar 14, 2018 at 11:58 AM, Rio wrote: > > > > While implementing a SOCKS proxy in Go, I ran into an issue which is better > > explained in details by Evan Klitzke in

Re: [go-nuts] Goroutines, Nonblocking I/O, And Memory Usage

What about something like: type TimedReader interface { TimedRead(out []byte,timeout int) (int,error) } so r.TimedRead(buf,0) becomes a non-blocking read? On Thu, Mar 15, 2018 at 11:23 AM, Michael Jones wrote: > What about: "wait, then get a buffer from a pool, and

Re: [go-nuts] Goroutines, Nonblocking I/O, And Memory Usage

What about: "wait, then get a buffer from a pool, and return. client uses buffer, then restores it." The present API could be used if the input buffer was presumed to go directly to that pool if not null, or, if a null pool argument means what i suggest and a new

Re: [go-nuts] Goroutines, Nonblocking I/O, And Memory Usage

On Thu, Mar 15, 2018 at 8:58 AM, Rio wrote: > > On Thursday, March 15, 2018 at 3:37:51 AM UTC+8, Ian Lance Taylor wrote: >> >> >> Even for TCP, that's an interesting point. I wonder if we should have >> a way to specify a number of bytes to read such that we only allocate >>

Re: [go-nuts] Goroutines, Nonblocking I/O, And Memory Usage

On Thursday, March 15, 2018 at 4:23:41 AM UTC+8, Bakul Shah wrote: > > > You will need 10K*64KiB*2 = 1.22GiB kernel bufferspace at a > minimum in any case. That is still one datagram's worth of > buffering per UDP socket so you can still lose some packets. > This is in addition to any user

Re: [go-nuts] Goroutines, Nonblocking I/O, And Memory Usage

On Thursday, March 15, 2018 at 3:37:51 AM UTC+8, Ian Lance Taylor wrote: > > > Even for TCP, that's an interesting point. I wonder if we should have > a way to specify a number of bytes to read such that we only allocate > the []byte when there is something to read. > I was thinking a

Re: [go-nuts] Goroutines, Nonblocking I/O, And Memory Usage

On Wed, Mar 14, 2018 at 11:58 AM, Rio wrote: > > While implementing a SOCKS proxy in Go, I ran into an issue which is better > explained in details by Evan Klitzke in this post > https://eklitzke.org/goroutines-nonblocking-io-and-memory-usage > > In my case, each proxied

[go-nuts] Goroutines, Nonblocking I/O, And Memory Usage

While implementing a SOCKS proxy in Go, I ran into an issue which is better explained in details by Evan Klitzke in this post https://eklitzke.org/goroutines-nonblocking-io-and-memory-usage In my case, each proxied connection costs two goroutines and two buffers in blocking read. For TCP