On Monday, April 27, 2020 at 10:00:52 PM UTC-7, Ian Lance Taylor wrote:
>
> On Mon, Apr 27, 2020 at 6:59 PM Liam <networ...@gmail.com <javascript:>>
> wrote:
> >
> > On Monday, April 27, 2020 at 5:56:52 PM UTC-7, Ian Lance Taylor wrote:
> >>
> >> On Mon, Apr 27, 2020 at 5:10 PM Liam <networ...@gmail.com> wrote:
> >> >
> >> > On Monday, April 27, 2020 at 4:22:41 PM UTC-7, Ian Lance Taylor
> wrote:
> >> >>
> >> >> On Sun, Apr 26, 2020 at 4:55 PM Liam <networ...@gmail.com> wrote:
> >> >> >
> >> >> > During an io.Copy() where the Writer is a TCPConn and the Reader
> is a 200K disk file, my code may concurrently Write() on the same TCPConn.
> >> >> >
> >> >> > I see the result of the Write() inserted into the result of the
> io.Copy(). I had the impression that was impossible, but I must be
> mistaken, as the sendfile(2) docs read:
> >> >> >
> >> >> > Note that a successful call to sendfile() may write fewer bytes
> than requested; the caller should be prepared to retry the call if there
> were unsent bytes.
> >> >> >
> >> >> > Could someone confirm that one must indeed synchronize concurrent
> use of tcpConn.Write() and io.Copy(tcpConn, file)?
> >> >>
> >> >> Synchronization should not be required. internal/poll.Sendfile
> >> >> acquires a write lock on dstFD, which is the TCP socket. That
> should
> >> >> ensure that the contents of an ordinary Write (which also acquires a
> >> >> write lock) should not interleave with the sendfile data.
> >> >>
> >> >> That said, if the sendfile system call cannot be used for whatever
> >> >> reason, the net package will fall back on doing ordinary Read and
> >> >> Write calls. And those Write calls can be interleaved with other
> >> >> Write calls done by a different goroutine. I think that is probably
> >> >> permitted, in that io.Copy doesn't promise to not interleave with
> >> >> simultaneous Write calls on the destination.
> >> >>
> >> >> So in the general case you should indeed use your own locking to
> avoid
> >> >> interleaving between io.Copy and a concurrent Write.
> >> >
> >> >
> >> > Thanks for the details. Where could I add a Println() to reveal why
> it doesn't call poll.Sendfile()?
> >> >
> >> > I expect this system to use sendfile(2). The file is a normal file on
> a local partition (running on a Digital Ocean Droplet).
> >> >
> >> >
> >> > /etc/fstab has:
> >> > UUID=[omitted] / ext4 defaults 1 1
> >> >
> >> >
> >> > $ df -h
> >> > Filesystem Size Used Avail Use% Mounted on
> >> > devtmpfs 981M 0 981M 0% /dev
> >> > tmpfs 996M 0 996M 0% /dev/shm
> >> > tmpfs 996M 436K 995M 1% /run
> >> > tmpfs 996M 0 996M 0% /sys/fs/cgroup
> >> > /dev/vda1 59G 5.7G 51G 11% /
> >> > tmpfs 200M 0 200M 0% /run/user/0
>
Well this is a surprise... Added some println()s
// my network setup
aCfgTcp := net.ListenConfig{KeepAlive: -1}
aListener, err := aCfgTcp.Listen(nil, iConf.Listen.Net,
iConf.Listen.Laddr)
if err != nil { return err }
aCert, err := tls.LoadX509KeyPair(iConf.Listen.CertPath,
iConf.Listen.KeyPath)
if err != nil { return err }
aCfgTls := tls.Config{Certificates: []tls.Certificate{aCert}}
aListener = tls.NewListener(aListener, &aCfgTls)
// conn writer goroutine, before io.Copy
println(".. io.Copy to net")
// conn reader goroutine, before io.CopyN
println(".. io.CopyN to file")
// package io
func copyBuffer(dst Writer, src Reader, buf []byte) (written int64, err
error) {
// If the reader has a WriteTo method, use it to do the copy.
// Avoids an allocation and a copy.
if wt, ok := src.(WriterTo); ok {
println(".. WriteTo")
return wt.WriteTo(dst)
}
// Similarly, if the writer has a ReadFrom method, use it to do the
copy.
if rt, ok := dst.(ReaderFrom); ok {
println(".. ReadFrom")
return rt.ReadFrom(src)
}
println(".. manual")
[etc]
// package net
func (c *TCPConn) readFrom(r io.Reader) (int64, error) {
if n, err, handled := splice(c.fd, r); handled {
println(".. spliced")
return n, err
}
if n, err, handled := sendFile(c.fd, r); handled {
println(".. sendFiled")
return n, err
}
println(".. generic")
return genericReadFrom(c, r)
LOG:
.. io.CopyN to file
.. manual
.. io.Copy to net
.. manual
.. io.CopyN to file
.. manual
.. io.Copy to net
.. manual
.. io.CopyN to file
.. manual
.. io.Copy to net
.. manual
.. io.Copy to net
.. manual
.. io.Copy to net
.. manual
.. io.Copy to net
.. manual
Not a sendfile(2) or splice(2) in sight :-(
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/45642e1e-a9d6-4cd2-9e1f-ea1499a530b5%40googlegroups.com.
