I still can't repro it with your code. But that doesn't mean it's not a real condition. It sounds like the kind of odd corner of entirely legitimate UDP behavior that is hard to provoke but which a robust app should handle.
Note that the default behavior in Tulip appears to be to ignore OSError coming out of sendto() -- the transport calls protocol.error_received(), which by default does nothing. Since there are many other cases where a packet may silently be dropped on the floor, this behavior is technically correct -- the question is whether it is the best default behavior we can imagine. Unfortunately turning it into a pause_protocol() call in your error_received() handler is a little tricky -- the transport remembers whether it has paused the protocol or not, but this state is not public. So you shouldn't call your own pause_writing(), since you'd never receive a resume_writing() call from the transport. Perhaps you can set a flag internal to your protocol that just causes you to back off for a brief period of time? The optimal back-off time should be tuned experimentally. --Guido On Sun, Feb 23, 2014 at 2:45 PM, Christopher Probst < [email protected]> wrote: > I made a simpler test, without using tulip, just using plain > sockets<http://stackoverflow.com/questions/21973661/os-x-udp-send-error-55-no-buffer-space-available/21973705?noredirect=1#comment33297277_21973705> > . > > from socket import * > > udp = socket(AF_INET, SOCK_DGRAM) > udp.setsockopt(SOL_SOCKET, SO_REUSEADDR, True) > > udp.bind(('0.0.0.0', 1337)) > udp.setblocking(False) > udp.setsockopt(SOL_IP, IP_TTL, 4) > udp.connect(('8.8.8.8', 12345)) > > buf = b'x' * 400for _ in range(1024 * 1024 * 10): > udp.send(buf) > > In this test I'm just writing a lot of udp packets to 8.8.8.8, which gets > dropped after 4 hops (it's just for testing anyway). But this code actually > causes the same error (so it has nothing to do with tulip, though > flow-control might be affected). > > The weird thing is, that the following code although causes the same error: > > from socket import * > > udp = socket(AF_INET, SOCK_DGRAM) > udp.connect(('8.8.8.8', 12345)) > > buf = b'x' * 400for _ in range(1024 * 1024 * 10): > udp.send(buf) > > In other words: The "blocking" mode is not so blocking apparently on my > machine. If you cannot reproduce this error(which is good!), than this > might be an issue on my local machine. But in any case it's not a tulip bug. > > I'm just curious what this can be... > > > Am Sonntag, 23. Februar 2014 23:34:39 UTC+1 schrieb Guido van Rossum: >> >> I haven't been able to repro this using the test you attached to issue >> 153. But I don't have another machine available, I've only tried localhost >> on OSX (10.9) and from that same box to a local virtual machine running >> Ubunu. The receiving end just throws away the data -- is that your test >> setup too? I used "while 1: s.recv(10000); n += 1" in an interactive Python >> shell. >> >> I do see packet loss (just counting packets received and packets sent) >> but no exceptions nor does pause_writing() ever get called. From putting >> some print()s in the code it looks like the sendto() operation always >> immediately succeeds. >> >> The errno has a name: errno.ENOBUFS -- it's 55 on OSX, but 105 on Ubuntu. >> >> Perhaps you can elaborate on your test setup? >> >> >> On Sun, Feb 23, 2014 at 11:29 AM, Christopher Probst < >> [email protected]> wrote: >> >>> Ok, I tried it now with 180 bytes packet and it does not occur, I guess >>> that my lan connection(gigabit) is too fast that those packets get queued >>> up. Or OSX decide whether or not to queue those packets based on the size. >>> But this seems to be heavily os dependent . >>> >>> Am Sonntag, 23. Februar 2014 20:12:46 UTC+1 schrieb Guido van Rossum: >>>> >>>> Have you tried reducing the write buffer size? >>>> On Feb 23, 2014 10:41 AM, "Christopher Probst" <foxnet.d...@googlemail. >>>> com> wrote: >>>> >>>>> I saw this in the official 3.4rc1 doc. >>>>> >>>>> The doc says, that flow-control callbacks are valid for Protocols and >>>>> SubprocessProtocols, but DatagramProtocol is not specified. >>>>> >>>>> Though I'm happy that Datagram control flow is officially supported I >>>>> have now an other problem which is directly connected with this issue. >>>>> Using OSX10.9.1 and python 3.3 sending a lot of udp packets actually >>>>> does not cause the flow-control to be activated but throws an OSError(55 >>>>> *No >>>>> buffer space available)* . >>>>> >>>>> After googling around for hours I found out that this is a >>>>> BSD(probably OS X only) thing. On linux the control-flow works as >>>>> expected. >>>>> >>>>> I listed an issue for this in your tulip repo: >>>>> Issue 153 <https://code.google.com/p/tulip/issues/detail?id=153> >>>>> >>>>> I think that this is really not a python bug, but the way BSD/OSX >>>>> handles too much udp packets (the socket sdnbuf option is kind of ignored >>>>> by OSX). I've read that windows actually handles udp overload in a >>>>> similiar >>>>> way as BSD does. Though i don't have a windows machine this should >>>>> probably >>>>> be tested to confirm or disprove this issue. >>>>> >>>>> PS: >>>>> 18.5.3.2.5. Flow control >>>>> callbacks<http://docs.python.org/3.4/library/asyncio-protocol.html#flow-control-callbacks> >>>>> >>>>> These callbacks may be called on >>>>> Protocol<http://docs.python.org/3.4/library/asyncio-protocol.html#asyncio.Protocol> >>>>> and >>>>> SubprocessProtocol<http://docs.python.org/3.4/library/asyncio-protocol.html#asyncio.SubprocessProtocol> >>>>> instances: >>>>> BaseProtocol.pause_writing()<http://docs.python.org/3.4/library/asyncio-protocol.html#asyncio.BaseProtocol.pause_writing> >>>>> >>>>> Called when the transport's buffer goes over the high-water mark. >>>>> BaseProtocol.resume_writing()<http://docs.python.org/3.4/library/asyncio-protocol.html#asyncio.BaseProtocol.resume_writing> >>>>> >>>>> Called when the transport's buffer drains below the low-water mark. >>>>> >>>>> >>>>> >>>>> >>>>> Am Sonntag, 23. Februar 2014 18:56:42 UTC+1 schrieb Guido van Rossum: >>>>>> >>>>>> This looks like a bug in the docs; the intention is that datagram >>>>>> protocols also support flow control. Where does it say so in the docs? Is >>>>>> it the PEP or the CPython Doc tree? >>>>>> >>>>>> >>>>>> On Sat, Feb 22, 2014 at 5:12 PM, Christopher Probst < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> after looking into the implementation I saw that, for instance, _ >>>>>>> SelectorDatagramTransport calls _maybe_pause_protocol and it's >>>>>>> counterpart, but the doc says that only Protocol and SubprocessProtocol >>>>>>> has >>>>>>> flow-control support and DatagramProtocol does not. >>>>>>> >>>>>>> I know that udp flow-control is not the same as tcp flow-control, >>>>>>> but I'm concerned about filling up the internal buffer when writing a >>>>>>> lot >>>>>>> of datagrams. If this is not supported, I would argue that the udp >>>>>>> support is pretty much broken for data intense application because how >>>>>>> would the writer know when the internal buffer (and/or kernel level >>>>>>> buffer) >>>>>>> are full ? >>>>>>> >>>>>>> So, is the doc just not up-to-date or is it an implementation detail >>>>>>> of tulip ? >>>>>>> >>>>>>> And other question that came up: Are there any plans for coroutine >>>>>>> methods for udp (like StreamWriter/Reader for TCP) ? >>>>>>> >>>>>>> Also, are there any "dirty" corners somebody heavily working with >>>>>>> udp have to know ? I'm implementing reliable udp and I would like to use >>>>>>> the coroutine style instead of callbacks. >>>>>>> >>>>>>> >>>>>>> Regards, >>>>>>> Chris >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> --Guido van Rossum (python.org/~guido) >>>>>> >>>>> >> >> >> -- >> --Guido van Rossum (python.org/~guido) >> > -- --Guido van Rossum (python.org/~guido)
