Thanks for your help so far, I really appreciate it. A manual backoff seems the best solution for this weird behavior for now, since reliable udp heavily depends on timing this is not such a bad thing anyway.
Meanwhile I try to figure out the cause for this issue. Am Montag, 24. Februar 2014 00:31:24 UTC+1 schrieb Guido van Rossum: > > I still can't repro it with your code. But that doesn't mean it's not a > real condition. It sounds like the kind of odd corner of entirely > legitimate UDP behavior that is hard to provoke but which a robust app > should handle. > > Note that the default behavior in Tulip appears to be to ignore OSError > coming out of sendto() -- the transport calls protocol.error_received(), > which by default does nothing. Since there are many other cases where a > packet may silently be dropped on the floor, this behavior is technically > correct -- the question is whether it is the best default behavior we can > imagine. > > Unfortunately turning it into a pause_protocol() call in your > error_received() handler is a little tricky -- the transport remembers > whether it has paused the protocol or not, but this state is not public. So > you shouldn't call your own pause_writing(), since you'd never receive a > resume_writing() call from the transport. Perhaps you can set a flag > internal to your protocol that just causes you to back off for a brief > period of time? The optimal back-off time should be tuned experimentally. > > --Guido > > On Sun, Feb 23, 2014 at 2:45 PM, Christopher Probst < > foxnet.d...@googlemail.com <javascript:>> wrote: > >> I made a simpler test, without using tulip, just using plain >> sockets<http://stackoverflow.com/questions/21973661/os-x-udp-send-error-55-no-buffer-space-available/21973705?noredirect=1#comment33297277_21973705> >> . >> >> from socket import * >> >> udp = socket(AF_INET, SOCK_DGRAM) >> udp.setsockopt(SOL_SOCKET, SO_REUSEADDR, True) >> >> udp.bind(('0.0.0.0', 1337)) >> udp.setblocking(False) >> udp.setsockopt(SOL_IP, IP_TTL, 4) >> udp.connect(('8.8.8.8', 12345)) >> >> buf = b'x' * 400for _ in range(1024 * 1024 * 10): >> udp.send(buf) >> >> In this test I'm just writing a lot of udp packets to 8.8.8.8, which gets >> dropped after 4 hops (it's just for testing anyway). But this code actually >> causes the same error (so it has nothing to do with tulip, though >> flow-control might be affected). >> >> The weird thing is, that the following code although causes the same >> error: >> >> from socket import * >> >> udp = socket(AF_INET, SOCK_DGRAM) >> udp.connect(('8.8.8.8', 12345)) >> >> buf = b'x' * 400for _ in range(1024 * 1024 * 10): >> udp.send(buf) >> >> In other words: The "blocking" mode is not so blocking apparently on my >> machine. If you cannot reproduce this error(which is good!), than this >> might be an issue on my local machine. But in any case it's not a tulip bug. >> >> I'm just curious what this can be... >> >> >> Am Sonntag, 23. Februar 2014 23:34:39 UTC+1 schrieb Guido van Rossum: >>> >>> I haven't been able to repro this using the test you attached to issue >>> 153. But I don't have another machine available, I've only tried localhost >>> on OSX (10.9) and from that same box to a local virtual machine running >>> Ubunu. The receiving end just throws away the data -- is that your test >>> setup too? I used "while 1: s.recv(10000); n += 1" in an interactive Python >>> shell. >>> >>> I do see packet loss (just counting packets received and packets sent) >>> but no exceptions nor does pause_writing() ever get called. From putting >>> some print()s in the code it looks like the sendto() operation always >>> immediately succeeds. >>> >>> The errno has a name: errno.ENOBUFS -- it's 55 on OSX, but 105 on Ubuntu. >>> >>> Perhaps you can elaborate on your test setup? >>> >>> >>> On Sun, Feb 23, 2014 at 11:29 AM, Christopher Probst < >>> foxnet.d...@googlemail.com> wrote: >>> >>>> Ok, I tried it now with 180 bytes packet and it does not occur, I guess >>>> that my lan connection(gigabit) is too fast that those packets get queued >>>> up. Or OSX decide whether or not to queue those packets based on the size. >>>> But this seems to be heavily os dependent . >>>> >>>> Am Sonntag, 23. Februar 2014 20:12:46 UTC+1 schrieb Guido van Rossum: >>>>> >>>>> Have you tried reducing the write buffer size? >>>>> On Feb 23, 2014 10:41 AM, "Christopher Probst" < >>>>> foxnet.d...@googlemail.com> wrote: >>>>> >>>>>> I saw this in the official 3.4rc1 doc. >>>>>> >>>>>> The doc says, that flow-control callbacks are valid for Protocols and >>>>>> SubprocessProtocols, but DatagramProtocol is not specified. >>>>>> >>>>>> Though I'm happy that Datagram control flow is officially supported I >>>>>> have now an other problem which is directly connected with this issue. >>>>>> Using OSX10.9.1 and python 3.3 sending a lot of udp packets actually >>>>>> does not cause the flow-control to be activated but throws an OSError(55 >>>>>> *No >>>>>> buffer space available)* . >>>>>> >>>>>> After googling around for hours I found out that this is a >>>>>> BSD(probably OS X only) thing. On linux the control-flow works as >>>>>> expected. >>>>>> >>>>>> I listed an issue for this in your tulip repo: >>>>>> Issue 153 <https://code.google.com/p/tulip/issues/detail?id=153> >>>>>> >>>>>> I think that this is really not a python bug, but the way BSD/OSX >>>>>> handles too much udp packets (the socket sdnbuf option is kind of >>>>>> ignored >>>>>> by OSX). I've read that windows actually handles udp overload in a >>>>>> similiar >>>>>> way as BSD does. Though i don't have a windows machine this should >>>>>> probably >>>>>> be tested to confirm or disprove this issue. >>>>>> >>>>>> PS: >>>>>> 18.5.3.2.5. Flow control >>>>>> callbacks<http://docs.python.org/3.4/library/asyncio-protocol.html#flow-control-callbacks> >>>>>> >>>>>> >>>>>> These callbacks may be called on >>>>>> Protocol<http://docs.python.org/3.4/library/asyncio-protocol.html#asyncio.Protocol> >>>>>> and >>>>>> SubprocessProtocol<http://docs.python.org/3.4/library/asyncio-protocol.html#asyncio.SubprocessProtocol> >>>>>> instances: >>>>>> BaseProtocol.pause_writing()<http://docs.python.org/3.4/library/asyncio-protocol.html#asyncio.BaseProtocol.pause_writing> >>>>>> >>>>>> >>>>>> Called when the transport’s buffer goes over the high-water mark. >>>>>> BaseProtocol.resume_writing()<http://docs.python.org/3.4/library/asyncio-protocol.html#asyncio.BaseProtocol.resume_writing> >>>>>> >>>>>> >>>>>> Called when the transport’s buffer drains below the low-water mark. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Am Sonntag, 23. Februar 2014 18:56:42 UTC+1 schrieb Guido van Rossum: >>>>>>> >>>>>>> This looks like a bug in the docs; the intention is that datagram >>>>>>> protocols also support flow control. Where does it say so in the docs? >>>>>>> Is >>>>>>> it the PEP or the CPython Doc tree? >>>>>>> >>>>>>> >>>>>>> On Sat, Feb 22, 2014 at 5:12 PM, Christopher Probst < >>>>>>> foxnet.d...@googlemail.com> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> after looking into the implementation I saw that, for instance, _ >>>>>>>> SelectorDatagramTransport calls _maybe_pause_protocol and it's >>>>>>>> counterpart, but the doc says that only Protocol and >>>>>>>> SubprocessProtocol has >>>>>>>> flow-control support and DatagramProtocol does not. >>>>>>>> >>>>>>>> I know that udp flow-control is not the same as tcp flow-control, >>>>>>>> but I'm concerned about filling up the internal buffer when writing a >>>>>>>> lot >>>>>>>> of datagrams. If this is not supported, I would argue that the udp >>>>>>>> support is pretty much broken for data intense application because how >>>>>>>> would the writer know when the internal buffer (and/or kernel level >>>>>>>> buffer) >>>>>>>> are full ? >>>>>>>> >>>>>>>> So, is the doc just not up-to-date or is it an implementation >>>>>>>> detail of tulip ? >>>>>>>> >>>>>>>> And other question that came up: Are there any plans for >>>>>>>> coroutine methods for udp (like StreamWriter/Reader for TCP) ? >>>>>>>> >>>>>>>> Also, are there any "dirty" corners somebody heavily working with >>>>>>>> udp have to know ? I'm implementing reliable udp and I would like to >>>>>>>> use >>>>>>>> the coroutine style instead of callbacks. >>>>>>>> >>>>>>>> >>>>>>>> Regards, >>>>>>>> Chris >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> --Guido van Rossum (python.org/~guido) >>>>>>> >>>>>> >>> >>> >>> -- >>> --Guido van Rossum (python.org/~guido) >>> >> > > > -- > --Guido van Rossum (python.org/~guido) >
