Re: [python-tulip] Does DatagramProtocol get notified for pause/resume (flow control) events ?

2014-02-23 Thread Guido van Rossum 
This looks like a bug in the docs; the intention is that datagram protocols
also support flow control. Where does it say so in the docs? Is it the PEP
or the CPython Doc tree?


On Sat, Feb 22, 2014 at 5:12 PM, Christopher Probst 
foxnet.develo...@googlemail.com wrote:

 Hi,

 after looking into the implementation I saw that, for instance,
 _SelectorDatagramTransport calls _maybe_pause_protocol and it's
 counterpart, but the doc says that only Protocol and SubprocessProtocol has
 flow-control support and DatagramProtocol does not.

 I know that udp flow-control is not the same as tcp flow-control, but I'm
 concerned about filling up the internal buffer when writing a lot of
 datagrams. If this is not supported, I would argue that the udp support
 is pretty much broken for data intense application because how would the
 writer know when the internal buffer (and/or kernel level buffer) are full ?

 So, is the doc just not up-to-date or is it an implementation detail of
 tulip ?

 And other question that came up: Are there any plans for coroutine methods
 for udp (like StreamWriter/Reader for TCP) ?

 Also, are there any dirty corners somebody heavily working with udp have
 to know ? I'm implementing reliable udp and I would like to use the
 coroutine style instead of callbacks.


 Regards,
 Chris




-- 
--Guido van Rossum (python.org/~guido)

Re: [python-tulip] Does DatagramProtocol get notified for pause/resume (flow control) events ?

2014-02-23 Thread Christopher Probst 
I saw this in the official 3.4rc1 doc.

The doc says, that flow-control callbacks are valid for Protocols and 
SubprocessProtocols, but DatagramProtocol is not specified.

Though I'm happy that Datagram control flow is officially supported I have 
now an other problem which is directly connected with this issue.
Using OSX10.9.1 and python 3.3 sending a lot of udp packets actually does 
not cause the flow-control to be activated but throws an OSError(55 *No 
buffer space available)* .

After googling around for hours I found out that this is a BSD(probably OS 
X only) thing. On linux the control-flow works as expected. 

I listed an issue for this in your tulip repo:
Issue 153 https://code.google.com/p/tulip/issues/detail?id=153

I think that this is really not a python bug, but the way BSD/OSX handles 
too much udp packets (the socket sdnbuf option is kind of ignored by OSX). 
I've read that windows actually handles udp overload in a similiar way as 
BSD does. Though i don't have a windows machine this should probably be 
tested to confirm or disprove this issue.

PS:
18.5.3.2.5. Flow control 
callbackshttp://docs.python.org/3.4/library/asyncio-protocol.html#flow-control-callbacks

These callbacks may be called on 
Protocolhttp://docs.python.org/3.4/library/asyncio-protocol.html#asyncio.Protocol
 and 
SubprocessProtocolhttp://docs.python.org/3.4/library/asyncio-protocol.html#asyncio.SubprocessProtocol
 instances:
BaseProtocol.pause_writing()http://docs.python.org/3.4/library/asyncio-protocol.html#asyncio.BaseProtocol.pause_writing

Called when the transport’s buffer goes over the high-water mark.
BaseProtocol.resume_writing()http://docs.python.org/3.4/library/asyncio-protocol.html#asyncio.BaseProtocol.resume_writing

Called when the transport’s buffer drains below the low-water mark.




Am Sonntag, 23. Februar 2014 18:56:42 UTC+1 schrieb Guido van Rossum:

 This looks like a bug in the docs; the intention is that datagram 
 protocols also support flow control. Where does it say so in the docs? Is 
 it the PEP or the CPython Doc tree?


 On Sat, Feb 22, 2014 at 5:12 PM, Christopher Probst 
 foxnet.d...@googlemail.com javascript: wrote:

 Hi,

 after looking into the implementation I saw that, for instance, 
 _SelectorDatagramTransport calls _maybe_pause_protocol and it's 
 counterpart, but the doc says that only Protocol and SubprocessProtocol has 
 flow-control support and DatagramProtocol does not.

 I know that udp flow-control is not the same as tcp flow-control, but I'm 
 concerned about filling up the internal buffer when writing a lot of 
 datagrams. If this is not supported, I would argue that the udp support 
 is pretty much broken for data intense application because how would the 
 writer know when the internal buffer (and/or kernel level buffer) are full ?

 So, is the doc just not up-to-date or is it an implementation detail of 
 tulip ?

  And other question that came up: Are there any plans for coroutine 
 methods for udp (like StreamWriter/Reader for TCP) ? 

 Also, are there any dirty corners somebody heavily working with udp 
 have to know ? I'm implementing reliable udp and I would like to use the 
 coroutine style instead of callbacks.


 Regards,
 Chris




 -- 
 --Guido van Rossum (python.org/~guido)

Re: [python-tulip] Does DatagramProtocol get notified for pause/resume (flow control) events ?

2014-02-23 Thread Guido van Rossum 
Have you tried reducing the write buffer size?
On Feb 23, 2014 10:41 AM, Christopher Probst 
foxnet.develo...@googlemail.com wrote:

 I saw this in the official 3.4rc1 doc.

 The doc says, that flow-control callbacks are valid for Protocols and
 SubprocessProtocols, but DatagramProtocol is not specified.

 Though I'm happy that Datagram control flow is officially supported I have
 now an other problem which is directly connected with this issue.
 Using OSX10.9.1 and python 3.3 sending a lot of udp packets actually does
 not cause the flow-control to be activated but throws an OSError(55 *No
 buffer space available)* .

 After googling around for hours I found out that this is a BSD(probably OS
 X only) thing. On linux the control-flow works as expected.

 I listed an issue for this in your tulip repo:
 Issue 153 https://code.google.com/p/tulip/issues/detail?id=153

 I think that this is really not a python bug, but the way BSD/OSX handles
 too much udp packets (the socket sdnbuf option is kind of ignored by OSX).
 I've read that windows actually handles udp overload in a similiar way as
 BSD does. Though i don't have a windows machine this should probably be
 tested to confirm or disprove this issue.

 PS:
 18.5.3.2.5. Flow control 
 callbackshttp://docs.python.org/3.4/library/asyncio-protocol.html#flow-control-callbacks

 These callbacks may be called on 
 Protocolhttp://docs.python.org/3.4/library/asyncio-protocol.html#asyncio.Protocol
  and 
 SubprocessProtocolhttp://docs.python.org/3.4/library/asyncio-protocol.html#asyncio.SubprocessProtocol
  instances:
 BaseProtocol.pause_writing()http://docs.python.org/3.4/library/asyncio-protocol.html#asyncio.BaseProtocol.pause_writing

 Called when the transport's buffer goes over the high-water mark.
 BaseProtocol.resume_writing()http://docs.python.org/3.4/library/asyncio-protocol.html#asyncio.BaseProtocol.resume_writing

 Called when the transport's buffer drains below the low-water mark.




 Am Sonntag, 23. Februar 2014 18:56:42 UTC+1 schrieb Guido van Rossum:

 This looks like a bug in the docs; the intention is that datagram
 protocols also support flow control. Where does it say so in the docs? Is
 it the PEP or the CPython Doc tree?


 On Sat, Feb 22, 2014 at 5:12 PM, Christopher Probst 
 foxnet.d...@googlemail.com wrote:

 Hi,

 after looking into the implementation I saw that, for instance, _
 SelectorDatagramTransport calls _maybe_pause_protocol and it's
 counterpart, but the doc says that only Protocol and SubprocessProtocol has
 flow-control support and DatagramProtocol does not.

 I know that udp flow-control is not the same as tcp flow-control, but
 I'm concerned about filling up the internal buffer when writing a lot of
 datagrams. If this is not supported, I would argue that the udp support
 is pretty much broken for data intense application because how would the
 writer know when the internal buffer (and/or kernel level buffer) are full ?

 So, is the doc just not up-to-date or is it an implementation detail of
 tulip ?

  And other question that came up: Are there any plans for coroutine
 methods for udp (like StreamWriter/Reader for TCP) ?

 Also, are there any dirty corners somebody heavily working with udp
 have to know ? I'm implementing reliable udp and I would like to use the
 coroutine style instead of callbacks.


 Regards,
 Chris




 --
 --Guido van Rossum (python.org/~guido)

Re: [python-tulip] Does DatagramProtocol get notified for pause/resume (flow control) events ?

2014-02-23 Thread Christopher Probst 
Yes, if I reduce the write buffer size the problem occurs later but it does 
occur. size ~ time to crash.
I read about a kernel setting on OSX called sysctl kern.ipc.maxsockbuf 
which actually relates to the real udp socket sndbuf size.

The only real fix I could imagine would be to catch the OSError and check 
for number 55 on OSX(probably BSD, too) to manage flow control, it's ugly I 
know.
But OSX does not seem to be very udp flow-control friendly.


Am Sonntag, 23. Februar 2014 20:12:46 UTC+1 schrieb Guido van Rossum:

 Have you tried reducing the write buffer size?
 On Feb 23, 2014 10:41 AM, Christopher Probst 
 foxnet.d...@googlemail.comjavascript: 
 wrote:

 I saw this in the official 3.4rc1 doc.

 The doc says, that flow-control callbacks are valid for Protocols and 
 SubprocessProtocols, but DatagramProtocol is not specified.

 Though I'm happy that Datagram control flow is officially supported I 
 have now an other problem which is directly connected with this issue.
 Using OSX10.9.1 and python 3.3 sending a lot of udp packets actually does 
 not cause the flow-control to be activated but throws an OSError(55 *No 
 buffer space available)* .

 After googling around for hours I found out that this is a BSD(probably 
 OS X only) thing. On linux the control-flow works as expected. 

 I listed an issue for this in your tulip repo:
 Issue 153 https://code.google.com/p/tulip/issues/detail?id=153

 I think that this is really not a python bug, but the way BSD/OSX handles 
 too much udp packets (the socket sdnbuf option is kind of ignored by OSX). 
 I've read that windows actually handles udp overload in a similiar way as 
 BSD does. Though i don't have a windows machine this should probably be 
 tested to confirm or disprove this issue.

 PS:
 18.5.3.2.5. Flow control 
 callbackshttp://docs.python.org/3.4/library/asyncio-protocol.html#flow-control-callbacks
  

 These callbacks may be called on 
 Protocolhttp://docs.python.org/3.4/library/asyncio-protocol.html#asyncio.Protocol
  and 
 SubprocessProtocolhttp://docs.python.org/3.4/library/asyncio-protocol.html#asyncio.SubprocessProtocol
  instances:
 BaseProtocol.pause_writing()http://docs.python.org/3.4/library/asyncio-protocol.html#asyncio.BaseProtocol.pause_writing
  

 Called when the transport’s buffer goes over the high-water mark.
 BaseProtocol.resume_writing()http://docs.python.org/3.4/library/asyncio-protocol.html#asyncio.BaseProtocol.resume_writing
  

 Called when the transport’s buffer drains below the low-water mark.




 Am Sonntag, 23. Februar 2014 18:56:42 UTC+1 schrieb Guido van Rossum:

 This looks like a bug in the docs; the intention is that datagram 
 protocols also support flow control. Where does it say so in the docs? Is 
 it the PEP or the CPython Doc tree?


 On Sat, Feb 22, 2014 at 5:12 PM, Christopher Probst 
 foxnet.d...@googlemail.com wrote:

 Hi,

 after looking into the implementation I saw that, for instance, _
 SelectorDatagramTransport calls _maybe_pause_protocol and it's 
 counterpart, but the doc says that only Protocol and SubprocessProtocol 
 has 
 flow-control support and DatagramProtocol does not.

 I know that udp flow-control is not the same as tcp flow-control, but 
 I'm concerned about filling up the internal buffer when writing a lot of 
 datagrams. If this is not supported, I would argue that the udp 
 support is pretty much broken for data intense application because how 
 would the writer know when the internal buffer (and/or kernel level 
 buffer) 
 are full ?

 So, is the doc just not up-to-date or is it an implementation detail of 
 tulip ?

  And other question that came up: Are there any plans for coroutine 
 methods for udp (like StreamWriter/Reader for TCP) ? 

 Also, are there any dirty corners somebody heavily working with udp 
 have to know ? I'm implementing reliable udp and I would like to use the 
 coroutine style instead of callbacks.


 Regards,
 Chris




 -- 
 --Guido van Rossum (python.org/~guido)

Re: [python-tulip] Does DatagramProtocol get notified for pause/resume (flow control) events ?

2014-02-23 Thread Guido van Rossum 
Reliable UDP? Isn't that a contradiction?

On Sunday, February 23, 2014, Christopher Probst 
foxnet.develo...@googlemail.com wrote:

 Thanks for your help so far, I really appreciate it.

 A manual backoff seems the best solution for this weird behavior for now,
 since reliable udp heavily depends on timing this is not such a bad thing
 anyway.

 Meanwhile I try to figure out the cause for this issue.


 Am Montag, 24. Februar 2014 00:31:24 UTC+1 schrieb Guido van Rossum:

 I still can't repro it with your code. But that doesn't mean it's not a
 real condition. It sounds like the kind of odd corner of entirely
 legitimate UDP behavior that is hard to provoke but which a robust app
 should handle.

 Note that the default behavior in Tulip appears to be to ignore OSError
 coming out of sendto() -- the transport calls protocol.error_received(),
 which by default does nothing. Since there are many other cases where a
 packet may silently be dropped on the floor, this behavior is technically
 correct -- the question is whether it is the best default behavior we can
 imagine.

 Unfortunately turning it into a pause_protocol() call in your
 error_received() handler is a little tricky -- the transport remembers
 whether it has paused the protocol or not, but this state is not public. So
 you shouldn't call your own pause_writing(), since you'd never receive a
 resume_writing() call from the transport. Perhaps you can set a flag
 internal to your protocol that just causes you to back off for a brief
 period of time? The optimal back-off time should be tuned experimentally.

 --Guido

 On Sun, Feb 23, 2014 at 2:45 PM, Christopher Probst 
 foxnet.d...@googlemail.com wrote:

 I made a simpler test, without using tulip, just using plain 
 socketshttp://stackoverflow.com/questions/21973661/os-x-udp-send-error-55-no-buffer-space-available/21973705?noredirect=1#comment33297277_21973705
 .

 from socket import *

 udp = socket(AF_INET, SOCK_DGRAM)
 udp.setsockopt(SOL_SOCKET, SO_REUSEADDR, True)

 udp.bind(('0.0.0.0', 1337))
 udp.setblocking(False)
 udp.setsockopt(SOL_IP, IP_TTL, 4)
 udp.connect(('8.8.8.8



-- 
--Guido van Rossum (on iPad)

Re: [python-tulip] Does DatagramProtocol get notified for pause/resume (flow control) events ?

2014-02-23 Thread Christopher Probst 
Well in a sense, yes. But even TCP uses unreliable packet based transport 
under the hood (IP). RUDP is basically reimplementing TCP on top of UDP (in 
a sense). It's even kind of 
definedhttp://tools.ietf.org/html/draft-ietf-sigtran-reliable-udp-00. 
But yes, it's a contradiction, I have the feeling that a lot of network 
related things are that way :D 

The reason why I need RUDP is NAT traversal  (UDP Hole Punching)  for a 
communication software, which is pretty difficult to achieve with TCP 
actually.

Am Montag, 24. Februar 2014 01:07:17 UTC+1 schrieb Guido van Rossum:

 Reliable UDP? Isn't that a contradiction?

 On Sunday, February 23, 2014, Christopher Probst 
 foxnet.d...@googlemail.com javascript: wrote:

 Thanks for your help so far, I really appreciate it.

 A manual backoff seems the best solution for this weird behavior for now, 
 since reliable udp heavily depends on timing this is not such a bad thing 
 anyway.

 Meanwhile I try to figure out the cause for this issue.


 Am Montag, 24. Februar 2014 00:31:24 UTC+1 schrieb Guido van Rossum:

 I still can't repro it with your code. But that doesn't mean it's not a 
 real condition. It sounds like the kind of odd corner of entirely 
 legitimate UDP behavior that is hard to provoke but which a robust app 
 should handle.

 Note that the default behavior in Tulip appears to be to ignore OSError 
 coming out of sendto() -- the transport calls protocol.error_received(), 
 which by default does nothing. Since there are many other cases where a 
 packet may silently be dropped on the floor, this behavior is technically 
 correct -- the question is whether it is the best default behavior we can 
 imagine.

 Unfortunately turning it into a pause_protocol() call in your 
 error_received() handler is a little tricky -- the transport remembers 
 whether it has paused the protocol or not, but this state is not public. So 
 you shouldn't call your own pause_writing(), since you'd never receive a 
 resume_writing() call from the transport. Perhaps you can set a flag 
 internal to your protocol that just causes you to back off for a brief 
 period of time? The optimal back-off time should be tuned experimentally.

 --Guido

 On Sun, Feb 23, 2014 at 2:45 PM, Christopher Probst 
 foxnet.d...@googlemail.com wrote:

 I made a simpler test, without using tulip, just using plain 
 socketshttp://stackoverflow.com/questions/21973661/os-x-udp-send-error-55-no-buffer-space-available/21973705?noredirect=1#comment33297277_21973705
 .

 from socket import *

 udp = socket(AF_INET, SOCK_DGRAM)
 udp.setsockopt(SOL_SOCKET, SO_REUSEADDR, True)

 udp.bind(('0.0.0.0', 1337))
 udp.setblocking(False)
 udp.setsockopt(SOL_IP, IP_TTL, 4)
 udp.connect(('8.8.8.8

  

 -- 
 --Guido van Rossum (on iPad)