[
https://issues.apache.org/jira/browse/THRIFT-1737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Will Pierce updated THRIFT-1737:
--------------------------------
Attachment: THRIFT-1737-UDP-v1.diff
patch attached (v1)
> UDP socket support for python
> -----------------------------
>
> Key: THRIFT-1737
> URL: https://issues.apache.org/jira/browse/THRIFT-1737
> Project: Thrift
> Issue Type: New Feature
> Components: Python - Library
> Reporter: Will Pierce
> Attachments: THRIFT-1737-UDP-v1.diff
>
>
> This patch adds support for UDP socket servers and clients in python. This
> reduces overhead and network latency due to TCP handshaking, _especially_ for
> "oneway" service methods.
> One useful feature of a UDP service is that the clients don't need to rebuild
> their connection to the server when a UDP packet is lost, so the "blast
> radius" of the timeout exception is limited to a single service call, not the
> entire "connection". Also, framing is not necessary because UDP packets have
> length encoded in their header.
> This transport is not suitable for large messages because UDP is inherently
> limited to 64 KB packet lengths, and often much smaller (500 - 1500 bytes)
> depending on intermediate links and whether UDP fragments are reassembled.
> Avoid large query/response payloads with this transport.
> h2. Implementation
> UDP support is implemented by subclassing TSocket and TServerSocket into
> TUDPSocket and TServerUDPSocket, and adding a TDatagramTransport. The
> server's accept() method actually receives an entire inbound request packet.
> An inbound packet is wrapped as a stream with StringIO, and the response
> "connection" records the sender's host+port so responses are delivered from
> the server's socket back to the client.
> The TDatagramTransport converts the EOFError raised after reaching the end of
> the packet into a TTransport exception, to accomodate TServers.
> h2. Testing:
> The unit tests now have a TestUDP.py script which runs a UDP server and
> client, and exercises several of the ThriftTest service calls, and verifies
> that responses match expectations. It ensures that "oneway" method calls are
> truly non-blocking, 1 packet "send and forget". It also forces a timeout in
> the middle of a sequence of blocking RPC calls, which confirms that a timeout
> only breaks a single RPC, not the entire client.
> I haven't used this with server types other than TThreadedServer, or in a big
> environment yet. There may be edge-cases that haven't surfaced yet.
> Tested with IPv4 and IPv6 on localhost and python2.7 (dev box is fedora17).
> h2. Minor bugfix:
> The python RunClientServer.py test script had a 1-line bug where it ran some
> other test scripts twice by mistake (probably a cut and paste error).
> h2. General warnings for posterity:
> * UDP packets are *easily*spoofed*!
> ** don't use this on public-internet facing interfaces
> ** spoofed client IP attacks may turn your server into an attack vector
> * UDP is not reliable
> ** clients will have to handle socket.timeout exceptions for every RPC call
> ** UDP may be _more_ unreliable during network congestion
> * No retries.
> ** this library doesn't do any retries
> ** there's only one timeout setting per client, which applies to every method
> call
> ** but the timeout may be changed with the existing .setTimeout(msec) call
> * Compression
> ** I haven't tested using TZlibTransport wrapping this to compress the
> packets, but it ought to work (unless there are bugs)
> h2. Tuning to avoid Timeouts:
> Linux hosts tend to have small default values for the kernel's memory buffers
> used to queue up UDP packets. When that buffer fills up with packets that
> the server process hasn't yet processed, then the kernel drops the packet,
> even though it's been fully decoded and pulled off the NIC.
> This would show up as lots of "socket.timeout" exceptions raised in client
> code, and no sign of an inbound method call at the server.
> If you run "netstat -s" and see increasing "packet receive errors" in the
> *Udp* section of output, that is strong evidence that you need to increase
> your hosts' receive buffers.
> As root, you can raise the UDP buffer receive (and send) space to 4MB with:
> {noformat}
> sysctl -w net.core.rmem_default=4194304
> sysctl -w net.core.wmem_default=4194304
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
