>
> hello,
>
some updates!
a) with queue then no UDP will be lost.
def datagram_received(self, data, addr):
asyncio_queue.put_nowait(data)
and read queue is running in 20 threads
async def queue_consume_g():
while True:
if asyncio_queue.qsize() != 0:
try:
data = asyncio_queue.get_nowait()
await process_report_from_queue(data)
except:
pass
else:
await asyncio.sleep(5)
our UDP traffic is constant, it will be send to queue and read from queue
and processed and send by tcp to server. 20 threads and 20 tcp
connections to server.
but UDP still fast than TCP. and 20 threads reading from queue , but I
cannot find a way to accelerate it. 40 threads make no difference.
our queue will keep growing and after reach 40k then we got crash (
mmap).
b) to avoid queue to increase to infinity , there is new sleep 1
second. during 1 seconds all UDP packets received by network will be
dropped.
def datagram_received(self, data, addr):
if asyncio_queue.qsize() > 10000:
time.sleep(1)
asyncio_queue.put_nowait(data)
else:
asyncio_queue.put_nowait(data)
our data is from a sensors , there is NO big problem to loose some samples
, sure we would like to have 100% perfect. but cannot find a way to
accelerate the read queue and send it by tcp.
able to receive 50 millions samples wihtout any crash.
c) still not able to change RECEIVE buffer on asncio.
socket.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, 2**30)
on graph. black line are udp received and red are tcp send to server.
if tcp is not processed faster than udp then queue will keep growing...
<https://lh3.googleusercontent.com/-X6tRRpeoUVc/V2su5ZwZ7nI/AAAAAAAACm4/iCfcG84tUds3SYOg4zwyJQo3I6MOmVXIgCLcB/s1600/received_udp_send_tcp.png>
regards!
Valdemar
