strace during the exception. ( errors 6 mmap2 and 3 ioctl )
[root@localhost ~]# strace -p 5023 -q -f -c
^C% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
67.22 16.287312 199 81737 poll
20.08 4.866613 92 52964 epoll_wait
12.18 2.951751 5 574504 98531 futex
0.15 0.037130 0 108647 gettimeofday
0.14 0.034103 0 505551 clock_gettime
0.07 0.017914 0 40868 send
0.05 0.011846 0 52897 mremap
0.03 0.008090 0 52900 recvfrom
0.03 0.006306 0 52904 6 mmap2
0.01 0.002889 0 20439 3 ioctl
0.01 0.002277 0 20436 recv
0.01 0.002231 0 20533 munmap
0.01 0.002000 1000 2 restart_syscall
0.00 0.001007 0 20447 stat64
0.00 0.000052 0 106 sendto
0.00 0.000000 0 20 read
0.00 0.000000 0 14 write
0.00 0.000000 0 4 open
0.00 0.000000 0 5 close
0.00 0.000000 0 1 time
0.00 0.000000 0 2 brk
0.00 0.000000 0 6 _llseek
0.00 0.000000 0 3 fstat64
0.00 0.000000 0 1 epoll_ctl
0.00 0.000000 0 1 socket
0.00 0.000000 0 1 1 connect
0.00 0.000000 0 1 setsockopt
0.00 0.000000 0 1 getsockopt
------ ----------- ----------- --------- --------- ----------------
100.00 24.231521 1604995 98541 total
[root@localhost ~]#
2016-06-10 11:30:30,756 [INFO] [576] Total udp from fns: 59995 , queue
size: 38296
2016-06-10 11:30:30,763 [INFO] [470] Got 21579 json report from traffica
server
2016-06-10 11:30:54,198 [ERROR] [1141] Fatal read error on datagram
transport
protocol: <__main__.UDPProtocolServerTraffica object at 0xb6ee8d2c>
transport: <_SelectorDatagramTransport fd=14 read=polling write=<idle,
bufsize=0>>
Traceback (most recent call last):
File "/usr/local/lib/python3.6/asyncio/selector_events.py", line 1001, in
_read_ready
data, addr = self._sock.recvfrom(self.max_size)
MemoryError
stop
2016-06-10 11:30:54,930 [ERROR] [1141] Exception in callback
UDPProtocolServerTraffica.timeout_heartbeat()
handle: <TimerHandle when=13662.108784672
UDPProtocolServerTraffica.timeout_heartbeat()>
Traceback (most recent call last):
File "/usr/local/lib/python3.6/asyncio/selector_events.py", line 1034, in
sendto
self._sock.sendto(data, addr)
AttributeError: 'NoneType' object has no attribute 'sendto'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/asyncio/events.py", line 125, in _run
self._callback(*self._args)
File "TrafficaElasticFNSasyncio.py", line 565, in timeout_heartbeat
self.transport.sendto(self.MSG_UDP_HEARTBEAT, (fns_remote_host,
fns_remote_port))
File "/usr/local/lib/python3.6/asyncio/selector_events.py", line 1043, in
sendto
'Fatal write error on datagram transport')
File "/usr/local/lib/python3.6/asyncio/selector_events.py", line 586, in
_fatal_error
self._loop.call_exception_handler({
AttributeError: 'NoneType' object has no attribute 'call_exception_handler'
2016-06-10 11:31:30,782 [INFO] [574]
__________________________________________
2016-06-10 11:31:30,782 [INFO] [575]
2016-06-10 11:31:30,782 [INFO] [576] Total udp from fns: 71716 , queue
size: 39029
2016-06-10 11:31:30,785 [INFO] [470] Got 32554 json report from traffica
server
2016-06-10 11:32:30,842 [INFO] [574]
__________________________________________
2016-06-10 11:32:30,842 [INFO] [575]
2016-06-10 11:32:30,842 [INFO] [576] Total udp from fns: 71716 , queue
size: 26767
2016-06-10 11:32:30,845 [INFO] [470] Got 44882 json report from traffica
server
++++
after get this exception then there are no more heartbeat and remote
machine will stops ending udp.
queue will be all processed.
2016-06-10 11:31:30,782 [INFO] [574]
__________________________________________
2016-06-10 11:31:30,782 [INFO] [575]
2016-06-10 11:31:30,782 [INFO] [576] Total udp from fns: 71716 , queue
size: 39029
2016-06-10 11:31:30,785 [INFO] [470] Got 32554 json report from traffica
server
2016-06-10 11:32:30,842 [INFO] [574]
__________________________________________
2016-06-10 11:32:30,842 [INFO] [575]
2016-06-10 11:32:30,842 [INFO] [576] Total udp from fns: 71716 , queue
size: 26767
2016-06-10 11:32:30,845 [INFO] [470] Got 44882 json report from traffica
server
2016-06-10 11:33:30,902 [INFO] [574]
__________________________________________
2016-06-10 11:33:30,902 [INFO] [575]
2016-06-10 11:33:30,903 [INFO] [576] Total udp from fns: 71716 , queue
size: 14777
2016-06-10 11:33:30,906 [INFO] [470] Got 56797 json report from traffica
server
2016-06-10 11:34:30,963 [INFO] [574]
__________________________________________
2016-06-10 11:34:30,963 [INFO] [575]
2016-06-10 11:34:30,963 [INFO] [576] Total udp from fns: 71716 , queue
size: 2650
2016-06-10 11:34:30,970 [INFO] [470] Got 68974 json report from traffica
server
2016-06-10 11:35:31,024 [INFO] [574]
__________________________________________
2016-06-10 11:35:31,024 [INFO] [575]
2016-06-10 11:35:31,024 [INFO] [576] Total udp from fns: 71716 , queue
size: 0
2016-06-10 11:35:31,029 [INFO] [470] Got 71716 json report from traffica
server
++++++
strace without problems:
[root@localhost ~]# strace -p 5047 -q -f -c
^C% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
65.98 8.198575 214 38329 poll
20.43 2.538010 95 26826 epoll_wait
13.12 1.630677 6 291448 47829 futex
0.13 0.015724 0 55899 gettimeofday
0.13 0.015583 0 248718 clock_gettime
0.08 0.010247 1 19165 send
0.05 0.006453 0 26799 mremap
0.03 0.003963 0 26799 mmap2
0.02 0.002766 0 26799 recvfrom
0.01 0.001310 0 9634 munmap
0.01 0.001306 0 9584 ioctl
0.01 0.000833 0 9582 recv
0.00 0.000282 0 9586 stat64
0.00 0.000037 37 1 1 connect
0.00 0.000000 0 1 restart_syscall
0.00 0.000000 0 5 write
0.00 0.000000 0 1 socket
0.00 0.000000 0 53 sendto
0.00 0.000000 0 1 setsockopt
0.00 0.000000 0 1 getsockopt
------ ----------- ----------- --------- --------- ----------------
100.00 12.425766 799231 47830 total
[root@localhost ~]#
