got it again , looks like there is a collision during read and send ( send will be send every 10 seconds , it is the heartbeat to tell remote machine to keep sending udp)
sendto data, addr = self._sock.recvfrom(self.max_size) [root@localhost /]# ./run_traf.sh 2016-06-10 10:38:19,980 [INFO] [1042] to exit press Ctrl+C 2016-06-10 10:38:19,980 [INFO] [1043] Started 2016-06-10 10:38:19,981 [INFO] [1046] 2016-06-10 10:38:19,981 [INFO] [1047] TrafficaElasticFNSasyncio 1.7 2016-06-10 10:38:19,981 [INFO] [1048] __________________________________________ 2016-06-10 10:38:19,981 [INFO] [1049] 2016-06-10 10:38:19,981 [INFO] [1050] fns: 2016-06-10 10:38:19,981 [INFO] [1051] ipaddr = 10.52.33.137 2016-06-10 10:38:19,982 [INFO] [1052] port udp = 60000 2016-06-10 10:38:19,982 [INFO] [1053] 2016-06-10 10:38:19,982 [INFO] [1054] __________________________________________ 2016-06-10 10:38:19,982 [INFO] [1055] 2016-06-10 10:38:19,982 [INFO] [1056] elasticsearch: 2016-06-10 10:38:19,982 [INFO] [1057] ipaddr = 10.203.168.55 2016-06-10 10:38:19,982 [INFO] [1058] port tcp = 9200 2016-06-10 10:38:19,982 [INFO] [1059] 2016-06-10 10:38:19,983 [INFO] [1060] __________________________________________ 2016-06-10 10:38:19,983 [INFO] [1061] me: 2016-06-10 10:38:19,983 [INFO] [1062] ipaddr = 0.0.0.0 2016-06-10 10:38:19,983 [INFO] [1063] port udp = 58634 2016-06-10 10:38:19,983 [INFO] [1064] __________________________________________ 2016-06-10 10:38:19,983 [INFO] [1065] 2016-06-10 10:38:19,983 [INFO] [1067] OS info: 2016-06-10 10:38:19,983 [INFO] [1068] Endianness = little 2016-06-10 10:38:19,988 [INFO] [1069] OS = Linux.3.12.11-201.fc19.i686 linux 2016-06-10 10:38:19,988 [INFO] [1070] Machine = i686 2016-06-10 10:38:19,988 [INFO] [1071] Node = localhost.localdomain 2016-06-10 10:38:19,989 [INFO] [1072] Processor = i686 2016-06-10 10:38:19,989 [INFO] [1073] Python version = 3.6.0a1 2016-06-10 10:38:19,989 [INFO] [1074] __________________________________________ 2016-06-10 10:38:19,989 [INFO] [1075] 2016-06-10 10:38:19,990 [INFO] [213] Starting new HTTP connection (1): 10.203.168.55 2016-06-10 10:38:19,997 [INFO] [63] GET http://10.203.168.55:9200/_nodes/_all/clear [status:200 request:0.007s] 2016-06-10 10:38:20,987 [INFO] [63] DELETE http://10.203.168.55:9200/fns_traffica_sm_mm_uscc [status:200 request:0.990s] 2016-06-10 10:38:21,628 [INFO] [63] PUT http://10.203.168.55:9200/fns_traffica_sm_mm_uscc [status:200 request:0.641s] 2016-06-10 10:38:21,998 [INFO] [63] PUT http://10.203.168.55:9200/fns_traffica_sm_mm_uscc/_mapping/fns_sm_mme [status:200 request:0.370s] 2016-06-10 10:38:22,038 [INFO] [63] PUT http://10.203.168.55:9200/fns_traffica_sm_mm_uscc/_mapping/fns_mm_mme [status:200 request:0.039s] 2016-06-10 10:38:22,038 [INFO] [1139] connect traffica 2016-06-10 10:38:22,039 [INFO] [1140] connect elasticsearch 2016-06-10 10:38:22,039 [INFO] [1141] __________________________________________ 2016-06-10 10:38:22,039 [INFO] [1142] 2016-06-10 10:39:03,089 [INFO] [213] Starting new HTTP connection (2): 10.203.168.55 2016-06-10 10:39:22,052 [INFO] [574] __________________________________________ 2016-06-10 10:39:22,052 [INFO] [575] 2016-06-10 10:39:22,052 [INFO] [576] Total udp from fns: 1420 , queue size: 0 2016-06-10 10:39:22,072 [INFO] [470] Got 1420 json report from traffica server 2016-06-10 10:40:22,060 [INFO] [574] __________________________________________ 2016-06-10 10:40:22,060 [INFO] [575] 2016-06-10 10:40:22,060 [INFO] [576] Total udp from fns: 4487 , queue size: 0 2016-06-10 10:40:22,087 [INFO] [470] Got 4487 json report from traffica server 2016-06-10 10:41:22,071 [INFO] [574] __________________________________________ 2016-06-10 10:41:22,071 [INFO] [575] 2016-06-10 10:41:22,071 [INFO] [576] Total udp from fns: 5165 , queue size: 0 2016-06-10 10:41:22,076 [INFO] [470] Got 5165 json report from traffica server 2016-06-10 10:42:22,071 [INFO] [574] __________________________________________ 2016-06-10 10:42:22,071 [INFO] [575] 2016-06-10 10:42:22,071 [INFO] [576] Total udp from fns: 15627 , queue size: 5206 2016-06-10 10:42:22,072 [INFO] [213] Starting new HTTP connection (3): 10.203.168.55 2016-06-10 10:42:22,081 [INFO] [470] Got 10321 json report from traffica server 2016-06-10 10:43:22,072 [INFO] [574] __________________________________________ 2016-06-10 10:43:22,072 [INFO] [575] 2016-06-10 10:43:22,072 [INFO] [576] Total udp from fns: 41640 , queue size: 20660 2016-06-10 10:43:22,106 [INFO] [470] Got 20918 json report from traffica server 2016-06-10 10:44:22,073 [INFO] [574] __________________________________________ 2016-06-10 10:44:22,074 [INFO] [575] 2016-06-10 10:44:22,074 [INFO] [576] Total udp from fns: 71635 , queue size: 38649 2016-06-10 10:44:22,083 [INFO] [470] Got 32878 json report from traffica server 2016-06-10 10:44:47,155 [ERROR] [1141] Fatal read error on datagram transport protocol: <__main__.UDPProtocolServerTraffica object at 0xb6edbd2c> transport: <_SelectorDatagramTransport fd=14 read=polling write=<idle, bufsize=0>> Traceback (most recent call last): File "/usr/local/lib/python3.6/asyncio/selector_events.py", line 1001, in _read_ready data, addr = self._sock.recvfrom(self.max_size) MemoryError stop 2016-06-10 10:44:52,187 [ERROR] [1141] Exception in callback UDPProtocolServerTraffica.timeout_heartbeat() handle: <TimerHandle when=10899.361216256 UDPProtocolServerTraffica.timeout_heartbeat()> Traceback (most recent call last): File "/usr/local/lib/python3.6/asyncio/selector_events.py", line 1034, in sendto self._sock.sendto(data, addr) AttributeError: 'NoneType' object has no attribute 'sendto' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.6/asyncio/events.py", line 125, in _run self._callback(*self._args) File "TrafficaElasticFNSasyncio.py", line 565, in timeout_heartbeat self.transport.sendto(self.MSG_UDP_HEARTBEAT, (fns_remote_host, fns_remote_port)) File "/usr/local/lib/python3.6/asyncio/selector_events.py", line 1043, in sendto 'Fatal write error on datagram transport') File "/usr/local/lib/python3.6/asyncio/selector_events.py", line 586, in _fatal_error self._loop.call_exception_handler({ AttributeError: 'NoneType' object has no attribute 'call_exception_handler' 2016-06-10 10:45:22,104 [INFO] [574] __________________________________________ 2016-06-10 10:45:22,105 [INFO] [575] 2016-06-10 10:45:22,105 [INFO] [576] Total udp from fns: 84176 , queue size: 39487 2016-06-10 10:45:22,108 [INFO] [470] Got 44544 json report from traffica server
