ryan4yin commented on issue #7934: URL: https://github.com/apache/apisix/issues/7934#issuecomment-1252403352
@tzssangglass I confirmed the problem is caused by nf_conntrack's table is full, many packets have been droped. Because all the monitoring from prometheus plugin and openresty-xray was normal, and the 504 errors only occurs when QPS upon 500, I suspected the 504 errors was caused by packets loss. Then I follow the steps describe in [Linux服务器丢包故障的解决思路及引申的TCP/IP协议栈理论](https://www.cnblogs.com/276815076/p/5736272.html) to find the core reason. I soon discovered the anomaly below: ``` $ dmesg |grep drop | tail [15374.608993] nf_conntrack: nf_conntrack: table full, dropping packet [15374.617074] nf_conntrack: nf_conntrack: table full, dropping packet [15374.621357] nf_conntrack: nf_conntrack: table full, dropping packet [15374.625732] nf_conntrack: nf_conntrack: table full, dropping packet [15374.639817] nf_conntrack: nf_conntrack: table full, dropping packet [15374.660076] nf_conntrack: nf_conntrack: table full, dropping packet [15374.664339] nf_conntrack: nf_conntrack: table full, dropping packet [15374.667106] nf_conntrack: nf_conntrack: table full, dropping packet [15374.672774] nf_conntrack: nf_conntrack: table full, dropping packet [15374.678729] nf_conntrack: nf_conntrack: table full, dropping packet $ $ cat /proc/sys/net/netfilter/nf_conntrack_max 131072 $ cat /proc/sys/net/netfilter/nf_conntrack_count 131072 ``` I'll try to adjust the `/proc/sys/net/netfilter/nf_conntrack_max` and other nf_conntrack arguments to see wether it's helpful tomorrow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
