Re: FreeBSD: syslog-ng: I/O error occurred while writing; fd='xx', error='No buffer space available (yy)'
On 22 Mar 2012, at 11:40, Traiano Welcome wrote: That's what I thought as well, but it's the details that evade me. Almost all traffic to and from this server is UDP (syslog), the graph I sent earlier shows the kind of volumes and trends that are typical: Peak traffic during the problem periods averages at about 1 Mbps outbound and 200 Kbps inbound to/from the interface. The interface itself is a Embedded Broadcom 5708 NIC on a Dell PowerEdge 1950. Here are a couple of netstat polls during one of the problem periods: [root@syslog2]# date;netstat -p udp -s |egrep -w (received|delivered|dropped) Thu Mar 22 12:11:34 SAST 2012 19969 datagrams received 2 dropped due to no socket 0 dropped due to full socket buffers 19967 delivered . . . [root@syslog2~]# date;netstat -p udp -s |egrep -w (received|delivered|dropped) Thu Mar 22 13:36:46 SAST 2012 662385 datagrams received 118 dropped due to no socket 0 dropped due to full socket buffers 662267 delivered --- Somehow this doesn't strike me as a large volume of throughput … Ok, fair enough. You might try simulating the problem by deliberately overloading the syslog UDP output and confirm the cause. - Mark___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: FreeBSD: syslog-ng: I/O error occurred while writing; fd='xx', error='No buffer space available (yy)'
Hi Mark On 22/03/2012 13:54, Mark Blackman m...@exonetric.com wrote: On 22 Mar 2012, at 11:40, Traiano Welcome wrote: That's what I thought as well, but it's the details that evade me. Almost all traffic to and from this server is UDP (syslog), the graph I sent earlier shows the kind of volumes and trends that are typical: Peak traffic during the problem periods averages at about 1 Mbps outbound and 200 Kbps inbound to/from the interface. The interface itself is a Embedded Broadcom 5708 NIC on a Dell PowerEdge 1950. Here are a couple of netstat polls during one of the problem periods: [root@syslog2]# date;netstat -p udp -s |egrep -w (received|delivered|dropped) Thu Mar 22 12:11:34 SAST 2012 19969 datagrams received 2 dropped due to no socket 0 dropped due to full socket buffers 19967 delivered . . . [root@syslog2~]# date;netstat -p udp -s |egrep -w (received|delivered|dropped) Thu Mar 22 13:36:46 SAST 2012 662385 datagrams received 118 dropped due to no socket 0 dropped due to full socket buffers 662267 delivered --- Somehow this doesn't strike me as a large volume of throughput Š Ok, fair enough. You might try simulating the problem by deliberately overloading the syslog UDP output and confirm the cause. Apparently this means that the network driver has filled up with packets. John Baldwin over at freebsd-net@ advises I up the number of descriptors assigned to igb to the maximum to workaround this using the hw.igb.maxtxd tunable you would set. So I've rebooted with the following in loader.conf: hw.igb.rxd=4096 hw.igb.txd=4096 This seems to be working so far. What I've noticed is that the system is using far less RAM than previously, and CPU utilisation is up to 100% of one core, load average is 1, which I would guess means that the system is now processing a lot more syslog data now that more packets are making it through the network driver. I'll keep monitoring over a 24 hour period though, to see how effective this is. - Mark ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: FreeBSD: syslog-ng: I/O error occurred while writing; fd='xx', error='No buffer space available (yy)'
On 23 Mar 2012, at 08:58, Traiano Welcome wrote: Hi Mark On 22/03/2012 13:54, Mark Blackman m...@exonetric.com wrote: On 22 Mar 2012, at 11:40, Traiano Welcome wrote: Somehow this doesn't strike me as a large volume of throughput Š Ok, fair enough. You might try simulating the problem by deliberately overloading the syslog UDP output and confirm the cause. Apparently this means that the network driver has filled up with packets. John Baldwin over at freebsd-net@ advises I up the number of descriptors assigned to igb to the maximum to workaround this using the hw.igb.maxtxd tunable you would set. So I've rebooted with the following in loader.conf: hw.igb.rxd=4096 hw.igb.txd=4096 This seems to be working so far. What I've noticed is that the system is using far less RAM than previously, and CPU utilisation is up to 100% of one core, load average is 1, which I would guess means that the system is now processing a lot more syslog data now that more packets are making it through the network driver. I'll keep monitoring over a 24 hour period though, to see how effective this is. Right, good news. Interesting that you need to tweak network drivers. - Mark ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: FreeBSD: syslog-ng: I/O error occurred while writing; fd='xx', error='No buffer space available (yy)'
On 03/22/12 19:00, Traiano Welcome wrote: Hi List I've been seeing the following in the messages log of my freebsd syslog server for quite some time now: --- Mar 20 12:19:12 syslog2 syslog-ng[35313]: I/O error occurred while writing; fd='12', error='No buffer space available (55)' Mar 20 12:19:12 syslog2 syslog-ng[35313]: Connection broken; time_reopen='60' Mar 20 12:19:12 syslog2 syslog-ng[35313]: I/O error occurred while writing; fd='13', error='No buffer space available (55)' Mar 20 12:19:12 syslog2 syslog-ng[35313]: Connection broken; time_reopen='60' --- These happen at a frequency of about 7 per minute on average. See attached trend graphs for an idea of the volume of traffic we're doing, as well as the memory and cpu utilisation trends on this server during this period. As can be seen from the graphs, load does not seem to be the issue. Occasionally during the week, the system freezes and requires a reboot, I think it's related to the above message, though I'm not sure. My question is: What does this error mean, and how can I resolve it? I have tried to frame this as an operating system kernel resource issue, and experimented with increasing the freebsd kernel sysctls for UDP performance: --- [root@syslog2mailto:r...@syslog2.ops.mtnbusiness.co.za /var/log]# sysctl kern.ipc.nmbclusters=102400 kern.ipc.nmbclusters: 25600 - 102400 [root@syslog2mailto:r...@syslog2.ops.mtnbusiness.co.za /var/log]# sysctl kern.ipc.maxsockbuf=201326592 kern.ipc.maxsockbuf: 100663296 - 201326592 [root@syslog2mailto:r...@syslog2.ops.mtnbusiness.co.za /var/log]# sysctl net.inet.udp.recvspace=33554432 net.inet.udp.recvspace: 16777216 - 33554432 --- This has reduced the frequency of the errors a little, but in general the problem still remains. Syslog version: -- [root@syslog2]# syslog-ng -V syslog-ng 2.0.10 -- FreeBSD version: -- FreeBSD 7.2-STABLE #0 -- Any help would be much appreciated! I'm sorry I can't shed some light on a solution, but this happens on ping and some other network related apps and tools for me too; just not often enough for me concern with atm due to higher priorities. Perhaps net@ might be a better resource for an answer to this one? ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: FreeBSD: syslog-ng: I/O error occurred while writing; fd='xx', error='No buffer space available (yy)'
On 22 Mar 2012, at 09:00, Traiano Welcome wrote: My question is: What does this error mean, and how can I resolve it? From a very casual inspection of the problem, I'd say you're pushing out syslog messages faster than the kernel can get them out the interface. How many syslog messages are going in (per second) and what kind of network interface are you trying to send them out through? I have tried to frame this as an operating system kernel resource issue, and experimented with increasing the freebsd kernel sysctls for UDP performance: I think you can push nmbclusters up to about 600k, but if your input is running faster than your output, no amount of buffering will permanently stave off this problem. - Mark ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
Re: FreeBSD: syslog-ng: I/O error occurred while writing; fd='xx', error='No buffer space available (yy)'
Hi Mark On 22/03/2012 11:52, Mark Blackman m...@exonetric.com wrote: On 22 Mar 2012, at 09:00, Traiano Welcome wrote: My question is: What does this error mean, and how can I resolve it? From a very casual inspection of the problem, I'd say you're pushing out syslog messages faster than the kernel can get them out the interface. How many syslog messages are going in (per second) and what kind of network interface are you trying to send them out through? That's what I thought as well, but it's the details that evade me. Almost all traffic to and from this server is UDP (syslog), the graph I sent earlier shows the kind of volumes and trends that are typical: Peak traffic during the problem periods averages at about 1 Mbps outbound and 200 Kbps inbound to/from the interface. The interface itself is a Embedded Broadcom 5708 NIC on a Dell PowerEdge 1950. Here are a couple of netstat polls during one of the problem periods: [root@syslog2]# date;netstat -p udp -s |egrep -w (received|delivered|dropped) Thu Mar 22 12:11:34 SAST 2012 19969 datagrams received 2 dropped due to no socket 0 dropped due to full socket buffers 19967 delivered . . . [root@syslog2~]# date;netstat -p udp -s |egrep -w (received|delivered|dropped) Thu Mar 22 13:36:46 SAST 2012 662385 datagrams received 118 dropped due to no socket 0 dropped due to full socket buffers 662267 delivered --- Somehow this doesn't strike me as a large volume of throughput ... I have tried to frame this as an operating system kernel resource issue, and experimented with increasing the freebsd kernel sysctls for UDP performance: I think you can push nmbclusters up to about 600k, but if your input is running faster than your output, no amount of buffering will permanently stave off this problem. I've done that just in the last 2 hours, though I agree with you that this is probably a (very) temporary imrovement. - Mark ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to freebsd-questions-unsubscr...@freebsd.org
