Re: FreeBSD: syslog-ng: I/O error occurred while writing; fd='xx', error='No buffer space available (yy)'

[Mark Blackman]

On 23 Mar 2012, at 08:58, Traiano Welcome wrote:

> Hi Mark
> 
> 
> On 22/03/2012 13:54, "Mark Blackman"  wrote:
> 
>> 
>> On 22 Mar 2012, at 11:40, Traiano Welcome wrote:
>>> 
>>> Somehow this doesn't strike  me as a large volume of throughput Š
>> 
>> Ok, fair enough. You might try simulating the problem by deliberately
>> overloading the syslog UDP output and confirm the cause.
> 
> 
> Apparently this means that the network driver has "filled" up with
> packets. John Baldwin over at freebsd-net@ advises I up the number of
> descriptors assigned to igb to the maximum
> to workaround this using the hw.igb.maxtxd tunable you would set. So I've
> rebooted with the following in loader.conf:
> 
> hw.igb.rxd=4096
> hw.igb.txd=4096
> 
> 
> This seems to be working so far. What I've noticed is that the system is
> using far  less RAM than previously, and CPU utilisation is up to 100% of
> one core, load average is 1, which I would guess means that the system is
> now processing a lot more syslog  data now that "more packets are making
> it through the network driver".
> 
> I'll keep monitoring over a 24 hour period though, to see how effective
> this is.

Right, good news. Interesting that you need to tweak network drivers.

- Mark

___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: FreeBSD: syslog-ng: I/O error occurred while writing; fd='xx', error='No buffer space available (yy)'

[Traiano Welcome]

Hi Mark


On 22/03/2012 13:54, "Mark Blackman"  wrote:

>
>On 22 Mar 2012, at 11:40, Traiano Welcome wrote:
>
>> That's what I thought as well, but it's the details that evade me.
>>Almost
>> all traffic to and from this server is UDP (syslog), the graph I sent
>> earlier shows the kind of volumes and trends that are typical: Peak
>> traffic during the problem periods averages at about 1 Mbps outbound and
>> 200 Kbps inbound to/from the interface. The interface itself is a
>> Embedded Broadcom 5708 NIC on a Dell PowerEdge 1950.
>> 
>> 
>> Here are a couple of netstat polls during one of the problem periods:
>> 
>> 
>> [root@syslog2]# date;netstat -p udp -s |egrep -w
>> "(received|delivered|dropped)"
>> Thu Mar 22 12:11:34 SAST 2012
>>   19969 datagrams received
>>   2 dropped due to no socket
>>   0 dropped due to full socket buffers
>>   19967 delivered
>> .
>> .
>> .
>> [root@syslog2~]# date;netstat -p udp -s |egrep -w
>> "(received|delivered|dropped)"
>> Thu Mar 22 13:36:46 SAST 2012
>>   662385 datagrams received
>>   118 dropped due to no socket
>>   0 dropped due to full socket buffers
>>   662267 delivered
>> ---
>> 
>> 
>> Somehow this doesn't strike  me as a large volume of throughput Š
>
>Ok, fair enough. You might try simulating the problem by deliberately
>overloading the syslog UDP output and confirm the cause.


Apparently this means that the network driver has "filled" up with
packets. John Baldwin over at freebsd-net@ advises I up the number of
descriptors assigned to igb to the maximum
to workaround this using the hw.igb.maxtxd tunable you would set. So I've
rebooted with the following in loader.conf:

hw.igb.rxd=4096
hw.igb.txd=4096


This seems to be working so far. What I've noticed is that the system is
using far  less RAM than previously, and CPU utilisation is up to 100% of
one core, load average is 1, which I would guess means that the system is
now processing a lot more syslog  data now that "more packets are making
it through the network driver".

I'll keep monitoring over a 24 hour period though, to see how effective
this is.



>
>- Mark

___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: FreeBSD: syslog-ng: I/O error occurred while writing; fd='xx', error='No buffer space available (yy)'

[Mark Blackman]

On 22 Mar 2012, at 11:40, Traiano Welcome wrote:

> That's what I thought as well, but it's the details that evade me. Almost
> all traffic to and from this server is UDP (syslog), the graph I sent
> earlier shows the kind of volumes and trends that are typical: Peak
> traffic during the problem periods averages at about 1 Mbps outbound and
> 200 Kbps inbound to/from the interface. The interface itself is a
> Embedded Broadcom 5708 NIC on a Dell PowerEdge 1950.
> 
> 
> Here are a couple of netstat polls during one of the problem periods:
> 
> 
> [root@syslog2]# date;netstat -p udp -s |egrep -w
> "(received|delivered|dropped)"
> Thu Mar 22 12:11:34 SAST 2012
>   19969 datagrams received
>   2 dropped due to no socket
>   0 dropped due to full socket buffers
>   19967 delivered
> .
> .
> .
> [root@syslog2~]# date;netstat -p udp -s |egrep -w
> "(received|delivered|dropped)"
> Thu Mar 22 13:36:46 SAST 2012
>   662385 datagrams received
>   118 dropped due to no socket
>   0 dropped due to full socket buffers
>   662267 delivered
> ---
> 
> 
> Somehow this doesn't strike  me as a large volume of throughput …

Ok, fair enough. You might try simulating the problem by deliberately 
overloading the syslog UDP output and confirm the cause.

- Mark___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: FreeBSD: syslog-ng: I/O error occurred while writing; fd='xx', error='No buffer space available (yy)'

[Traiano Welcome]

Hi Mark


On 22/03/2012 11:52, "Mark Blackman"  wrote:

>
>On 22 Mar 2012, at 09:00, Traiano Welcome wrote:
>
>> 
>> 
>> My question is: What does this error mean, and how can I resolve it?
>
>
>From a very casual inspection of the problem, I'd say you're pushing out
>syslog messages faster than the kernel can get them out the interface.
>How many syslog messages are going in (per second) and what kind of
>network interface are you trying to send them out through?


That's what I thought as well, but it's the details that evade me. Almost
all traffic to and from this server is UDP (syslog), the graph I sent
earlier shows the kind of volumes and trends that are typical: Peak
traffic during the problem periods averages at about 1 Mbps outbound and
200 Kbps inbound to/from the interface. The interface itself is a
Embedded Broadcom 5708 NIC on a Dell PowerEdge 1950.


Here are a couple of netstat polls during one of the problem periods:


[root@syslog2]# date;netstat -p udp -s |egrep -w
"(received|delivered|dropped)"
Thu Mar 22 12:11:34 SAST 2012
19969 datagrams received
2 dropped due to no socket
0 dropped due to full socket buffers
19967 delivered
.
.
.
[root@syslog2~]# date;netstat -p udp -s |egrep -w
"(received|delivered|dropped)"
Thu Mar 22 13:36:46 SAST 2012
662385 datagrams received
118 dropped due to no socket
0 dropped due to full socket buffers
662267 delivered
---


Somehow this doesn't strike  me as a large volume of throughput ...





>
>> 
>> I have tried to frame this as an operating system kernel resource issue,
>> and experimented with increasing the freebsd kernel sysctls for UDP
>> performance:
>
>
>I think you can push nmbclusters up to about 600k, but if your input is
>running faster than your output, no amount of buffering will permanently
>stave off this problem.



I've done that just in the last 2 hours, though I agree with you that this
is probably a (very) temporary imrovement.



>
>- Mark
>
>
>

___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: FreeBSD: syslog-ng: I/O error occurred while writing; fd='xx', error='No buffer space available (yy)'

[Mark Blackman]

On 22 Mar 2012, at 09:00, Traiano Welcome wrote:

> 
> 
> My question is: What does this error mean, and how can I resolve it?


From a very casual inspection of the problem, I'd say you're pushing out
syslog messages faster than the kernel can get them out the interface.
How many syslog messages are going in (per second) and what kind of
network interface are you trying to send them out through?

> 
> I have tried to frame this as an operating system kernel resource issue,
> and experimented with increasing the freebsd kernel sysctls for UDP
> performance:


I think you can push nmbclusters up to about 600k, but if your input is
running faster than your output, no amount of buffering will permanently
stave off this problem.

- Mark



___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

Re: FreeBSD: syslog-ng: I/O error occurred while writing; fd='xx', error='No buffer space available (yy)'

[Da Rock]

On 03/22/12 19:00, Traiano Welcome wrote:

Hi List

I've been seeing the following in the messages log of my freebsd syslog
server for quite some time now:

---
Mar 20 12:19:12 syslog2 syslog-ng[35313]: I/O error occurred while
writing; fd='12', error='No buffer space available (55)'
Mar 20 12:19:12 syslog2 syslog-ng[35313]: Connection broken;
time_reopen='60'
Mar 20 12:19:12 syslog2 syslog-ng[35313]: I/O error occurred while
writing; fd='13', error='No buffer space available (55)'
Mar 20 12:19:12 syslog2 syslog-ng[35313]: Connection broken;
time_reopen='60'
---

These happen at a frequency of about 7 per minute on average. See attached
trend graphs for an idea of the volume of traffic we're doing, as well as
the memory and cpu utilisation trends on this server during this period.
As can be seen from the graphs, load does not seem to be the issue.
Occasionally during the week, the system freezes and requires a reboot, I
think it's related to the above message, though I'm not sure.

My question is: What does this error mean, and how can I resolve it?

I have tried to frame this as an operating system kernel resource issue,
and experimented with increasing the freebsd kernel sysctls for UDP
performance:

---
[root@syslog2  /var/log]#
sysctl kern.ipc.nmbclusters=102400
kern.ipc.nmbclusters: 25600 ->  102400

[root@syslog2  /var/log]#
sysctl kern.ipc.maxsockbuf=201326592
kern.ipc.maxsockbuf: 100663296 ->  201326592

[root@syslog2  /var/log]#
sysctl net.inet.udp.recvspace=33554432
net.inet.udp.recvspace: 16777216 ->  33554432
---

This has reduced the frequency of the errors a little, but in general the
problem still remains.

Syslog version:

--
[root@syslog2]# syslog-ng -V
syslog-ng 2.0.10

--

FreeBSD version:

--
FreeBSD 7.2-STABLE #0
--

Any help would be much appreciated!
I'm sorry I can't shed some light on a solution, but this happens on 
ping and some other network related apps and tools for me too; just not 
often enough for me concern with atm due to higher priorities.


Perhaps net@ might be a better resource for an answer to this one?
___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"

FreeBSD: syslog-ng: I/O error occurred while writing; fd='xx', error='No buffer space available (yy)'

[Traiano Welcome]

Hi List

I've been seeing the following in the messages log of my freebsd syslog
server for quite some time now:

---
Mar 20 12:19:12 syslog2 syslog-ng[35313]: I/O error occurred while
writing; fd='12', error='No buffer space available (55)'
Mar 20 12:19:12 syslog2 syslog-ng[35313]: Connection broken;
time_reopen='60'
Mar 20 12:19:12 syslog2 syslog-ng[35313]: I/O error occurred while
writing; fd='13', error='No buffer space available (55)'
Mar 20 12:19:12 syslog2 syslog-ng[35313]: Connection broken;
time_reopen='60'
---

These happen at a frequency of about 7 per minute on average. See attached
trend graphs for an idea of the volume of traffic we're doing, as well as
the memory and cpu utilisation trends on this server during this period.
As can be seen from the graphs, load does not seem to be the issue.
Occasionally during the week, the system freezes and requires a reboot, I
think it's related to the above message, though I'm not sure.

My question is: What does this error mean, and how can I resolve it?

I have tried to frame this as an operating system kernel resource issue,
and experimented with increasing the freebsd kernel sysctls for UDP
performance:

---
[root@syslog2  /var/log]#
sysctl kern.ipc.nmbclusters=102400
kern.ipc.nmbclusters: 25600 -> 102400

[root@syslog2  /var/log]#
sysctl kern.ipc.maxsockbuf=201326592
kern.ipc.maxsockbuf: 100663296 -> 201326592

[root@syslog2  /var/log]#
sysctl net.inet.udp.recvspace=33554432
net.inet.udp.recvspace: 16777216 -> 33554432
---

This has reduced the frequency of the errors a little, but in general the
problem still remains.

Syslog version:

--
[root@syslog2]# syslog-ng -V
syslog-ng 2.0.10

--

FreeBSD version:

--
FreeBSD 7.2-STABLE #0
--

Any help would be much appreciated!
Traiano


___
freebsd-questions@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to "freebsd-questions-unsubscr...@freebsd.org"