[jira] [Commented] (LOG4NET-415) RemoteSyslogAppender may block for ARP resolution + Improvement Strict RFC3164
[ https://issues.apache.org/jira/browse/LOG4NET-415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15419927#comment-15419927 ] Stefan Bodewig commented on LOG4NET-415: Sorry, got side-tracked for far too long, again. It would be good if you could either re-create a patch or provide a PR against the current trunk - when you created the patch the current trunk used to be the 1.2.x branch. > RemoteSyslogAppender may block for ARP resolution + Improvement Strict RFC3164 > -- > > Key: LOG4NET-415 > URL: https://issues.apache.org/jira/browse/LOG4NET-415 > Project: Log4net > Issue Type: Bug > Components: Appenders >Affects Versions: 1.2.13 > Environment: Any Windows environment >Reporter: Jose Luis Pedrosa > Labels: RemoteSyslogAppender > Attachments: LOG4NET-415.patch, MessageLostTest.cs, > MessageLostTest.cs, MessageLostTestAsync.cs, RemoteSyslogNonBlockingV3.patch, > RemoteSyslogNonBlockingv2.patch > > > Sending UDP packages may block for some time in specific circumstances: > 1) Next hop in network level 3 can't be resolved by ARP. > 2) Datagram size exceeds FastSendDatagramThreshold configured size. > When sending packets bigger than FastSendDatagramThreshold, the OS waits > until the packet is actually sent, if the If the syslog (or the next hop to > reach the syslog) is in the same VLAN/Subnet the OS tries to resolve by ARP > the Ip of the configured syslog, this may take up to 3 seconds, slowing down > the whole application, which in some cases can lead to outages (timeouts, DB > locks...). > Also the fact that each carriage return generates the headers of the packet > again, that can lead to a significant overhead in some scenarios, for > instance when loggign HTTP requests to a remote syslog, every header will go > in a different message. Also some logging may require characters that are now > skipped in patch: https://issues.apache.org/jira/browse/LOG4NET-370 > I'm adding a patch that > 1) Moves the use of UDPClient to Non blocking sockets, which eliminates the > blocking. > 2) Adds a configuration field to decide if you want Strict RFC Behaviour or > not (with default Strict). > Please your feedback, thanks in advance > Jose Luis -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (LOG4NET-415) RemoteSyslogAppender may block for ARP resolution + Improvement Strict RFC3164
[ https://issues.apache.org/jira/browse/LOG4NET-415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15304149#comment-15304149 ] Jose Luis Pedrosa commented on LOG4NET-415: --- Hi Is there any way to get this patch applied to log4net? should I do a PR or something? would it help? Rgds JL > RemoteSyslogAppender may block for ARP resolution + Improvement Strict RFC3164 > -- > > Key: LOG4NET-415 > URL: https://issues.apache.org/jira/browse/LOG4NET-415 > Project: Log4net > Issue Type: Bug > Components: Appenders >Affects Versions: 1.2.13 > Environment: Any Windows environment >Reporter: Jose Luis Pedrosa > Labels: RemoteSyslogAppender > Attachments: LOG4NET-415.patch, MessageLostTest.cs, > MessageLostTest.cs, MessageLostTestAsync.cs, RemoteSyslogNonBlockingV3.patch, > RemoteSyslogNonBlockingv2.patch > > > Sending UDP packages may block for some time in specific circumstances: > 1) Next hop in network level 3 can't be resolved by ARP. > 2) Datagram size exceeds FastSendDatagramThreshold configured size. > When sending packets bigger than FastSendDatagramThreshold, the OS waits > until the packet is actually sent, if the If the syslog (or the next hop to > reach the syslog) is in the same VLAN/Subnet the OS tries to resolve by ARP > the Ip of the configured syslog, this may take up to 3 seconds, slowing down > the whole application, which in some cases can lead to outages (timeouts, DB > locks...). > Also the fact that each carriage return generates the headers of the packet > again, that can lead to a significant overhead in some scenarios, for > instance when loggign HTTP requests to a remote syslog, every header will go > in a different message. Also some logging may require characters that are now > skipped in patch: https://issues.apache.org/jira/browse/LOG4NET-370 > I'm adding a patch that > 1) Moves the use of UDPClient to Non blocking sockets, which eliminates the > blocking. > 2) Adds a configuration field to decide if you want Strict RFC Behaviour or > not (with default Strict). > Please your feedback, thanks in advance > Jose Luis -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (LOG4NET-415) RemoteSyslogAppender may block for ARP resolution + Improvement Strict RFC3164
[ https://issues.apache.org/jira/browse/LOG4NET-415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13869607#comment-13869607 ] Stefan Bodewig commented on LOG4NET-415: Sorry for not responding for so long - and in addition I don't really have my brain with me right now, so I may be suggesting stupid things. * without Jose Luis' changes SyslogAppender had guaranteed delivery - if we add an option to use non-blocking sockets we need to make this change explicit (or rather state what the choice of the parameter entails). It is quite possible people using a syslog appender don't actually know they are using UDP under the covers. * doing things asynchronously requires some extra coding - you need to fix thread-local state that may contribute to the log message. I think there is some kind of AsyncAppender similar to BufferingForwardingAppender inside this JIRA (and if there isn't, then we still should provide one). I'd prefer a generic async appender (using ,NET 2.0 compatible code if possible) over appender specific async solutions. RemoteSyslogAppender may block for ARP resolution + Improvement Strict RFC3164 -- Key: LOG4NET-415 URL: https://issues.apache.org/jira/browse/LOG4NET-415 Project: Log4net Issue Type: Bug Components: Appenders Affects Versions: 1.2.13 Environment: Any Windows environment Reporter: Jose Luis Pedrosa Labels: RemoteSyslogAppender Attachments: LOG4NET-415.patch, MessageLostTest.cs, MessageLostTest.cs, MessageLostTestAsync.cs, RemoteSyslogNonBlockingv2.patch Sending UDP packages may block for some time in specific circumstances: 1) Next hop in network level 3 can't be resolved by ARP. 2) Datagram size exceeds FastSendDatagramThreshold configured size. When sending packets bigger than FastSendDatagramThreshold, the OS waits until the packet is actually sent, if the If the syslog (or the next hop to reach the syslog) is in the same VLAN/Subnet the OS tries to resolve by ARP the Ip of the configured syslog, this may take up to 3 seconds, slowing down the whole application, which in some cases can lead to outages (timeouts, DB locks...). Also the fact that each carriage return generates the headers of the packet again, that can lead to a significant overhead in some scenarios, for instance when loggign HTTP requests to a remote syslog, every header will go in a different message. Also some logging may require characters that are now skipped in patch: https://issues.apache.org/jira/browse/LOG4NET-370 I'm adding a patch that 1) Moves the use of UDPClient to Non blocking sockets, which eliminates the blocking. 2) Adds a configuration field to decide if you want Strict RFC Behaviour or not (with default Strict). Please your feedback, thanks in advance Jose Luis -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (LOG4NET-415) RemoteSyslogAppender may block for ARP resolution + Improvement Strict RFC3164
[
https://issues.apache.org/jira/browse/LOG4NET-415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13866439#comment-13866439
]
Dongsheng Song commented on LOG4NET-415:
When I send 100k messages via log4net on a super fast machine with 1G link, use
Blocking=false only get 33k mesages:
{quote}
33683
33733
33732
{quote}
Use BeginSend can get 50k messages:
{quote}
51238
50794
50979
{quote}
{code:borderStyle=solid}
using System;
using System.Diagnostics;
using System.Reflection;
using log4net;
class MessageLostTestViaLog4net
{
private static void Main()
{
System.IO.Directory.SetCurrentDirectory(AppDomain.CurrentDomain.BaseDirectory);
log4net.Config.XmlConfigurator.Configure(new
System.IO.FileInfo(AppDomain.CurrentDomain.SetupInformation.ConfigurationFile));
ILog logger =
LogManager.GetLogger(MethodBase.GetCurrentMethod().DeclaringType);
const int messageCount = 100 * 1000;
Stopwatch hpet = Stopwatch.StartNew();
long t1 = hpet.ElapsedTicks;
for(int i = 1; i = messageCount; i++)
logger.InfoFormat(Message {0} send, i);
long t2 = hpet.ElapsedTicks;
Console.WriteLine(Use log4net send {0} messages, use {1:0.000}
seconds, please check messages on the server.,
messageCount, (t2 - t1) / (double)Stopwatch.Frequency);
}
}
{code}
RemoteSyslogAppender may block for ARP resolution + Improvement Strict RFC3164
--
Key: LOG4NET-415
URL: https://issues.apache.org/jira/browse/LOG4NET-415
Project: Log4net
Issue Type: Bug
Components: Appenders
Affects Versions: 1.2.13
Environment: Any Windows environment
Reporter: Jose Luis Pedrosa
Labels: RemoteSyslogAppender
Attachments: LOG4NET-415.patch, MessageLostTest.cs,
MessageLostTest.cs, MessageLostTestAsync.cs, RemoteSyslogNonBlockingv2.patch
Sending UDP packages may block for some time in specific circumstances:
1) Next hop in network level 3 can't be resolved by ARP.
2) Datagram size exceeds FastSendDatagramThreshold configured size.
When sending packets bigger than FastSendDatagramThreshold, the OS waits
until the packet is actually sent, if the If the syslog (or the next hop to
reach the syslog) is in the same VLAN/Subnet the OS tries to resolve by ARP
the Ip of the configured syslog, this may take up to 3 seconds, slowing down
the whole application, which in some cases can lead to outages (timeouts, DB
locks...).
Also the fact that each carriage return generates the headers of the packet
again, that can lead to a significant overhead in some scenarios, for
instance when loggign HTTP requests to a remote syslog, every header will go
in a different message. Also some logging may require characters that are now
skipped in patch: https://issues.apache.org/jira/browse/LOG4NET-370
I'm adding a patch that
1) Moves the use of UDPClient to Non blocking sockets, which eliminates the
blocking.
2) Adds a configuration field to decide if you want Strict RFC Behaviour or
not (with default Strict).
Please your feedback, thanks in advance
Jose Luis
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (LOG4NET-415) RemoteSyslogAppender may block for ARP resolution + Improvement Strict RFC3164
[ https://issues.apache.org/jira/browse/LOG4NET-415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13866538#comment-13866538 ] Jose Luis Pedrosa commented on LOG4NET-415: --- Hi Cauchy, Because of your result tests, I am quite convinced that non-blocking is better for this scenario: Use Non-Blocking send 10 message, 3733 failed (3.73 %), use 1.396 seconds. Vs Use BeginSend send 10 message, 0 failed (.00 %), use 2.576 seconds. The behaviour I want to achieve is fire and forget and don't block the main thread. The async took double of time! Also I think we need to understand the use case, it's for logging over UDP. In extreme scenarios as the one we are using here, I'd radther higher message lost rate than longer execution. Also I would expect a higher CPU using async vs non-blocking, I think that is the reason why you loose less messages in async mode, because it takes more time to do the same. But I don't think anybody wants to log from a single application 100K messages per second, Imagine that in any real application (some of the one we know). I think even for really high values 10k per second, the lost would be almost 0. Also in your test you receive more messages because async is slower! so you take double of time, then double send. Blocking false maximizes what you can send per second and loosing messages if it exceeds your capacity, I think that is the best behaviour. If you want to do a final test for the good shake of testing, you can try log at fixed rate, (100, 1000, 2500, 5000 ) per second, and see when you start loosing. But for what I've seen now, non-blocking achieves better the behaviour that I'd want. Rgds JL RemoteSyslogAppender may block for ARP resolution + Improvement Strict RFC3164 -- Key: LOG4NET-415 URL: https://issues.apache.org/jira/browse/LOG4NET-415 Project: Log4net Issue Type: Bug Components: Appenders Affects Versions: 1.2.13 Environment: Any Windows environment Reporter: Jose Luis Pedrosa Labels: RemoteSyslogAppender Attachments: LOG4NET-415.patch, MessageLostTest.cs, MessageLostTest.cs, MessageLostTestAsync.cs, RemoteSyslogNonBlockingv2.patch Sending UDP packages may block for some time in specific circumstances: 1) Next hop in network level 3 can't be resolved by ARP. 2) Datagram size exceeds FastSendDatagramThreshold configured size. When sending packets bigger than FastSendDatagramThreshold, the OS waits until the packet is actually sent, if the If the syslog (or the next hop to reach the syslog) is in the same VLAN/Subnet the OS tries to resolve by ARP the Ip of the configured syslog, this may take up to 3 seconds, slowing down the whole application, which in some cases can lead to outages (timeouts, DB locks...). Also the fact that each carriage return generates the headers of the packet again, that can lead to a significant overhead in some scenarios, for instance when loggign HTTP requests to a remote syslog, every header will go in a different message. Also some logging may require characters that are now skipped in patch: https://issues.apache.org/jira/browse/LOG4NET-370 I'm adding a patch that 1) Moves the use of UDPClient to Non blocking sockets, which eliminates the blocking. 2) Adds a configuration field to decide if you want Strict RFC Behaviour or not (with default Strict). Please your feedback, thanks in advance Jose Luis -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (LOG4NET-415) RemoteSyslogAppender may block for ARP resolution + Improvement Strict RFC3164
[ https://issues.apache.org/jira/browse/LOG4NET-415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13867469#comment-13867469 ] Dongsheng Song commented on LOG4NET-415: JL, you only see BeginSend use double times (14 us vs 31 us), but forget log4net use more times. e.g. In my testing, log4net use about 50us to format messages to byte array, then the difference is 64 us vs 81 us, a little difference. If you can not accept 17 us extra cost, you should not log messages, then you can gain more times (64 us). PS: UDP has best efforts nature, but set Blocking=failse, without resend when OS buffer is full, is not best efforts. RemoteSyslogAppender may block for ARP resolution + Improvement Strict RFC3164 -- Key: LOG4NET-415 URL: https://issues.apache.org/jira/browse/LOG4NET-415 Project: Log4net Issue Type: Bug Components: Appenders Affects Versions: 1.2.13 Environment: Any Windows environment Reporter: Jose Luis Pedrosa Labels: RemoteSyslogAppender Attachments: LOG4NET-415.patch, MessageLostTest.cs, MessageLostTest.cs, MessageLostTestAsync.cs, RemoteSyslogNonBlockingv2.patch Sending UDP packages may block for some time in specific circumstances: 1) Next hop in network level 3 can't be resolved by ARP. 2) Datagram size exceeds FastSendDatagramThreshold configured size. When sending packets bigger than FastSendDatagramThreshold, the OS waits until the packet is actually sent, if the If the syslog (or the next hop to reach the syslog) is in the same VLAN/Subnet the OS tries to resolve by ARP the Ip of the configured syslog, this may take up to 3 seconds, slowing down the whole application, which in some cases can lead to outages (timeouts, DB locks...). Also the fact that each carriage return generates the headers of the packet again, that can lead to a significant overhead in some scenarios, for instance when loggign HTTP requests to a remote syslog, every header will go in a different message. Also some logging may require characters that are now skipped in patch: https://issues.apache.org/jira/browse/LOG4NET-370 I'm adding a patch that 1) Moves the use of UDPClient to Non blocking sockets, which eliminates the blocking. 2) Adds a configuration field to decide if you want Strict RFC Behaviour or not (with default Strict). Please your feedback, thanks in advance Jose Luis -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (LOG4NET-415) RemoteSyslogAppender may block for ARP resolution + Improvement Strict RFC3164
[ https://issues.apache.org/jira/browse/LOG4NET-415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13865410#comment-13865410 ] Jose Luis Pedrosa commented on LOG4NET-415: --- Hi, Cauchy/Dongsheng, please send the patch in attached code, otherwise the comments are harder to follow. 1) I did this test using Log4net log.info (instead of sending just on a raw socket) and 0 packet lost or exceptions over millions of messages. Anyway I don't think this would happen in any other scenario than a thread sending messages continously, actually i did that test, only printing one byte to the console every 1 messages, and 0 lost. In any case, a while loop for extraordinary long messages blocks is a good idea. Despite the current log4net version does not check that. 2) Stefan: Any feedback about the CF framework and the strict option? Rgds JL RemoteSyslogAppender may block for ARP resolution + Improvement Strict RFC3164 -- Key: LOG4NET-415 URL: https://issues.apache.org/jira/browse/LOG4NET-415 Project: Log4net Issue Type: Bug Components: Appenders Affects Versions: 1.2.13 Environment: Any Windows environment Reporter: Jose Luis Pedrosa Labels: RemoteSyslogAppender Attachments: LOG4NET-415.patch, MessageLostTest.cs, RemoteSyslogNonBlockingv2.patch Sending UDP packages may block for some time in specific circumstances: 1) Next hop in network level 3 can't be resolved by ARP. 2) Datagram size exceeds FastSendDatagramThreshold configured size. When sending packets bigger than FastSendDatagramThreshold, the OS waits until the packet is actually sent, if the If the syslog (or the next hop to reach the syslog) is in the same VLAN/Subnet the OS tries to resolve by ARP the Ip of the configured syslog, this may take up to 3 seconds, slowing down the whole application, which in some cases can lead to outages (timeouts, DB locks...). Also the fact that each carriage return generates the headers of the packet again, that can lead to a significant overhead in some scenarios, for instance when loggign HTTP requests to a remote syslog, every header will go in a different message. Also some logging may require characters that are now skipped in patch: https://issues.apache.org/jira/browse/LOG4NET-370 I'm adding a patch that 1) Moves the use of UDPClient to Non blocking sockets, which eliminates the blocking. 2) Adds a configuration field to decide if you want Strict RFC Behaviour or not (with default Strict). Please your feedback, thanks in advance Jose Luis -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (LOG4NET-415) RemoteSyslogAppender may block for ARP resolution + Improvement Strict RFC3164
[
https://issues.apache.org/jira/browse/LOG4NET-415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13865444#comment-13865444
]
Dongsheng Song commented on LOG4NET-415:
{code:title=Bar.java|borderStyle=solid}
buffer = this.Encoding.GetBytes(builder.ToString());
Int32 offset = 0;
while (offset != buffer.Length)
{
offset += this.Client.SendTo(buffer, offset,
buffer.Length - offset, SocketFlags.None,
this.RemoteEndPoint);
}
{code}
RemoteSyslogAppender may block for ARP resolution + Improvement Strict RFC3164
--
Key: LOG4NET-415
URL: https://issues.apache.org/jira/browse/LOG4NET-415
Project: Log4net
Issue Type: Bug
Components: Appenders
Affects Versions: 1.2.13
Environment: Any Windows environment
Reporter: Jose Luis Pedrosa
Labels: RemoteSyslogAppender
Attachments: LOG4NET-415.patch, MessageLostTest.cs,
RemoteSyslogNonBlockingv2.patch
Sending UDP packages may block for some time in specific circumstances:
1) Next hop in network level 3 can't be resolved by ARP.
2) Datagram size exceeds FastSendDatagramThreshold configured size.
When sending packets bigger than FastSendDatagramThreshold, the OS waits
until the packet is actually sent, if the If the syslog (or the next hop to
reach the syslog) is in the same VLAN/Subnet the OS tries to resolve by ARP
the Ip of the configured syslog, this may take up to 3 seconds, slowing down
the whole application, which in some cases can lead to outages (timeouts, DB
locks...).
Also the fact that each carriage return generates the headers of the packet
again, that can lead to a significant overhead in some scenarios, for
instance when loggign HTTP requests to a remote syslog, every header will go
in a different message. Also some logging may require characters that are now
skipped in patch: https://issues.apache.org/jira/browse/LOG4NET-370
I'm adding a patch that
1) Moves the use of UDPClient to Non blocking sockets, which eliminates the
blocking.
2) Adds a configuration field to decide if you want Strict RFC Behaviour or
not (with default Strict).
Please your feedback, thanks in advance
Jose Luis
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (LOG4NET-415) RemoteSyslogAppender may block for ARP resolution + Improvement Strict RFC3164
[ https://issues.apache.org/jira/browse/LOG4NET-415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13865462#comment-13865462 ] Jose Luis Pedrosa commented on LOG4NET-415: --- Hi Cauchy If you apply the patch to the code, you would realize that it is already inside a try/catch block with log4net error handling this exception. It was already there in the trunk. Rgds JL RemoteSyslogAppender may block for ARP resolution + Improvement Strict RFC3164 -- Key: LOG4NET-415 URL: https://issues.apache.org/jira/browse/LOG4NET-415 Project: Log4net Issue Type: Bug Components: Appenders Affects Versions: 1.2.13 Environment: Any Windows environment Reporter: Jose Luis Pedrosa Labels: RemoteSyslogAppender Attachments: LOG4NET-415.patch, MessageLostTest.cs, RemoteSyslogNonBlockingv2.patch Sending UDP packages may block for some time in specific circumstances: 1) Next hop in network level 3 can't be resolved by ARP. 2) Datagram size exceeds FastSendDatagramThreshold configured size. When sending packets bigger than FastSendDatagramThreshold, the OS waits until the packet is actually sent, if the If the syslog (or the next hop to reach the syslog) is in the same VLAN/Subnet the OS tries to resolve by ARP the Ip of the configured syslog, this may take up to 3 seconds, slowing down the whole application, which in some cases can lead to outages (timeouts, DB locks...). Also the fact that each carriage return generates the headers of the packet again, that can lead to a significant overhead in some scenarios, for instance when loggign HTTP requests to a remote syslog, every header will go in a different message. Also some logging may require characters that are now skipped in patch: https://issues.apache.org/jira/browse/LOG4NET-370 I'm adding a patch that 1) Moves the use of UDPClient to Non blocking sockets, which eliminates the blocking. 2) Adds a configuration field to decide if you want Strict RFC Behaviour or not (with default Strict). Please your feedback, thanks in advance Jose Luis -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (LOG4NET-415) RemoteSyslogAppender may block for ARP resolution + Improvement Strict RFC3164
[
https://issues.apache.org/jira/browse/LOG4NET-415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13865469#comment-13865469
]
Dongsheng Song commented on LOG4NET-415:
Again, simple set Blocking to false is not correct async program, the prefer
async on windows is:
Do not set Blocking to false, just use Socket.Begin* API, e.g. use
{code:borderStyle=solid}
socket.BeginSendTo(buffer, 0, buffer.Length,
SocketFlags.Truncated,
RemoteEndPoint,
delegate(IAsyncResult ar)
{
Socket client = (Socket)ar.AsyncState;
client.EndSendTo(ar);
},
socket);
{code}
instead of:
{code:borderStyle=solid}
Socket.SendTo(buffer, this.RemoteEndPoint);
{code}
RemoteSyslogAppender may block for ARP resolution + Improvement Strict RFC3164
--
Key: LOG4NET-415
URL: https://issues.apache.org/jira/browse/LOG4NET-415
Project: Log4net
Issue Type: Bug
Components: Appenders
Affects Versions: 1.2.13
Environment: Any Windows environment
Reporter: Jose Luis Pedrosa
Labels: RemoteSyslogAppender
Attachments: LOG4NET-415.patch, MessageLostTest.cs,
RemoteSyslogNonBlockingv2.patch
Sending UDP packages may block for some time in specific circumstances:
1) Next hop in network level 3 can't be resolved by ARP.
2) Datagram size exceeds FastSendDatagramThreshold configured size.
When sending packets bigger than FastSendDatagramThreshold, the OS waits
until the packet is actually sent, if the If the syslog (or the next hop to
reach the syslog) is in the same VLAN/Subnet the OS tries to resolve by ARP
the Ip of the configured syslog, this may take up to 3 seconds, slowing down
the whole application, which in some cases can lead to outages (timeouts, DB
locks...).
Also the fact that each carriage return generates the headers of the packet
again, that can lead to a significant overhead in some scenarios, for
instance when loggign HTTP requests to a remote syslog, every header will go
in a different message. Also some logging may require characters that are now
skipped in patch: https://issues.apache.org/jira/browse/LOG4NET-370
I'm adding a patch that
1) Moves the use of UDPClient to Non blocking sockets, which eliminates the
blocking.
2) Adds a configuration field to decide if you want Strict RFC Behaviour or
not (with default Strict).
Please your feedback, thanks in advance
Jose Luis
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (LOG4NET-415) RemoteSyslogAppender may block for ARP resolution + Improvement Strict RFC3164
[ https://issues.apache.org/jira/browse/LOG4NET-415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13865472#comment-13865472 ] Dongsheng Song commented on LOG4NET-415: When WSAENOBUFS happened, you must resend. This is not network error, just OS buffer is full temporary. It may be disappeared immediately. Use BeginSendTo is the most simplest way. RemoteSyslogAppender may block for ARP resolution + Improvement Strict RFC3164 -- Key: LOG4NET-415 URL: https://issues.apache.org/jira/browse/LOG4NET-415 Project: Log4net Issue Type: Bug Components: Appenders Affects Versions: 1.2.13 Environment: Any Windows environment Reporter: Jose Luis Pedrosa Labels: RemoteSyslogAppender Attachments: LOG4NET-415.patch, MessageLostTest.cs, RemoteSyslogNonBlockingv2.patch Sending UDP packages may block for some time in specific circumstances: 1) Next hop in network level 3 can't be resolved by ARP. 2) Datagram size exceeds FastSendDatagramThreshold configured size. When sending packets bigger than FastSendDatagramThreshold, the OS waits until the packet is actually sent, if the If the syslog (or the next hop to reach the syslog) is in the same VLAN/Subnet the OS tries to resolve by ARP the Ip of the configured syslog, this may take up to 3 seconds, slowing down the whole application, which in some cases can lead to outages (timeouts, DB locks...). Also the fact that each carriage return generates the headers of the packet again, that can lead to a significant overhead in some scenarios, for instance when loggign HTTP requests to a remote syslog, every header will go in a different message. Also some logging may require characters that are now skipped in patch: https://issues.apache.org/jira/browse/LOG4NET-370 I'm adding a patch that 1) Moves the use of UDPClient to Non blocking sockets, which eliminates the blocking. 2) Adds a configuration field to decide if you want Strict RFC Behaviour or not (with default Strict). Please your feedback, thanks in advance Jose Luis -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (LOG4NET-415) RemoteSyslogAppender may block for ARP resolution + Improvement Strict RFC3164
[ https://issues.apache.org/jira/browse/LOG4NET-415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13865497#comment-13865497 ] Jose Luis Pedrosa commented on LOG4NET-415: --- Cauchy This is UDP, Fire and forget, you don't block your bussiness for logging. Failures are better than retries, TCP vs UDP, And again this is a very very exotic scenario that It's not possible to reproduce using the whole .log4net (only raw socket send in a loop to make it happen) Async involves more threads going on on your application and more memory consumption. Async is good when you need to know when your invokation has completed and you need to do something afterwards, which is not the case. Rgds RemoteSyslogAppender may block for ARP resolution + Improvement Strict RFC3164 -- Key: LOG4NET-415 URL: https://issues.apache.org/jira/browse/LOG4NET-415 Project: Log4net Issue Type: Bug Components: Appenders Affects Versions: 1.2.13 Environment: Any Windows environment Reporter: Jose Luis Pedrosa Labels: RemoteSyslogAppender Attachments: LOG4NET-415.patch, MessageLostTest.cs, RemoteSyslogNonBlockingv2.patch Sending UDP packages may block for some time in specific circumstances: 1) Next hop in network level 3 can't be resolved by ARP. 2) Datagram size exceeds FastSendDatagramThreshold configured size. When sending packets bigger than FastSendDatagramThreshold, the OS waits until the packet is actually sent, if the If the syslog (or the next hop to reach the syslog) is in the same VLAN/Subnet the OS tries to resolve by ARP the Ip of the configured syslog, this may take up to 3 seconds, slowing down the whole application, which in some cases can lead to outages (timeouts, DB locks...). Also the fact that each carriage return generates the headers of the packet again, that can lead to a significant overhead in some scenarios, for instance when loggign HTTP requests to a remote syslog, every header will go in a different message. Also some logging may require characters that are now skipped in patch: https://issues.apache.org/jira/browse/LOG4NET-370 I'm adding a patch that 1) Moves the use of UDPClient to Non blocking sockets, which eliminates the blocking. 2) Adds a configuration field to decide if you want Strict RFC Behaviour or not (with default Strict). Please your feedback, thanks in advance Jose Luis -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (LOG4NET-415) RemoteSyslogAppender may block for ARP resolution + Improvement Strict RFC3164
[
https://issues.apache.org/jira/browse/LOG4NET-415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13866321#comment-13866321
]
Dongsheng Song commented on LOG4NET-415:
Here is the test result:
{quote}
C:\MessageLostTest.exe
Use Non-Blocking send 10 message, 3733 failed (3.73 %), use 1.396 seconds.
C:\MessageLostTest.exe
Use Non-Blocking send 10 message, 5912 failed (5.91 %), use 1.458 seconds.
C:\MessageLostTest
Use Non-Blocking send 10 message, 7237 failed (7.24 %), use 1.382 seconds.
C:\MessageLostTestAsync
Use BeginSend send 10 message, 0 failed (.00 %), use 2.576 seconds.
C:\MessageLostTestAsync
Use BeginSend send 10 message, 0 failed (.00 %), use 3.470 seconds.
C:\MessageLostTestAsync
Use BeginSend send 10 message, 0 failed (.00 %), use 3.273 seconds.
{quote}
From the test result, we can see use BeginSend is the best choice.
If we can do the best, why not ?
RemoteSyslogAppender may block for ARP resolution + Improvement Strict RFC3164
--
Key: LOG4NET-415
URL: https://issues.apache.org/jira/browse/LOG4NET-415
Project: Log4net
Issue Type: Bug
Components: Appenders
Affects Versions: 1.2.13
Environment: Any Windows environment
Reporter: Jose Luis Pedrosa
Labels: RemoteSyslogAppender
Attachments: LOG4NET-415.patch, MessageLostTest.cs,
MessageLostTest.cs, MessageLostTestAsync.cs, RemoteSyslogNonBlockingv2.patch
Sending UDP packages may block for some time in specific circumstances:
1) Next hop in network level 3 can't be resolved by ARP.
2) Datagram size exceeds FastSendDatagramThreshold configured size.
When sending packets bigger than FastSendDatagramThreshold, the OS waits
until the packet is actually sent, if the If the syslog (or the next hop to
reach the syslog) is in the same VLAN/Subnet the OS tries to resolve by ARP
the Ip of the configured syslog, this may take up to 3 seconds, slowing down
the whole application, which in some cases can lead to outages (timeouts, DB
locks...).
Also the fact that each carriage return generates the headers of the packet
again, that can lead to a significant overhead in some scenarios, for
instance when loggign HTTP requests to a remote syslog, every header will go
in a different message. Also some logging may require characters that are now
skipped in patch: https://issues.apache.org/jira/browse/LOG4NET-370
I'm adding a patch that
1) Moves the use of UDPClient to Non blocking sockets, which eliminates the
blocking.
2) Adds a configuration field to decide if you want Strict RFC Behaviour or
not (with default Strict).
Please your feedback, thanks in advance
Jose Luis
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (LOG4NET-415) RemoteSyslogAppender may block for ARP resolution + Improvement Strict RFC3164
[ https://issues.apache.org/jira/browse/LOG4NET-415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13864040#comment-13864040 ] Jose Luis Pedrosa commented on LOG4NET-415: --- Hi all, Please also note that in the http://tools.ietf.org/html/rfc5424 (a newer syslog specification) specifies that all encoding should be UTF8. Maybe we should change the parameter name to control which specification is followed. RemoteSyslogAppender may block for ARP resolution + Improvement Strict RFC3164 -- Key: LOG4NET-415 URL: https://issues.apache.org/jira/browse/LOG4NET-415 Project: Log4net Issue Type: Bug Components: Appenders Affects Versions: 1.2.13 Environment: Any Windows environment Reporter: Jose Luis Pedrosa Labels: RemoteSyslogAppender Attachments: LOG4NET-415.patch Sending UDP packages may block for some time in specific circumstances: 1) Next hop in network level 3 can't be resolved by ARP. 2) Datagram size exceeds FastSendDatagramThreshold configured size. When sending packets bigger than FastSendDatagramThreshold, the OS waits until the packet is actually sent, if the If the syslog (or the next hop to reach the syslog) is in the same VLAN/Subnet the OS tries to resolve by ARP the Ip of the configured syslog, this may take up to 3 seconds, slowing down the whole application, which in some cases can lead to outages (timeouts, DB locks...). Also the fact that each carriage return generates the headers of the packet again, that can lead to a significant overhead in some scenarios, for instance when loggign HTTP requests to a remote syslog, every header will go in a different message. Also some logging may require characters that are now skipped in patch: https://issues.apache.org/jira/browse/LOG4NET-370 I'm adding a patch that 1) Moves the use of UDPClient to Non blocking sockets, which eliminates the blocking. 2) Adds a configuration field to decide if you want Strict RFC Behaviour or not (with default Strict). Please your feedback, thanks in advance Jose Luis -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (LOG4NET-415) RemoteSyslogAppender may block for ARP resolution + Improvement Strict RFC3164
[ https://issues.apache.org/jira/browse/LOG4NET-415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13864056#comment-13864056 ] Stefan Bodewig commented on LOG4NET-415: I'm not a syslog expert, far from it. I'm afraid if we start talking about RFC5424 then we'd signal way more support than we actually offer (structured data, TLS, maybe even TCP based transport - I know the latter are technically different RFCs). As for your patch, I'm not sure what the non-strict mode would offer - does it work for RFC5424 compliant targets? I'm more than happy with using non-blocking sockets, BTW, will only need to look up the used APIs to verify they are available for NETCF as well. RemoteSyslogAppender may block for ARP resolution + Improvement Strict RFC3164 -- Key: LOG4NET-415 URL: https://issues.apache.org/jira/browse/LOG4NET-415 Project: Log4net Issue Type: Bug Components: Appenders Affects Versions: 1.2.13 Environment: Any Windows environment Reporter: Jose Luis Pedrosa Labels: RemoteSyslogAppender Attachments: LOG4NET-415.patch Sending UDP packages may block for some time in specific circumstances: 1) Next hop in network level 3 can't be resolved by ARP. 2) Datagram size exceeds FastSendDatagramThreshold configured size. When sending packets bigger than FastSendDatagramThreshold, the OS waits until the packet is actually sent, if the If the syslog (or the next hop to reach the syslog) is in the same VLAN/Subnet the OS tries to resolve by ARP the Ip of the configured syslog, this may take up to 3 seconds, slowing down the whole application, which in some cases can lead to outages (timeouts, DB locks...). Also the fact that each carriage return generates the headers of the packet again, that can lead to a significant overhead in some scenarios, for instance when loggign HTTP requests to a remote syslog, every header will go in a different message. Also some logging may require characters that are now skipped in patch: https://issues.apache.org/jira/browse/LOG4NET-370 I'm adding a patch that 1) Moves the use of UDPClient to Non blocking sockets, which eliminates the blocking. 2) Adds a configuration field to decide if you want Strict RFC Behaviour or not (with default Strict). Please your feedback, thanks in advance Jose Luis -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (LOG4NET-415) RemoteSyslogAppender may block for ARP resolution + Improvement Strict RFC3164
[ https://issues.apache.org/jira/browse/LOG4NET-415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13864131#comment-13864131 ] Jose Luis Pedrosa commented on LOG4NET-415: --- Hi Stefan, Thanks for your answer, About CF: In the MS page i looked before sending the patch it said it was supported, that is why I removed the ifdef markers, but indeed i did not test against that platform. About Strict mode, the issue it is trying to address is: avoid a lot of extra data when logging things that contains new line (IE: HTTP SOAP requests with headers), indeed the patch does not provide RFC5424 compliance. But I am quite confident this can be useful for multiple scenarios (An stack trace of an exception for isntance). So my opinion is: This Strict mode (maybe rename the variable to, SplitByNewLine) is useful in various scenarios, and later create another SyslogAppender for RFC5424 or add a parameter/variable to specify the RFC compliance it should use, Like SyslogVersion = RFC5424|RFC3164. I think the option of RFC3164 and split (or not) by lines would be still usefull. What do you think about it? Rgds JL RemoteSyslogAppender may block for ARP resolution + Improvement Strict RFC3164 -- Key: LOG4NET-415 URL: https://issues.apache.org/jira/browse/LOG4NET-415 Project: Log4net Issue Type: Bug Components: Appenders Affects Versions: 1.2.13 Environment: Any Windows environment Reporter: Jose Luis Pedrosa Labels: RemoteSyslogAppender Attachments: LOG4NET-415.patch Sending UDP packages may block for some time in specific circumstances: 1) Next hop in network level 3 can't be resolved by ARP. 2) Datagram size exceeds FastSendDatagramThreshold configured size. When sending packets bigger than FastSendDatagramThreshold, the OS waits until the packet is actually sent, if the If the syslog (or the next hop to reach the syslog) is in the same VLAN/Subnet the OS tries to resolve by ARP the Ip of the configured syslog, this may take up to 3 seconds, slowing down the whole application, which in some cases can lead to outages (timeouts, DB locks...). Also the fact that each carriage return generates the headers of the packet again, that can lead to a significant overhead in some scenarios, for instance when loggign HTTP requests to a remote syslog, every header will go in a different message. Also some logging may require characters that are now skipped in patch: https://issues.apache.org/jira/browse/LOG4NET-370 I'm adding a patch that 1) Moves the use of UDPClient to Non blocking sockets, which eliminates the blocking. 2) Adds a configuration field to decide if you want Strict RFC Behaviour or not (with default Strict). Please your feedback, thanks in advance Jose Luis -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (LOG4NET-415) RemoteSyslogAppender may block for ARP resolution + Improvement Strict RFC3164
[
https://issues.apache.org/jira/browse/LOG4NET-415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13865013#comment-13865013
]
Dongsheng Song commented on LOG4NET-415:
Just Set Socket.Blocking to false is not correct non-block programing,
this will cause huge unexpected message lost.
Here is a demo:
Message lost rate: 32.61 %, 32.61 %
Message lost rate: 32.90 %, 32.76 %
Message lost rate: 27.28 %, 30.81 %
Message lost rate: 55.46 %, 36.56 %
...
Here is my test code:
class MessageLostTest
{
private static long _messageCount;
private static long _errorCount;
private static void Main()
{
Socket socket = new Socket(AddressFamily.InterNetwork,
SocketType.Dgram, ProtocolType.Udp);
socket.Connect(192.168.30.233, 514);
socket.Blocking = false;
new Thread(delegate(object obj)
{
long m = _messageCount, e = _errorCount;
while (true)
{
Thread.Sleep(5000);
long m2 = Interlocked.Read(ref _messageCount);
long e2 = Interlocked.Read(ref _errorCount);
Console.WriteLine(Message lost rate: {0:##.00 %}, {1:
##.00 %},
(e2 - e) / (double)(m2 - m), e2 /
(double)m2);
m = m2; e = e2;
}
}).Start();
byte[] message = new byte[400];
while (true)
{
try
{
if (socket.Send(message) != message.Length)
Interlocked.Increment(ref _errorCount);
Interlocked.Increment(ref _messageCount);
}
catch (Exception e)
{
Interlocked.Increment(ref _errorCount);
//Console.WriteLine(Socket.Send failed: {0}{1}{2},
e.Message, Environment.NewLine, e.StackTrace);
}
}
}
}
RemoteSyslogAppender may block for ARP resolution + Improvement Strict RFC3164
--
Key: LOG4NET-415
URL: https://issues.apache.org/jira/browse/LOG4NET-415
Project: Log4net
Issue Type: Bug
Components: Appenders
Affects Versions: 1.2.13
Environment: Any Windows environment
Reporter: Jose Luis Pedrosa
Labels: RemoteSyslogAppender
Attachments: LOG4NET-415.patch
Sending UDP packages may block for some time in specific circumstances:
1) Next hop in network level 3 can't be resolved by ARP.
2) Datagram size exceeds FastSendDatagramThreshold configured size.
When sending packets bigger than FastSendDatagramThreshold, the OS waits
until the packet is actually sent, if the If the syslog (or the next hop to
reach the syslog) is in the same VLAN/Subnet the OS tries to resolve by ARP
the Ip of the configured syslog, this may take up to 3 seconds, slowing down
the whole application, which in some cases can lead to outages (timeouts, DB
locks...).
Also the fact that each carriage return generates the headers of the packet
again, that can lead to a significant overhead in some scenarios, for
instance when loggign HTTP requests to a remote syslog, every header will go
in a different message. Also some logging may require characters that are now
skipped in patch: https://issues.apache.org/jira/browse/LOG4NET-370
I'm adding a patch that
1) Moves the use of UDPClient to Non blocking sockets, which eliminates the
blocking.
2) Adds a configuration field to decide if you want Strict RFC Behaviour or
not (with default Strict).
Please your feedback, thanks in advance
Jose Luis
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
