Hello, I did some performance tests with RELP recently and I had a nice surprise regarding performance. I used rsyslog 7.4.3 and, depending on the conditions (network latency, no of concurrent sender threads) it went from a few times slower than TCP up to pretty much the same throughput under high load.
I will post the full results and test procedure somewhere as soon as I have some spare time. Note the fuzzy deadline :p On Aug 14, 2013 3:44 PM, "Rainer Gerhards" <[email protected]> wrote: > On Wed, Aug 14, 2013 at 7:31 AM, David Lang <[email protected]> wrote: > > > The application level acknowledgement takes care of just about > everything. > > > > The one remaining gap is that if the sender gets restarted while the > > recipent is down, the sender gets confused and can loose some messages > > > > > I'll try to look into this anyhow. A "fix" would be really nice to have. > But that may take some time, so probably end of month. > > > > I don't know what performance RELP will provide, unfortunantly it does > not > > allow overlapping sent messages, so you have to do a round trip to > > acknoledge after every message. > > > > > It's definitely slower. > > On the round-trips: it uses a Window, just like TCP does. So the round-trip > is not required after each message. HOWEVER, all librelp versions up until > roughly 2 month ago had a very small default window and code that made it > fill up quickly, so it boiled down to one round trip per message in steady > state. This has been much improved with recent librelp and rsyslog > versions. The performance got considerably up thanks to these changes (but > still does not match plain TCP). With the new code, the user can also > specify the window size. The default is still at 128, which is not much for > typical syslog messages (and relatively slow lines). > > HTH > Rainer > > > David Lang > > > > > > On Tue, 13 Aug 2013, Erik Steffl wrote: > > > > thanks, that makes sense. BTW switching to RELP :) Just trying to > >> understand different failure scenarios to make sure we don't miss > anything > >> thatr might be relevant even when using RELP. > >> > >> erik > >> > >> On 08/12/2013 07:16 PM, David Lang wrote: > >> > >>> On Mon, 12 Aug 2013, Erik Steffl wrote: > >>> > >>> On 08/12/2013 03:53 PM, David Lang wrote: > >>>> > >>>>> On Mon, 12 Aug 2013, Erik Steffl wrote: > >>>>> > >>>>> On 08/12/2013 01:21 PM, David Lang wrote: > >>>>>> > >>>>>>> Eric, with your particular load balancer, if you telnet to the ip > on > >>>>>>> port 514 does telnet fail with an error, or does it establish a > >>>>>>> connection that then gets close almost immediatly? > >>>>>>> > >>>>>> > >>>>>> it opens connection that is closed immediately (to Amazon elastic > load > >>>>>> balancer with no hosts behind it): > >>>>>> > >>>>> > >>>>> Ok, I think what is happening is that rsyslog is opening the > connection > >>>>> successfully, and is then sending the log message before it gets > >>>>> closed. > >>>>> > >>>>> When you use netcat to do your test, does it show the same behavior > as > >>>>> telnet? (connect then close), or is it somehow failing the connect? > >>>>> > >>>> > >>>> in same scenario it behaves in similar way, it connects, sends one > >>>> message (which appears to be sent ok) then closes connection (it does > >>>> not retry), the difference is that connect return -1 and errno is > >>>> EINPROGRESS (unlike rsyslog connects successfully). > >>>> > >>> > >>> different configuration options, something for Rainer to look at when > he > >>> gets back from vacation. Somehow detecting that things aren't in good > >>> shape would be a good thing, but I suspect that what's happening is > that > >>> there is some data being sent in the nc case (negotiation of some sort) > >>> that's not being sent in the rsyslog case that's how the problem is > >>> being detected. > >>> > >>> As far as I can tell it's the load balancer behaving badly, the > >>>> connect and sendto/write should not succeed if load balancer has > >>>> nowhere to send it (looking at Amazon console I see that load balancer > >>>> has no hosts). > >>>> > >>> > >>> yes and no. > >>> > >>> There are two ways that a load balancer can operate > >>> > >>> 1. the load balancer does not terminate the connection. It just > >>> redirects it via NAT (or MAC rewriting) to the appropriate destination. > >>> > >>> In this case, the load balancer must make it's decision before any data > >>> is sent at all. It can't terminate the SSL on the load balancer, and it > >>> can't direct sessions based on data in the message (such as cookie > >>> values) > >>> > >>> In this case, if there is no active server, the sending system will not > >>> succeed in making a connection. > >>> > >>> > >>> 2. the load balancer does terminate the connenction, then it makes a > new > >>> connection to the destination server. > >>> > >>> In this case, the load balancer has much more flexibility in it's > >>> decision, and it can terminate the SSL session on the load balancer. > >>> > >>> But when the load balancer acts like this, you get the behavior that > you > >>> are seeing, where the connection is established even if there is no > >>> back-end system available. In theory the load balancer could stop > >>> listening on the VIP, but disconnecting and reconnecting like that is > >>> expensive, and you aren't supposed to be in a situation hwere you don't > >>> have any available servers, right? :-) > >>> > >>> I've seen this same behavior with several different brands of load > >>> balancers, it's not just the AWS ELB that does this, I see the same > >>> thing when you hit a radware or F5 load balancer that is in the mode > >>> that it terminates the connection. > >>> > >>> > >>> > >>> As far as the sendto/write failing, that isn't going to happen because > >>> the sendto/write doesn't actually involve the remote machine. All that > >>> happens at the sendto/write command is that the program hands the data > >>> to the kernel on the local machine for the kernel to send sometime > >>> later. The kernel will only return an error if there is no connection, > >>> or the queues are full on the sending system. > >>> > >>> In this case, the connection is made (syn, syn/ack, ack exchange), then > >>> when the sending machine first trys to send actual data, it gets an > >>> error and terminates the connection. But this isn't happening until > >>> after rsyslog has 'sent' the log. > >>> > >>> the same thing would happen if something happened to the recieving > >>> machine after the connection is established. > >>> > >>> If you think about this a bit. Suppose that you have a high speed > >>> connection across the country. Since your data can only travel at the > >>> speed of light, you can have a LOT of data in flight (64K was the old > >>> configured max, but with newer, high speed lines, it's common to end up > >>> with a few MB of data in flight). The sending software thinks the data > >>> has been delivered as soon as it hands it to the kernel to send, but > >>> when the recieving system dies (or a firewall cuts the connection), all > >>> the data in flight will not be recieved, but the sending software has > no > >>> way of knowing this fact. > >>> > >>> This is why TCP is not really reliable when something can happen to the > >>> connection or the machine at the far end. > >>> > >>> The way around this is to have the recieving application acknowlege > that > >>> it has the log message. This is _exactly_ what RELP does. And this is > >>> one of the reasons that it does this. > >>> > >>> David Lang > >>> > >>> erik > >>>> > >>>> > >>>>> David Lang > >>>>> > >>>>> erik@yummly-ubuntu-erik-**gazelle:~$ telnet $elbHost 5140 > >>>>>> Trying 54.243.148.203... > >>>>>> Connected to ...elb host... > >>>>>> Escape character is '^]'. > >>>>>> Connection closed by foreign host. > >>>>>> > >>>>>> strace (part from opening to closing connection, the connection > opens > >>>>>> successfully)): > >>>>>> > >>>>>> socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 3 > >>>>>> setsockopt(3, SOL_IP, IP_TOS, [16], 4) = 0 > >>>>>> connect(3, {sa_family=AF_INET, sin_port=htons(5140), > >>>>>> sin_addr=inet_addr("54.243.**148.203")}, 16) = 0 > >>>>>> open("/etc/telnetrc", O_RDONLY) = -1 ENOENT (No such file or > >>>>>> directory) > >>>>>> open("/home/erik/.telnetrc", O_RDONLY) = -1 ENOENT (No such file or > >>>>>> directory) > >>>>>> write(1, "Connected to > >>>>>> erik-tcp-test-2132493349.us-**east-1.elb.amazonaws.com< > http://erik-tcp-test-2132493349.us-east-1.elb.amazonaws.com>.\n", > >>>>>> 67) = 67 > >>>>>> write(1, "Escape character is '^]'.\n", 26) = 26 > >>>>>> rt_sigprocmask(SIG_BLOCK, NULL, [], 8) = 0 > >>>>>> rt_sigaction(SIGINT, {0x407970, [INT], SA_RESTORER|SA_RESTART, > >>>>>> 0x7f0b9e3fc0b0}, {SIG_DFL, [], 0}, 8) = 0 > >>>>>> rt_sigaction(SIGQUIT, {0x407920, [QUIT], SA_RESTORER|SA_RESTART, > >>>>>> 0x7f0b9e3fc0b0}, {SIG_DFL, [], 0}, 8) = 0 > >>>>>> rt_sigaction(SIGWINCH, {0x407900, [WINCH], SA_RESTORER|SA_RESTART, > >>>>>> 0x7f0b9e3fc0b0}, {SIG_DFL, [], 0}, 8) = 0 > >>>>>> rt_sigaction(SIGTSTP, {0x40b6c0, [TSTP], SA_RESTORER|SA_RESTART, > >>>>>> 0x7f0b9e3fc0b0}, {0x40b6c0, [TSTP], SA_RESTORER|SA_RESTART, > >>>>>> 0x7f0b9e3fc0b0}, 8) = 0 > >>>>>> ioctl(0, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon > echo > >>>>>> ...}) = 0 > >>>>>> ioctl(0, SNDCTL_TMR_STOP or TCSETSW, {B38400 opost isig icanon echo > >>>>>> ...}) = 0 > >>>>>> ioctl(0, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon > echo > >>>>>> ...}) = 0 > >>>>>> ioctl(0, FIONBIO, [1]) = 0 > >>>>>> ioctl(1, FIONBIO, [1]) = 0 > >>>>>> ioctl(3, FIONBIO, [1]) = 0 > >>>>>> setsockopt(3, SOL_SOCKET, SO_OOBINLINE, [1], 4) = 0 > >>>>>> select(4, [0 3], [], [3], {0, 0}) = 0 (Timeout) > >>>>>> select(4, [0 3], [], [3], NULL) = 1 (in [3]) > >>>>>> recvfrom(3, "", 8191, 0, NULL, NULL) = 0 > >>>>>> rt_sigaction(SIGTSTP, {SIG_DFL, [TSTP], SA_RESTORER|SA_RESTART, > >>>>>> 0x7f0b9e3fc0b0}, {0x40b6c0, [TSTP], SA_RESTORER|SA_RESTART, > >>>>>> 0x7f0b9e3fc0b0}, 8) = 0 > >>>>>> ioctl(0, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon > echo > >>>>>> ...}) = 0 > >>>>>> ioctl(0, SNDCTL_TMR_STOP or TCSETSW, {B38400 opost isig icanon echo > >>>>>> ...}) = 0 > >>>>>> ioctl(0, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon > echo > >>>>>> ...}) = 0 > >>>>>> ioctl(0, FIONBIO, [0]) = 0 > >>>>>> ioctl(1, FIONBIO, [0]) = 0 > >>>>>> close(3) = 0 > >>>>>> > >>>>>> erik > >>>>>> > >>>>>> David Lang > >>>>>>> > >>>>>>> On Fri, 9 Aug 2013, Erik Steffl wrote: > >>>>>>> > >>>>>>> Date: Fri, 09 Aug 2013 19:10:29 -0700 > >>>>>>>> From: Erik Steffl <[email protected]> > >>>>>>>> Reply-To: rsyslog-users <[email protected]> > >>>>>>>> To: rsyslog-users <[email protected]> > >>>>>>>> Subject: Re: [rsyslog] rsyslogd keeps sending data to closed > >>>>>>>> connection > >>>>>>>> > >>>>>>>> On 07/26/2013 04:33 PM, David Lang wrote: > >>>>>>>> > >>>>>>>>> On Fri, 26 Jul 2013, Erik Steffl wrote: > >>>>>>>>> > >>>>>>>>> While testing rsyslogd sending logs to a remote server I > >>>>>>>>>> encountered > >>>>>>>>>> this scenario: > >>>>>>>>>> > >>>>>>>>>> - remote server (which is behind amazon elastic load balancer) > >>>>>>>>>> closes > >>>>>>>>>> connection (the host is up but nobody listens on the port) > >>>>>>>>>> > >>>>>>>>>> - rsyslogd seems to be sending data to the closed connection > >>>>>>>>>> > >>>>>>>>>> - connection is in CLOSED_WAIT state > >>>>>>>>>> > >>>>>>>>>> strace of rsyslogd reveals (replaced IP and content of message > by > >>>>>>>>>> XXX): > >>>>>>>>>> > >>>>>>>>>> [pid 14188] socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 2 > >>>>>>>>>> [pid 14188] connect(2, {sa_family=AF_INET, sin_port=htons(5140), > >>>>>>>>>> sin_addr=inet_addr("XXX")}, 16) = 0 > >>>>>>>>>> [pid 14188] recvfrom(2, 0x7f44d42e765f, 1, 66, 0, 0) = -1 EAGAIN > >>>>>>>>>> (Resource temporarily unavailable) > >>>>>>>>>> [pid 14188] sendto(2, "<135>2013-07-26T20:43:08+00:**00 XXX", > >>>>>>>>>> 679, 0, > >>>>>>>>>> NULL, 0) = 679 > >>>>>>>>>> > >>>>>>>>>> And everybody thinks the connection is in CLOSE_WAIT state: > >>>>>>>>>> > >>>>>>>>>> $ sudo lsof | grep rsyslogd | grep TCP > >>>>>>>>>> rsyslogd 14184 syslog 2u IPv4 > 188760 > >>>>>>>>>> 0t0 > >>>>>>>>>> TCP > >>>>>>>>>> ip-10-2-35-151.ec2.internal:**48228->ec2-54-225-181-82.** > >>>>>>>>>> compute-1.amazonaws.com:5140< > http://ec2-54-225-181-82.compute-1.amazonaws.com:5140> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> (CLOSE_WAIT) > >>>>>>>>>> > >>>>>>>>>> $ netstat -a | grep 5140 > >>>>>>>>>> tcp 1 0 XXX:48229 ELB:5140 CLOSE_WAIT > >>>>>>>>>> > >>>>>>>>>> ELB in above is name/IP of Amazon elastic load balancer, it > seems > >>>>>>>>>> that it behaves slightly suspiciously (why does connect > >>>>>>>>>> succeed?, why > >>>>>>>>>> does sendto succeed?) > >>>>>>>>>> > >>>>>>>>>> Any ideas why rsyslogd does not close the connection that is in > >>>>>>>>>> CLOSE_WAIT state? The connection remains in CLOSE_WAIT state > even > >>>>>>>>>> after the remote server starts listening on 5140 port. This does > >>>>>>>>>> not > >>>>>>>>>> happen everytime there is no listener on remote host but when it > >>>>>>>>>> happens it doesn't seem to be fixed until I restart rsyslogd. > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> rsyslog is closing the connection, but then it's opening the > >>>>>>>>> connection > >>>>>>>>> again for the next message. > >>>>>>>>> > >>>>>>>>> My guess is that the ELB is accepting the connection (which is > >>>>>>>>> why the > >>>>>>>>> connect succeeds), and then discovering that there's no way for > >>>>>>>>> it to > >>>>>>>>> send the connection to one of the servers that would handle the > >>>>>>>>> traffic, > >>>>>>>>> so it turns around and closes the connection. > >>>>>>>>> > >>>>>>>>> While it is closing the connection, rsyslog is sending data out > >>>>>>>>> through > >>>>>>>>> that connection, because it thinks it's open. This results in > some > >>>>>>>>> data > >>>>>>>>> being lost. but since rsyslog thinks some data got through (and > it > >>>>>>>>> successfully opened the connection), it doesn't mark that > >>>>>>>>> destination as > >>>>>>>>> being down, and keeps trying to send data there. > >>>>>>>>> > >>>>>>>>> If you were to use RELP, it would properly detect that the > messages > >>>>>>>>> are > >>>>>>>>> not being delivered and no messages would be lost. This is > >>>>>>>>> exactly the > >>>>>>>>> type of thing that triggered the creation of RELP. > >>>>>>>>> > >>>>>>>> > >>>>>>>> it seems that using RELP should work, will try that, > >>>>>>>> > >>>>>>>> investigating this I figured out that netcat does not suffer the > >>>>>>>> same > >>>>>>>> problem for some reason. Not saying there is a problem with > rsyslog > >>>>>>>> (seem load balancer problem), just wondering if anybody has any > >>>>>>>> insights. > >>>>>>>> > >>>>>>>> I created a simple setup of client machine -> ELB (load balancer) > >>>>>>>> -> > >>>>>>>> server machine that works like this: > >>>>>>>> > >>>>>>>> client runs: > >>>>>>>> > >>>>>>>> perl -e '$| = 1 ; for($i=0;$i<100000;$i++) {print localtime() . " > >>>>>>>> [$i]\n"; sleep(1);}' | nc $elb_hostname 5140 > >>>>>>>> > >>>>>>>> server runs: > >>>>>>>> > >>>>>>>> nc -k -l $ip_address 5140 > >>>>>>>> > >>>>>>>> In this setup client nc recognies all errors, strace example (when > >>>>>>>> the > >>>>>>>> server is down): > >>>>>>>> > >>>>>>>> socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 3 > >>>>>>>> fcntl(3, F_GETFL) = 0x2 (flags O_RDWR) > >>>>>>>> fcntl(3, F_SETFL, O_RDWR|O_NONBLOCK) = 0 > >>>>>>>> connect(3, {sa_family=AF_INET, sin_port=htons(5140), > >>>>>>>> sin_addr=inet_addr("10.151.83.**76")}, 16) = -1 EINPROGRESS > >>>>>>>> (Operation > >>>>>>>> now in progress) > >>>>>>>> select(4, NULL, [3], NULL, NULL) = 1 (out [3]) > >>>>>>>> getsockopt(3, SOL_SOCKET, SO_ERROR, [111], [4]) = 0 > >>>>>>>> fcntl(3, F_SETFL, O_RDWR) = 0 > >>>>>>>> close(3) = 0 > >>>>>>>> > >>>>>>>> On the other hand rsyslog is doing this (same scenario, sending > data > >>>>>>>> to load balancer that has nothing on the other side): > >>>>>>>> > >>>>>>>> 15218 socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 6 > >>>>>>>> 15218 connect(6, {sa_family=AF_INET, sin_port=htons(5140), > >>>>>>>> sin_addr=inet_addr("54.225.**181.82")}, 16) = 0 > >>>>>>>> 15218 recvfrom(6, 0x7f009427065f, 1, 66, 0, 0) = -1 EAGAIN > (Resource > >>>>>>>> temporarily unavailable) > >>>>>>>> 15218 sendto(6, "<135>2013-08-10T00:11:10+00:**00 ...data > >>>>>>>> removed...\n", > >>>>>>>> 679, 0, NULL, 0) = 679 > >>>>>>>> 15218 gettimeofday({1376093470, 69447}, NULL) = 0 > >>>>>>>> 15218 gettimeofday({1376093470, 69515}, NULL) = 0 > >>>>>>>> 15218 futex(0xd80b94, FUTEX_WAIT_PRIVATE, 497273, NULL <unfinished > >>>>>>>> ...> > >>>>>>>> 15217 <... epoll_wait resumed> {{EPOLLIN, {u32=14173920, > >>>>>>>> u64=14173920}}}, 10, 4294967295) = 1 > >>>>>>>> 15217 recvfrom(4, "<135>Aug 10 00:11:11 ...data removed... \n", > >>>>>>>> 8096, > >>>>>>>> 0, {sa_family=AF_INET, sin_port=htons(57437), > >>>>>>>> sin_addr=inet_addr("127.0.0.1"**)}, [16]) = 399 > >>>>>>>> 15217 gettimeofday({1376093471, 560833}, NULL) = 0 > >>>>>>>> 15217 recvfrom(4, 0xd9e370, 8096, 0, 0x7f0094a714b0, > >>>>>>>> 0x7f0094a6f364) = > >>>>>>>> -1 EAGAIN (Resource temporarily unavailable) > >>>>>>>> 15217 futex(0xd80b94, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xd80b90, > >>>>>>>> {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1} <unfinished ...> > >>>>>>>> 15218 <... futex resumed> ) = 0 > >>>>>>>> 15217 <... futex resumed> ) = 1 > >>>>>>>> 15218 futex(0xd80dc0, FUTEX_WAKE_PRIVATE, 1 <unfinished ...> > >>>>>>>> 15217 epoll_wait(2, <unfinished ...> > >>>>>>>> 15218 <... futex resumed> ) = 0 > >>>>>>>> 15218 gettimeofday({1376093471, 561248}, NULL) = 0 > >>>>>>>> 15218 gettimeofday({1376093471, 561367}, NULL) = 0 > >>>>>>>> 15218 recvfrom(6, "", 1, MSG_PEEK|MSG_DONTWAIT, NULL, NULL) = 0 > >>>>>>>> 15218 close(6) = 0 > >>>>>>>> > >>>>>>>> This repeats so efectively rsyslog keeps sending messages (sendto > >>>>>>>> returns success), closing connection, opening connection etc. As > far > >>>>>>>> as I can tell load balancer should not successfully connect and > even > >>>>>>>> receive data... > >>>>>>>> > >>>>>>>> Any comments/insights? > >>>>>>>> > >>>>>>>> erik > >>>>>>>> ______________________________**_________________ > >>>>>>>> rsyslog mailing list > >>>>>>>> http://lists.adiscon.net/**mailman/listinfo/rsyslog< > http://lists.adiscon.net/mailman/listinfo/rsyslog> > >>>>>>>> http://www.rsyslog.com/**professional-services/< > http://www.rsyslog.com/professional-services/> > >>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >>>>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT > >>>>>>>> POST > >>>>>>>> if you DON'T LIKE THAT. > >>>>>>>> > >>>>>>>> ______________________________**_________________ > >>>>>>> rsyslog mailing list > >>>>>>> http://lists.adiscon.net/**mailman/listinfo/rsyslog< > http://lists.adiscon.net/mailman/listinfo/rsyslog> > >>>>>>> http://www.rsyslog.com/**professional-services/< > http://www.rsyslog.com/professional-services/> > >>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >>>>>>> myriad > >>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > >>>>>>> you > >>>>>>> DON'T LIKE THAT. > >>>>>>> > >>>>>> > >>>>>> ______________________________**_________________ > >>>>>> rsyslog mailing list > >>>>>> http://lists.adiscon.net/**mailman/listinfo/rsyslog< > http://lists.adiscon.net/mailman/listinfo/rsyslog> > >>>>>> http://www.rsyslog.com/**professional-services/< > http://www.rsyslog.com/professional-services/> > >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >>>>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT > POST > >>>>>> if you DON'T LIKE THAT. > >>>>>> > >>>>>> ______________________________**_________________ > >>>>> rsyslog mailing list > >>>>> http://lists.adiscon.net/**mailman/listinfo/rsyslog< > http://lists.adiscon.net/mailman/listinfo/rsyslog> > >>>>> http://www.rsyslog.com/**professional-services/< > http://www.rsyslog.com/professional-services/> > >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >>>>> myriad > >>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you > >>>>> DON'T LIKE THAT. > >>>>> > >>>> > >>>> ______________________________**_________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/**mailman/listinfo/rsyslog< > http://lists.adiscon.net/mailman/listinfo/rsyslog> > >>>> http://www.rsyslog.com/**professional-services/< > http://www.rsyslog.com/professional-services/> > >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST > >>>> if you DON'T LIKE THAT. > >>>> > >>>> ______________________________**_________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/**mailman/listinfo/rsyslog< > http://lists.adiscon.net/mailman/listinfo/rsyslog> > >>> http://www.rsyslog.com/**professional-services/< > http://www.rsyslog.com/professional-services/> > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >>> DON'T LIKE THAT. > >>> > >> > >> ______________________________**_________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/**mailman/listinfo/rsyslog< > http://lists.adiscon.net/mailman/listinfo/rsyslog> > >> http://www.rsyslog.com/**professional-services/< > http://www.rsyslog.com/professional-services/> > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >> DON'T LIKE THAT. > >> > >> ______________________________**_________________ > > rsyslog mailing list > > http://lists.adiscon.net/**mailman/listinfo/rsyslog< > http://lists.adiscon.net/mailman/listinfo/rsyslog> > > http://www.rsyslog.com/**professional-services/< > http://www.rsyslog.com/professional-services/> > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > DON'T LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
