Hi Janmejay, thanks for being so persistent with this. I am currently working on libfastjson and hope to have a much different version available shortly. In theory, it could give quite a performance boost, but this is not totally clear, a lot is really experimental. Would you be willing to give the new code a try when it is ready? That would definitely help me get quicker feedback and thus a quicker path towards a final solution.
The main thing of concern is this: https://github.com/rsyslog/libfastjson/issues/35 Rainer 2016-04-12 16:39 GMT+02:00 singh.janmejay <[email protected]>: > I cooked up this little systemtap script to track cause of off-cpu > latency (with latency quantification). It identifies latencies by > userspace-backtrace, kernel-backtrace and thread-id. > > It'll give us very useful data when you see low-TP with > processing-backlog while CPU runs cold. > > Run it as $ sudo stap -v <script> -x $(pgrep rsyslogd) > > Let it run for several seconds (say ~ 2 minutes) and interrupt it after that. > > It requires libc(and other libraries being used by Rsyslog) to be > built with frame-pointers (-fno-omit-frame-pointer) and will require > debug-symbols. > > Here is the script: > ===========================START > > global u_traces, k_traces, off_start, latencies, ubt_cache, kbt_cache > > probe syscall.* { > if (task_current()->tgid == target()) { > ubt_cache[task_current()->pid] = ubacktrace(); > } > } > > probe kernel.trace("sched_switch") { > if ($prev->tgid == target()) { > off_start[$prev->pid] = gettimeofday_us(); > kbt_cache[$prev->pid] = backtrace(); > } else if ($next->tgid == target()) { > start = off_start[$next->pid]; > if (start > 0) { > lat = gettimeofday_us() - start; > latencies[$next->pid] += lat; > if (ubt_cache[$next->pid] != "") { > u_traces[ubt_cache[$next->pid]] += lat; > } > cached_kbt = kbt_cache[$next->pid] > if (cached_kbt != "") { > k_traces[cached_kbt] += lat; > } > } > } > } > > probe end { > printf("\nU-Traces(with micro-sec off-cpu time, remember this is not > accurate in case when you see high-enough of non-voluntary > ctx-sw):\n") > foreach (stack in u_traces) { > printf("%d:\n", u_traces[stack]) > print_usyms(stack); > } > printf("\nK-Traces(with micro-sec off-cpu time):\n") > foreach (stack in k_traces) { > printf("%d:\n", k_traces[stack]) > print_syms(stack); > } > printf("\nThread-wise micro-sec off-cpu time:\n") > foreach (l in latencies) { > printf("tid=%d : %d\n", l, latencies[l]) > } > } > > ============================END > > This should help you triangulate data with sched_switch event: > > sudo perf stat -e sched:sched_switch -a -g -p $(pgrep rsyslogd) sleep 120 > > Here is the comparison of cpu-profile of my branch with (left side) > and without (right side) core and edge relay stats gathering. Both > runs took ~24 - 26 seconds. > > https://drive.google.com/open?id=0B_XhUZLNFT4dWWl4R2ZrU0w2aFE > > Here is it with my old patched branch on the left side and master on right: > > https://drive.google.com/open?id=0B_XhUZLNFT4dOHo2QXdGYjFGVTQ > > It seems master is utilizing CPU noticeably better. Again, because > this is an EC2 machine, I don't have control over hardware and other > tenants, so I can't be sure (I was seeing high io-waits, the pattern > was different from yesterday). But all in all, master seems to be > working better (better CPU util, lower batch-posting time etc). I had > a run with master rsyslog again just to check overall time to handle > -C 100 -I <good json messages> flood and it still reports master > taking ~ 20s and my branch taking ~26s. > > Reduced CPU on processDataRcvd clearly indicates some back-pressure. > fjson_tokener_parse_ex seems to be making it much better than old > json_tokener_parse_ex. > > I guess I am happy with what I see both from the PoV of moving to > master (ditching my branch without fear of perf issues, 300k to 200k > jump you talked about) and from dynstats + rscript handling variables > that need higher number of de-references. We can only go so far on a > synthetic benchmark. > > Please do let us know what stap tells you, it'll be interesting input > while working on future perf related tasks. > > On Tue, Apr 12, 2016 at 2:30 AM, David Lang <[email protected]> wrote: >> At this point I have things working. I found that in some cases I had logic >> errors where the variable being used in the update did not exist, or was >> otherwise not a valid json tag (spaces, etc) >> >> David Lang >> >> On Mon, 11 Apr 2016, singh.janmejay wrote: >> >>> Date: Mon, 11 Apr 2016 23:04:48 +0530 >>> >>> From: singh.janmejay <[email protected]> >>> Reply-To: rsyslog-users <[email protected]> >>> To: rsyslog-users <[email protected]> >>> Subject: Re: [rsyslog] dynastats problems with user defined variables? >>> >>> David, >>> >>> Did a few runs with those lines present / commented out with >>> rsyslog-master. I see no difference in either on-cpu or off-cpu >>> performance. The left-hand-side is with edge-relay and core-relay >>> lines being active, and the right-side has them commented out. Here is >>> the cpu profile: >>> https://drive.google.com/open?id=0B_XhUZLNFT4dWDFOZlB6VW1hc0E >>> The hypercall on top is cpu-idle call. This was on a c1.4large box on ec2. >>> >>> Both were on the same version of Rsyslog (built off master). I'll do >>> another round of tests with the version I run on production (which is >>> patched fork of 8.12, afair) to identify any large change in cpu >>> profile. >>> >>> This is a much more simplified version of config though. I extracted >>> valid (non garbled) json messages from debug log(got 9089 lines) and >>> pumped them in 100 times using tcpflood. >>> >>> tcpflood call was timed, it took between 19.7 - 20.5 seconds for each >>> run regardless of edge and core relay counters being incremented. It >>> was ~ 45k / sec (but again, neither the load profile, not the machine >>> is not exactly like yours), so its not apples to apples. >>> >>> >>> On Fri, Apr 8, 2016 at 12:15 PM, singh.janmejay >>> <[email protected]> wrote: >>>> >>>> It doesn't treat them any different. It takes the key and puts it in a >>>> hash-table and set an accumulator as its value. Then it'll look it up >>>> every time you ask it to increment the counter followed by actual >>>> increment. That is about it. >>>> >>>> The only way I can foresee the contents affecting its behavior is the >>>> hash-fn takes more CPU time when given a large string. But that should >>>> be on-cpu and not off-cpu time. >>>> >>>> Today im working on a getting an Rsyslog build running with config >>>> into which I inject messages from your debug-log (i'll extract >>>> messages and tcpflood them in) and try to understand why it behaves >>>> different between those lines commented out or not. >>>> >>>> If I fail to reproduce it in this simple setup, we'll have to track >>>> backtraces that are taking Rsyslog off-cpu in your environment. >>>> >>>> On Fri, Mar 25, 2016 at 12:57 AM, David Lang <[email protected]> wrote: >>>>> >>>>> doing a little more digging (and some accidental stuff), is it possible >>>>> that >>>>> it's running into grief if the contents of the variable are not a simple >>>>> word (spaces or other funny characters in the value)? >>>>> >>>>> David Lang >>>>> >>>>> On Wed, 23 Mar 2016, singh.janmejay wrote: >>>>> >>>>>> Date: Wed, 23 Mar 2016 01:07:49 +0530 >>>>>> >>>>>> From: singh.janmejay <[email protected]> >>>>>> Reply-To: rsyslog-users <[email protected]> >>>>>> To: rsyslog-users <[email protected]> >>>>>> Subject: Re: [rsyslog] dynastats problems with user defined variables? >>>>>> >>>>>> Yep, it looked like that. Its interesting though, its 100% off CPU >>>>>> (very unique). >>>>>> >>>>>> On Wed, Mar 23, 2016 at 1:02 AM, David Lang <[email protected]> wrote: >>>>>>> >>>>>>> >>>>>>> Yes, the first one is with the user variables commented out, exactly >>>>>>> what >>>>>>> I >>>>>>> pasted in from the grep. >>>>>>> >>>>>>> the second is enabling the dyn_inc for edge relays. >>>>>>> >>>>>>> David Lang >>>>>>> >>>>>>> On Wed, 23 Mar 2016, singh.janmejay wrote: >>>>>>> >>>>>>>> Date: Wed, 23 Mar 2016 00:51:18 +0530 >>>>>>>> From: singh.janmejay <[email protected]> >>>>>>>> Reply-To: rsyslog-users <[email protected]> >>>>>>>> To: rsyslog-users <[email protected]> >>>>>>>> Subject: Re: [rsyslog] dynastats problems with user defined >>>>>>>> variables? >>>>>>>> >>>>>>>> You meant the first one was with "commented out" version right? (you >>>>>>>> said uncommenting edge_relay, so double-checking). >>>>>>>> >>>>>>>> On Wed, Mar 23, 2016 at 12:38 AM, David Lang <[email protected]> wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> here is vmstat 1 running >>>>>>>>> >>>>>>>>> # vmstat 1 >>>>>>>>> procs -----------memory---------- ---swap-- -----io---- -system-- >>>>>>>>> ------cpu----- >>>>>>>>> r b swpd free buff cache si so bi bo in cs us >>>>>>>>> sy >>>>>>>>> id >>>>>>>>> wa st >>>>>>>>> 2 0 62280 2363804 584 27280376 0 1 21 596 0 1 >>>>>>>>> 46 >>>>>>>>> 3 >>>>>>>>> 51 0 0 >>>>>>>>> 1 0 62280 2349476 584 27280612 0 0 0 87042 979 891 >>>>>>>>> 4 >>>>>>>>> 0 >>>>>>>>> 95 0 0 >>>>>>>>> 1 0 62280 2330488 584 27280908 0 0 0 0 388 258 >>>>>>>>> 4 >>>>>>>>> 0 >>>>>>>>> 96 0 0 >>>>>>>>> 1 0 62280 2314744 584 27281148 0 0 0 0 396 255 >>>>>>>>> 4 >>>>>>>>> 0 >>>>>>>>> 96 0 0 >>>>>>>>> 1 0 62280 2297736 584 27281388 0 0 0 0 376 245 >>>>>>>>> 4 >>>>>>>>> 0 >>>>>>>>> 96 0 0 >>>>>>>>> 2 0 62280 2546260 584 27245272 0 0 0 4 639 568 >>>>>>>>> 6 >>>>>>>>> 1 >>>>>>>>> 94 0 0 >>>>>>>>> 2 0 62280 2464300 584 27245928 0 0 0 1716 936 1146 >>>>>>>>> 8 >>>>>>>>> 0 >>>>>>>>> 91 0 0 >>>>>>>>> 2 0 62280 2394452 584 27246444 0 0 0 8 687 333 >>>>>>>>> 8 >>>>>>>>> 0 >>>>>>>>> 92 0 0 >>>>>>>>> >>>>>>>>> >>>>>>>>> here is vmstat 1 uncommenting edge_relay >>>>>>>>> >>>>>>>>> # vmstat 1 >>>>>>>>> procs -----------memory---------- ---swap-- -----io---- -system-- >>>>>>>>> ------cpu----- >>>>>>>>> r b swpd free buff cache si so bi bo in cs us >>>>>>>>> sy >>>>>>>>> id >>>>>>>>> wa st >>>>>>>>> 1 0 62280 3198672 584 26770624 0 1 21 596 0 1 >>>>>>>>> 46 >>>>>>>>> 3 >>>>>>>>> 51 0 0 >>>>>>>>> 0 0 62280 3198904 584 26770664 0 0 0 0 151 247 >>>>>>>>> 0 >>>>>>>>> 0 >>>>>>>>> 100 0 0 >>>>>>>>> 0 0 62280 3200332 584 26770664 0 0 0 0 128 233 >>>>>>>>> 0 >>>>>>>>> 0 >>>>>>>>> 100 0 0 >>>>>>>>> 0 0 62280 3202244 584 26770664 0 0 0 4 172 315 >>>>>>>>> 0 >>>>>>>>> 0 >>>>>>>>> 100 0 0 >>>>>>>>> 0 0 62280 3203132 584 26770664 0 0 0 0 132 218 >>>>>>>>> 0 >>>>>>>>> 0 >>>>>>>>> 100 0 0 >>>>>>>>> 0 0 62280 3203912 584 26770668 0 0 0 8 203 347 >>>>>>>>> 0 >>>>>>>>> 0 >>>>>>>>> 100 0 0 >>>>>>>>> >>>>>>>>> >>>>>>>>> things just are not running with it uncommented. >>>>>>>>> >>>>>>>>> I'm going to send you my config directly, it is a very heavy config. >>>>>>>>> >>>>>>>>> David Lang >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, 23 Mar 2016, singh.janmejay wrote: >>>>>>>>> >>>>>>>>>> On my cluster, each node (those 66 nodes) have 4 threads for >>>>>>>>>> ruleset >>>>>>>>>> (it does everything, lognorm based parsing, lookups using loopup >>>>>>>>>> table, re_extract, string concat, several if-else based conditions >>>>>>>>>> and >>>>>>>>>> queues up for egress using omkafka and I generally see 3 being >>>>>>>>>> utilized). This setup runs at ~37k msg/s (2.5M / 66 = 37k). >>>>>>>>>> >>>>>>>>>> 300k/Min is still 5k/s which should be doable on a single-thd >>>>>>>>>> (unless >>>>>>>>>> your ruleset is much heavier than mine). Im thinking for a >>>>>>>>>> equivalently complex ruleset 37k / 4 = 9k should be achievable. >>>>>>>>>> >>>>>>>>>> I'll try to reproduce this. Can you share out the exact commit-id >>>>>>>>>> you >>>>>>>>>> are working with? >>>>>>>>>> >>>>>>>>>> On Tue, Mar 22, 2016 at 8:38 PM, David Lang <[email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Tue, 22 Mar 2016, singh.janmejay wrote: >>>>>>>>>>> >>>>>>>>>>>> On Tue, Mar 22, 2016 at 5:01 AM, David Lang <[email protected]> >>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> I added this to my configs >>>>>>>>>>>>> >>>>>>>>>>>>> # grep -B 1 -A 1 dyn_ /etc/rsyslog.conf >>>>>>>>>>>>> # custom stats to track in rsyslog >>>>>>>>>>>>> dyn_stats(name="msgs_per_host" resettable="on" >>>>>>>>>>>>> maxCardinality="3000" >>>>>>>>>>>>> unusedMetricLife="1200") >>>>>>>>>>>>> dyn_stats(name="msgs_per_edge_relay" resettable="on" >>>>>>>>>>>>> maxCardinality="3000" >>>>>>>>>>>>> unusedMetricLife="1200") >>>>>>>>>>>>> dyn_stats(name="msgs_per_core_relay" resettable="on" >>>>>>>>>>>>> maxCardinality="3000" >>>>>>>>>>>>> unusedMetricLife="1200") >>>>>>>>>>>>> dyn_stats(name="msgs_per_program" resettable="on" >>>>>>>>>>>>> maxCardinality="3000" >>>>>>>>>>>>> unusedMetricLife="1200") >>>>>>>>>>>>> dyn_stats(name="msgs_per_tag" resettable="on" >>>>>>>>>>>>> maxCardinality="3000" >>>>>>>>>>>>> unusedMetricLife="1200") >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> /var/log/sources-messages;sources >>>>>>>>>>>>> set $.inc = dyn_inc("msgs_per_host", $hostname); >>>>>>>>>>>>> set $.inc = dyn_inc("msgs_per_program", $programname); >>>>>>>>>>>>> # if $!trusted!edge!relay != "" then { >>>>>>>>>>>>> # set $.inc = dyn_inc("msgs_per_edge_relay", >>>>>>>>>>>>> $!trusted!edge!relay); >>>>>>>>>>>>> # } >>>>>>>>>>>>> # if $!trusted!core!relay != "" then { >>>>>>>>>>>>> # set $.inc = dyn_inc("msgs_per_edge_relay", >>>>>>>>>>>>> $!trusted!core!relay); >>>>>>>>>>>>> # } >>>>>>>>>>>>> -- >>>>>>>>>>>>> } >>>>>>>>>>>>> #set $.inc = dyn_inc("msgs_per_tag", $.custommessage); >>>>>>>>>>>>> /var/log/messages-tags;manual >>>>>>>>>>>>> >>>>>>>>>>>>> if I uncomment any of the lines that refer to $! or $. >>>>>>>>>>>>> variables, >>>>>>>>>>>>> rsyslog >>>>>>>>>>>>> basically stops processing messages (a few messages seem to be >>>>>>>>>>>>> processed >>>>>>>>>>>>> in >>>>>>>>>>>>> spurts, bit a lot of processing never happens) >>>>>>>>>>>>> >>>>>>>>>>>>> any chance that this has problems with locking around user >>>>>>>>>>>>> defined >>>>>>>>>>>>> variables? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Not sure, I run it in production (66 6-cpu(HT) haswell VMs >>>>>>>>>>>> handling >>>>>>>>>>>> an >>>>>>>>>>>> aggregate of ~2.5 M-msg / sec (~1.5kB each)) and haven't seen >>>>>>>>>>>> such >>>>>>>>>>>> jitters, but then I don't use 3 level deep keys either and in my >>>>>>>>>>>> case, >>>>>>>>>>>> the feature is backported to 8.12 branch), so still uses json-c >>>>>>>>>>>> (not >>>>>>>>>>>> libfastjson). I do use it with user-defined variables too, just >>>>>>>>>>>> not >>>>>>>>>>>> nested as deep. >>>>>>>>>>>> >>>>>>>>>>>> We don't treat variables differently after dereferencing them in >>>>>>>>>>>> the >>>>>>>>>>>> top-most layer handling rainerscript fn-calls, so $hostname and >>>>>>>>>>>> $!x!y!z, as far as any rainerscript fn invocation is considered, >>>>>>>>>>>> are >>>>>>>>>>>> the same (performance-wise) except for the top-level dereference >>>>>>>>>>>> (outside of fn-impl-body). >>>>>>>>>>>> >>>>>>>>>>>> Some more info may come handy in reproducing it: >>>>>>>>>>>> - What is the message throughput (in msg/s) and what is the avg >>>>>>>>>>>> msg >>>>>>>>>>>> size? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> with these commented out as above, ~50k messages/min normal, ~200K >>>>>>>>>>> message/min if I'm replaying queued logs (box saturated, see the >>>>>>>>>>> other >>>>>>>>>>> thread where 8.17 final seems to have slowed down from ~300K >>>>>>>>>>> messages/min >>>>>>>>>>> that I was getting with a 8.16+ git) >>>>>>>>>>> >>>>>>>>>>>> - Do you see messages being written to sources-messages file, >>>>>>>>>>>> while >>>>>>>>>>>> they don't get written to messages-tags file? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Yes, but not all messages. I have an alarm that goes off if >>>>>>>>>>> sources-messages >>>>>>>>>>> isn't written to in two 35 second checks (70 seconds total), that >>>>>>>>>>> alarm >>>>>>>>>>> never fired. But there would be a burst of messages show up in >>>>>>>>>>> sources-messages and then 20 or so seconds before another burst >>>>>>>>>>> would >>>>>>>>>>> show >>>>>>>>>>> up. >>>>>>>>>>> >>>>>>>>>>>> - Do you see a significant jump in voluntary context switches(say >>>>>>>>>>>> per >>>>>>>>>>>> 10 seconds or even per second) when you uncomment those lines? >>>>>>>>>>>> How >>>>>>>>>>>> big >>>>>>>>>>>> is the jump? >>>>>>>>>>>> - How does the observation change if you replace it with this >>>>>>>>>>>> kind >>>>>>>>>>>> of >>>>>>>>>>>> variable access pattern: >>>>>>>>>>>> >>>>>>>>>>>> set $.x = $!trusted!edge!relay; >>>>>>>>>>>> if $.x != "" then { >>>>>>>>>>>> set $.inc = dyn_inc("msgs_per_edge_relay", $.x); >>>>>>>>>>>> } >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> will check. >>>>>>>>>>> >>>>>>>>>>>> - Is this stock 8.17 build? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> stock config, but against git as of yesterday. I needed to get >>>>>>>>>>> liblognorm >>>>>>>>>>> 2.0 and the current libjsonfast >>>>>>>>>>> >>>>>>>>>>>> - Is this x86-64 Linux? What distro (and version)? >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> ubuntu trusty (14.04) 64 bit. >>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Also, the documentation doesn't say why we are setting $.inc to >>>>>>>>>>>>> the >>>>>>>>>>>>> result >>>>>>>>>>>>> of the command, what does $.inc contain after the command is >>>>>>>>>>>>> run? >>>>>>>>>>>>> from >>>>>>>>>>>>> the >>>>>>>>>>>>> documentation, it looks like it contains an error code (at least >>>>>>>>>>>>> the >>>>>>>>>>>>> way >>>>>>>>>>>>> it's used is similar to error code returns in other languages), >>>>>>>>>>>>> but >>>>>>>>>>>>> I >>>>>>>>>>>>> don't >>>>>>>>>>>>> see what the errors are in the doc page. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Yes, it is the error-code. It has value 0 when successful, an >>>>>>>>>>>> non-zero >>>>>>>>>>>> when it fails to increment (it may fail due to maxCardinality >>>>>>>>>>>> being >>>>>>>>>>>> hit or due to a contended for lock etc (it doesn't wait for the >>>>>>>>>>>> lock, >>>>>>>>>>>> it uses try-lock and doesn't increment the counter if it can't >>>>>>>>>>>> get a >>>>>>>>>>>> shared-lock over the bucket). >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> While I do have multiple threads running on the main queue, the >>>>>>>>>>> places >>>>>>>>>>> I >>>>>>>>>>> am >>>>>>>>>>> calling this are on action queues with only one worker, so the >>>>>>>>>>> only >>>>>>>>>>> lock >>>>>>>>>>> conflict should be when the data is reported. >>>>>>>>>>> >>>>>>>>>>>> These are rsyslog error-codes. It does an error pass-thru. >>>>>>>>>>>> >>>>>>>>>>>> I'll add this to documentation (didn't know this was missed). >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Thanks. >>>>>>>>>>> >>>>>>>>>>> Sorry I didn't get a chance to run this pre-release, this is >>>>>>>>>>> exactly >>>>>>>>>>> why >>>>>>>>>>> I >>>>>>>>>>> try to use features like this pre-release to spot the things that >>>>>>>>>>> the >>>>>>>>>>> person >>>>>>>>>>> developign them misses because they are too close to the problem. >>>>>>>>>>> :-) >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> David Lang >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> rsyslog mailing list >>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>>>>>>> myriad >>>>>>>>>>> of >>>>>>>>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>>>>>>>>>> you >>>>>>>>>>> DON'T >>>>>>>>>>> LIKE THAT. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> rsyslog mailing list >>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>>>>> myriad >>>>>>>>> of >>>>>>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>>>>>> DON'T >>>>>>>>> LIKE THAT. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> _______________________________________________ >>>>>>> rsyslog mailing list >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>>> http://www.rsyslog.com/professional-services/ >>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>>> myriad >>>>>>> of >>>>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>>>> DON'T >>>>>>> LIKE THAT. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com/professional-services/ >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>>>> of >>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>> DON'T >>>>> LIKE THAT. >>>> >>>> >>>> >>>> >>>> -- >>>> Regards, >>>> Janmejay >>>> http://codehunk.wordpress.com >>> >>> >>> >>> >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >> LIKE THAT. > > > > -- > Regards, > Janmejay > http://codehunk.wordpress.com > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T > LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
