Works like a champ... I'm running 2 instances of listeners, one running imptcp with stream compression on 1514, and another without compression on 514. Does imptcp accept uncompressed messages as well? Or is it the right thing to run 2 different listeners?
One last question. Let's say i'm bound by network traffic, say 30mb/s forwarding 'upstream'. If i add compression and some threads, is it possible to push more EPS through the same network bottleneck using compression? I've got around a 40k EPS ceiling that I think i'm hitting. For pure forwarding throughput (EPS wise) what is the most performant way to send data from one rsyslog server to another (assuming there is plenty of memory and CPU on each)? Thanks again for all the help. Cheers, JB On Tue, Mar 10, 2015 at 3:59 AM, Rainer Gerhards <[email protected]> wrote: > 2015-03-10 3:37 GMT+01:00 David Lang <[email protected]>: > > > On Mon, 9 Mar 2015, Joe Blow wrote: > > > > Hey folks, > >> > >> I'm using 2 rsyslog servers and am trying to get this stream compression > >> working, as it sounds quite magical (1/20th of the data sounds > fantastic). > >> Sending box: > >> > >> [root@sender ~]# rsyslogd -v > >> rsyslogd 8.8.0, compiled with: > >> PLATFORM: x86_64-redhat-linux-gnu > >> PLATFORM (lsb_release -d): > >> FEATURE_REGEXP: Yes > >> GSSAPI Kerberos 5 support: No > >> FEATURE_DEBUG (debug build, slow code): No > >> 32bit Atomic operations supported: Yes > >> 64bit Atomic operations supported: Yes > >> memory allocator: system default > >> Runtime Instrumentation (slow code): No > >> uuid support: Yes > >> Number of Bits in RainerScript integers: 64 > >> > >> [root@receiver ~]# rsyslogd -v > >> rsyslogd 8.8.0.ad1, compiled with: > >> PLATFORM: x86_64-redhat-linux-gnu > >> PLATFORM (lsb_release -d): > >> FEATURE_REGEXP: Yes > >> GSSAPI Kerberos 5 support: No > >> FEATURE_DEBUG (debug build, slow code): No > >> 32bit Atomic operations supported: Yes > >> 64bit Atomic operations supported: Yes > >> memory allocator: system default > >> Runtime Instrumentation (slow code): No > >> uuid support: Yes > >> Number of Bits in RainerScript integers: 64 > >> > >> Sending setup: > >> $template RAW, "%rawmsg:1:20480%\n" > >> if $rawmsg contains "myHeaderString|" then {action( > >> Type="omfwd" > >> Target="127.0.0.1" > >> Port="1514" > >> Protocol="tcp" > >> Template="RAW" > >> RebindInterval="250" > >> compression.mode="stream:always" > >> compression.stream.flushOnTXEnd="off" > >> queue.dequeuebatchsize="10000" > >> queue.type="fixedarray" > >> queue.filename="output.rsq" > >> queue.highwatermark="120000000" > >> queue.lowwatermark="15000000" > >> queue.discardmark="150000000" > >> queue.maxdiskspace="50g" > >> queue.size="150000000" > >> queue.saveonshutdown="on" > >> action.resumeretrycount="-1")stop} > >> > >> Receiving setup: > >> > >> $ModLoad imptcp > >> input(type="imptcp" port="514" address="0.0.0.0" Threads="16") > >> > > > > start with much lower thread counts, only increase past 1 if you find you > > need to (too many threads will actually slow you down as the batch sized > > approach 1 there is a lot more contention for the locks on the main > queue. > > > > also, you are sending to port 1514 and listening to port 514, is this a > > copy/paste error? > > > > > >> When i turn this on, i get garbage spewing to all consoles on both the > >> receiving and sending system. > >> > > > > the sending system should not be sending anything out to the console, > what > > else is in your config > > > > Am i missing any gotchas with this? Is > >> stream compression supported/tested? Would it make any difference that > >> i'm > >> shoving this over an SSH tunnel? > >> > > > > it shouldn't. > > > > I would start by simplifying the config to the minimum (eliminate queues, > > leave threads at the default > > > > Anyone run into this before? Could these 2 different 8.8.0 versions > >> having > >> compatibility issues? > >> > > > > the version you compiled yourself, was it from the v8.8.0 tag or a later > > checkout from git that didn't have that tag changed yet? > > > The .adX versions are Adiscon interim builds. When the project switched to > the 6-weekly release cycle, Adiscon got some trouble because support > customers often require new features or a bugfix in package form more > timely. In this case, a special .adX version is released. They are very > close to the "regular" version. There are not .adX tags in the public git > repos because these versions are specially patched. > > I guess I didn't give that information before, sorry for that. > > Rainer > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
