Thanks for the quick response Dave! There is a lot more within the config(s) than just those lines. The garbage that is getting spat to console is probably because i have a lot of 'stock' rsyslog lines at the bottom of my config. Strangely enough it only happens when i enable stream compression, otherwise i've got probably a 150 line rsyslog file on the sender and 350 lines on the receiver.
I'll have to take this off onto 2 boxes that aren't doing anything at all to get this working i'm guessing? I've got around 40k EPS going through the boxes without a problem so there isn't much sense in messing with queues that are working as expected. I guess i was just hoping that if i was running extremely similar versions of rsyslog i could just enable this wonderful sounding stream compression. In a case like mine, with very large amounts of data, the 2 line stream compression sounded like a fantastic idea idea. Are the configs i posted all i *should* need to get both the sender and the receiver working? In theory, i'd really like to run a hybrid queue with compression to my other syslog boxes. The isolated box route i think will be my next step. Cheers, JB On Mon, Mar 9, 2015 at 10:37 PM, David Lang <[email protected]> wrote: > On Mon, 9 Mar 2015, Joe Blow wrote: > > Hey folks, >> >> I'm using 2 rsyslog servers and am trying to get this stream compression >> working, as it sounds quite magical (1/20th of the data sounds fantastic). >> Sending box: >> >> [root@sender ~]# rsyslogd -v >> rsyslogd 8.8.0, compiled with: >> PLATFORM: x86_64-redhat-linux-gnu >> PLATFORM (lsb_release -d): >> FEATURE_REGEXP: Yes >> GSSAPI Kerberos 5 support: No >> FEATURE_DEBUG (debug build, slow code): No >> 32bit Atomic operations supported: Yes >> 64bit Atomic operations supported: Yes >> memory allocator: system default >> Runtime Instrumentation (slow code): No >> uuid support: Yes >> Number of Bits in RainerScript integers: 64 >> >> [root@receiver ~]# rsyslogd -v >> rsyslogd 8.8.0.ad1, compiled with: >> PLATFORM: x86_64-redhat-linux-gnu >> PLATFORM (lsb_release -d): >> FEATURE_REGEXP: Yes >> GSSAPI Kerberos 5 support: No >> FEATURE_DEBUG (debug build, slow code): No >> 32bit Atomic operations supported: Yes >> 64bit Atomic operations supported: Yes >> memory allocator: system default >> Runtime Instrumentation (slow code): No >> uuid support: Yes >> Number of Bits in RainerScript integers: 64 >> >> Sending setup: >> $template RAW, "%rawmsg:1:20480%\n" >> if $rawmsg contains "myHeaderString|" then {action( >> Type="omfwd" >> Target="127.0.0.1" >> Port="1514" >> Protocol="tcp" >> Template="RAW" >> RebindInterval="250" >> compression.mode="stream:always" >> compression.stream.flushOnTXEnd="off" >> queue.dequeuebatchsize="10000" >> queue.type="fixedarray" >> queue.filename="output.rsq" >> queue.highwatermark="120000000" >> queue.lowwatermark="15000000" >> queue.discardmark="150000000" >> queue.maxdiskspace="50g" >> queue.size="150000000" >> queue.saveonshutdown="on" >> action.resumeretrycount="-1")stop} >> >> Receiving setup: >> >> $ModLoad imptcp >> input(type="imptcp" port="514" address="0.0.0.0" Threads="16") >> > > start with much lower thread counts, only increase past 1 if you find you > need to (too many threads will actually slow you down as the batch sized > approach 1 there is a lot more contention for the locks on the main queue. > > also, you are sending to port 1514 and listening to port 514, is this a > copy/paste error? > > >> When i turn this on, i get garbage spewing to all consoles on both the >> receiving and sending system. >> > > the sending system should not be sending anything out to the console, what > else is in your config > > Am i missing any gotchas with this? Is >> stream compression supported/tested? Would it make any difference that >> i'm >> shoving this over an SSH tunnel? >> > > it shouldn't. > > I would start by simplifying the config to the minimum (eliminate queues, > leave threads at the default > > Anyone run into this before? Could these 2 different 8.8.0 versions >> having >> compatibility issues? >> > > the version you compiled yourself, was it from the v8.8.0 tag or a later > checkout from git that didn't have that tag changed yet? > > I'm about to throw the sending system into debug >> mode to see if i can figure this out. >> > > that would be my next step after simplifying the config. > > David Lang > > Cheers, >> >> JB >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> >> _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
