I meant to put David, not Dave. Cheers,
JB On Mon, Mar 9, 2015 at 11:30 PM, Joe Blow <[email protected]> wrote: > Thanks for the quick response Dave! > > There is a lot more within the config(s) than just those lines. The > garbage that is getting spat to console is probably because i have a lot of > 'stock' rsyslog lines at the bottom of my config. Strangely enough it only > happens when i enable stream compression, otherwise i've got probably a 150 > line rsyslog file on the sender and 350 lines on the receiver. > > I'll have to take this off onto 2 boxes that aren't doing anything at all > to get this working i'm guessing? I've got around 40k EPS going through > the boxes without a problem so there isn't much sense in messing with > queues that are working as expected. I guess i was just hoping that if i > was running extremely similar versions of rsyslog i could just enable this > wonderful sounding stream compression. In a case like mine, with very > large amounts of data, the 2 line stream compression sounded like a > fantastic idea idea. > > Are the configs i posted all i *should* need to get both the sender and > the receiver working? In theory, i'd really like to run a hybrid queue > with compression to my other syslog boxes. The isolated box route i think > will be my next step. > > Cheers, > > JB > > On Mon, Mar 9, 2015 at 10:37 PM, David Lang <[email protected]> wrote: > >> On Mon, 9 Mar 2015, Joe Blow wrote: >> >> Hey folks, >>> >>> I'm using 2 rsyslog servers and am trying to get this stream compression >>> working, as it sounds quite magical (1/20th of the data sounds >>> fantastic). >>> Sending box: >>> >>> [root@sender ~]# rsyslogd -v >>> rsyslogd 8.8.0, compiled with: >>> PLATFORM: x86_64-redhat-linux-gnu >>> PLATFORM (lsb_release -d): >>> FEATURE_REGEXP: Yes >>> GSSAPI Kerberos 5 support: No >>> FEATURE_DEBUG (debug build, slow code): No >>> 32bit Atomic operations supported: Yes >>> 64bit Atomic operations supported: Yes >>> memory allocator: system default >>> Runtime Instrumentation (slow code): No >>> uuid support: Yes >>> Number of Bits in RainerScript integers: 64 >>> >>> [root@receiver ~]# rsyslogd -v >>> rsyslogd 8.8.0.ad1, compiled with: >>> PLATFORM: x86_64-redhat-linux-gnu >>> PLATFORM (lsb_release -d): >>> FEATURE_REGEXP: Yes >>> GSSAPI Kerberos 5 support: No >>> FEATURE_DEBUG (debug build, slow code): No >>> 32bit Atomic operations supported: Yes >>> 64bit Atomic operations supported: Yes >>> memory allocator: system default >>> Runtime Instrumentation (slow code): No >>> uuid support: Yes >>> Number of Bits in RainerScript integers: 64 >>> >>> Sending setup: >>> $template RAW, "%rawmsg:1:20480%\n" >>> if $rawmsg contains "myHeaderString|" then {action( >>> Type="omfwd" >>> Target="127.0.0.1" >>> Port="1514" >>> Protocol="tcp" >>> Template="RAW" >>> RebindInterval="250" >>> compression.mode="stream:always" >>> compression.stream.flushOnTXEnd="off" >>> queue.dequeuebatchsize="10000" >>> queue.type="fixedarray" >>> queue.filename="output.rsq" >>> queue.highwatermark="120000000" >>> queue.lowwatermark="15000000" >>> queue.discardmark="150000000" >>> queue.maxdiskspace="50g" >>> queue.size="150000000" >>> queue.saveonshutdown="on" >>> action.resumeretrycount="-1")stop} >>> >>> Receiving setup: >>> >>> $ModLoad imptcp >>> input(type="imptcp" port="514" address="0.0.0.0" Threads="16") >>> >> >> start with much lower thread counts, only increase past 1 if you find you >> need to (too many threads will actually slow you down as the batch sized >> approach 1 there is a lot more contention for the locks on the main queue. >> >> also, you are sending to port 1514 and listening to port 514, is this a >> copy/paste error? >> >> >>> When i turn this on, i get garbage spewing to all consoles on both the >>> receiving and sending system. >>> >> >> the sending system should not be sending anything out to the console, >> what else is in your config >> >> Am i missing any gotchas with this? Is >>> stream compression supported/tested? Would it make any difference that >>> i'm >>> shoving this over an SSH tunnel? >>> >> >> it shouldn't. >> >> I would start by simplifying the config to the minimum (eliminate queues, >> leave threads at the default >> >> Anyone run into this before? Could these 2 different 8.8.0 versions >>> having >>> compatibility issues? >>> >> >> the version you compiled yourself, was it from the v8.8.0 tag or a later >> checkout from git that didn't have that tag changed yet? >> >> I'm about to throw the sending system into debug >>> mode to see if i can figure this out. >>> >> >> that would be my next step after simplifying the config. >> >> David Lang >> >> Cheers, >>> >>> JB >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >>> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >> DON'T LIKE THAT. >> > > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
