Oh one last thing, the port mismatch you noticed is because it's being port forwarded. 1514 on the local stack is actually a remote system.
Cheers, JB On Mon, Mar 9, 2015 at 11:30 PM, Joe Blow <[email protected]> wrote: > I meant to put David, not Dave. > > Cheers, > > JB > > On Mon, Mar 9, 2015 at 11:30 PM, Joe Blow <[email protected]> wrote: > >> Thanks for the quick response Dave! >> >> There is a lot more within the config(s) than just those lines. The >> garbage that is getting spat to console is probably because i have a lot of >> 'stock' rsyslog lines at the bottom of my config. Strangely enough it only >> happens when i enable stream compression, otherwise i've got probably a 150 >> line rsyslog file on the sender and 350 lines on the receiver. >> >> I'll have to take this off onto 2 boxes that aren't doing anything at all >> to get this working i'm guessing? I've got around 40k EPS going through >> the boxes without a problem so there isn't much sense in messing with >> queues that are working as expected. I guess i was just hoping that if i >> was running extremely similar versions of rsyslog i could just enable this >> wonderful sounding stream compression. In a case like mine, with very >> large amounts of data, the 2 line stream compression sounded like a >> fantastic idea idea. >> >> Are the configs i posted all i *should* need to get both the sender and >> the receiver working? In theory, i'd really like to run a hybrid queue >> with compression to my other syslog boxes. The isolated box route i think >> will be my next step. >> >> Cheers, >> >> JB >> >> On Mon, Mar 9, 2015 at 10:37 PM, David Lang <[email protected]> wrote: >> >>> On Mon, 9 Mar 2015, Joe Blow wrote: >>> >>> Hey folks, >>>> >>>> I'm using 2 rsyslog servers and am trying to get this stream compression >>>> working, as it sounds quite magical (1/20th of the data sounds >>>> fantastic). >>>> Sending box: >>>> >>>> [root@sender ~]# rsyslogd -v >>>> rsyslogd 8.8.0, compiled with: >>>> PLATFORM: x86_64-redhat-linux-gnu >>>> PLATFORM (lsb_release -d): >>>> FEATURE_REGEXP: Yes >>>> GSSAPI Kerberos 5 support: No >>>> FEATURE_DEBUG (debug build, slow code): No >>>> 32bit Atomic operations supported: Yes >>>> 64bit Atomic operations supported: Yes >>>> memory allocator: system default >>>> Runtime Instrumentation (slow code): No >>>> uuid support: Yes >>>> Number of Bits in RainerScript integers: 64 >>>> >>>> [root@receiver ~]# rsyslogd -v >>>> rsyslogd 8.8.0.ad1, compiled with: >>>> PLATFORM: x86_64-redhat-linux-gnu >>>> PLATFORM (lsb_release -d): >>>> FEATURE_REGEXP: Yes >>>> GSSAPI Kerberos 5 support: No >>>> FEATURE_DEBUG (debug build, slow code): No >>>> 32bit Atomic operations supported: Yes >>>> 64bit Atomic operations supported: Yes >>>> memory allocator: system default >>>> Runtime Instrumentation (slow code): No >>>> uuid support: Yes >>>> Number of Bits in RainerScript integers: 64 >>>> >>>> Sending setup: >>>> $template RAW, "%rawmsg:1:20480%\n" >>>> if $rawmsg contains "myHeaderString|" then {action( >>>> Type="omfwd" >>>> Target="127.0.0.1" >>>> Port="1514" >>>> Protocol="tcp" >>>> Template="RAW" >>>> RebindInterval="250" >>>> compression.mode="stream:always" >>>> compression.stream.flushOnTXEnd="off" >>>> queue.dequeuebatchsize="10000" >>>> queue.type="fixedarray" >>>> queue.filename="output.rsq" >>>> queue.highwatermark="120000000" >>>> queue.lowwatermark="15000000" >>>> queue.discardmark="150000000" >>>> queue.maxdiskspace="50g" >>>> queue.size="150000000" >>>> queue.saveonshutdown="on" >>>> action.resumeretrycount="-1")stop} >>>> >>>> Receiving setup: >>>> >>>> $ModLoad imptcp >>>> input(type="imptcp" port="514" address="0.0.0.0" Threads="16") >>>> >>> >>> start with much lower thread counts, only increase past 1 if you find >>> you need to (too many threads will actually slow you down as the batch >>> sized approach 1 there is a lot more contention for the locks on the main >>> queue. >>> >>> also, you are sending to port 1514 and listening to port 514, is this a >>> copy/paste error? >>> >>> >>>> When i turn this on, i get garbage spewing to all consoles on both the >>>> receiving and sending system. >>>> >>> >>> the sending system should not be sending anything out to the console, >>> what else is in your config >>> >>> Am i missing any gotchas with this? Is >>>> stream compression supported/tested? Would it make any difference that >>>> i'm >>>> shoving this over an SSH tunnel? >>>> >>> >>> it shouldn't. >>> >>> I would start by simplifying the config to the minimum (eliminate >>> queues, leave threads at the default >>> >>> Anyone run into this before? Could these 2 different 8.8.0 versions >>>> having >>>> compatibility issues? >>>> >>> >>> the version you compiled yourself, was it from the v8.8.0 tag or a later >>> checkout from git that didn't have that tag changed yet? >>> >>> I'm about to throw the sending system into debug >>>> mode to see if i can figure this out. >>>> >>> >>> that would be my next step after simplifying the config. >>> >>> David Lang >>> >>> Cheers, >>>> >>>> JB >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com/professional-services/ >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if >>>> you DON'T LIKE THAT. >>>> >>>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>> DON'T LIKE THAT. >>> >> >> > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
