Re: [rsyslog] imptcp - compression.mode="stream:always"

David Lang Mon, 09 Mar 2015 21:27:07 -0700

I don't see anything obviously wrong with your config, but I haven't used thisfeature myself. I've just had too many cases where something else had an effectbecause I didn't fully understand things, so when I'm faced with somethinghappening that I don't understand, I work to simplify things and find thesmallest config that breaks. That almost always lets me solve the problem, andwhere it doesn't it lets others see the point of failure without being confusedor distracted by other things.

Do I expect that removing the queue from the action will make any difference?No, but I've been surprised before.

I expect that there is something with the interaction of the other things in theconfig file, or possibly a bug that causes the omfwd module to overwrite avariable that it shouldn't when it's doing compression, something that would notbe noticed on the sender side if it's the last (or only) action in the file.

If simplifying the config on the sending side doesn't show anything, setup twooutput files, one before and one after the omfwd file, both logging inRSYSLOG_DebugFormat and see if the output changes.

If you have the default stuff included after that point, and the compression isoverriding the known variables, it could be setting the severity to emergency,which is sent out to all users by default.


While I go by David, I answer to Dave as well :-)

David Lang

On Mon, 9 Mar 2015, Joe Blowwrote:

Oh one last thing, the port mismatch you noticed is because it's being port
forwarded.  1514 on the local stack is actually a remote system.

Cheers,

JB

On Mon, Mar 9, 2015 at 11:30 PM, Joe Blow <[email protected]> wrote:

I meant to put David, not Dave.

Cheers,

JB

On Mon, Mar 9, 2015 at 11:30 PM, Joe Blow <[email protected]> wrote:

Thanks for the quick response Dave!

There is a lot more within the config(s) than just those lines.  The
garbage that is getting spat to console is probably because i have a lot of
'stock' rsyslog lines at the bottom of my config.  Strangely enough it only
happens when i enable stream compression, otherwise i've got probably a 150
line rsyslog file on the sender and 350 lines on the receiver.

I'll have to take this off onto 2 boxes that aren't doing anything at all
to get this working i'm guessing?  I've got around 40k EPS going through
the boxes without a problem so there isn't much sense in messing with
queues that are working as expected.  I guess i was just hoping that if i
was running extremely similar versions of rsyslog i could just enable this
wonderful sounding stream compression.  In a case like mine, with very
large amounts of data, the 2 line stream compression sounded like a
fantastic idea idea.

Are the configs i posted all i *should* need to get both the sender and
the receiver working?  In theory, i'd really like to run a hybrid queue
with compression to my other syslog boxes. The isolated box route i think
will be my next step.

Cheers,

JB

On Mon, Mar 9, 2015 at 10:37 PM, David Lang <[email protected]> wrote:

On Mon, 9 Mar 2015, Joe Blow wrote:

 Hey folks,


I'm using 2 rsyslog servers and am trying to get this stream compression
working, as it sounds quite magical (1/20th of the data sounds
fantastic).
Sending box:

[root@sender ~]# rsyslogd -v
rsyslogd 8.8.0, compiled with:
 PLATFORM:             x86_64-redhat-linux-gnu
 PLATFORM (lsb_release -d):
 FEATURE_REGEXP:              Yes
 GSSAPI Kerberos 5 support:    No
 FEATURE_DEBUG (debug build, slow code):  No
 32bit Atomic operations supported: Yes
 64bit Atomic operations supported: Yes
 memory allocator:            system default
 Runtime Instrumentation (slow code):  No
 uuid support:          Yes
 Number of Bits in RainerScript integers: 64

[root@receiver ~]# rsyslogd -v
rsyslogd 8.8.0.ad1, compiled with:
 PLATFORM:             x86_64-redhat-linux-gnu
 PLATFORM (lsb_release -d):
 FEATURE_REGEXP:              Yes
 GSSAPI Kerberos 5 support:    No
 FEATURE_DEBUG (debug build, slow code):  No
 32bit Atomic operations supported: Yes
 64bit Atomic operations supported: Yes
 memory allocator:            system default
 Runtime Instrumentation (slow code):  No
 uuid support:          Yes
 Number of Bits in RainerScript integers: 64

Sending setup:
$template RAW, "%rawmsg:1:20480%\n"
if $rawmsg contains "myHeaderString|" then {action(
                       Type="omfwd"
                       Target="127.0.0.1"
                       Port="1514"
                       Protocol="tcp"
                       Template="RAW"
                       RebindInterval="250"
                       compression.mode="stream:always"
                       compression.stream.flushOnTXEnd="off"
                       queue.dequeuebatchsize="10000"
                       queue.type="fixedarray"
                       queue.filename="output.rsq"
                       queue.highwatermark="120000000"
                       queue.lowwatermark="15000000"
                       queue.discardmark="150000000"
                       queue.maxdiskspace="50g"
                       queue.size="150000000"
                       queue.saveonshutdown="on"
                       action.resumeretrycount="-1")stop}

Receiving setup:

$ModLoad imptcp
input(type="imptcp" port="514" address="0.0.0.0" Threads="16")


start with much lower thread counts, only increase past 1 if you find
you need to (too many threads will actually slow you down as the batch
sized approach 1 there is a lot more contention for the locks on the main
queue.

also, you are sending to port 1514 and listening to port 514, is this a
copy/paste error?

When i turn this on, i get garbage spewing to all consoles on both the
receiving and sending system.


the sending system should not be sending anything out to the console,
what else is in your config

   Am i missing any gotchas with this?  Is

stream compression supported/tested?  Would it make any difference that
i'm
shoving this over an SSH tunnel?


it shouldn't.

I would start by simplifying the config to the minimum (eliminate
queues, leave threads at the default

 Anyone run into this before?  Could these 2 different 8.8.0 versions

having
compatibility issues?


the version you compiled yourself, was it from the v8.8.0 tag or a later
checkout from git that didn't have that tag changed yet?

  I'm about to throw the sending system into debug

mode to see if i can figure this out.


that would be my next step after simplifying the config.

David Lang

 Cheers,


JB
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
you DON'T LIKE THAT.

 _______________________________________________

rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] imptcp - compression.mode="stream:always"

Reply via email to