On Mon, 9 Mar 2015, Joe Blow wrote:
Hey folks,
I'm using 2 rsyslog servers and am trying to get this stream compression
working, as it sounds quite magical (1/20th of the data sounds fantastic).
Sending box:
[root@sender ~]# rsyslogd -v
rsyslogd 8.8.0, compiled with:
PLATFORM: x86_64-redhat-linux-gnu
PLATFORM (lsb_release -d):
FEATURE_REGEXP: Yes
GSSAPI Kerberos 5 support: No
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
memory allocator: system default
Runtime Instrumentation (slow code): No
uuid support: Yes
Number of Bits in RainerScript integers: 64
[root@receiver ~]# rsyslogd -v
rsyslogd 8.8.0.ad1, compiled with:
PLATFORM: x86_64-redhat-linux-gnu
PLATFORM (lsb_release -d):
FEATURE_REGEXP: Yes
GSSAPI Kerberos 5 support: No
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
memory allocator: system default
Runtime Instrumentation (slow code): No
uuid support: Yes
Number of Bits in RainerScript integers: 64
Sending setup:
$template RAW, "%rawmsg:1:20480%\n"
if $rawmsg contains "myHeaderString|" then {action(
Type="omfwd"
Target="127.0.0.1"
Port="1514"
Protocol="tcp"
Template="RAW"
RebindInterval="250"
compression.mode="stream:always"
compression.stream.flushOnTXEnd="off"
queue.dequeuebatchsize="10000"
queue.type="fixedarray"
queue.filename="output.rsq"
queue.highwatermark="120000000"
queue.lowwatermark="15000000"
queue.discardmark="150000000"
queue.maxdiskspace="50g"
queue.size="150000000"
queue.saveonshutdown="on"
action.resumeretrycount="-1")stop}
Receiving setup:
$ModLoad imptcp
input(type="imptcp" port="514" address="0.0.0.0" Threads="16")
start with much lower thread counts, only increase past 1 if you find you need
to (too many threads will actually slow you down as the batch sized approach 1
there is a lot more contention for the locks on the main queue.
also, you are sending to port 1514 and listening to port 514, is this a
copy/paste error?
When i turn this on, i get garbage spewing to all consoles on both the
receiving and sending system.
the sending system should not be sending anything out to the console, what else
is in your config
Am i missing any gotchas with this? Is
stream compression supported/tested? Would it make any difference that i'm
shoving this over an SSH tunnel?
it shouldn't.
I would start by simplifying the config to the minimum (eliminate queues, leave
threads at the default
Anyone run into this before? Could these 2 different 8.8.0 versions having
compatibility issues?
the version you compiled yourself, was it from the v8.8.0 tag or a later
checkout from git that didn't have that tag changed yet?
I'm about to throw the sending system into debug
mode to see if i can figure this out.
that would be my next step after simplifying the config.
David Lang
Cheers,
JB
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
