Dear community,
sorry if I'm re-posting here the GitHub issue
https://github.com/rsyslog/rsyslog/issues/1771 , I'm try to reach other
people that maybe had my same problem.
I'trying to build a centralized logging system using:
omkafka -> kafka -> imkafka -> omfile
The rsyslog sender configuration is:
module(load="omkafka")
$WorkDirectory /var/spool/rsyslog # where to place spool files
$ActionQueueFileName fwdRule1 # unique name prefix for spool files
$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
$ActionQueueType LinkedList # run asynchronously
$ActionResumeRetryCount -1 # infinite retries if host is down
if $programname contains 'backend' then
action(type="omkafka" topic="logging" broker=["IP:9092"]
confParam=["compression.codec=snappy",
"socket.timeout.ms=15",
"socket.keepalive.enable=true"]
)
stop
The sender is working properly i can see the messages in the kafka topic
using:
./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic
logging --from-beginning
On the receiver part i configured rsyslog with the following:
module(load="imkafka")
input(type="imkafka" topic="logging" broker=["IP:9092"] ruleset="pRuleset"
consumergroup="default"
confParam=["compression.codec=snappy",
"socket.timeout.ms=15",
"socket.keepalive.enable=true"]
)
ruleset(name="pRuleset") {
action(type="omfile" file="/var/log/backend")
}
I'm using for sender and receiver:
rsyslogd 8.29.0, compiled with:
PLATFORM: x86_64-redhat-linux-gnu
PLATFORM (lsb_release -d):
FEATURE_REGEXP: Yes
GSSAPI Kerberos 5 support: No
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
memory allocator: system default
Runtime Instrumentation (slow code): No
uuid support: Yes
Number of Bits in RainerScript integers: 64
and Kafka version : 0.11.0.0
I did a quick look at the code, and matching the logs with the code I think
the thread is having a problem on the subscription here:
https://github.com/rsyslog/rsyslog/blob/master/plugins/imkafka/imkafka.c#L459-L464
The debug logs for the Kafka library are in the GitHub case and they don't
show any additional info that helps in solving the issue.
Does someone use imkafka? If so can please share the conf and the versions
that is using?
Looking forward to your reply.
Regards
L.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
