Re: [rsyslog] getting error on IncludeConfig

[Peter Viskup via rsyslog]

It is bug of main_queue() not supporting division into more files. Calling
action() does support it.
Opened bug report: https://github.com/rsyslog/rsyslog/issues/4484

-- 
Peter

On Tue, Dec 8, 2020 at 4:34 PM Peter Viskup  wrote:

> Seeing errors
> rsyslogd: error during parsing file /etc/rsyslog.d/lin/03-main-queue.conf,
> on or before line 2: invalid character '$' in object definition - is there
> an invalid escape sequence somewhere? [v8.1901.0 try
> https://www.rsyslog.com/e/2207 ]
> rsyslogd: error during parsing file /etc/rsyslog.d/lin/03-main-queue.conf,
> on or before line 2: invalid character '/' in object definition - is there
> an invalid escape sequence somewhere? [v8.1901.0 try
> https://www.rsyslog.com/e/2207 ]
> rsyslogd: error during parsing file /etc/rsyslog.d/lin/03-main-queue.conf,
> on or before line 2: syntax error on token 'etc' [v8.1901.0 try
> https://www.rsyslog.com/e/2207 ]
>
> on file
> # ls -la lin/03-main-queue.conf
> lrwxrwxrwx 1 root root 27 Dec  8 09:30 lin/03-main-queue.conf ->
> ../host/lin/main-queue.conf
> where the file contains
> # cat host/lin/main-queue.conf
> $IncludeConfig /etc/rsyslog.d/global/includes/target-main.inc
> $IncludeConfig /etc/rsyslog.d/lin/includes/queue-lin-size-small.inc
> $IncludeConfig /etc/rsyslog.d/lin/includes/queue-lin-disk-1k-small.inc
> )
> and all configuration parts are accessible and formatted properly.
>
> Change of host/lin/main-queue.conf to contain content of all the parts in
> the file is an workaround for the issue.
>
> What can be the root cause for this issue? Is there any limitation in
> nesting of IncludeConfig?
>
> Running Debian10 with rsyslog 8.1901.0-1.
>
> --
> Peter
>
___
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] getting error on IncludeConfig

[Peter Viskup via rsyslog]

Seeing errors
rsyslogd: error during parsing file /etc/rsyslog.d/lin/03-main-queue.conf,
on or before line 2: invalid character '$' in object definition - is there
an invalid escape sequence somewhere? [v8.1901.0 try
https://www.rsyslog.com/e/2207 ]
rsyslogd: error during parsing file /etc/rsyslog.d/lin/03-main-queue.conf,
on or before line 2: invalid character '/' in object definition - is there
an invalid escape sequence somewhere? [v8.1901.0 try
https://www.rsyslog.com/e/2207 ]
rsyslogd: error during parsing file /etc/rsyslog.d/lin/03-main-queue.conf,
on or before line 2: syntax error on token 'etc' [v8.1901.0 try
https://www.rsyslog.com/e/2207 ]

on file
# ls -la lin/03-main-queue.conf
lrwxrwxrwx 1 root root 27 Dec  8 09:30 lin/03-main-queue.conf ->
../host/lin/main-queue.conf
where the file contains
# cat host/lin/main-queue.conf
$IncludeConfig /etc/rsyslog.d/global/includes/target-main.inc
$IncludeConfig /etc/rsyslog.d/lin/includes/queue-lin-size-small.inc
$IncludeConfig /etc/rsyslog.d/lin/includes/queue-lin-disk-1k-small.inc
)
and all configuration parts are accessible and formatted properly.

Change of host/lin/main-queue.conf to contain content of all the parts in
the file is an workaround for the issue.

What can be the root cause for this issue? Is there any limitation in
nesting of IncludeConfig?

Running Debian10 with rsyslog 8.1901.0-1.

-- 
Peter
___
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] handling Windows Event Messages

[Peter Viskup via rsyslog]

Hello Rainer,
just curious about the resources rsyslog windows agent requires. Maybe in
comparison to nxlog or others.
We are facing issues with getting the IIS logs from Windows hosts. They log
to plain text files as writing to Windows EventLog caused performance
issues.
How much resources does the rsyslog windows agent consume? How does it
perform for this type of plain text file processing? Do you have some
numbers to count on?

Did someone compare rsyslog windows to nxlog or other syslog forwarding
tool?

-- 
Peter

On Thu, Aug 27, 2020 at 2:18 PM Rainer Gerhards via rsyslog <
rsyslog@lists.adiscon.com> wrote:

> That's one of the reasons why I recommend rsyslog windows Agent: you
> have full control over the output format. Also, it's default format
> (Adiscon EventReporter) is known by many systems because it was the
> first tool ever to perform that type of work.
>
> Rainer
>
> El jue., 27 ago. 2020 a las 13:41, Mariusz Kruk via rsyslog
> () escribió:
> >
> > Strange thing, because in my "Sent" folder the message is full of
> > content whereas I see the posting on the list empty.
> >
> > Anyways, I'll repost the contents of the original message:
> >
> > "I've seen Kiwi and Solarwinds in use and the main problem is not in
> > generating log events as such or forwarding them later with rsyslog or
> > any other solution. The problem in the end is that when you receive the
> > events at the destination, you're probably want to parse it into some
> > kind of log management software.
> >
> > And here is where it gets tricky because your solution might not be very
> > happy with the format of the message. I suggest you take a look at both
> > of them if you're interested and see for yourself whether it's parseable
> > on your end.
> > If I remember correctly, kiwi sends some part of the data as xml and
> > some as key-value part of the syslog message but Solarwinds sends the
> > events rendered to a simple text message. (But I haven't seen the for
> > quite a while so this is just my vague recollection)."
> >
> > Mariusz Kruk
> > Ekspert ds. Bezpieczeństwa IT
> > COMP S.A.
> > Pion Cyberbezpieczeństwa i Zarządzania Ryzykiem
> > e-mail: mariusz.k...@comp.com.pl
> > e-mail: mariusz.k...@safecomp.com
> > tel: +48 608 623 299
> >
> > On 27.08.2020 09:03, mariusz.kruk--- via rsyslog wrote:
> > > ___
> > > rsyslog mailing list
> > > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com/professional-services/
> > > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
> you DON'T LIKE THAT.
> > ___
> > rsyslog mailing list
> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
> ___
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
___
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Rsyslog issue - when imptcp & imtcp/TLS on same system - imptcp messages received in Rsyslogd not added to log file

[Peter Viskup via rsyslog]

At first you may have a look into /etc/rsyslog.d/*.conf whether the
messages are not processed and filtered somewhere in those configuration
snippets.

You can try to log all messages flowing through the rsyslog to one file
with debug format:
*.*   /var/log/debug;RSYSLOG_DebugFormat
Put the line on top of the $IncludeConfig statement.

More on different formatting templates available in documentation.
https://www.rsyslog.com/doc/v8-stable/configuration/templates.html
and rsyslog configuration in general
https://www.rsyslog.com/doc/v8-stable/configuration/index.html

Peter

On Sat, Oct 24, 2020 at 2:06 AM Walton, Glenn  wrote:

> Hello, thank you for any suggestions as to why the data is not captured in
> /var/log/messages.
>
>
>
> Data sent from a separate host on same subnet via:
>
> logger -p daemon.warn "to cpsyslog01 testing-d1023-t1855
> - on tcp 601" --tcp --port 601  --server 172.16.130.19
>
>
>
> attachment shows data received on the syslog host port 601. Including here
> the raw pcap file and also as viewed in wireshark.   Regards,
>
>
>
> glenn
>
>
>
>
>
>
>
> *From:* Peter Viskup 
> *Sent:* Friday, October 23, 2020 12:23 AM
> *To:* rsyslog-users 
> *Cc:* Walton, Glenn 
> *Subject:* Re: [rsyslog] Rsyslog issue - when imptcp & imtcp/TLS on same
> system - imptcp messages received in Rsyslogd not added to log file
>
>
>
> Hello Glenn,
>
>
>
> On Thu, Oct 22, 2020 at 11:26 PM Walton, Glenn via rsyslog <
> rsyslog@lists.adiscon.com> wrote:
>
> Questions:
>
>   1.  Its my understanding when configuring TLS with imtcp module that
> imptcp should be used to provide a plain unencrypted TCP listener; is there
> a better alternative, or any specific guidelines for this scenario  ?
>
> Yes - you are right.  It was already discussed some time ago.
>
>
> http://rsyslog-users.1305293.n2.nabble.com/Mix-of-GTLS-and-PTCP-listeners-running-same-instance-tc7591434.html
> 
>
>
> Following bugreport is related.
>
> https://github.com/rsyslog/rsyslog/issues/3727
> 
>
>
>
>
>   2.  With imptcp in place, is there some extra configuration needed to
> cause these incoming events to be written to the log file
> (/var/log/messages) ?
>
> No extra configuration options are required.
>
>
>
> One of the reasons why you do not see the messages in /var/log/messages is
> they are of debug syslog priority. Send the message examples you see on the
> wire (running tcpdump).
>
>
>
> --
>
> Peter
> --
> This message is intended only for the person(s) to which it is addressed
> and may contain privileged, confidential and/or insider information.
> If you have received this communication in error, please notify us
> immediately by replying to the message and deleting it from your computer.
> Any disclosure, copying, distribution, or the taking of any action
> concerning
> the contents of this message and any attachment(s) by anyone other
> than the named recipient(s) is strictly prohibited.
>
___
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Rsyslog issue - when imptcp & imtcp/TLS on same system - imptcp messages received in Rsyslogd not added to log file

[Peter Viskup via rsyslog]

Hello Glenn,

On Thu, Oct 22, 2020 at 11:26 PM Walton, Glenn via rsyslog <
rsyslog@lists.adiscon.com> wrote:

> Questions:
>
>   1.  Its my understanding when configuring TLS with imtcp module that
> imptcp should be used to provide a plain unencrypted TCP listener; is there
> a better alternative, or any specific guidelines for this scenario  ?

Yes - you are right.  It was already discussed some time ago.
http://rsyslog-users.1305293.n2.nabble.com/Mix-of-GTLS-and-PTCP-listeners-running-same-instance-tc7591434.html

Following bugreport is related.
https://github.com/rsyslog/rsyslog/issues/3727


>   2.  With imptcp in place, is there some extra configuration needed to
> cause these incoming events to be written to the log file
> (/var/log/messages) ?
>
No extra configuration options are required.

One of the reasons why you do not see the messages in /var/log/messages is
they are of debug syslog priority. Send the message examples you see on the
wire (running tcpdump).

-- 
Peter
___
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] split messages

[Peter Viskup via rsyslog]

Hi,
the LF seems to be not missing:
0x0540:  4e6b 4a54 5a57 7836 4d6d 5934 6130 4933  NkJTZWx6MmY4a0I3
0x0550:  6454 6478 556d 6734 546b 4a6f 4b32 704a  dTdxUmg4TkJoK2pJ
0x0560:  51*0a* 3c31 333e 3120 3230 3230 2d30   Q.<13>1.2020-0
13:19:34.689926 IP 10.x.y.z.49938 > 10.a.b.c.2514: Flags [.], seq
5404:6742, ack 1, win 502, options [nop,nop,TS val 2066170096 ecr
1503586084], length 1338
0x:  4500 056e 1f64 4000 3c06 d2b2 0a67 1982  E..n.d@.<g..
0x0010:  0a01 198a c312 09d2 f6f5 ddac 769b fcaf  v...
0x0020:  8010 01f6 6d15  0101 080a 7b27 40f0  m...{'@.
0x0030:  599e e724 392d 3033 5431 333a 3139 3a33  Y..$9-03T13:19:3
0x0040:  342e 3638 3532 3535 2b30 303a 3030 2068  4.685255+00:00.h
What else to check?

Peter

On Fri, Sep 18, 2020 at 10:03 AM Rainer Gerhards 
wrote:

> mhhh... when rsyslog forwards, it should add an \n AFTER the message.
> Can you check what is outgoing (e.g. via Wireshark)? If the LF is
> missing, can you post the client's config (and maybe a debug log)?
>
> Rainer
>
> El vie., 18 sept. 2020 a las 9:56, Peter Viskup
> () escribió:
> >
> > Hi Rainer,
> > confirm it is related to messages not having LF on the end.
> > Problem is reported on the second syslog relay. It is caused by messages
> being split on the first row relay due to exceeding size limit. The first
> part of the splitted message is not terminated with LF. Both relays use the
> same size limit for received messages, therefore all the messages reported
> on relay2 start with .
> >
> > Is there any possibility to let rsyslog add the LF on the end of the
> message once splitted?
> > Thank you.
> >
> > --
> > Peter
> >
> > On Tue, Sep 8, 2020 at 10:07 AM Rainer Gerhards <
> rgerha...@hq.adiscon.com> wrote:
> >>
> >> This smells like incorrect framing (no LF at end of message).
> >>
> >> Rainer
> >>
> >> El mar., 8 sept. 2020 a las 9:48, Peter Viskup via rsyslog
> >> () escribió:
> >> >
> >> > Getting following strange messages on our syslog servers:
> >> >
> >> >
> >> > Sep  8 06:02:03 syslog01 rsyslogd: imptcp bo-t: message received is at
> >> > least 2001 byte larger than max msg size; message will be split
> starting
> >> > at: " <13>1 2020-09-08T06:02:03.25764"  [v8.1901.0]
> >> >
> >> >
> >> > Not getting them all the time.
> >> >
> >> > Any idea what could be causing this?
> >> >
> >> > Thank you.
> >> >
> >> >
> >> > --
> >> >
> >> > Peter
> >> > ___
> >> > rsyslog mailing list
> >> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> >> > http://www.rsyslog.com/professional-services/
> >> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> >> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
> you DON'T LIKE THAT.
>
___
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] split messages

[Peter Viskup via rsyslog]

Hi Rainer,
confirm it is related to messages not having LF on the end.
Problem is reported on the second syslog relay. It is caused by messages
being split on the first row relay due to exceeding size limit. The first
part of the splitted message is not terminated with LF. Both relays use the
same size limit for received messages, therefore all the messages reported
on relay2 start with .

Is there any possibility to let rsyslog add the LF on the end of the
message once splitted?
Thank you.

-- 
Peter

On Tue, Sep 8, 2020 at 10:07 AM Rainer Gerhards 
wrote:

> This smells like incorrect framing (no LF at end of message).
>
> Rainer
>
> El mar., 8 sept. 2020 a las 9:48, Peter Viskup via rsyslog
> () escribió:
> >
> > Getting following strange messages on our syslog servers:
> >
> >
> > Sep  8 06:02:03 syslog01 rsyslogd: imptcp bo-t: message received is at
> > least 2001 byte larger than max msg size; message will be split starting
> > at: " <13>1 2020-09-08T06:02:03.25764"  [v8.1901.0]
> >
> >
> > Not getting them all the time.
> >
> > Any idea what could be causing this?
> >
> > Thank you.
> >
> >
> > --
> >
> > Peter
> > ___
> > rsyslog mailing list
> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] split messages

[Peter Viskup via rsyslog]

Getting following strange messages on our syslog servers:


Sep  8 06:02:03 syslog01 rsyslogd: imptcp bo-t: message received is at
least 2001 byte larger than max msg size; message will be split starting
at: " <13>1 2020-09-08T06:02:03.25764"  [v8.1901.0]


Not getting them all the time.

Any idea what could be causing this?

Thank you.


-- 

Peter
___
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] howto rsyslog HA with PCS (and keep local logs)

[Peter Viskup via rsyslog]

Hi David,
we also run more instances of chrooted rsyslog in cluster setup. ATM just
simple hearbeat+ldirector+IPVS, but plan to upgrade to a full
featured cluster stack soon for better balancing of the load.
More information on our current setup available in my previous post [1].
For running more instances of rsyslog find the attached systemd service
file. Feel free to use it as the basis for your setup (you should remove
all the chrooting and pre/post exec's). Decide whether to read about the
instantioned services under systemd [2].

[1]
http://rsyslog-users.1305293.n2.nabble.com/rsyslog-in-HA-mode-tc7595885.html#a7595894

[2]
https://www.freedesktop.org/software/systemd/man/systemd.service.html#Service%20Templates

-- 
Peter

On Mon, Aug 31, 2020 at 11:35 AM Mariusz Kruk via rsyslog <
rsyslog@lists.adiscon.com> wrote:

> You can always use two instances of rsyslog. One for local logs and
> another for forwarding logs from remote.
>
> Of course you'll need to create a unit file for systemd to start
> rsyslogd with your own config file.
>
> I think that's the easiest approach.
>
> Mariusz Kruk
> Ekspert ds. Bezpieczeństwa IT
> COMP S.A.
> Pion Cyberbezpieczeństwa i Zarządzania Ryzykiem
> e-mail: mariusz.k...@comp.com.pl
> e-mail: mariusz.k...@safecomp.com
> tel: +48 608 623 299
>
> On 31.08.2020 10:27, David CAPG via rsyslog wrote:
> > Hi,
> >
> > I've found a pcs cluster for rsyslog at work (quite old but it's not the
> > topic :) ), in active/passive mode (HA)
> >
> > One resource on the cluster manage the rsyslog service (lsb:rsyslog), and
> > ... the service rsyslog is configured too to run at start.
> >
> > For me, with PCS if a service is managed by pcs, it must be disable at
> > system level.
> >
> > But, if i disable rsyslog at system level, on the slave inactive node, i
> > got no more logs for local services, like /var/log/secure, messages,
> >
> > I try to find a how-to for this case, but hard even for google :) (find
> > only how to log PCS clustrs, ...)
> >
> > Thx for any help.
> > David
> > ___
> > rsyslog mailing list
> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
> ___
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.


debian_rsyslog-chroot@.service
Description: Binary data
___
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] handling Windows Event Messages

[Peter Viskup via rsyslog]

Understand. It is one of our candidates.
Just discovered one of your latest posts regarding Windows Events
forwarding. :-)
https://rainer.gerhards.net/2019/10/rsyslog-integrating-windows-event-log-via-udp.html

Still interesting whether some other users have experience with other
software.

-- 
Peter

On Mon, Aug 24, 2020 at 4:47 PM Rainer Gerhards 
wrote:

> For obvious reasons, I recommend the rsyslog Windows Agent ;-)
>
> https://www.rsyslog.com/windows-agent/
>
> Rainer
>
> El lun., 24 ago. 2020 a las 16:17, Peter Viskup via rsyslog
> () escribió:
> >
> > Does anyone have experience of handling WEC messages from Windows clients
> > in (r)syslog infrastructure?
> > The standard way is to install some Windows syslog agent which forwards
> > Windows events to syslog infrastructure. What Windows syslog agent do you
> > use?
> >
> > Might be interesting to see something like the imwec module.
> >
> https://docs.microsoft.com/en-us/windows/win32/wec/using-windows-event-collector
> > The same way the syslog-ng PE implemented it.
> >
> https://support.oneidentity.com/technical-documents/syslog-ng-premium-edition/7.0.17/windows-event-collector-administration-guide/log
> > They switch from developing Windows Syslog agent to WEC input module for
> > syslog-ng server which I find the best way of handling this type of data
> > flow.
> >
> > --
> > Peter
> > ___
> > rsyslog mailing list
> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] queue files handling question

[Peter Viskup via rsyslog]

Running rsyslog 8.2001 on Debian 10 we do monitor file changes with use of
iWatch [1].
Within monitoring we occasionally see a lot of messages
Aug 26 00:01:10 loco iWatch[3575]: *
/chroot/net/var/spool/rsyslog/mainq.qi.tmp is moved to
/chroot/net/var/spool/rsyslog/mainq.qi
Aug 26 00:01:10 loco iWatch[3575]: *
/chroot/net/var/spool/rsyslog/mainq.qi.tmp is moved to
/chroot/net/var/spool/rsyslog/mainq.qi
and sometimes
Aug 26 00:01:10 loco iWatch[3575]: *
/chroot/net/var/spool/rsyslog/mainq.qi.tmp is closed
or other messages regarding the closing and deleting the message queue
files.

There are 1564 messages logged within one second causing the filesystem
being full. Where 1427 are related to main.qi.tmp being moved to main.qi
file.

Seems the queue index file is always open in "sync" mode and the
queue.checkpointinterval [2] and queue.syncqueuefiles [3] are used for the
message queue files only.
Is that assumption correct? Is there any way to modify the queue index file
handling?

Thank you.

[1] http://iwatch.sourceforge.net/index.html
[2]
https://www.rsyslog.com/doc/master/rainerscript/queue_parameters.html#queue-checkpointinterval
[3]
https://www.rsyslog.com/doc/master/rainerscript/queue_parameters.html#queue-syncqueuefiles

-- 
Peter
___
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] handling Windows Event Messages

[Peter Viskup via rsyslog]

Does anyone have experience of handling WEC messages from Windows clients
in (r)syslog infrastructure?
The standard way is to install some Windows syslog agent which forwards
Windows events to syslog infrastructure. What Windows syslog agent do you
use?

Might be interesting to see something like the imwec module.
https://docs.microsoft.com/en-us/windows/win32/wec/using-windows-event-collector
The same way the syslog-ng PE implemented it.
https://support.oneidentity.com/technical-documents/syslog-ng-premium-edition/7.0.17/windows-event-collector-administration-guide/log
They switch from developing Windows Syslog agent to WEC input module for
syslog-ng server which I find the best way of handling this type of data
flow.

-- 
Peter
___
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] SSL for ommysql

[Peter Viskup via rsyslog]

Hi Andrea,
try to read and use
https://www.rsyslog.com/doc/v8-stable/configuration/modules/ommysql.html#mysqlconfig-file
https://www.rsyslog.com/doc/v8-stable/configuration/modules/ommysql.html#mysqlconfig-section
with
https://dev.mysql.com/doc/refman/8.0/en/option-files.html
That might help you to find the appropriate configuration.

Peter

On Wed, Jul 22, 2020 at 5:36 PM Andrea Gabellini via rsyslog <
rsyslog@lists.adiscon.com> wrote:

> Hello,
>
> I am sending logs to a remote mysql server using the ommysql module.
>
> The remote server requires SSL. How can I configure rsyslog and the
> ommysql module to use SSL?
>
> Thanks,
> Andrea
>
> --
> __
> A program is never finished until the programmer dies.
> __
>
> TIM San Marino S.p.A.
> Andrea Gabellini
> Engineering R
> TIM San Marino S.p.A. - https://www.telecomitalia.sm
> Via Ventotto Luglio, 212 - Piano -2
> 47893 - Borgo Maggiore - Republic of San Marino
> Tel: (+378) 0549 886237
> Fax: (+378) 0549 886188
>
>
> --
> Informativa Privacy
>
> Questa email ha per destinatari dei contatti presenti negli archivi di TIM
> San Marino S.p.A.. Tutte le informazioni vengono trattate e tutelate nel
> rispetto della normativa vigente sulla protezione dei dati personali (Reg.
> EU 2016/679). Per richiedere informazioni e/o variazioni e/o la
> cancellazione dei vostri dati presenti nei nostri archivi potete inviare
> una email a priv...@telecomitalia.sm.
>
> Avviso di Riservatezza
>
> Il contenuto di questa e-mail e degli eventuali allegati e' strettamente
> confidenziale e destinato alla/e persona/e a cui e' indirizzato. Se avete
> ricevuto per errore questa e-mail, vi preghiamo di segnalarcelo
> immediatamente e di cancellarla dal vostro computer. E' fatto divieto di
> copiare e divulgare il contenuto di questa e-mail. Ogni utilizzo abusivo
> delle informazioni qui contenute da parte di persone terze o comunque non
> indicate nella presente e-mail potra' essere perseguito ai sensi di legge.
> ___
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Error message in message logs and systemctl status rsyslogd

[Peter Viskup via rsyslog]

Hi Ren,
seems FW or some other device invalidated/closed the connection. It could
happen when no data is sent within configured timeout.
Make sure to have TCP KeepAlive enabled on both sides (including OS and
rsyslog omfwd/im[p]tcp settings).
That might help.

Peter

On Sat, Jul 25, 2020 at 8:43 PM Ren You via rsyslog <
rsyslog@lists.adiscon.com> wrote:

> Hi,
> I am seeing lots of messages as following in message logs and “systemctl
> status rsyslog.service” on my syslog server, my question is this will
> affect the performance or potentially case issues?
>
> # systemctl status rsyslog.service
> ● rsyslog.service - System Logging Service
>Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled;
> vendor preset: enabled)
>Active: active (running) since Sat 2020-07-25 11:24:49 PDT; 11min ago
>  Docs: man:rsyslogd(8)
>http://www.rsyslog.com/doc/
>  Main PID: 1037 (rsyslogd)
>CGroup: /system.slice/rsyslog.service
>└─1037 /usr/sbin/rsyslogd -n
>
> Jul 25 11:36:40   rsyslogd[1037]: unexpected GnuTLS error -110 in
> nsd_gtls.c:536: The TLS connection was non-properly terminated.  ...e/2078 ]
> Jul 25 11:36:40   rsyslogd[1037]: netstream session 0x7f29a425a060 from
> ***.***.***.*** will be closed due to error  [v8.24.0-34.el7 t...e/2078 ]
> Jul 25 11:36:42   rsyslogd[1037]: unexpected GnuTLS error -110 in
> nsd_gtls.c:536: The TLS connection was non-properly terminated.  ...e/2078 ]
> Jul 25 11:36:42   rsyslogd[1037]: netstream session 0x7f29a4174ff0 from
> ***.***.***.***will be closed due to error  [v8.24.0-34.el7...e/2078 ]
> Jul 25 11:36:42   rsyslogd[1037]: unexpected GnuTLS error -110 in
> nsd_gtls.c:536: The TLS connection was non-properly terminated.  ...e/2078 ]
> Jul 25 11:36:42   rsyslogd[1037]: netstream session 0x7f29a4116cd0 from
> ***.***.***.*** will be closed due to error  [v8.24.0-34.el7 ...e/2078 ]
> Jul 25 11:36:42   rsyslogd[1037]: unexpected GnuTLS error -110 in
> nsd_gtls.c:536: The TLS connection was non-properly terminated.  ...e/2078 ]
> Jul 25 11:36:42   rsyslogd[1037]: netstream session 0x7f29a45e1190 from
> ***.***.***.***will be closed due to error  [v8.24.0-34.el7 ...e/2078 ]
> Jul 25 11:36:43   rsyslogd[1037]: unexpected GnuTLS error -110 in
> nsd_gtls.c:536: The TLS connection was non-properly terminated.  ...e/2078 ]
> Jul 25 11:36:43   rsyslogd[1037]: netstream session 0x7f29a404ea00 from
> ***.***.***.***will be closed due to error  [v8.24.0-34.el7 ...e/2078 ]
> Hint: Some lines were ellipsized, use -l to show in full.
>
>
>
>
>
> Thanks.
>
>
>
> -Ren
> ___
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
___
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Double imupd stats entries

[Peter Viskup via rsyslog]

Opened bugreport https://github.com/rsyslog/rsyslog/issues/4364
will post updates there.

Peter

On Sat, Jul 18, 2020 at 11:48 AM Rainer Gerhards 
wrote:

> thx - looks like I need to dig a bit deeper, I am sure there is a
> valid explanation - which then should be reflected by some name
> mangling.
>
> Rainer
>
> El vie., 17 jul. 2020 a las 20:21, Peter Viskup
> () escribió:
> >
> > Not related to IPv4 vs. IPv6 nor rulesets
> >
> > On server with IPv6 disabled with only one public iface and IP
> > root@server:~# sysctl -a |grep .disable_ipv6
> > net.ipv6.conf.all.disable_ipv6 = 1
> > net.ipv6.conf.default.disable_ipv6 = 1
> > net.ipv6.conf.ens192.disable_ipv6 = 1
> > net.ipv6.conf.lo.disable_ipv6 = 1
> >
> > Related configuration
> > root@server:~# cat /chroot/net/etc/rsyslog.d/host/net/listeners/*.conf
> > input(type="imptcp" port="1514" KeepAlive="on")
> > input(type="imudp" port="1514")
> > root@server:~# cat /chroot/net/etc/rsyslog.d/global/03-modules.conf
> > module(load="imudp")
> > module(load="imptcp")
> > root@server:~# cat /chroot/net/etc/rsyslog.d/global/00-stats.conf
> > module(load="impstats"
> >   interval="15"
> >   severity="7"
> >   ResetCounters="on"
> >   log.syslog="off"
> >   # need to turn log stream logging off!
> >   log.file="/var/spool/rsyslog/rsyslog.stats")
> >
> > We do see similar "doubled" stats:
> > Fri Jul 17 18:06:46 2020: imudp(*:1514): origin=imudp submitted=1216
> disallowed=0
> > Fri Jul 17 18:06:46 2020: imudp(*:1514): origin=imudp submitted=0
> disallowed=0
> >
> > Running rsyslog 8.1901.0-1 from Debian 10 stable.
> >
> > With input name configured, the output shows same name in both lines
> > root@server:~# cat /etc/rsyslog-eset/host/net/listeners/input-udp.conf
> > input(type="imudp" port="1514" name="udp1514")
> >
> > Fri Jul 17 18:17:27 2020: udp1514(*:1514): origin=imudp submitted=1354
> disallowed=0
> > Fri Jul 17 18:17:27 2020: udp1514(*:1514): origin=imudp submitted=0
> disallowed=0
> >
> > Looks like a bug being here with us for a long time. :-)
> >
> > Peter
> >
> >
> > On Fri, Jul 17, 2020 at 9:19 AM Rainer Gerhards <
> rgerha...@hq.adiscon.com> wrote:
> >>
> >> El jue., 16 jul. 2020 a las 9:00, Peter Viskup ()
> escribió:
> >> >
> >> > Just discovered the same on our infra.
> >> > Will test by disabling IPV6 and confirm if Ángel will not answer
> sooner.
> >>
> >> Thx - I guess if it is that way, it would make sense to automatically
> >> append"ipv4" or "v6" to the configured name.
> >>
> >> Rainer
> >> >
> >> > Peter
> >> >
> >> > On Tue, Jul 14, 2020 at 4:02 PM Rainer Gerhards <
> rgerha...@hq.adiscon.com> wrote:
> >> >>
> >> >> Sorry for being late to the discussion.
> >> >>
> >> >> I would need to check, but I guess this is ipv4 and ipv6, which
> >> >> possibly are not clearly indicated. Could this be the case?
> >> >>
> >> >> Rainer
> >> >>
> >> >> El mar., 14 jul. 2020 a las 15:49, Peter Viskup via rsyslog
> >> >> () escribió:
> >> >> >
> >> >> > Hi Ángel,
> >> >> > might be related to the ruleset in input configuration.
> >> >> > Use the Name and Name.appendPort options to specify the name of
> that input
> >> >> > for your ruleset.
> >> >> >
> https://www.rsyslog.com/doc/v8-stable/configuration/modules/imudp.html#name
> >> >> >
> >> >> > The other input reporting stats could be initialized by default
> ruleset
> >> >> > which is always defined.
> >> >> > https://www.rsyslog.com/doc/v8-stable/concepts/multi_ruleset.html
> >> >> >
> >> >> > According to the docu
> >> >> > <
> https://www.rsyslog.com/doc/v8-stable/concepts/multi_ruleset.html#what-does-to-bind-to-a-ruleset-mean
> >
> >> >> > - is the ruleset already defined when input is being initialised?
> >> >> > That might lead to this behaviour.
> >> >> >
> >> >> > Peter
> >> >> >
> >> >> > On Thu, Jul 9, 2020 at 1:43 PM Ángel L. Mateo v

Re: [rsyslog] Behavior during shutdown

[Peter Viskup via rsyslog]

Just had a look on this.
Running rsyslog 8.1901 version we do see this behavior.
Rsyslog receiver restarts, sends the FIN packet, client responds with ACK,
but after a while it sends another message to the already closed socket.
Not sure whether the behavior is correct or not. There is FIN ACK at
09:44:36.597404 and another message sent at 09:44:37.899188.

(tcpdump from client/relay 10.0.0.130, 10.0.0.138 is the receiver)
09:44:19.614994 IP 10.0.0.130.41498 > 10.0.0.138.2514: Flags [P.], seq
7629:7822, ack 1, win 502, options [nop,nop,TS val 2488259817 ecr
1259647639], length 193
.O..K...<30>1 2020-07-20T09:44:19+00:00 loco01-10.0.0.130 snmpd 629 -
[syslogTimes@29171 10.0.0.130="2020-07-20T09:44:19.610564+00:00"]
Connection from UDP: [10.0.0.20]:42624->[10.0.0.130]:161
09:44:19.615009 IP 10.0.0.138.2514 > 10.0.0.130.41498: Flags [.], ack 7822,
win 8190, options [nop,nop,TS val 1259654040 ecr 2488259817], length 0
09:44:36.588199 IP 10.0.0.138.2514 > 10.0.0.130.41498: Flags [F.], seq 1,
ack 7822, win 8190, options [nop,nop,TS val 1259671013 ecr 2488259817],
length 0
09:44:36.597404 IP 10.0.0.130.41498 > 10.0.0.138.2514: Flags [.], ack 2,
win 502, options [nop,nop,TS val 2488276799 ecr 1259671013], length 0
09:44:37.899188 IP 10.0.0.130.41498 > 10.0.0.138.2514: Flags [P.], seq
7822:8024, ack 2, win 502, options [nop,nop,TS val 2488278101 ecr
1259671013], length 202
.P.UK...<30>1 2020-07-20T11:44:37.894029+02:00 services-10.0.0.34 snmpd
1714 - [syslogTimes@29171 10.0.0.130="2020-07-20T09:44:37.894927+00:00"]
Connection from UDP: [10.0.0.20]:47156->[10.0.0.34]:161
09:44:37.899213 IP 10.0.0.138.2514 > 10.0.0.130.41498: Flags [R], seq
3960804835, win 0, length 0

Peter

On Wed, Apr 29, 2020 at 2:30 PM Rainer Gerhards 
wrote:

> oh, that's a good question - maybe I was on the wrong path. I need to
> investigate.
>
> Rainer
>
> El mié., 29 abr. 2020 a las 13:45, Peter Viskup
> () escribió:
> >
> > What's the purpose of  inputs.timeout.shutdown then.
> > Thought it should cover this scenario in a way that the clients will
> have enough time to send the data from buffers before closing the socket.
> > Shouldn't the listener wait the time defined in inputs.timeout.shutdown
> for client's response with FIN,ACK? Would expect that.
> >
> > Peter
> >
> > On Wed, Apr 29, 2020 at 1:03 PM Rainer Gerhards <
> rgerha...@hq.adiscon.com> wrote:
> >>
> >> no, the receiver shuts down as soon as possible. This is intended.
> >> Otherwise you get even longer shutdown times.
> >>
> >> Rainer
> >>
> >> El mié., 29 abr. 2020 a las 13:00, Peter Viskup via rsyslog
> >> () escribió:
> >> >
> >> > Just testing the message forwarding and reliability of plain TCP. Am
> aware
> >> > of the un-reliability of that forwarding.
> >> > See some missing messages after the rsyslog proper process restart on
> >> > destination side.
> >> > Configured simple TCP omfwd action with keepalive enabled.
> >> > On receiving side imptcp input with keepalive enabled and
> >> > global(inputs.timeout.shutdown="1").
> >> > Unfortunately see rsyslog going down and closing listeners too fast -
> not
> >> > accepting the timeout.shutdown value. After that the receiver does
> not wait
> >> > for FIN,ACK from client side and just close the socket. The data being
> >> > flushed from buffer on client side are responded with RST packet.
> Would
> >> > expect the receiver to wait up to 10 seconds to let the clients flush
> the
> >> > data. Have some remote sites with slow link - due to long distance -
> >> > causing the socket sending queue being occupied most of the time.
> >> >
> >> > Do not see this behavior as appropriate. Could anybody review the
> code? Is
> >> > it bug or configuration issue?
> >> >
> >> > Found this code:
> >> >
> https://github.com/rsyslog/rsyslog/blob/69f8e1d1f7fe62fd2c5f38a81d4102a9a62d1722/plugins/imptcp/imptcp.c#L2381
> >> >
> >> > According to the documentation the two shutdown()s can be called
> before
> >> > close(), but are not strictly required.
> >> > Digging a little deeper discovered SO_LINGER is referenced, but with
> value
> >> > of 0
> >> >
> https://github.com/rsyslog/rsyslog/blob/6f74f7e7b43eb32ab165c5975a0fcbbf0f21/runtime/nsd_ptcp.c#L359
> >> > which might be ok as with plain TCP there is no data transferred to
> >> > client from the listener. And the SO_LINGER covers only flush buffered
> >> > output (does not 

Re: [rsyslog] Double imupd stats entries

[Peter Viskup via rsyslog]

Not related to IPv4 vs. IPv6 nor rulesets

On server with IPv6 disabled with only one public iface and IP
root@server:~# sysctl -a |grep .disable_ipv6
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.ens192.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

Related configuration
root@server:~# cat /chroot/net/etc/rsyslog.d/host/net/listeners/*.conf
input(type="imptcp" port="1514" KeepAlive="on")
input(type="imudp" port="1514")
root@server:~# cat /chroot/net/etc/rsyslog.d/global/03-modules.conf
module(load="imudp")
module(load="imptcp")
root@server:~# cat /chroot/net/etc/rsyslog.d/global/00-stats.conf
module(load="impstats"
  interval="15"
  severity="7"
  ResetCounters="on"
  log.syslog="off"
  # need to turn log stream logging off!
  log.file="/var/spool/rsyslog/rsyslog.stats")

We do see similar "doubled" stats:
Fri Jul 17 18:06:46 2020: imudp(*:1514): origin=imudp submitted=1216
disallowed=0
Fri Jul 17 18:06:46 2020: imudp(*:1514): origin=imudp submitted=0
disallowed=0

Running rsyslog 8.1901.0-1 from Debian 10 stable.

With input name configured, the output shows same name in both lines
root@server:~# cat /etc/rsyslog-eset/host/net/listeners/input-udp.conf
input(type="imudp" port="1514" name="udp1514")

Fri Jul 17 18:17:27 2020: udp1514(*:1514): origin=imudp submitted=1354
disallowed=0
Fri Jul 17 18:17:27 2020: udp1514(*:1514): origin=imudp submitted=0
disallowed=0

Looks like a bug being here with us for a long time. :-)

Peter


On Fri, Jul 17, 2020 at 9:19 AM Rainer Gerhards 
wrote:

> El jue., 16 jul. 2020 a las 9:00, Peter Viskup ()
> escribió:
> >
> > Just discovered the same on our infra.
> > Will test by disabling IPV6 and confirm if Ángel will not answer sooner.
>
> Thx - I guess if it is that way, it would make sense to automatically
> append"ipv4" or "v6" to the configured name.
>
> Rainer
> >
> > Peter
> >
> > On Tue, Jul 14, 2020 at 4:02 PM Rainer Gerhards <
> rgerha...@hq.adiscon.com> wrote:
> >>
> >> Sorry for being late to the discussion.
> >>
> >> I would need to check, but I guess this is ipv4 and ipv6, which
> >> possibly are not clearly indicated. Could this be the case?
> >>
> >> Rainer
> >>
> >> El mar., 14 jul. 2020 a las 15:49, Peter Viskup via rsyslog
> >> () escribió:
> >> >
> >> > Hi Ángel,
> >> > might be related to the ruleset in input configuration.
> >> > Use the Name and Name.appendPort options to specify the name of that
> input
> >> > for your ruleset.
> >> >
> https://www.rsyslog.com/doc/v8-stable/configuration/modules/imudp.html#name
> >> >
> >> > The other input reporting stats could be initialized by default
> ruleset
> >> > which is always defined.
> >> > https://www.rsyslog.com/doc/v8-stable/concepts/multi_ruleset.html
> >> >
> >> > According to the docu
> >> > <
> https://www.rsyslog.com/doc/v8-stable/concepts/multi_ruleset.html#what-does-to-bind-to-a-ruleset-mean
> >
> >> > - is the ruleset already defined when input is being initialised?
> >> > That might lead to this behaviour.
> >> >
> >> > Peter
> >> >
> >> > On Thu, Jul 9, 2020 at 1:43 PM Ángel L. Mateo via rsyslog <
> >> > rsyslog@lists.adiscon.com> wrote:
> >> >
> >> > > Hi,
> >> > >
> >> > > I have activated the stats in rsyslog to log to syslog stats
> >> > > entries.
> >> > > My problem is that udp stats are doubled twice.
> >> > >
> >> > > My configuration is:
> >> > >
> >> > > module(load="impstats"
> >> > >interval="60"
> >> > >format="json"
> >> > >
> >> > > )
> >> > > module(load="imudp")
> >> > > input(type="imudp"
> >> > >address="*"
> >> > >port="514"
> >> > >ruleset="remote_udp"
> >> > > )
> >> > > ...
> >> > >
> >> > > I don't have any other udp input.
> >> > >
> >> > > With this configuration, anytime that stats are recorded I
> get:
> >> > >
> >> > > Jul  9 13:35:56 pitufo41 rsyslogd-pstats: { &qu

Re: [rsyslog] Double imupd stats entries

[Peter Viskup via rsyslog]

Just discovered the same on our infra.
Will test by disabling IPV6 and confirm if Ángel will not answer sooner.

Peter

On Tue, Jul 14, 2020 at 4:02 PM Rainer Gerhards 
wrote:

> Sorry for being late to the discussion.
>
> I would need to check, but I guess this is ipv4 and ipv6, which
> possibly are not clearly indicated. Could this be the case?
>
> Rainer
>
> El mar., 14 jul. 2020 a las 15:49, Peter Viskup via rsyslog
> () escribió:
> >
> > Hi Ángel,
> > might be related to the ruleset in input configuration.
> > Use the Name and Name.appendPort options to specify the name of that
> input
> > for your ruleset.
> >
> https://www.rsyslog.com/doc/v8-stable/configuration/modules/imudp.html#name
> >
> > The other input reporting stats could be initialized by default ruleset
> > which is always defined.
> > https://www.rsyslog.com/doc/v8-stable/concepts/multi_ruleset.html
> >
> > According to the docu
> > <
> https://www.rsyslog.com/doc/v8-stable/concepts/multi_ruleset.html#what-does-to-bind-to-a-ruleset-mean
> >
> > - is the ruleset already defined when input is being initialised?
> > That might lead to this behaviour.
> >
> > Peter
> >
> > On Thu, Jul 9, 2020 at 1:43 PM Ángel L. Mateo via rsyslog <
> > rsyslog@lists.adiscon.com> wrote:
> >
> > > Hi,
> > >
> > > I have activated the stats in rsyslog to log to syslog stats
> > > entries.
> > > My problem is that udp stats are doubled twice.
> > >
> > > My configuration is:
> > >
> > > module(load="impstats"
> > >interval="60"
> > >format="json"
> > >
> > > )
> > > module(load="imudp")
> > > input(type="imudp"
> > >address="*"
> > >port="514"
> > >ruleset="remote_udp"
> > > )
> > > ...
> > >
> > > I don't have any other udp input.
> > >
> > > With this configuration, anytime that stats are recorded I get:
> > >
> > > Jul  9 13:35:56 pitufo41 rsyslogd-pstats: { "name": "imudp(*:514)",
> > > "origin": "imudp", "submitted": 64559362, "disallowed": 0 }
> > > Jul  9 13:35:56 pitufo41 rsyslogd-pstats: { "name": "imudp(*:514)",
> > > "origin": "imudp", "submitted": 0, "disallowed": 0 }
> > > Jul  9 13:35:56 pitufo41 rsyslogd-pstats: { "name": "imudp(w0)",
> > > "origin": "imudp", "called.recvmmsg": 42316004, "called.recvmsg": 0,
> > > "msgs.received": 64559362 }
> > >
> > > The imupd(w0) is correctly documented in
> > >
> > >
> https://www.rsyslog.com/doc/v8-stable/configuration/modules/imudp.html#imudp-statistic-counter
> > > as the worker statistics.
> > >
> > > But I don't know I'm getting two records for input
> imudp(*:514).
> > > For
> > > other inputs like tcp or relp (I'm using too) I don't have such
> > > duplicity. For example:
> > >
> > > Jul  9 13:39:58 pitufo31 rsyslogd-pstats: { "name": "imrelp(20514)",
> > > "origin": "imrelp", "submitted": 35531697 }
> > > Jul  9 13:39:58 pitufo31 rsyslogd-pstats: { "name": "imtcp(514)",
> > > "origin": "imtcp", "submitted": 0 }
> > > Jul  9 13:40:58 pitufo31 rsyslogd-pstats: { "name": "imrelp(20514)",
> > > "origin": "imrelp", "submitted": 35619726 }
> > > Jul  9 13:40:58 pitufo31 rsyslogd-pstats: { "name": "imtcp(514)",
> > > "origin": "imtcp", "submitted": 0 }
> > >
> > > I'm running rsyslog 8.2006.0-0adiscon2bionic1.
> > >
> > > Any idea of why this?
> > >
> > > --
> > > Angel L. Mateo Martínez
> > > Sección de Telemática
> > > Área de Tecnologías de la Información
> > > y las Comunicaciones Aplicadas (ATICA)
> > > http://www.um.es/atica
> > > Tfo: 868889150
> > > Fax: 86337
> > > ___
> > > rsyslog mailing list
> > > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com/professional-services/
> > > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad
> > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> > > DON'T LIKE THAT.
> > ___
> > rsyslog mailing list
> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Double imupd stats entries

[Peter Viskup via rsyslog]

Hi Ángel,
might be related to the ruleset in input configuration.
Use the Name and Name.appendPort options to specify the name of that input
for your ruleset.
https://www.rsyslog.com/doc/v8-stable/configuration/modules/imudp.html#name

The other input reporting stats could be initialized by default ruleset
which is always defined.
https://www.rsyslog.com/doc/v8-stable/concepts/multi_ruleset.html

According to the docu

- is the ruleset already defined when input is being initialised?
That might lead to this behaviour.

Peter

On Thu, Jul 9, 2020 at 1:43 PM Ángel L. Mateo via rsyslog <
rsyslog@lists.adiscon.com> wrote:

> Hi,
>
> I have activated the stats in rsyslog to log to syslog stats
> entries.
> My problem is that udp stats are doubled twice.
>
> My configuration is:
>
> module(load="impstats"
>interval="60"
>format="json"
>
> )
> module(load="imudp")
> input(type="imudp"
>address="*"
>port="514"
>ruleset="remote_udp"
> )
> ...
>
> I don't have any other udp input.
>
> With this configuration, anytime that stats are recorded I get:
>
> Jul  9 13:35:56 pitufo41 rsyslogd-pstats: { "name": "imudp(*:514)",
> "origin": "imudp", "submitted": 64559362, "disallowed": 0 }
> Jul  9 13:35:56 pitufo41 rsyslogd-pstats: { "name": "imudp(*:514)",
> "origin": "imudp", "submitted": 0, "disallowed": 0 }
> Jul  9 13:35:56 pitufo41 rsyslogd-pstats: { "name": "imudp(w0)",
> "origin": "imudp", "called.recvmmsg": 42316004, "called.recvmsg": 0,
> "msgs.received": 64559362 }
>
> The imupd(w0) is correctly documented in
>
> https://www.rsyslog.com/doc/v8-stable/configuration/modules/imudp.html#imudp-statistic-counter
> as the worker statistics.
>
> But I don't know I'm getting two records for input imudp(*:514).
> For
> other inputs like tcp or relp (I'm using too) I don't have such
> duplicity. For example:
>
> Jul  9 13:39:58 pitufo31 rsyslogd-pstats: { "name": "imrelp(20514)",
> "origin": "imrelp", "submitted": 35531697 }
> Jul  9 13:39:58 pitufo31 rsyslogd-pstats: { "name": "imtcp(514)",
> "origin": "imtcp", "submitted": 0 }
> Jul  9 13:40:58 pitufo31 rsyslogd-pstats: { "name": "imrelp(20514)",
> "origin": "imrelp", "submitted": 35619726 }
> Jul  9 13:40:58 pitufo31 rsyslogd-pstats: { "name": "imtcp(514)",
> "origin": "imtcp", "submitted": 0 }
>
> I'm running rsyslog 8.2006.0-0adiscon2bionic1.
>
> Any idea of why this?
>
> --
> Angel L. Mateo Martínez
> Sección de Telemática
> Área de Tecnologías de la Información
> y las Comunicaciones Aplicadas (ATICA)
> http://www.um.es/atica
> Tfo: 868889150
> Fax: 86337
> ___
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
___
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog in HA mode

[Peter Viskup via rsyslog]

Hi Ren,
let me share information about our setup:

Basics:
~22 relays, thousands clients worldwide, 3 worker nodes, Linux Virtual
Server managed by heartbeat+ldirectord+home grown script for LVS UDP
monitoring, 3 service_ip x 3 service_port x 2 (tcp+udp) = 18 balancing
services

Limitations:
 - HB does support two node cluster only, third is just the worker (will
upgrade to 3node cluster with peacemaker soon)
 - one LVS instance for all service IPs and instance types (will separate
service IP balancing within peacemaker upgrade)
 - UDP listener needs to use the same port number like TCP listener for LVS
UDP monitoring script to work

Example:
ldirectord.cf snippet for TCP and UDP listeners (they are open by the same
rsyslog instance on worker nodes):
virtual=10.0.0.10:1514
real=10.0.0.1:1514 gate 4
real=10.0.0.2:1514 gate 4
real=10.0.0.3:1514 gate 4
service=none
scheduler=lc
protocol=tcp
checktype=connect

virtual=10.0.0.10:1514
real=10.0.0.1:1514 gate
real=10.0.0.2:1514 gate
real=10.0.0.3:1514 gate
service=none
scheduler=rr
protocol=udp
checktype=external-perl
checkcommand=/usr/local/sbin/ldirector_port_check

The script is attached. Feel free to reuse, the code should be cleaned a
little, but working fine.
Our setup is combination of
http://www.linuxvirtualserver.org/docs/ha/heartbeat_ldirectord.html
setup with DR
http://www.linuxvirtualserver.org/VS-DRouting.html
and we share workers as LB nodes which needs to be properly handled.

Would also recommend reading about the NFTLB as all modern Linux
distributions already use nftables as the default packet handling.
https://www.zevenet.com/knowledge-base/nftlb/what-is-nftlb/
https://github.com/zevenet/nftlb
We plan to focus on the NFTLB setup in near future.

Do not forget to enable TCP KeepAlive on both sides. We do not use
rebindinterval, but that might be good option.
Keep in mind, the server does not send any data to the client except TCP
ACKs. For plain TCP the data flow is one way only, thus consider the
balancing design accordingly. This makes the syslog communication different
to HTTP.

Good luck!

Peter

On Thu, Jul 9, 2020 at 8:25 AM David Lang via rsyslog <
rsyslog@lists.adiscon.com> wrote:

> yep, one thing to keep in mind when making your logs HA is that the fewer
> dependencies you have, the less likely you are to have your logs fail when
> you
> need them to troubleshot why things aren't working :-)
>
> keepalived and corosync/pacemaker both do the job by managing a virtual IP
> on
> the syslog servers themselves, so nothing else (other than the switch)
> needs to
> be working and they can both be set to very fast failovers. They also give
> you
> the ability to get a health check on rsyslog itself (have some source of
> logs
> that is frequent and watch for them to stop arriving). Keepalived is
> simpler,
> Corocync/Pacemaker allows for load sharing and multi-site clusters (only
> send
> alerts from one system, even across datacenters for example)
>
> external load balancers (HAProxy, F5, etc) have a much harder time dealing
> with
> health checks for non-webservers.
>
> David Lang
>
> On Thu, 9 Jul 2020, Benoit DOLEZ via rsyslog wrote:
>
> > HAProxy is a good solution for tcp/relp LB but do not manage HA part of
> > himself nor UDP LB. For HA, it is easier to use keepalived. Add script
> > check to look for rsyslog process. For UDP LB, you can use LVS.
> >
> > Benoit
> >
> > Le 09/07/2020 à 02:46, David Lang via rsyslog a écrit :
> >> on the sending side, enble rebindinterval so that the sender disconnects
> >> periodically to let the load balancer have a chance of doing it's job.
> >>
> >> also be aware that tcp syslog can loose data in a failover (see
> >>
> https://rainer.gerhards.net/2008/04/on-unreliability-of-plain-tcp-syslog.html
> >> ) rsyslog supports the RELP protocol to be reliable in the face of
> >> network failures (relp has one known failure mode that can loose a log,
> >> but so far nobody has cared enough to sponsor a fix for it)
> >>
> >> David Lang
> >> ___
> >> rsyslog mailing list
> >> https://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com/professional-services/
> >> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> >> DON'T LIKE THAT.
> >
> > --
> > Benoit DOLEZ
> > GSM: +33 6 21 05 91 69mailto:bdo...@ant-computing.com
> > ___
> > rsyslog mailing list
> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond 

[rsyslog] Getting audit logs from DB

[Peter Viskup via rsyslog]

What is the best way to get audit logs from SQL DB into syslog using
rsyslog?
Had a look on the input modules
https://www.rsyslog.com/doc/v8-stable/configuration/modules/improg.html
https://www.rsyslog.com/doc/v8-stable/configuration/modules/imbatchreport.html
https://www.rsyslog.com/doc/v8-stable/configuration/modules/imfile.html
Does not have experience with them except the imfile.

Another not-rsyslog related question. What would be the best way for
getting the data from DB?
The solution might support:
 - all/most SQL engines
 - compose data from more tables
 - use templating

Any comments are welcome.

Peter
___
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog does not forward logs to new IP address, when DNS A record is updated

[Peter Viskup via rsyslog]

Just to make a note.

On Sat, May 30, 2020 at 8:05 PM David Lang via rsyslog <
rsyslog@lists.adiscon.com> wrote:

> when you issue a HUP, rsyslog creates new connections.
>

This does not seems to be right. AFAIK the re-opening omfwd connections was
not implemented or am I wrong?

Peter
___
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] lookup table wildcard

[Peter Viskup via rsyslog]

Consider opening feature request on rsyslog's GitHub in that case.

On Mon, Jun 1, 2020 at 9:21 AM Naoum, A. (Alexandros) <
alexandros.na...@nn.cz> wrote:

> Hi Peter,
>
>
>
> My scenario has the index as hostnames and to minimize the total amount of
> entries, a wildcard (in contrast with ‘contains’ if I wasn’t a lookup
> table) would be wonderful.
>
> Property replacer as I read don’t see to fit, unfortunately.
>
>
>
>
>
> Regards,
>
> Alexandros
>
>
>
> *From:* Peter Viskup 
> *Sent:* Saturday, May 30, 2020 8:40 AM
> *To:* rsyslog-users 
> *Cc:* Naoum, A. (Alexandros) 
> *Subject:* Re: [rsyslog] lookup table wildcard
>
>
>
> Hi Alexandros,
>
> no, this is not supported.
>
> https://www.rsyslog.com/doc/master/configuration/lookup_tables.html
> 
>
>
>
> Consider having a look on property replacer with regular expressions
>
> https://www.rsyslog.com/doc/v8-stable/configuration/property_replacer.html
> 
>
>
>
> --
>
> Peter
>
>
>
> On Fri, May 29, 2020 at 5:38 PM Naoum, A. (Alexandros) via rsyslog <
> rsyslog@lists.adiscon.com> wrote:
>
>
> Hello,
>
> I would like to ask, is it possible to have a lookup table with index
> something with wildcard?
>
> Example
>
> { "nomatch" : "none",
>   "type" : "string",
>   "table":[
> {"index" : "foo*", "value" : "bar" },
> {"index" : "barr*", "value" : "quux" }]}
>
>
> Regards,
> Alexandros
>
> This message (including any attachments) may contain confidential
> information. It is intended for use by the recipient only. Any
> dissemination, copying or distribution to third parties without the express
> consent of the sender is strictly prohibited. If you have received this
> message in error, please delete it immediately and notify the sender. Thank
> you for your collaboration.
> ___
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> 
> http://www.rsyslog.com/professional-services/
> 
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> 
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
> This message (including any attachments) may contain confidential
> information. It is intended for use by the recipient only. Any
> dissemination, copying or distribution to third parties without the express
> consent of the sender is strictly prohibited. If you have received this
> message in error, please delete it immediately and notify the sender. Thank
> you for your collaboration.
>
___
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] lookup table wildcard

[Peter Viskup via rsyslog]

Hi Alexandros,
no, this is not supported.
https://www.rsyslog.com/doc/master/configuration/lookup_tables.html

Consider having a look on property replacer with regular expressions
https://www.rsyslog.com/doc/v8-stable/configuration/property_replacer.html

-- 
Peter

On Fri, May 29, 2020 at 5:38 PM Naoum, A. (Alexandros) via rsyslog <
rsyslog@lists.adiscon.com> wrote:

>
> Hello,
>
> I would like to ask, is it possible to have a lookup table with index
> something with wildcard?
>
> Example
>
> { "nomatch" : "none",
>   "type" : "string",
>   "table":[
> {"index" : "foo*", "value" : "bar" },
> {"index" : "barr*", "value" : "quux" }]}
>
>
> Regards,
> Alexandros
>
> This message (including any attachments) may contain confidential
> information. It is intended for use by the recipient only. Any
> dissemination, copying or distribution to third parties without the express
> consent of the sender is strictly prohibited. If you have received this
> message in error, please delete it immediately and notify the sender. Thank
> you for your collaboration.
> ___
> rsyslog mailing list
> https://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] shared name for imptcp and imudp inputs

[Peter Viskup via rsyslog]

On Mon, May 11, 2020 at 12:57 PM Rainer Gerhards 
wrote:

> > Can the imptcp and imudp inputs share the same inputname?
>
> I am not sure if it is checked, but the idea was that they are unique.
> So I wouldn't try it that way...
>
> > Thinking of using the different IPs with same port numbers for
> > listener pairs imptcp+imudp. Need to process messages based on it. One
> > input pair (tcp+udp) will use lookup table, the other one the static
> > variable assignment.
> >
> > Would like to use this config snippet:
>
> How about "startswith"?
>
> e.g. with names bo-tcp and bo-udp
>
>   if ($inputname startswith  'bo') then {
>
>
Make sense, will try it that way.

Rainer
>
> >   if ($inputname == 'bo') then {
> > set $.location = lookup("location", $fromhost-ip);
> >   } else {
> > set $.location = $inputname;
> >   }
> >
> > Peter
>

Thank you.

Peter
___
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] shared name for imptcp and imudp inputs

[Peter Viskup via rsyslog]

Can the imptcp and imudp inputs share the same inputname?
Thinking of using the different IPs with same port numbers for
listener pairs imptcp+imudp. Need to process messages based on it. One
input pair (tcp+udp) will use lookup table, the other one the static
variable assignment.

Would like to use this config snippet:
  if ($inputname == 'bo') then {
set $.location = lookup("location", $fromhost-ip);
  } else {
set $.location = $inputname;
  }

Peter
___
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] omfile thread terminated too quick

[Peter Viskup via rsyslog]

It was related to size of dynaFileCache.
Strange is that the omfile thread was stopped.

Might make sense to update documentation in a way that the consequences of
invalidating cache entry to omfile threading will be more clear. It is not
mentioned at all.

Peter

On Tue, May 5, 2020 at 10:33 AM Peter Viskup  wrote:

> Reported bug for Debian package
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=959774
>
> Following is the evidence of the rotated thread PIDs:
> root@fwd01:~# date; pstree -t -sap 9276
> Tue 05 May 2020 08:05:27 AM UTC
> systemd,1
>   └─rsyslogd-local,9276 -n -f /etc/rsyslog.d/rsyslog-local.conf
>   ├─{in:impstats},9279
>   ├─{in:imptcp},9281
>   ├─{in:imptcp},9282
>   ├─{in:imptcp},9283
>   ├─{in:imudp},9280
>   ├─{rs:ESP01-SURI q},6817
>   ├─{rs:ESP03-WEB qu},9438
>   ├─{rs:SIEMEP2 queu},6818
>   ├─{rs:apache-site-},1001
>   ├─{rs:apache-site-},1002
>   ├─{rs:apache-site-},1003
>   ├─{rs:haproxy.log},729
>   ├─{rs:haproxy.log},992
>   ├─{rs:local-all.lo},999
>   ├─{rs:main Q:Reg},9437
>   ├─{rs:nginx-site-a},914
>   ├─{rs:nginx-site-a},987
>   ├─{rs:nginx-site-e},986
>   ├─{rs:nginx-site-e},989
>   ├─{rsyslogd-local},9277
>   └─{rsyslogd-local},9278
> root@fwd01:~# date; pstree -t -sap 9276
> Tue 05 May 2020 08:05:31 AM UTC
> systemd,1
>   └─rsyslogd-local,9276 -n -f /etc/rsyslog.d/rsyslog-local.conf
>   ├─{in:impstats},9279
>   ├─{in:imptcp},9281
>   ├─{in:imptcp},9282
>   ├─{in:imptcp},9283
>   ├─{in:imudp},9280
>   ├─{rs:ESP01-SURI q},6817
>   ├─{rs:ESP03-WEB qu},9438
>   ├─{rs:SIEMEP2 queu},6818
>   ├─{rs:apache-site-},1281
>   ├─{rs:haproxy.log},1034
>   ├─{rs:haproxy.log},1266
>   ├─{rs:local-all.lo},1277
>   ├─{rs:local-all.lo},1278
>   ├─{rs:main Q:Reg},9437
>   ├─{rs:nginx-site-a},1261
>   ├─{rs:nginx-site-a},1273
>   ├─{rs:nginx-site-e},1223
>   ├─{rs:nginx-site-e},1260
>   ├─{rsyslogd-local},9277
>   ├─{rsyslogd-local},9278
>   └─{rsyslogd-local},1276
>
>
> And stable list of thread PIDs after rsyslog restart
>
> root@fwd01:~# date; pstree -t -sap 12417
> Tue 05 May 2020 08:15:06 AM UTC
> systemd,1
>   └─rsyslogd-local,12417 -n -f /etc/rsyslog.d/rsyslog-local.conf
>   ├─{in:impstats},12420
>   ├─{in:imptcp},12422
>   ├─{in:imptcp},12423
>   ├─{in:imptcp},12424
>   ├─{in:imudp},12421
>   ├─{rs:ESP03-GWS qu},14096
>   ├─{rs:ESP03-WAF qu},14094
>   ├─{rs:ESP03-WEB qu},14092
>   ├─{rs:local-all.lo},14089
>   ├─{rs:local-all.lo},14090
>   ├─{rs:local-all.lo},14091
>   ├─{rs:main Q:Reg},14088
>   ├─{rs:nginx-server},14097
>   ├─{rs:nginx-site-a},14093
>   ├─{rs:nginx-site-e},14095
>   ├─{rsyslogd-local},12418
>   └─{rsyslogd-local},12419
>  root@fwd01:~# date; pstree -t -sap 12417
> Tue 05 May 2020 08:15:36 AM UTC
> systemd,1
>   └─rsyslogd-local,12417 -n -f /etc/rsyslog.d/rsyslog-local.conf
>   ├─{in:impstats},12420
>   ├─{in:imptcp},12422
>   ├─{in:imptcp},12423
>   ├─{in:imptcp},12424
>   ├─{in:imudp},12421
>   ├─{rs:ESP03-GWS qu},14096
>   ├─{rs:ESP03-WAF qu},14094
>   ├─{rs:ESP03-WEB qu},14092
>   ├─{rs:local-all.lo},14089
>   ├─{rs:local-all.lo},14090
>   ├─{rs:local-all.lo},14091
>   ├─{rs:main Q:Reg},14088
>   ├─{rs:nginx-server},14097
>   ├─{rs:nginx-site-a},14093
>   ├─{rs:nginx-site-e},14095
>   ├─{rsyslogd-local},12418
>   └─{rsyslogd-local},12419
>
> Peter
>
> On Mon, May 4, 2020 at 5:28 PM Peter Viskup  wrote:
>
>> For some weeks there are a lot of closing logfile notification via
>> inotify seen on one syslog relay running rsyslog 8.1901 version.
>>
>> The messages like these
>>
>> May  4 15:10:04 fwd01 iWatch[31831]: *
>> /chroot/local/var/log/h1/local-all.log is closed
>> May  4 15:10:04 fwd01 iWatch[31831]: *
>> /chroot/local/var/log/h3/haproxy.log is closed
>> May  4 15:10:04 fwd01 iWatch[31831]: *
>> /chroot/local/var/log/h1/haproxy.log is closed
>> May  4 15:10:04 fwd01 iWatch[31831]: *
>> /chroot/local/var/log/h5/nginx-site-access.log is closed
>> May  4 15:10:04 fwd01 iWatch[31831]: *
>> /chroot/local/var/log/h1/apache-site-error.log is closed
>> May  4 15:10:04 fwd01 iWatch[31831]: *
>> /chroot/local/var/log/h1/apache-site-access.log is closed
>>
>> are seen. With simple check of top -p PID I see that the thread PIDs of
>> omfile are changing more times in second. Seems the logfiles are just
>> updated. After restarting the instance, the situation is solved. The issue
>> is maybe caused by logrotation job.
>>
>> Seems this bug is hit
>> https://github.com/rsyslog/rsyslog/blob/master/ChangeLog#L533
>>
>> Is there any way to prove it somehow?
>> In that case would like to open bug report at Debian to make an patch
>> backport to 8.1901 version which is Debian10 base.
>>
>> Peter
>>
>
___
rsyslog mailing list

Re: [rsyslog] omfile thread terminated too quick

[Peter Viskup via rsyslog]

Reported bug for Debian package
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=959774

Following is the evidence of the rotated thread PIDs:
root@fwd01:~# date; pstree -t -sap 9276
Tue 05 May 2020 08:05:27 AM UTC
systemd,1
  └─rsyslogd-local,9276 -n -f /etc/rsyslog.d/rsyslog-local.conf
  ├─{in:impstats},9279
  ├─{in:imptcp},9281
  ├─{in:imptcp},9282
  ├─{in:imptcp},9283
  ├─{in:imudp},9280
  ├─{rs:ESP01-SURI q},6817
  ├─{rs:ESP03-WEB qu},9438
  ├─{rs:SIEMEP2 queu},6818
  ├─{rs:apache-site-},1001
  ├─{rs:apache-site-},1002
  ├─{rs:apache-site-},1003
  ├─{rs:haproxy.log},729
  ├─{rs:haproxy.log},992
  ├─{rs:local-all.lo},999
  ├─{rs:main Q:Reg},9437
  ├─{rs:nginx-site-a},914
  ├─{rs:nginx-site-a},987
  ├─{rs:nginx-site-e},986
  ├─{rs:nginx-site-e},989
  ├─{rsyslogd-local},9277
  └─{rsyslogd-local},9278
root@fwd01:~# date; pstree -t -sap 9276
Tue 05 May 2020 08:05:31 AM UTC
systemd,1
  └─rsyslogd-local,9276 -n -f /etc/rsyslog.d/rsyslog-local.conf
  ├─{in:impstats},9279
  ├─{in:imptcp},9281
  ├─{in:imptcp},9282
  ├─{in:imptcp},9283
  ├─{in:imudp},9280
  ├─{rs:ESP01-SURI q},6817
  ├─{rs:ESP03-WEB qu},9438
  ├─{rs:SIEMEP2 queu},6818
  ├─{rs:apache-site-},1281
  ├─{rs:haproxy.log},1034
  ├─{rs:haproxy.log},1266
  ├─{rs:local-all.lo},1277
  ├─{rs:local-all.lo},1278
  ├─{rs:main Q:Reg},9437
  ├─{rs:nginx-site-a},1261
  ├─{rs:nginx-site-a},1273
  ├─{rs:nginx-site-e},1223
  ├─{rs:nginx-site-e},1260
  ├─{rsyslogd-local},9277
  ├─{rsyslogd-local},9278
  └─{rsyslogd-local},1276


And stable list of thread PIDs after rsyslog restart

root@fwd01:~# date; pstree -t -sap 12417
Tue 05 May 2020 08:15:06 AM UTC
systemd,1
  └─rsyslogd-local,12417 -n -f /etc/rsyslog.d/rsyslog-local.conf
  ├─{in:impstats},12420
  ├─{in:imptcp},12422
  ├─{in:imptcp},12423
  ├─{in:imptcp},12424
  ├─{in:imudp},12421
  ├─{rs:ESP03-GWS qu},14096
  ├─{rs:ESP03-WAF qu},14094
  ├─{rs:ESP03-WEB qu},14092
  ├─{rs:local-all.lo},14089
  ├─{rs:local-all.lo},14090
  ├─{rs:local-all.lo},14091
  ├─{rs:main Q:Reg},14088
  ├─{rs:nginx-server},14097
  ├─{rs:nginx-site-a},14093
  ├─{rs:nginx-site-e},14095
  ├─{rsyslogd-local},12418
  └─{rsyslogd-local},12419
 root@fwd01:~# date; pstree -t -sap 12417
Tue 05 May 2020 08:15:36 AM UTC
systemd,1
  └─rsyslogd-local,12417 -n -f /etc/rsyslog.d/rsyslog-local.conf
  ├─{in:impstats},12420
  ├─{in:imptcp},12422
  ├─{in:imptcp},12423
  ├─{in:imptcp},12424
  ├─{in:imudp},12421
  ├─{rs:ESP03-GWS qu},14096
  ├─{rs:ESP03-WAF qu},14094
  ├─{rs:ESP03-WEB qu},14092
  ├─{rs:local-all.lo},14089
  ├─{rs:local-all.lo},14090
  ├─{rs:local-all.lo},14091
  ├─{rs:main Q:Reg},14088
  ├─{rs:nginx-server},14097
  ├─{rs:nginx-site-a},14093
  ├─{rs:nginx-site-e},14095
  ├─{rsyslogd-local},12418
  └─{rsyslogd-local},12419

Peter

On Mon, May 4, 2020 at 5:28 PM Peter Viskup  wrote:

> For some weeks there are a lot of closing logfile notification via inotify
> seen on one syslog relay running rsyslog 8.1901 version.
>
> The messages like these
>
> May  4 15:10:04 fwd01 iWatch[31831]: *
> /chroot/local/var/log/h1/local-all.log is closed
> May  4 15:10:04 fwd01 iWatch[31831]: *
> /chroot/local/var/log/h3/haproxy.log is closed
> May  4 15:10:04 fwd01 iWatch[31831]: *
> /chroot/local/var/log/h1/haproxy.log is closed
> May  4 15:10:04 fwd01 iWatch[31831]: *
> /chroot/local/var/log/h5/nginx-site-access.log is closed
> May  4 15:10:04 fwd01 iWatch[31831]: *
> /chroot/local/var/log/h1/apache-site-error.log is closed
> May  4 15:10:04 fwd01 iWatch[31831]: *
> /chroot/local/var/log/h1/apache-site-access.log is closed
>
> are seen. With simple check of top -p PID I see that the thread PIDs of
> omfile are changing more times in second. Seems the logfiles are just
> updated. After restarting the instance, the situation is solved. The issue
> is maybe caused by logrotation job.
>
> Seems this bug is hit
> https://github.com/rsyslog/rsyslog/blob/master/ChangeLog#L533
>
> Is there any way to prove it somehow?
> In that case would like to open bug report at Debian to make an patch
> backport to 8.1901 version which is Debian10 base.
>
> Peter
>
___
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] omfile thread terminated too quick

[Peter Viskup via rsyslog]

For some weeks there are a lot of closing logfile notification via inotify
seen on one syslog relay running rsyslog 8.1901 version.

The messages like these

May  4 15:10:04 fwd01 iWatch[31831]: *
/chroot/local/var/log/h1/local-all.log is closed
May  4 15:10:04 fwd01 iWatch[31831]: * /chroot/local/var/log/h3/haproxy.log
is closed
May  4 15:10:04 fwd01 iWatch[31831]: * /chroot/local/var/log/h1/haproxy.log
is closed
May  4 15:10:04 fwd01 iWatch[31831]: *
/chroot/local/var/log/h5/nginx-site-access.log is closed
May  4 15:10:04 fwd01 iWatch[31831]: *
/chroot/local/var/log/h1/apache-site-error.log is closed
May  4 15:10:04 fwd01 iWatch[31831]: *
/chroot/local/var/log/h1/apache-site-access.log is closed

are seen. With simple check of top -p PID I see that the thread PIDs of
omfile are changing more times in second. Seems the logfiles are just
updated. After restarting the instance, the situation is solved. The issue
is maybe caused by logrotation job.

Seems this bug is hit
https://github.com/rsyslog/rsyslog/blob/master/ChangeLog#L533

Is there any way to prove it somehow?
In that case would like to open bug report at Debian to make an patch
backport to 8.1901 version which is Debian10 base.

Peter
___
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Behavior during shutdown

[Peter Viskup via rsyslog]

What's the purpose of  inputs.timeout.shutdown then.
Thought it should cover this scenario in a way that the clients will have
enough time to send the data from buffers before closing the socket.
Shouldn't the listener wait the time defined in inputs.timeout.shutdown for
client's response with FIN,ACK? Would expect that.

Peter

On Wed, Apr 29, 2020 at 1:03 PM Rainer Gerhards 
wrote:

> no, the receiver shuts down as soon as possible. This is intended.
> Otherwise you get even longer shutdown times.
>
> Rainer
>
> El mié., 29 abr. 2020 a las 13:00, Peter Viskup via rsyslog
> () escribió:
> >
> > Just testing the message forwarding and reliability of plain TCP. Am
> aware
> > of the un-reliability of that forwarding.
> > See some missing messages after the rsyslog proper process restart on
> > destination side.
> > Configured simple TCP omfwd action with keepalive enabled.
> > On receiving side imptcp input with keepalive enabled and
> > global(inputs.timeout.shutdown="1").
> > Unfortunately see rsyslog going down and closing listeners too fast - not
> > accepting the timeout.shutdown value. After that the receiver does not
> wait
> > for FIN,ACK from client side and just close the socket. The data being
> > flushed from buffer on client side are responded with RST packet. Would
> > expect the receiver to wait up to 10 seconds to let the clients flush the
> > data. Have some remote sites with slow link - due to long distance -
> > causing the socket sending queue being occupied most of the time.
> >
> > Do not see this behavior as appropriate. Could anybody review the code?
> Is
> > it bug or configuration issue?
> >
> > Found this code:
> >
> https://github.com/rsyslog/rsyslog/blob/69f8e1d1f7fe62fd2c5f38a81d4102a9a62d1722/plugins/imptcp/imptcp.c#L2381
> >
> > According to the documentation the two shutdown()s can be called before
> > close(), but are not strictly required.
> > Digging a little deeper discovered SO_LINGER is referenced, but with
> value
> > of 0
> >
> https://github.com/rsyslog/rsyslog/blob/6f74f7e7b43eb32ab165c5975a0fcbbf0f21/runtime/nsd_ptcp.c#L359
> > which might be ok as with plain TCP there is no data transferred to
> > client from the listener. And the SO_LINGER covers only flush buffered
> > output (does not wait for incoming data)
> >
> https://www.gnu.org/software/libc/manual/html_node/Opening-and-Closing-Files.html#Opening-and-Closing-Files
> >
> > Was not able to find the LINGERing on client side code. Traced socket
> > handling in omfwd
> >
> https://github.com/rsyslog/rsyslog/blob/1f8f621a97df6b1989e1aebd8cb15cd6a552fa9c/tools/omfwd.c
> > was able to find the abort data in netstrm driver
> >
> https://github.com/rsyslog/rsyslog/blob/6f74f7e7b43eb32ab165c5975a0fcbbf0f21/runtime/netstrm.c#L83
> > which seems related.
> > Hopefully from the tcpdump that part of the code seems to be working as
> it
> > is seen the client is trying to flush the data, all of which are
> responded
> > with RST packet.
> > Both sides are running Debian10 with Debian's rsyslog 8.1901.0-1.
> >
> > Any help to sort this out is appreciated.
> >
> > Peter
> > ___
> > rsyslog mailing list
> > https://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] Behavior during shutdown

[Peter Viskup via rsyslog]

Just testing the message forwarding and reliability of plain TCP. Am aware
of the un-reliability of that forwarding.
See some missing messages after the rsyslog proper process restart on
destination side.
Configured simple TCP omfwd action with keepalive enabled.
On receiving side imptcp input with keepalive enabled and
global(inputs.timeout.shutdown="1").
Unfortunately see rsyslog going down and closing listeners too fast - not
accepting the timeout.shutdown value. After that the receiver does not wait
for FIN,ACK from client side and just close the socket. The data being
flushed from buffer on client side are responded with RST packet. Would
expect the receiver to wait up to 10 seconds to let the clients flush the
data. Have some remote sites with slow link - due to long distance -
causing the socket sending queue being occupied most of the time.

Do not see this behavior as appropriate. Could anybody review the code? Is
it bug or configuration issue?

Found this code:
https://github.com/rsyslog/rsyslog/blob/69f8e1d1f7fe62fd2c5f38a81d4102a9a62d1722/plugins/imptcp/imptcp.c#L2381

According to the documentation the two shutdown()s can be called before
close(), but are not strictly required.
Digging a little deeper discovered SO_LINGER is referenced, but with value
of 0
https://github.com/rsyslog/rsyslog/blob/6f74f7e7b43eb32ab165c5975a0fcbbf0f21/runtime/nsd_ptcp.c#L359
which might be ok as with plain TCP there is no data transferred to
client from the listener. And the SO_LINGER covers only flush buffered
output (does not wait for incoming data)
https://www.gnu.org/software/libc/manual/html_node/Opening-and-Closing-Files.html#Opening-and-Closing-Files

Was not able to find the LINGERing on client side code. Traced socket
handling in omfwd
https://github.com/rsyslog/rsyslog/blob/1f8f621a97df6b1989e1aebd8cb15cd6a552fa9c/tools/omfwd.c
was able to find the abort data in netstrm driver
https://github.com/rsyslog/rsyslog/blob/6f74f7e7b43eb32ab165c5975a0fcbbf0f21/runtime/netstrm.c#L83
which seems related.
Hopefully from the tcpdump that part of the code seems to be working as it
is seen the client is trying to flush the data, all of which are responded
with RST packet.
Both sides are running Debian10 with Debian's rsyslog 8.1901.0-1.

Any help to sort this out is appreciated.

Peter
___
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog's programname

[Peter Viskup via rsyslog]

[Replying with mailing list address in recipients.]
Thank you, Rainer, for quick answer.

On Wed, Feb 12, 2020 at 3:31 PM Rainer Gerhards 
wrote:

> El mié., 12 feb. 2020 a las 15:26, Peter Viskup via rsyslog
> () escribió:
> >
> > In other case it seems those internal rsyslog messages are duplicite
> (once
> > logged with full syslog-tag with PID and secondly with msg only). Is
> there
> > some way to supress duplicite logging of these messages?
>
> May this be config induced?  If not, a debug log would be useful.
>

There are instances in chroot having disabled processing of internal
messages (they should log to /dev/log inside the chroot - and they
definitely do).
~# cat /chroot/lin/etc/rsyslog.d/global/01-global.conf
global(workDirectory="/var/spool/rsyslog")
global(action.reportSuspensionContinuation="on")
global(processInternalMessages="off")
global(abortOnUncleanConfig="on")
$EscapeControlCharactersOnReceive off

And the standard operating system instance running on the host is reading
the /dev/log sockets in chroots:
~# cat /etc/rsyslog.d/chroot-lin-imuxsock.conf
input(type="imuxsock"
  Socket="/chroot/lin/dev/log"
  CreatePath="on")

Do not see any other configuration options related to that message
processing.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] rsyslog's programname

[Peter Viskup via rsyslog]

Is there way to configure rsyslog instance to use its own programname?
For example rsyslog-net or rsyslog-lin for appropriate instances which have
different listen ports open.
As those usually run on the same host, the error messages are logged under
"rsyslog" and it is hard to decide what message is from which instance.
One way for dealing could be to create link /usr/sbin/rsyslogd-lin pointing
to /usr/sbin/rsyslogd and starting instance with calling this link. That
will result in rsyslogd-lin being used as programname. Is there other way
for dealing with it in rsyslog configuration only?

In other case it seems those internal rsyslog messages are duplicite (once
logged with full syslog-tag with PID and secondly with msg only). Is there
some way to supress duplicite logging of these messages?

Feb 12 12:12:13 syslog01 rsyslogd[10891]: rsyslogd: imptcp imptcp: message
received is at least 1536 byte larger than max msg size; message will be
split starting at: "1011322,"UserFingerprint":null,""  [v8.1901.0]
Feb 12 12:12:13 syslog01 rsyslogd: imptcp imptcp: message received is at
least 1536 byte larger than max msg size; message will be split starting
at: "1011322,"UserFingerprint":null,""  [v8.1901.0]

Thank you.

-- 
Peter
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] recovery.qi.pl update

[Peter Viskup via rsyslog]

Let me share the patch for recovery.qi.pl script with you.
It does automatically create $basename.qi file (no STDOUT redirection
required) and initiate $digits and $spool with defaults (they are optional).
One of other improvement is the queue files are reordered when broken queue
is detected.

In that case the call is just
/var/spool/rsyslog# /usr/local/sbin/recover.qi2.pl -f forward_queue
instead of
/var/spool/rsyslog# /usr/local/sbin/recover.qi2.pl -f forward_queue -d 8 -w
. >  forward_queue.qi

One of the planned improvement could be change of ownership and file mode
according to the queue files.

Peter


recover.qi.pl.patch
Description: Binary data
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Hostname field in 5424 header Parsing

[Peter Viskup via rsyslog]

Hi Harish,
good for reading and understanding
https://en.wikipedia.org/wiki/Hostname
https://tools.ietf.org/html/rfc5424#section-6.2.4
https://tools.ietf.org/html/rfc3164#section-4.1.2

On Tue, Jan 28, 2020 at 9:01 AM Harish Patil via rsyslog <
rsyslog@lists.adiscon.com> wrote:

> Ok, thanks for the clarification.
>
> On Mon, Jan 27, 2020 at 11:58 PM Rainer Gerhards  >
> wrote:
>
> > no, hostnames cannot contain spaces. It's against all RFCs.
> >
> > HTH
> > Rainer
> >
> > El lun., 27 ene. 2020 a las 23:40, Harish Patil via rsyslog
> > () escribió:
> > >
> > > Hi,
> > > What is the parsing logic of hostname field with
> > > RSYSLOG_SyslogProtocol23Format?
> > > Can the hostname contains spaces?
> > > Pls let me know.
> > >
> > > Thanks,
> > > Harish
> > > ___
> > > rsyslog mailing list
> > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > > http://www.rsyslog.com/professional-services/
> > > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad
> > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> > DON'T LIKE THAT.
> >
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] Debugging rsyslog segfault

[Peter Viskup via rsyslog]

Experience regular segfaults on one rsyslog 8.15 instance. I know it is old
version, but still would like to trace it as am not able to upgrade ATM.
Seems it is caused by writing some message to DA cache (or by reading it
from).

Would it be possible to find it in debug log (already got it)? What should
I look for?

Error message:
Jan 24 08:54:25 hostname kernel: [38302481.667884] rs:FWD queue[DA[6533]:
segfault at 20 ip 560e6b9c834c sp 7f877ae83a40 error 4 in
rsyslogd-lin[560e6b9a2000+9]

Thank you.

-- 
Peter
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] string match filter 'contains' vs. '=='

[Peter Viskup via rsyslog]

Running rsyslog 8.1901.0-1 and it seems there is some difference in
processing these two filters.

On the input there is message which is parsed with hostname property set to
the IP address exactly. The match with use of 'contains' is not effective,
while '==' is.
Is this expected result?

Message example (message is forwarded):
<133>1 2019-12-13T14:57:36.227429+01:00 10.1.2.5  - - 2019 Dec 13 13:57:36
UTC: %AUTHPRIV-5-SYSTEM_MSG: root : TTY=unknown ;
PWD=/var/sysmgr/sysmgr-subproc ; USER=root ; COMMAND=/sbin/sysctl -q -w
vm.drop_caches=3 - sudo
# with debug
Debug line with all properties:
FROMHOST: '10.1.2.3', fromhost-ip: '10.1.2.3', HOSTNAME: '10.1.2.5', PRI:
133,
syslogtag '', programname: '', APP-NAME: '', PROCID: '-', MSGID: '-',

filters:
# does not work
if $hostname contains ['10.1.2.4', '10.1.2.5'] then
# does work
if $hostname contains ['10.1.2.4', '10.1.2.5'] or $hostname == '10.1.2.5'
then

Following issue is experienced on this message samples. Other properly
formatted messages from 10.1.2.4 are matched with no issues. There are no
other type of messages coming from 10.1.2.5 address.

Peter
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] MainQ workerthreads not effective

[Peter Viskup via rsyslog]

There is not much in configuration:
 - impstats module configuration
 - one imtcp input (one udp input - not used atm)
 - main queue configuration
 - two omfwd:tcp forwards with DA queues
 - local file store
There is some msg regex parsing and manipulation which might look strange
and complicated, and maybe could be simplified one day.

How could/might the multithreading of main queue help to improve
performance?
Thought the rsyslog's 'main queue preprocessor' will spread the messages
from imtcp input between two main queue threads proportionally. (according
the data flow diagram on
https://www.rsyslog.com/doc/v8-stable/whitepapers/queues_analogy.html).
Also thinking that the main queue thread itself is responsible for the
message parsing and manipulation.

Attached is the most of the configuration. It is divided into more files,
thus merged them in order into one file.

Peter

On Thu, Nov 28, 2019 at 2:52 PM David Lang  wrote:

> first off, adding additional threads is probably not going to help, and
> can
> actually hurt (locking contention between the threads)
>
> I would look at increasing the batch size before adding additional threads.
>
> It's hard to know what's going on without seeing your config.
>
> David Lang
>
> On Thu, 28 Nov 2019, Peter Viskup via rsyslog wrote:
>
> > Date: Thu, 28 Nov 2019 09:22:43 +0100
> > From: Peter Viskup via rsyslog 
> > To: rsyslog-users 
> > Cc: Peter Viskup 
> > Subject: [rsyslog] MainQ workerthreads not effective
> >
> > Experiencing high load on some rsyslog instances.
> > Status of threads showed the mainQ thread consumed 50-100% CPU.
> > Change of queue.workerthreads to 2 enabled the second workerthread, but
> > this does not consume any CPU.
> > How are the workerthreads for main queue loaded?
> >
> > Running on Debian 10 with rsyslog 8.1901.0-1.
> >
> > Top output:
> >
> > top - 08:12:50 up 6 days, 17:12,  2 users,  load average: 1.25, 0.50,
> 0.46
> > Threads:  14 total,   0 running,  14 sleeping,   0 stopped,   0 zombie
> > %Cpu(s): 14.0 us,  2.9 sy,  0.0 ni, 81.5 id,  0.2 wa,  0.0 hi,  1.4 si,
> > 0.0 st
> > MiB Mem :   7973.2 total,119.5 free,341.9 used,   7511.9
> buff/cache
> > MiB Swap:952.0 total,825.7 free,126.2 used.   7273.8 avail
> Mem
> >
> >  PID USER  PR  NIVIRTRESSHR S  %CPU  %MEM TIME+
> COMMAND
> >
> > *30437 lognet20   0  815648  28612   6320 S  66.1   0.4   0:46.40
> > rs:main Q:Reg*30438 lognet20   0  815648  28612   6320 S   7.0   0.4
> > 0:05.58 rs:ESP02 queue:
> > 30439 lognet20   0  815648  28612   6320 S   6.3   0.4   0:04.58
> > rs:SIEMEP1 queu
> > 28705 lognet20   0  815648  28612   6320 S   6.0   0.4   0:04.31
> > in:imtcp
> > 30440 lognet20   0  815648  28612   6320 S   1.0   0.4   0:00.40
> > rs:net-all.log
> > 30461 lognet20   0  815648  28612   6320 S   1.0   0.4   0:00.56
> > rs:ESP01-IPS qu
> > .
> > *30507 lognet20   0  815648  28612   6320 S   0.0   0.4   0:00.83
> > rs:main Q:Reg*
> >
> > Peter
> > ___
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
> >
>
module(load="impstats"
  interval="15"
  severity="7"
  log.syslog="off"
  # need to turn log stream logging off!
  log.file="/var/spool/rsyslog/rsyslog.stats")
global(workDirectory="/var/spool/rsyslog")
global(action.reportSuspensionContinuation="on")
global(processInternalMessages="off")
global(abortOnUncleanConfig="on")
$EscapeControlCharactersOnReceive off
$Umask 0027
$FileCreateMode 0640
$DirCreateMode 0750
$CreateDirs off
module(load="imudp")
module(load="imtcp")
# set local variable .localip
$IncludeConfig /etc/rsyslog.d/host/global/includes/localip.inc

$template 
getOrigIpOnly,"%hostname:R,ERE,1,BLANK:(^[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})$--end%"
$template 
getOrigHostOnly,"%hostname:R,ERE,1,BLANK:(^[a-zA-Z]+[a-zA-Z0-9-]+[a-zA-Z0-9.-]*)$--end%"
$template 
getOrigHost,"%hostname:R,ERE,1,BLANK:(^[a-zA-Z]+[a-zA-Z0-9-]+[a-zA-Z0-9.-]*)-([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})$--end%"
$template 
getOrigIp,"%hostname:R,ERE,2,BLANK:(^[a-zA-Z]+[a-zA-Z0-9-]+[a-zA-Z0-9.-]*)-([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})$--end%"

[rsyslog] MainQ workerthreads not effective

[Peter Viskup via rsyslog]

Experiencing high load on some rsyslog instances.
Status of threads showed the mainQ thread consumed 50-100% CPU.
Change of queue.workerthreads to 2 enabled the second workerthread, but
this does not consume any CPU.
How are the workerthreads for main queue loaded?

Running on Debian 10 with rsyslog 8.1901.0-1.

Top output:

top - 08:12:50 up 6 days, 17:12,  2 users,  load average: 1.25, 0.50, 0.46
Threads:  14 total,   0 running,  14 sleeping,   0 stopped,   0 zombie
%Cpu(s): 14.0 us,  2.9 sy,  0.0 ni, 81.5 id,  0.2 wa,  0.0 hi,  1.4 si,
 0.0 st
MiB Mem :   7973.2 total,119.5 free,341.9 used,   7511.9 buff/cache
MiB Swap:952.0 total,825.7 free,126.2 used.   7273.8 avail Mem

  PID USER  PR  NIVIRTRESSHR S  %CPU  %MEM TIME+ COMMAND

*30437 lognet20   0  815648  28612   6320 S  66.1   0.4   0:46.40
rs:main Q:Reg*30438 lognet20   0  815648  28612   6320 S   7.0   0.4
0:05.58 rs:ESP02 queue:
30439 lognet20   0  815648  28612   6320 S   6.3   0.4   0:04.58
rs:SIEMEP1 queu
28705 lognet20   0  815648  28612   6320 S   6.0   0.4   0:04.31
in:imtcp
30440 lognet20   0  815648  28612   6320 S   1.0   0.4   0:00.40
rs:net-all.log
30461 lognet20   0  815648  28612   6320 S   1.0   0.4   0:00.56
rs:ESP01-IPS qu
.
*30507 lognet20   0  815648  28612   6320 S   0.0   0.4   0:00.83
rs:main Q:Reg*

Peter
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Hostname resolution updates (remote logging) not picked up

[Peter Viskup via rsyslog]

We had a little discussion about TCP reopening (which might include name
resolution) in following bug report (Reopen TCP sockets on HUP signal).
https://github.com/rsyslog/rsyslog/issues/3683

The outcome is to use rebindinterval omfwd config option which makes the
same, but cannot be enforced by user.

The name resolution takes effect within establishing the TCP connection.
Once established, there is no easy way to propagate the change to the
application. The POSIX name resolver just do not count with DNS TTL.
https://curl.haxx.se/mail/lib-2017-06/0022.html

-- 
Peter

On Tue, Oct 15, 2019 at 8:57 PM David Lang via rsyslog <
rsyslog@lists.adiscon.com> wrote:

> so there are a few things going on here
>
> with rsyslog, a HUP signal does not reload the config, it closes outputs
> to
> support log rotation
>
> I'm not sure if it closes network connections or not, it not there is not
> going
> to be a name lookup, I'm not sure if a name lookup would happen anyway as
> I
> think rsyslog is farily aggressive in caching the results of a lookup.
>
> We have the config option to close and re-open the connections after every
> X
> messages so that any load balancing can take place.
>
> David Lang
>
>
> On Tue, 15 Oct 2019, Adam Chalkley via rsyslog wrote:
>
> > Date: Tue, 15 Oct 2019 17:35:02 +
> > From: Adam Chalkley via rsyslog 
> > To: rsyslog-users 
> > Cc: Adam Chalkley 
> > Subject: Re: [rsyslog] Hostname resolution updates (remote logging) not
> picked
> >  up
> >
> > FWIW, we use FQDN to forward messages, but whenever our campus DNS
> servers experience issues our clients will backup and Nagios will start
> screaming about stuck items in the forward queue.
> >
> > IP Address appear to make a more resilient forwarding target.
> >
> > In our case we had good success with migrating a central receiver
> between subnets (i.e., IP change) and the clients picked up the change. I
> don't know whether this is because the receiver was down for a sufficient
> amount of time to force disconnect/reconnect behavior on the clients or if
> it's because we used the newer configuration format where you configure
> forwarding as an "action". To further stir mud in the water we are also
> using RELP, so that could have a bearing.
> >
> > I recall seeing on the list somewhere some discussion about
> load-balancers and how forced disconnections can be used to switch targets.
> I might be thinking of forwarding into elasticsearch, so take that for what
> it's worth.
> >
> > -Original Message-
> > From: rsyslog  On Behalf Of Marki
> via rsyslog
> > Sent: Tuesday, October 15, 2019 11:31 AM
> > To: rsyslog@lists.adiscon.com
> > Cc: Marki 
> > Subject: [rsyslog] Hostname resolution updates (remote logging) not
> picked up
> >
> >
> > Hey,
> >
> > When using remote logging (*.* @syslog.example.com) "syslog" is an alias
> > (CNAME with low TTL) in our DNS, like all service names.
> >
> > Now it seems when we change this alias' destination in DNS, the change
> > is never picked up. Not even on reload, only on restart. On reload would
> > at least make it use the new IP address after logrotation for example.
> >
> > I don't even think it's about rsyslog. Seems to be how all syslog
> > implementations usually behave. But it is still a topic of discussion:
> >
> > Are people just not using hostnames? I understand that for example on
> > network equipment you would rather hardcode IPs than use hostnames. But
> > what do you do on the servers?
> >
> > Is there a best practice with valid reasons why it should be done that
> > way? What do you think?
> >
> > Cheers.
> > ___
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
> > ___
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
> >
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog

[rsyslog] imptcp maximum TCP sessions

[Peter Viskup via rsyslog]

What is the limit of TCP sessions the imptcp can handle?
There is no option like MaxSessions of imtcp. Was not able to find the
information in documentation.

Discovered code which might point to that limit, but do not understand it.
https://github.com/rsyslog/rsyslog/blob/master/plugins/imptcp/imptcp.c#L532

Does it rely on nofile ulimit only?

-- 
Peter
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] replace carriage return

[Peter Viskup via rsyslog]

What should be the best way to handle carriage return character on the end
of message?

Without setting the $EscapeControlCharactersOnReceive to off, the messages
end with #015 and are also forwarded that way.
With setting $EscapeControlCharactersOnReceive to off the messages are
forwarded with \r on the end, but this also affects whole message
processing.

Is there any other way to deal with this? Thought about possibility to
remove the character by regular expression, which might raise performance
issues.

-- 
Peter
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Issue with Disk Assisted queues

[Peter Viskup via rsyslog]

Hi Malhar,
try to enable impstats [1] which will provide you the evidence of the
rsyslog runtime statistics and queue sizes. Also read about the rsyslog
queues [2][3] a little.
That might help you to understand the queuing in rsyslog.

[1] https://www.rsyslog.com/how-to-use-impstats/
[2] https://www.rsyslog.com/doc/v8-stable/whitepapers/queues_analogy.html
[3] https://www.rsyslog.com/doc/v8-stable/concepts/queues.html

-- 
Peter

On Wed, Sep 25, 2019 at 10:03 AM Malhar vora via rsyslog <
rsyslog@lists.adiscon.com> wrote:

> Thanks for reply.
>
> I tried TCP using following Rsyslog configuration but no luck.
> if ($msg contains "IEC:" ) then
> {
> set $!message = $msg;
> action(
> queue.filename="iem_queue"
> type="omfwd"
> Target="172.16.13.12"
> Port="10515"
> Protocol="tcp"
> Device="ens33"
> queue.type="linkedlist"
> name="action_sspl_iem_fwd"
> action.resumeRetryCount="-1"
> )
> }
>
> This time instead of my own script I tried "ncat -l 10515" command to start
> a TCP listener.
> Basically I believe that Rsyslog will accumulate messages when my
> script/app is down and send all the messages right away when my app starts
> but it doesn't do so. Instead of that it sends all those messages only when
> new message comes up. It sends all the old ones with that new message. So
> the thing is my script will not get any of those messages which were
> forwarded to Rsyslog during it was not running, until some new message
> arrives.
>
> Don't know I am missing something or is this a normal behaviour.
>
>
>
>
>
> Regds,
> *Malhar Vora*
> Blog : http://malhar2010.blogspot.com
> Blog : http://byteofcloud.blogspot.in/
> Twitter : https://twitter.com/mlvora
> Github :  https://github.com/vbmade2000 
>
>
>
> On Wed, Sep 25, 2019 at 11:47 AM Rainer Gerhards  >
> wrote:
>
> > With datagram protocol you cannot detect that the remote side is down.
> Use
> > tcp.
> >
> > HTH
> > Rainer
> >
> > Sent from phone, thus brief.
> >
> > Malhar vora via rsyslog  schrieb am Mi., 25.
> > Sep. 2019, 09:05:
> >
> >> Hello Experts,
> >>
> >> I am experimenting with Rsyslog. I am trying to redirect Rsyslog log to
> an
> >> Rsyslog server I have created using Python. I am using Disk Assisted
> >> queue.
> >>
> >> The problem is when my server is running Rsyslog sends logs properly but
> >> problem occurs when I follow these steps.
> >>
> >> 1. Stop my rsyslog server.
> >> 2. Send some logs
> >> 3. Start my rsyslog server.
> >> Here after starting my rsyslog server script I expect messages from
> >> rsyslog
> >> which I sent during stopped server. I believe that rsyslog enqueues
> those
> >> messages if destination not reachable or available but it doesn't work
> >> that
> >> way. It doesn't send those logs when server starts. It sends whole bunch
> >> of
> >> those pending logs when I generate one more log message.
> >>
> >> *Following is my Rsyslog server script.*
> >> import SocketServer
> >> HOST, PORT = "0.0.0.0", 10514
> >>  class SyslogUDPHandler(SocketServer.BaseRequestHandler):
> >>
> >> def handle(self):
> >> data = bytes.decode(self.request[0].strip())
> >> print(self.request)
> >> if __name__ == "__main__":
> >> try:
> >> server = SocketServer.UDPServer((HOST,PORT), SyslogUDPHandler)
> >> server.serve_forever(poll_interval=0.5)
> >> except (IOError, SystemExit):
> >> raise
> >> except KeyboardInterrupt:
> >> print("Crtl+C Pressed. Shutting down.")
> >>
> >>
> >> *Following is my configuration file for Rsyslog.*
> >> if ($msg contains "IEC:" ) then
> >> {
> >> action(
> >> queue.filename="iem_queue"
> >> type="omfwd"
> >> Target="172.16.13.12"
> >> Port="10514"
> >> Protocol="udp"
> >> Device="ens33"
> >> queue.type="linkedlist"
> >> name="action_sspl_iem_fwd"
> >> action.resumeRetryCount="-1"
> >> )
> >> }
> >>
> >> I have another python script and rsyslog conf file that use named pipe
> for
> >> same purpose. I am facing same issue with that too so I believe that
> issue
> >> is at Rsyslog side. I could be wrong.
> >>
> >> I need help to solve this issue. I am not able to understand what is
> >> missing/wrong at Rsyslog side.
> >>
> >>
> >>
> >>
> >> Regds,
> >> *Malhar Vora*
> >> Twitter : https://twitter.com/mlvora
> >> Github :  https://github.com/vbmade2000 
> >> ___
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com/professional-services/
> >> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> >> DON'T 

Re: [rsyslog] rainerscript control structures

[Peter Viskup via rsyslog]

For my use case, yes.
Just build configuration using 'global' templates with configuration
snippets and this just simplify the things.

One of the forward action as example:
root@syslog:/etc/rsyslog-global/host/lin/forwards# ls -la 11*
lrwxrwxrwx 1 root root 51 Sep 19 14:13 111-fwd-esparser03-lin-target.conf
-> ../../../lin/includes/fwd-esparser03-lin-target.inc
lrwxrwxrwx 1 root root 47 Sep 19 14:13 112-fwd-esparser03-lin-queue.conf ->
../../../global/includes/fwd-queue-1k-large.inc
lrwxrwxrwx 1 root root 48 Sep 19 14:13 113-fwd-esparser03-lin-template.conf
-> ../../../global/includes/fwd-template-relay2.inc
root@syslog:/etc/rsyslog-global/host/lin/forwards# cat
111-fwd-esparser03-lin-target.conf
action(type="omfwd" protocol="tcp" target="10.0.0.3" port="2514"
  name="ESP03"
  queue.FileName="fq_esp03"
root@syslog:/etc/rsyslog-global/host/lin/forwards# cat
112-fwd-esparser03-lin-queue.conf
  queue.spoolDirectory="/var/spool/rsyslog"
  queue.size="1000"
  queue.MaxDiskSpace="5000m"
  queue.Type="LinkedList"
  queue.HighWaterMark="50"
  queue.LowWaterMark="40"
root@syslog:/etc/rsyslog-global/host/lin/forwards# cat
113-fwd-esparser03-lin-template.conf
  template="relay2ForwardTemplate"
)
root@syslog:/etc/rsyslog-global/host/lin/forwards# cat 11*
action(type="omfwd" protocol="tcp" target="10.0.0.3" port="2514"
  name="ESP03"
  queue.FileName="fq_esp03"
  queue.spoolDirectory="/var/spool/rsyslog"
  queue.size="1000"
  queue.MaxDiskSpace="5000m"
  queue.Type="LinkedList"
  queue.HighWaterMark="50"
  queue.LowWaterMark="40"
  template="relay2ForwardTemplate"
)

Creating 110-fwd-filter.conf file with simple 'if property' check make the
filter and action work as expected.

On Thu, Sep 19, 2019 at 4:41 PM Илья Рассадин via rsyslog <
rsyslog@lists.adiscon.com> wrote:

> Is there any sense to not use brackets always?
>
> On 19/09/2019 17:36, Peter Viskup via rsyslog wrote:
> > Want to be sure the following configurations are the same
> >
> > if $hostname contains "text" then {
> >action(type="omfwd" .)
> > }
> >
> > and without curly brackets
> >
> > if $hostname contains "text" then
> >action(type="omfwd" ..)
> >
> > The first option with brackets has to be used in case of more actions
> > following the filter.
> > There is only one action following the filter. Is this my
> > assumption correct?
> >
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] rainerscript control structures

[Peter Viskup via rsyslog]

Want to be sure the following configurations are the same

if $hostname contains "text" then {
  action(type="omfwd" .)
}

and without curly brackets

if $hostname contains "text" then
  action(type="omfwd" ..)

The first option with brackets has to be used in case of more actions
following the filter.
There is only one action following the filter. Is this my
assumption correct?

-- 
Peter
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] imtcp performance

[Peter Viskup via rsyslog]

Would like to know your experience with imtcp and/or imptcp.

With +1100 established TCP connection we get ~100% CPU usage on imtcp
thread causing the TCP stack/connections being stalled/not possible to
establish.

TOP screen:
Threads: 295 total,   3 running, 292 sleeping,   0 stopped,   0 zombie
%Cpu0  : 21.6 us, 76.5 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  2.0 si,
 0.0 st
%Cpu1  :  7.5 us,  5.7 sy,  0.0 ni, 83.0 id,  0.0 wa,  0.0 hi,  3.8 si,
 0.0 st
KiB Mem:   8197852 total,  7946872 used,   250980 free,33520 buffers
KiB Swap:  1048572 total,   164648 used,   883924 free.  7067060 cached Mem

  PID USER  PR  NIVIRTRESSHR S %CPU %MEM TIME+ COMMAND
13760 loglin20   0  826836  24972   4676 R 90.6  0.3  64:33.11 in:imtcp
13761 loglin20   0  826836  24972   4676 R  5.9  0.3   8:09.49 rs:main
Q:Reg
  857 root  20   0   25764   3024   2492 S  2.0  0.0   0:02.72 top
12871 loglocal  20   0  757816  19452712 S  2.0  0.2 150:04.44 in:imtcp
13762 loglin20   0  826836  24972   4676 S  2.0  0.3   1:59.07 in:imtcp

Is that expected behavior? What is the suitable number of connections and
throughput imtcp can handle?

-- 
Peter
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] funding specific features (was Re: Making sure I understand execOnlyWhenPreviousIsSuspended correctly, )

[Peter Viskup via rsyslog]

The list of open improvements waiting for funding might help.
Can ask in our company about funding rsyslog project if some feature will
be interesting for our deployment.

Peter

On Thu, Sep 5, 2019 at 9:39 PM David Lang via rsyslog <
rsyslog@lists.adiscon.com> wrote:

> On Thu, 5 Sep 2019, Rainer Gerhards wrote:
>
> >> PS. I still think it would be a good idea to be able to "save" those
> messages stored temporarily in the "internal RELP-queue". In that way, one
> wouldn't need to worry about "loosing messages" if the client
> reboots/crashes before the primary logserver comes back up again.
> >
> > Just as an exercise: ask you company if they are willing to fund ~4000
> > Euros to make this happen? I guess the answer is no - and that says a
> > bit on priorities
>
> as a thought, I suspect that there are a fair number of such improvements
> that
> you have thought about enough to price out, but the companies involved
> have not
> funded (I've generated a few such situations)
>
> would it make sense to have a page somewhere to list such projects? either
> for
> companies to fund or for crowdfunding of such features? (crowdfunding
> would add
> costs, so the 'price' would have to go up a bit)
>
> David Lang
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Rsyslog HA-style redirect

[Peter Viskup via rsyslog]

Yes it is.
https://www.rsyslog.com/doc/master/tutorials/failover_syslog_server.html

Peter

On Fri, Aug 30, 2019 at 12:24 PM rsyslog--- via rsyslog <
rsyslog@lists.adiscon.com> wrote:

> Hello,
>
> When using TCP redirects (@@), is it possible to configure multiple
> servers but only send to one only? (If that one fails, then try another
> one from the list.)
>
> How would I do this?
>
> Thanks,
> Marki
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] Processing SQL audit messages

[Peter Viskup via rsyslog]

There are some application which write audit logs to SQL database only.
Might be interesting to process them with rsyslog for the distribution to
SIEM and/or archiving.

Does anybody work on similar use case?
Do you think input alternative of omlibdbi will make sense?

-- 
Peter
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] local variables not printed in debug format

[Peter Viskup via rsyslog]

FYI

Found the problem.
The listener used local ruleset, while the variables resided in
RSYSLOG_DefaultRuleset. Moving the omfile action out of ruleset definition
resolved the issue.

Reading sentence
"As such, any modifications made to the message object (e.g. message or
local variables that are set) or discarding of the message object have no
effect outside that ruleset."
on https://www.rsyslog.com/doc/v8-stable/concepts/multi_ruleset.html make
it more clear.

-- 
Peter

On Fri, Aug 9, 2019 at 2:51 PM Peter Viskup  wrote:

> Running rsyslog 8.1901 on fresh Debian10 the $.localvars are not printed
> in debug format.
> Starting rsyslog by
> /usr/sbin/rsyslogd -d -n -f /etc/rsyslog-2/rsyslog-lin.conf
> Config files are processed without any error.
> The message looks like this:
> Debug line with all properties:
> FROMHOST: 'hostname.domain', fromhost-ip: '10.1.1.3', HOSTNAME:
> 'server-10.1.1.4', PRI: 86,
> syslogtag 'sudo', programname: 'sudo', APP-NAME: 'sudo', PROCID: '-',
> MSGID: '-',
> TIMESTAMP: 'Aug  9 14:07:50', STRUCTURED-DATA: '[syslogTimes@29171
> 10.1.1.1="2019-08-09T14:07:50.497819+02:00"]',
> msg: 'pam_unix(sudo:session): session closed for user root'
> escaped msg: 'pam_unix(sudo:session): session closed for user root'
> inputname: imtcp rawmsg: '<86>1 2019-08-09T14:07:50.497129+02:00
> server-10.1.1.4 sudo - - [syslogTimes@29171
> 10.1.1.3="2019-08-09T14:07:50.497819+02:00"] pam_unix(sudo:session):
> session closed for user root'
> $!:
> $.:
> $/:
>
> No local variables listed. Also all the variables used in templates are
> empty.
> Is there any known bug which might be related? What to check? Option
> abortonuncleanconfig is enabled.
>
> Config snippets:
>
> /etc/rsyslog-2# cat rsyslog-lin.conf
> # Include global configuration
> $IncludeConfig /etc/rsyslog-2/global/*.conf
>
> # Include instance configuration
> $IncludeConfig /etc/rsyslog-2/lin/*.conf
> /etc/rsyslog-2# ls -la /etc/rsyslog-2/global/*.conf
> -rw-r--r-- 1 root root  151 May 30 13:38
> /etc/rsyslog-2/global/00-stats.conf
> -rw-r--r-- 1 root root  164 Jun 12 12:26
> /etc/rsyslog-2/global/01-global.conf
> -rw-r--r-- 1 root root   69 May 30 13:46
> /etc/rsyslog-2/global/02-permissions.conf
> -rw-r--r-- 1 root root   42 May 31 11:51
> /etc/rsyslog-2/global/03-modules.conf
> -rw-r--r-- 1 root root 1026 Aug  9 10:22
> /etc/rsyslog-2/global/09-variables.conf
> -rw-r--r-- 1 root root 3237 Aug  5 08:41
> /etc/rsyslog-2/global/10-templates.conf
> /etc/rsyslog-2# cat /etc/rsyslog-2/global/09-variables.conf
> # set local variables
> set $.localip="1.1.1.3";
> set $.host=$$myhostname;
>
> $template
> getOrigip,"%hostname:R,ERE,1,ZERO:([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})--end%"
> set $.origip=exec_template("getOrigip");
> if ( $.origip == "0" ) then {
>   if ( $fromhost-ip == "127.0.0.1" ) then {
> set $.origip=$.localip;
>   }
>   else {
> set $.origip=$fromhost-ip;
>   }
> }
> 
>
> --
> Peter
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] local variables not printed in debug format

[Peter Viskup via rsyslog]

Running rsyslog 8.1901 on fresh Debian10 the $.localvars are not printed in
debug format.
Starting rsyslog by
/usr/sbin/rsyslogd -d -n -f /etc/rsyslog-2/rsyslog-lin.conf
Config files are processed without any error.
The message looks like this:
Debug line with all properties:
FROMHOST: 'hostname.domain', fromhost-ip: '10.1.1.3', HOSTNAME:
'server-10.1.1.4', PRI: 86,
syslogtag 'sudo', programname: 'sudo', APP-NAME: 'sudo', PROCID: '-',
MSGID: '-',
TIMESTAMP: 'Aug  9 14:07:50', STRUCTURED-DATA: '[syslogTimes@29171
10.1.1.1="2019-08-09T14:07:50.497819+02:00"]',
msg: 'pam_unix(sudo:session): session closed for user root'
escaped msg: 'pam_unix(sudo:session): session closed for user root'
inputname: imtcp rawmsg: '<86>1 2019-08-09T14:07:50.497129+02:00
server-10.1.1.4 sudo - - [syslogTimes@29171
10.1.1.3="2019-08-09T14:07:50.497819+02:00"] pam_unix(sudo:session):
session closed for user root'
$!:
$.:
$/:

No local variables listed. Also all the variables used in templates are
empty.
Is there any known bug which might be related? What to check? Option
abortonuncleanconfig is enabled.

Config snippets:

/etc/rsyslog-2# cat rsyslog-lin.conf
# Include global configuration
$IncludeConfig /etc/rsyslog-2/global/*.conf

# Include instance configuration
$IncludeConfig /etc/rsyslog-2/lin/*.conf
/etc/rsyslog-2# ls -la /etc/rsyslog-2/global/*.conf
-rw-r--r-- 1 root root  151 May 30 13:38 /etc/rsyslog-2/global/00-stats.conf
-rw-r--r-- 1 root root  164 Jun 12 12:26
/etc/rsyslog-2/global/01-global.conf
-rw-r--r-- 1 root root   69 May 30 13:46
/etc/rsyslog-2/global/02-permissions.conf
-rw-r--r-- 1 root root   42 May 31 11:51
/etc/rsyslog-2/global/03-modules.conf
-rw-r--r-- 1 root root 1026 Aug  9 10:22
/etc/rsyslog-2/global/09-variables.conf
-rw-r--r-- 1 root root 3237 Aug  5 08:41
/etc/rsyslog-2/global/10-templates.conf
/etc/rsyslog-2# cat /etc/rsyslog-2/global/09-variables.conf
# set local variables
set $.localip="1.1.1.3";
set $.host=$$myhostname;

$template
getOrigip,"%hostname:R,ERE,1,ZERO:([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})--end%"
set $.origip=exec_template("getOrigip");
if ( $.origip == "0" ) then {
  if ( $fromhost-ip == "127.0.0.1" ) then {
set $.origip=$.localip;
  }
  else {
set $.origip=$fromhost-ip;
  }
}


-- 
Peter
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] Rsyslog regex test page not working

[Peter Viskup via rsyslog]

The page  https://www.rsyslog.com/regex/ does not show Regexp results.
Please check.

-- 
Peter
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] set myhostname property value into local variable

[Peter Viskup via rsyslog]

Configuration with lines:
# set local variables
set $.localip = "1.1.1.1";
set $.host = $myhostname;

seems not be working. Getting these errors.

Jul 23 14:25:41 HOST-LOCO rsyslogd[6024]: rsyslogd: error during parsing
file /etc/rsyslog.d/global/09-variables.conf, on or before line 3: invalid
property 'myhostname' [v8.1901.0 try https://www.rsyslog.com/e/2207 ]
Jul 23 14:25:41 HOST-LOCO rsyslogd[6024]: rsyslogd: error during parsing
file /etc/rsyslog.d/global/09-variables.conf, on or before line 3: did you
mean '$myhostname' instead of 'myhostname'? See also:
https://www.rsyslog.com/rsyslog-info-1/ [v8.1901.0 try
https://www.rsyslog.com/e/2207 ]

Running rsyslog version 8.1901.0-1 on clean Debian 10 installation.
What's wrong here?

-- 
Peter
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Debian packages and what we can do better

[Peter Viskup via rsyslog]

On Thu, Jul 4, 2019 at 1:35 PM Michael Biebl  wrote:

> Am Do., 4. Juli 2019 um 13:30 Uhr schrieb Peter Viskup via rsyslog
> :
> > The syslog infra is something which most of admins do not want to update
> on
> > daily basis.
> > I think this is not something we should expect from admins - and as you
> > see, it was just proven. Also some bugs might occur after a while.
> > Find it not appropriate to follow agile development principles on such
> > crucial subsystem as syslog still is. This is user's point of view.
>
>
> So if I understand you correctly you don't want the latest and
> greatest but you would actually prefer the version that is shipped in
> $stable and only apply targetted fixes?
>

Yes, Michael, that would make sense. Not all bug fixes, but at least those,
which will be considered critical, making the software not work as expected
or failing.
E.g. that one in 8.24 version causing DA queues not being processed.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Debian packages and what we can do better

[Peter Viskup via rsyslog]

On Thu, Jul 4, 2019 at 11:51 AM Rainer Gerhards 
wrote:

> Hijacking the thread just slightly...
>
> El jue., 4 jul. 2019 a las 9:51, Peter Viskup via rsyslog
> () escribió:
> >
>
> > The use of package from backports is not always the best option as those
> > versions also come with new bugs and regressions.
> > For example in 8.1905 release there was important regression in core
> > - core bugfix: segfault on startup depending on queue file names
> >   rsyslog will segfault on startup when a main queue file name has
> >   been set and at least on other queue contains a file name. This
> >   was cased by too-early freeing config error-detection data
> >   structures. It is a regression caused by commit e22fb205a3.
> >   Thanks to Wade Simmons for reporting this issue and providing
> >   detailled analysis. That greatly helps fixing it quickly.
> >   closes https://github.com/rsyslog/rsyslog/issues/3681
> >
> > Maybe, with help of community, Debian would be able to provide "real
> > stable" rsyslog release.
>
> A main problem from our PoV is that almost nobody uses the daily
> stable releases. If at least a couple of folks would do, we could
> usually iron out regressions like the above very quickly. As this does
> not happen, it propagates to the 6-week stable, is deteced rather
> quickly, and folks need to wait another 6 weeks to get the patch
> (daily stable users have it immediately).
>

The syslog infra is something which most of admins do not want to update on
daily basis.
I think this is not something we should expect from admins - and as you
see, it was just proven. Also some bugs might occur after a while.
Find it not appropriate to follow agile development principles on such
crucial subsystem as syslog still is. This is user's point of view.

Similar situation is with systemd. Was just trying to create chrooted
systemd rsyslog services and fall down on knees searching why this and that
super-duper feature does not work. In some cases it was bug in version
available in Debian, in others this or that feature was not available in
the Debian version, or it was architectural issue in vanilla where those
features just does not work together.
On the end the services were built upon systemd features available since
the very beginning, with calling Start/StopExecPre/Post commands.

-- 
Peter
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Debian packages and what we can do better

[Peter Viskup via rsyslog]

Hello Michael,
at first, thank you for your work done.

Propose rsyslog-ossl (OpenSSL driver for TLS encryption) being built and
put into non-free if possible. Just to let people test or use it if they
want.
The libssl-dev is listed in BuildDepends list. Are there other parts of
rsyslog which are dependent on OpenSSL libraries? These are not
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=930816

Maybe backporting of some bugfixes for rsyslog in stable release should be
made.
At the moment the 8.24.0-1 in current stable means there were no changes
from original source. Due to rsyslog release policies bugfixes are
primarily included in new releases only. At least some important bug might
be backported in my opinion.
Just list from 8.25 release:
 - bugfix imtcp: fix very small (cosmetic) memory leak
 - bugfix rainerscript: set/unset statement do not check variable name
validity
 - bugfix core: str2num mishandling empty strings
 - bugfix queue subsystem: queue corrupted if certain msg props are used
 - core: fix potential message loss in old-style transactional interface

The bug "queue corrupted if certain msg props are used" caused DA queues
not being dequeued when some variables used in message processing (most
semi-advanced setups do).
At least this and some of others might be backported to Debian stable
release package.

The use of package from backports is not always the best option as those
versions also come with new bugs and regressions.
For example in 8.1905 release there was important regression in core
- core bugfix: segfault on startup depending on queue file names
  rsyslog will segfault on startup when a main queue file name has
  been set and at least on other queue contains a file name. This
  was cased by too-early freeing config error-detection data
  structures. It is a regression caused by commit e22fb205a3.
  Thanks to Wade Simmons for reporting this issue and providing
  detailled analysis. That greatly helps fixing it quickly.
  closes https://github.com/rsyslog/rsyslog/issues/3681

Maybe, with help of community, Debian would be able to provide "real
stable" rsyslog release.

-- 
Peter

On Tue, Jul 2, 2019 at 9:13 PM Michael Biebl via rsyslog <
rsyslog@lists.adiscon.com> wrote:

> Hi everyone,
>
> in case you don't know me, I'm the (official) maintainer of rsyslog in
> Debian.
> I put the official in parenthesis as I know there are deb packages as
> well provided by Adiscon directly.
> While I appreciate the service that is done by Rainer and his folks, I
> wonder if there is something we can improve on the Debian side. I try
> to keep the Debian packages up-to-date [1] as well as I can given the
> constraints that a distro like Debian has.
> Is there anything else that you are missing?
> Any recommendations how the Debian packages can be improved?
>
> I'm happy to receive feedback here. Just keep in mind, that I have to
> balance here, that rsyslog is installed on basically everyone's
> (Debian) system.
>
> Regards,
> Michael
>
> [1] https://tracker.debian.org/pkg/rsyslog
>
> --
> Why is it that all of the instruments seeking intelligent life in the
> universe are pointed away from Earth?
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] UDP syslog load balancer healthcheck workaround

[Peter Viskup via rsyslog]

Small remark for ldirectord config.
The UDP syslog service work much better with scheduler=sh (source hash) and
quiescent=yes.
That will let the LVS balance across real servers with source-ip going to
the same destination if available. For UDP service the LVS does not route
packets for unavailable real servers (with weight set to 0). With
quiescent=no the real server is removed from LVS configuration and the
source-hash mapping is lost. For source-ip hash sheduler, the weight needs
to be setup as it does mean "maximum allowed connections" for real server.
http://kb.linuxvirtualserver.org/wiki/Source_Hashing_Scheduling

On the contrary, for TCP the quiescent=yes causing the new connections are
still balanced to the unavailable real servers causing the connection to
fail. In combination with source-ip hash might cause general unavailability
for part of the environment.

Then the configuration might look like this
virtual=10.0.x.a:5514
real=10.0.x.b:5514 gate 50
real=10.0.x.c:5514 gate 50
real=10.0.x.d:5514 gate 50
quiescent=no
service=none
scheduler=sh
protocol=tcp
checktype=connect

virtual=10.1.x.a:5514
real=10.0.x.b:5514 gate 100
real=10.0.x.c:5514 gate 100
real=10.0.x.d:5514 gate 100
quiescent=yes
service=none
scheduler=sh
protocol=udp
checktype=external-perl
checkcommand=/usr/local/sbin/ldirector_port_check


On Wed, Jun 26, 2019 at 2:11 PM Peter Viskup  wrote:

> Want to share the ldirector_port_check script based on check_port.pl
> script [1] which can be used to perform the remote healthcheck for listen
> ports.
> The remote monitoring UDP listen ports is not possible. Ldirector use
> simple ping of remote host for UDP services, which is not sufficient. To
> let the ldirector manage the real servers for UDP services, the external
> check script needs to be used.
> The attached script can be used to monitor remote TCP port as an health
> check for remote UDP port. In case the remote TCP port become unavailable,
> also the UDP real server will be removed from LVS balancing configuration.
> The only requirement is that the rsyslog instance should have both UDP and
> TCP ports open with same port number.
>
> Script can be used with ldirectord configuration as follows:
> ~# cat /etc/heartbeat/ldirectord.cf
> checktimeout=10
> checkinterval=2
> autoreload=yes
> logfile="/var/log/ldirectord.log"
> quiescent=no
> readdquiescent=no
>
> virtual=10.0.x.a:5514
> real=10.0.x.b:5514 gate
> real=10.0.x.c:5514 gate
> service=none
> scheduler=rr
> protocol=tcp
> checktype=connect
>
> virtual=10.0.x.a:5514
> real=10.0.x.b:5514 gate
> real=10.0.x.c:5514 gate
> service=none
> scheduler=rr
> protocol=udp
> checktype=external-perl
> checkcommand=/usr/local/sbin/ldirector_port_check
>
> [1]
> https://exchange.nagios.org/directory/Plugins/Network-Protocols/*-TCP-and-UDP-(Generic)/check_port-2Epl/details
>
> --
> Peter
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] UDP syslog load balancer healthcheck workaround

[Peter Viskup via rsyslog]

Want to share the ldirector_port_check script based on check_port.pl script
[1] which can be used to perform the remote healthcheck for listen ports.
The remote monitoring UDP listen ports is not possible. Ldirector use
simple ping of remote host for UDP services, which is not sufficient. To
let the ldirector manage the real servers for UDP services, the external
check script needs to be used.
The attached script can be used to monitor remote TCP port as an health
check for remote UDP port. In case the remote TCP port become unavailable,
also the UDP real server will be removed from LVS balancing configuration.
The only requirement is that the rsyslog instance should have both UDP and
TCP ports open with same port number.

Script can be used with ldirectord configuration as follows:
~# cat /etc/heartbeat/ldirectord.cf
checktimeout=10
checkinterval=2
autoreload=yes
logfile="/var/log/ldirectord.log"
quiescent=no
readdquiescent=no

virtual=10.0.x.a:5514
real=10.0.x.b:5514 gate
real=10.0.x.c:5514 gate
service=none
scheduler=rr
protocol=tcp
checktype=connect

virtual=10.0.x.a:5514
real=10.0.x.b:5514 gate
real=10.0.x.c:5514 gate
service=none
scheduler=rr
protocol=udp
checktype=external-perl
checkcommand=/usr/local/sbin/ldirector_port_check

[1]
https://exchange.nagios.org/directory/Plugins/Network-Protocols/*-TCP-and-UDP-(Generic)/check_port-2Epl/details

-- 
Peter


ldirector_port_check
Description: Binary data
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] OBS repositories for Debian 10

[Peter Viskup via rsyslog]

When it is planned to make Debian 10 repositories on openSUSE build service?
Debian 10 release is planned on 6.7.2019 and would be good to have some
time to test it in advance.

-- 
Peter
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] rsyslog with TLS on Debian

[Peter Viskup via rsyslog]

What is the actual status of building rsyslog with TLS on Debian.
Just remember there were some issues with ossl driver caused the Debian
package cannot be built with it.
Is this still the case? Should ossl driver be preferred? What is the
quality of both ossl and gtls drivers in latest versions?

-- 
Peter
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog 8.1904 not built with systemd in OBS repos

[Peter Viskup via rsyslog]

Confirm the sd_notify interface working now for rsyslog from OBS Debian9
repositories.
Thank you.

On Tue, Jun 18, 2019 at 1:26 PM Rainer Gerhards 
wrote:

> Peter,
>
> I guess I won my fight with OBS and, if so, new packages with systemd
> support are now building. Should be available soon. Would be great if
> you could check.
>
> Rainer
>
> El mar., 18 jun. 2019 a las 9:01, Peter Viskup via rsyslog
> () escribió:
> >
> > Tried to start rsyslog 8.1904 in chrooted environment, but got the
> systemd
> > service timeout error.
> > The sd_notify in rsyslog 8.1901 version from Debian repositories is
> working
> > fine with just bind mounting host /run/systemd/notify into the chroot
> under
> > the same path.
> >
> > The root cause seems to be the rsyslog 8.1904 in OBS Debian9 repository
> is
> > not built with systemd support for some reason.
> >
> > ~# /usr/sbin/rsyslogd -vvv
> > rsyslogd  8.1904.0 (aka 2019.04) compiled with:
> > PLATFORM:   x86_64-pc-linux-gnu
> > PLATFORM (lsb_release -d):
> > FEATURE_REGEXP: Yes
> > GSSAPI Kerberos 5 support:  No
> > FEATURE_DEBUG (debug build, slow code): No
> > 32bit Atomic operations supported:  Yes
> > 64bit Atomic operations supported:  Yes
> > memory allocator:   system default
> > Runtime Instrumentation (slow code):No
> > uuid support:   Yes
> > systemd support:No
> > Config file:/etc/rsyslog.conf
> > PID file:   /var/run/rsyslogd.pid
> > Number of Bits in RainerScript integers: 64
> >
> > Peter
> > ___
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] rsyslog 8.1904 not built with systemd in OBS repos

[Peter Viskup via rsyslog]

Tried to start rsyslog 8.1904 in chrooted environment, but got the systemd
service timeout error.
The sd_notify in rsyslog 8.1901 version from Debian repositories is working
fine with just bind mounting host /run/systemd/notify into the chroot under
the same path.

The root cause seems to be the rsyslog 8.1904 in OBS Debian9 repository is
not built with systemd support for some reason.

~# /usr/sbin/rsyslogd -vvv
rsyslogd  8.1904.0 (aka 2019.04) compiled with:
PLATFORM:   x86_64-pc-linux-gnu
PLATFORM (lsb_release -d):
FEATURE_REGEXP: Yes
GSSAPI Kerberos 5 support:  No
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported:  Yes
64bit Atomic operations supported:  Yes
memory allocator:   system default
Runtime Instrumentation (slow code):No
uuid support:   Yes
systemd support:No
Config file:/etc/rsyslog.conf
PID file:   /var/run/rsyslogd.pid
Number of Bits in RainerScript integers: 64

Peter
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Lookup table does not set variable

[Peter Viskup via rsyslog]

Opened https://github.com/rsyslog/rsyslog/issues/3706

On Fri, Jun 14, 2019 at 2:08 PM Rainer Gerhards 
wrote:

> I suggest to open a GitHub issues, as the code is most likely the same as
> current. I can than see if the contributor steps in.
>
> Rainer
>
> Sent from phone, thus brief.
>
> Peter Viskup  schrieb am Fr., 14. Juni 2019, 13:37:
>
>> Can just confirm it is the same with 8.1901 from official Debian
>> backports repository.
>> Not able to confirm on 8.1904 version available in your OBS repositories
>> as it is not working in our chrooted environment by default.
>> Seeing errors like
>>
>> Jun 14 13:25:56 HOST-LOCO rsyslogd: rsyslogd's groupid changed to 115
>> Jun 14 13:25:56 HOST-LOCO rsyslogd: rsyslogd's userid changed to 111
>> Jun 14 13:25:56 HOST-LOCO rsyslogd:  [origin software="rsyslogd"
>> swVersion="8.1904.0" x-pid="15495" x-info="https://www.rsyslog.com;]
>> start
>> Jun 14 13:26:11 HOST-LOCO rsyslogd[15495]: rsyslogd: impstats: error
>> reading /proc/15495/fd : No such file or directory [v8.1904.0]
>> Jun 14 13:26:11 HOST-LOCO rsyslogd: impstats: error reading
>> /proc/15495/fd : No such file or directory [v8.1904.0]
>> Jun 14 13:26:26 HOST-LOCO rsyslogd[15495]: rsyslogd: impstats: error
>> reading /proc/15495/fd : No such file or directory [v8.1904.0]
>> Jun 14 13:26:26 HOST-LOCO rsyslogd: impstats: error reading
>> /proc/15495/fd : No such file or directory [v8.1904.0]
>>
>> and the systemd notify not working as with 8.24 and 8.1901 version.
>>
>> Jun 14 13:27:26 HOST-LOCO systemd[1]: rsyslog-chroot@local.service:
>> Start operation timed out. Terminating.
>> Jun 14 13:27:26 HOST-LOCO rsyslogd:  [origin software="rsyslogd"
>> swVersion="8.1904.0" x-pid="15495" x-info="https://www.rsyslog.com;]
>> exiting on signal 15.
>> Jun 14 13:27:26 HOST-LOCO rsyslogd[15495]: rsyslog internal message
>> (3,-3000): impstats: error reading /proc/15495/fd
>> Jun 14 13:27:26 HOST-LOCO rsyslogd[15495]: : No such file or directory
>> [v8.1904.0]
>> Jun 14 13:27:26 HOST-LOCO systemd[1]: Failed to start Syslog Service
>> local instance under /chroot/local.
>> Jun 14 13:27:26 HOST-LOCO systemd[1]: rsyslog-chroot@local.service: Unit
>> entered failed state.
>> Jun 14 13:27:26 HOST-LOCO systemd[1]: rsyslog-chroot@local.service:
>> Failed with result 'timeout'.
>>
>> --
>> Peter
>>
>> On Fri, Jun 14, 2019 at 1:09 PM Rainer Gerhards 
>> wrote:
>>
>>> does this also happen with current 8.1905.0?
>>> Rainer
>>>
>>> El vie., 14 jun. 2019 a las 12:29, Peter Viskup via rsyslog
>>> () escribió:
>>> >
>>> > Running rsyslog 8.24 on Debian9.
>>> >
>>> > The lookup table
>>> > ~# cat /etc/rsyslog.d/local/programnames.lookup
>>> > { "version" : 1,
>>> >   "nomatch" : "local-all",
>>> >   "type" : "string",
>>> >   "table" : [
>>> > {"index" : "apache_site_access", "value" : "apache-site-access" },
>>> > {"index" : "apache_site_error", "value" : "apache-site-error" }
>>> > ]}
>>> >
>>> > does not set variable for DynaFile and all logs go to /var/log/.log
>>> >
>>> > ~# cat /etc/rsyslog.d/local/02-rulesets.conf
>>> > lookup_table(name="programname"
>>> > file="/etc/rsyslog.d/local/programnames.lookup")
>>> > set $.filedest = lookup("programname", $programname);
>>> > template(name="programnameFileStoreTemplate" type="string"
>>> > string="/var/log/%$.filedest%.log")
>>> >
>>> > ruleset(name="ruleset-local"){
>>> >   call ruleset-localstore
>>> >   call ruleset-forwards-last
>>> > }
>>> >
>>> > ruleset(name="ruleset-localstore"){
>>> >   action(type="omfile" DynaFile="programnameFileStoreTemplate"
>>> > template="RSYSLOG_DebugFormat" asyncWriting="on" ioBufferSize="128K")
>>> > }
>>> >
>>> > Any thoughts what could be wrong? Running debug does not show any
>>> errors.
>>> > Just variable not set.
>>> >
>>> > Jun 14 12:16:19 HOST-LOCO rsyslogd[29154]: 7379.801914690:imudp.c
>>>   :
>>> >

Re: [rsyslog] Lookup table does not set variable

[Peter Viskup via rsyslog]

Can just confirm it is the same with 8.1901 from official Debian backports
repository.
Not able to confirm on 8.1904 version available in your OBS repositories as
it is not working in our chrooted environment by default.
Seeing errors like

Jun 14 13:25:56 HOST-LOCO rsyslogd: rsyslogd's groupid changed to 115
Jun 14 13:25:56 HOST-LOCO rsyslogd: rsyslogd's userid changed to 111
Jun 14 13:25:56 HOST-LOCO rsyslogd:  [origin software="rsyslogd"
swVersion="8.1904.0" x-pid="15495" x-info="https://www.rsyslog.com;] start
Jun 14 13:26:11 HOST-LOCO rsyslogd[15495]: rsyslogd: impstats: error
reading /proc/15495/fd : No such file or directory [v8.1904.0]
Jun 14 13:26:11 HOST-LOCO rsyslogd: impstats: error reading /proc/15495/fd
: No such file or directory [v8.1904.0]
Jun 14 13:26:26 HOST-LOCO rsyslogd[15495]: rsyslogd: impstats: error
reading /proc/15495/fd : No such file or directory [v8.1904.0]
Jun 14 13:26:26 HOST-LOCO rsyslogd: impstats: error reading /proc/15495/fd
: No such file or directory [v8.1904.0]

and the systemd notify not working as with 8.24 and 8.1901 version.

Jun 14 13:27:26 HOST-LOCO systemd[1]: rsyslog-chroot@local.service: Start
operation timed out. Terminating.
Jun 14 13:27:26 HOST-LOCO rsyslogd:  [origin software="rsyslogd"
swVersion="8.1904.0" x-pid="15495" x-info="https://www.rsyslog.com;]
exiting on signal 15.
Jun 14 13:27:26 HOST-LOCO rsyslogd[15495]: rsyslog internal message
(3,-3000): impstats: error reading /proc/15495/fd
Jun 14 13:27:26 HOST-LOCO rsyslogd[15495]: : No such file or directory
[v8.1904.0]
Jun 14 13:27:26 HOST-LOCO systemd[1]: Failed to start Syslog Service local
instance under /chroot/local.
Jun 14 13:27:26 HOST-LOCO systemd[1]: rsyslog-chroot@local.service: Unit
entered failed state.
Jun 14 13:27:26 HOST-LOCO systemd[1]: rsyslog-chroot@local.service: Failed
with result 'timeout'.

-- 
Peter

On Fri, Jun 14, 2019 at 1:09 PM Rainer Gerhards 
wrote:

> does this also happen with current 8.1905.0?
> Rainer
>
> El vie., 14 jun. 2019 a las 12:29, Peter Viskup via rsyslog
> () escribió:
> >
> > Running rsyslog 8.24 on Debian9.
> >
> > The lookup table
> > ~# cat /etc/rsyslog.d/local/programnames.lookup
> > { "version" : 1,
> >   "nomatch" : "local-all",
> >   "type" : "string",
> >   "table" : [
> > {"index" : "apache_site_access", "value" : "apache-site-access" },
> > {"index" : "apache_site_error", "value" : "apache-site-error" }
> > ]}
> >
> > does not set variable for DynaFile and all logs go to /var/log/.log
> >
> > ~# cat /etc/rsyslog.d/local/02-rulesets.conf
> > lookup_table(name="programname"
> > file="/etc/rsyslog.d/local/programnames.lookup")
> > set $.filedest = lookup("programname", $programname);
> > template(name="programnameFileStoreTemplate" type="string"
> > string="/var/log/%$.filedest%.log")
> >
> > ruleset(name="ruleset-local"){
> >   call ruleset-localstore
> >   call ruleset-forwards-last
> > }
> >
> > ruleset(name="ruleset-localstore"){
> >   action(type="omfile" DynaFile="programnameFileStoreTemplate"
> > template="RSYSLOG_DebugFormat" asyncWriting="on" ioBufferSize="128K")
> > }
> >
> > Any thoughts what could be wrong? Running debug does not show any errors.
> > Just variable not set.
> >
> > Jun 14 12:16:19 HOST-LOCO rsyslogd[29154]: 7379.801914690:imudp.c
> :
> > imudp: epoll_wait() returned with 1 fds
> > Jun 14 12:16:19 HOST-LOCO rsyslogd[29154]: 7379.801973745:imudp.c
> :
> > imudp: recvmmsg returned 1
> > Jun 14 12:16:19 HOST-LOCO rsyslogd[29154]: 7379.801991366:imudp.c
> :
> > recv(3,149),acl:1,msg:<13>1 2019-06-14T12:16:19.801660+02:00 HOST-LOCO
> > apache_site_access 29180 - [timeQuality tzKn
> > own="1" isSynced="1" syncAccuracy="990465"] test message
> > Jun 14 12:16:19 HOST-LOCO rsyslogd[29154]: 7379.802015589:imudp.c
> :
> > msg parser: flags 70, from '~NOTRESOLVED~', msg '<13>1
> > 2019-06-14T12:16:19.801660+02:00 HOST-LOCO apache_site'
> > Jun 14 12:16:19 HOST-LOCO rsyslogd[29154]: 7379.802026839:imudp.c
> :
> > parse using parser list 0x55abe7d25090 (the default list).
> > Jun 14 12:16:19 HOST-LOCO rsyslogd[29154]: 7379.802041233:imudp.c
> :
> > Message has RFC5424/syslog-protocol format.
> > Jun 14 12:16:19 HOST-LOCO rsyslogd[29154]: 7379.802059418:imudp.c
> :
> > Parser 'rsyslog.rfc5424' returne

[rsyslog] Lookup table does not set variable

[Peter Viskup via rsyslog]

Running rsyslog 8.24 on Debian9.

The lookup table
~# cat /etc/rsyslog.d/local/programnames.lookup
{ "version" : 1,
  "nomatch" : "local-all",
  "type" : "string",
  "table" : [
{"index" : "apache_site_access", "value" : "apache-site-access" },
{"index" : "apache_site_error", "value" : "apache-site-error" }
]}

does not set variable for DynaFile and all logs go to /var/log/.log

~# cat /etc/rsyslog.d/local/02-rulesets.conf
lookup_table(name="programname"
file="/etc/rsyslog.d/local/programnames.lookup")
set $.filedest = lookup("programname", $programname);
template(name="programnameFileStoreTemplate" type="string"
string="/var/log/%$.filedest%.log")

ruleset(name="ruleset-local"){
  call ruleset-localstore
  call ruleset-forwards-last
}

ruleset(name="ruleset-localstore"){
  action(type="omfile" DynaFile="programnameFileStoreTemplate"
template="RSYSLOG_DebugFormat" asyncWriting="on" ioBufferSize="128K")
}

Any thoughts what could be wrong? Running debug does not show any errors.
Just variable not set.

Jun 14 12:16:19 HOST-LOCO rsyslogd[29154]: 7379.801914690:imudp.c:
imudp: epoll_wait() returned with 1 fds
Jun 14 12:16:19 HOST-LOCO rsyslogd[29154]: 7379.801973745:imudp.c:
imudp: recvmmsg returned 1
Jun 14 12:16:19 HOST-LOCO rsyslogd[29154]: 7379.801991366:imudp.c:
recv(3,149),acl:1,msg:<13>1 2019-06-14T12:16:19.801660+02:00 HOST-LOCO
apache_site_access 29180 - [timeQuality tzKn
own="1" isSynced="1" syncAccuracy="990465"] test message
Jun 14 12:16:19 HOST-LOCO rsyslogd[29154]: 7379.802015589:imudp.c:
msg parser: flags 70, from '~NOTRESOLVED~', msg '<13>1
2019-06-14T12:16:19.801660+02:00 HOST-LOCO apache_site'
Jun 14 12:16:19 HOST-LOCO rsyslogd[29154]: 7379.802026839:imudp.c:
parse using parser list 0x55abe7d25090 (the default list).
Jun 14 12:16:19 HOST-LOCO rsyslogd[29154]: 7379.802041233:imudp.c:
Message has RFC5424/syslog-protocol format.
Jun 14 12:16:19 HOST-LOCO rsyslogd[29154]: 7379.802059418:imudp.c:
Parser 'rsyslog.rfc5424' returned 0
Jun 14 12:16:19 HOST-LOCO rsyslogd[29154]: 7379.802074210:imudp.c:
imudp: recvmmsg returned -1
Jun 14 12:16:19 HOST-LOCO rsyslogd[29154]: 7379.802091320:imudp.c:
main Q: qqueueAdd: entry added, size now log 1, phys 1 entries
Jun 14 12:16:19 HOST-LOCO rsyslogd[29154]: 7379.802103168:imudp.c:
main Q:Reg: high activity - starting 1 additional worker thread(s).
Jun 14 12:16:19 HOST-LOCO rsyslogd[29154]: 7379.802241185:imudp.c:
main Q:Reg: started with state 0, num workers now 1
Jun 14 12:16:19 HOST-LOCO rsyslogd[29154]: 7379.802936660:7fd6c50ab700:
thread created, tid 29181, name 'rs:main Q:Reg'
Jun 14 12:16:19 HOST-LOCO rsyslogd[29154]: 7379.803136562:main Q:Reg/w0  :
wti 0x55abe7d40600: worker starting
Jun 14 12:16:19 HOST-LOCO rsyslogd[29154]: 7379.803157598:main Q:Reg/w0  :
DeleteProcessedBatch: we deleted 0 objects and enqueued 0 objects
Jun 14 12:16:19 HOST-LOCO rsyslogd[29154]: 7379.803167833:main Q:Reg/w0  :
doDeleteBatch: delete batch from store, new sizes: log 1, phys 1
Jun 14 12:16:19 HOST-LOCO rsyslogd[29154]: 7379.803180562:main Q:Reg/w0  :
processBATCH: batch of 1 elements must be processed
Jun 14 12:16:19 HOST-LOCO rsyslogd[29154]: 7379.803190204:main Q:Reg/w0  :
processBATCH: next msg 0: <13>1 2019-06-14T12:16:19.801660+02:00 HOST-LOCO
apache_site_access 29180 - [timeQuality
tzKnown="1" isSynced="1" syncAccuracy="
Jun 14 12:16:19 HOST-LOCO rsyslogd[29154]: 7379.803212631:main Q:Reg/w0  :
CALL [ruleset-forwards-first, queue:0]
Jun 14 12:16:19 HOST-LOCO rsyslogd[29154]: 7379.803239267:main Q:Reg/w0  :
CALL [ruleset-localstore, queue:0]
Jun 14 12:16:19 HOST-LOCO rsyslogd[29154]: 7379.803266139:main Q:Reg/w0  :
ACTION 0 [builtin:omfile:action(type="builtin:omfile" ...)]
Jun 14 12:16:19 HOST-LOCO rsyslogd[29154]: 7379.803293252:main Q:Reg/w0  :
executing action 0
Jun 14 12:16:19 HOST-LOCO rsyslogd[29154]: 7379.803304519:main Q:Reg/w0  :
action 'action 0': called, logging to builtin:omfile (susp 0/0, direct q 1)
Jun 14 12:16:19 HOST-LOCO rsyslogd[29154]: 7379.803338450:main Q:Reg/w0  :
dnscache: entry (nil) found
Jun 14 12:16:19 HOST-LOCO rsyslogd[29154]: 7379.807241319:imudp.c:
main Q: MultiEnqObj advised worker start
Jun 14 12:16:19 HOST-LOCO rsyslogd[29154]: 7379.807929989:main Q:Reg/w0  :
action 'action 0': is transactional - executing in commit phase
Jun 14 12:16:19 HOST-LOCO rsyslogd[29154]: 7379.807951211:main Q:Reg/w0  :
wti 0x55abe7d40600: we need to create a new action worker instance for
action 0
Jun 14 12:16:19 HOST-LOCO rsyslogd[29154]: 7379.807966193:main Q:Reg/w0  :
wti 0x55abe7d40600: created action worker instance 1 for action 0
Jun 14 12:16:19 HOST-LOCO rsyslogd[29154]: 7379.807978331:main Q:Reg/w0  :
Action 0 transitioned to state: itx
Jun 14 12:16:19 HOST-LOCO rsyslogd[29154]: 7379.807990882:main Q:Reg/w0  :
action 'action 0': set suspended state to 0
Jun 14 12:16:19 HOST-LOCO rsyslogd[29154]: 7379.808004265:main Q:Reg/w0  :

Re: [rsyslog] imuxsock needs UseSpecialParser='off" to parse /var/run/log correctly on FreeBSD

[Peter Viskup via rsyslog]

Hello David,

On Wed, Jun 5, 2019 at 7:08 PM David Lang via rsyslog <
rsyslog@lists.adiscon.com> wrote:

> I think I've seen this before and the problem is that the timestamp being
> provided has too many digits after the .
>
> can you try to rig up a test where you send 3 digits after the . instead
> of 6?
>
>
According the RFC5424 [1] up to the 6 digit seconds fraction should be
recognized as valid.
If this is the case, then it should be marked as rsyslog bug.

[1] https://tools.ietf.org/html/rfc5424#section-6.2.3

Peter
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] call ruleset

[Peter Viskup via rsyslog]

>From reading the call documentation [1] I understand the call ruleset can
be used to independent parallel message processing bypassing the standard
queue-lanes behavior [2].
Is this my assumption correct?
Want to come with configuration that will prevent unavailability of one
destination to block message processing of other actions.
Will the ruleset-linux use the default mainQ on input if not configured
with queue.* options?

something like this (one input only):
input(type="imtcp" ... ruleset="ruleset-linux")
ruleset(name="ruleset-linux"){
  call ruleset-localstore
  call ruleset-forward1
  call ruleset-forward2
}
ruleset(name="ruleset-localstore"){
  action(type="omfile" asyncWriting="on" file="/var/log/lin-all.log"
template="fileStoreTemplate")
}
ruleset(name="ruleset-forward1"){
  action(type="omfwd" 
queue.*=...
  )
}
ruleset(name="ruleset-forward2"){
  action(type="omfwd" 
queue.*=...
  )
}

[1]
https://www.rsyslog.com/doc/v8-stable/rainerscript/rainerscript_call.html
[2] https://www.rsyslog.com/doc/v8-stable/whitepapers/queues_analogy.html

-- 
Peter
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] Relaying queue design

[Peter Viskup via rsyslog]

Want to come with final design of two level relays for syslog flow:
client -> relay11 -> -> dest1
client -> relay12 -> relay20 -> dest2
client -> relay13 -> -> dest3

Thought about the possibility to use mainQ in DA mode and omfwdQs (3 omfwd
over TCP) as small in-memory or direct only on relay20.

 - mainQ stores "clean" messages as retrieved
 - more efficient for disk space, no need for du/triplicate store
 - more reliable as there are no additional metadata in Q files which
might cause issue during dequeuing
 - actionQs store messages after templates were processed, including the
local variables
- less efficient for disk space
- no template processing during dequeuing
- previous issues with dequeuing when stored with local variables
 - actionQ in direct mode can block all subsequent forwards
 - actionQ in in-memory are not reliable in case of process/system crash
 - actionQ in in-memory can discard messages
 - avg. data troughput on input ~13GB/hour on relay20

Questions:
 - does all this make sense?
 - how actionQs before blocked direct Q will be processed?
 - may we expect message loss on actionQs after blocked direct Q once made
available again or will all messages in mainQ reach all the subsequent
actionQs?
 - do you have any other recommendations?

Thanks for all comments.

Peter
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] imfile state file changes

[Peter Viskup via rsyslog]

 The information with good explanation is available in the documentation
[1].

[1]
https://www.rsyslog.com/doc/v8-stable/configuration/modules/imfile.html#persiststateinterval

Peter

On Fri, Apr 12, 2019 at 2:29 PM John Chivian via rsyslog <
rsyslog@lists.adiscon.com> wrote:

> Hello Maintainers:
>
> I noticed that after going from v8.1901 to v8.1903 that imfile-state
> files are created almost immediately (within seconds or minutes) of the
> monitored file appearing.  I like this better than having to wait at
> shutdown for potentially hundreds (or thousands) of state files to be
> created, but I would like to understand when and under what
> circumstances are these files are created or updated.  Even a high level
> synopsis would be appreciated.
>
> Thanks and regards,
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] Status of imgssapi

[Peter Viskup via rsyslog]

Just looked for secured syslog transport in rsyslog other than TLS. Found
the imgssapi module [1].

Does the module support 'advanced' format configuration? It is not
mentioned in documentation. What is the experience from using this module?
Does it perform well?

[1]
https://www.rsyslog.com/doc/v8-stable/configuration/modules/imgssapi.html

-- 
Peter
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog 5.8 and ssh issue

[Peter Viskup via rsyslog]

We have been facing the same issue. It is related to "full buffer" for
/dev/log device, which is used by sudo, PAM, SSH and other services to log
authentication messages. The "unavailability" is caused by SSH not able to
write to /dev/log.
The same issue might appear with use of any other syslog server.

Peter

On Tue, Mar 12, 2019 at 6:32 PM Ani Sinha via rsyslog <
rsyslog@lists.adiscon.com> wrote:

> Hi guys,
>
> We rsyslog 5.8  in our centOS 6 based systems and we have bumped into the
> much discussed syslog issue where when log forwarding is enabled using tcp
> and the remote server is unavailable, after some time, the ssh connection
> to the host dies. I have been able to reproduce it and I believe the issue
> arises from the 2 sec default timeout interval set for messages to wait
> when the spooled queue (In-memory or disk or disk assisted) is completely
> full. Restarting the rsyslog service promptly bring ssh connectivity back.
>
> I have also experimented with rsyslog7 on centOS 6 system and I do not see
> the same issue.
>
> Can someone please throw some light as to what changed between rsyslog 5.8
> and rsyslog 7.10 that the issue has been fixed.  Are there other issues
> with rsyslog7 which I should be aware of?
>
> thanks
> ani
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Syslog Output File Generation Frequency (HOURLY) at Syslog Server

[Peter Viskup via rsyslog]

Copying logrotate to /etc/cron.daily is correct. Then logrotate will check
the configuration files and rotate only those logs which should be rotated
according the configuration.
Value of maxage is in days and you should probably change the value to 3 to
correspond with rotate value. Read the manpage for more information [1].

[1] https://manpages.debian.org/stretch/logrotate/logrotate.8.en.html

On Tue, Mar 12, 2019 at 7:01 AM sarjit yadav via rsyslog <
rsyslog@lists.adiscon.com> wrote:

> Thanks @Flo, it worked like charm.
>
> Also I would like to do housekeeping(*compression/rotation/deletion*) of
> specific syslog files with the help of *cron *& *logrotate *(*on hourly
> basis*).
>
>
> I thought of copying *logrotate *from* /etc/cron.daily/ *to
> */etc/cron.hourl*y but this will also impact every log in the system.
>
> And I want to rotate specific syslog files as hourly and rest to remain as
> usual way (daily), like -
>
> 1. Specific logs (var/log/NIPFW/MX480/*.log) on hourly basis
> 2. Remaining logs (/var/log/* etc.) as usual way (daily)
>
> Below is the config file for the same /etc/logrotate.d/syslog
> /var/log/NIPFW/MX480/*.log
> {
> rotate 50
> hourly
> copytruncate
> missingok
> notifempty
> compress
> delaycompress
> maxage 30
> sharedscripts
> postrotate
> reload rsyslog >/dev/null 2>&1 || true
> endscript
> }
>
> Any guidance will be much helpful.
> Thank You.
>
>
> On Tue, Mar 5, 2019 at 7:30 PM Flo Rance  wrote:
>
> > Did you try to simply add %$hour% ?
> >
> > E.g. "/var/log/NIPFW/MX480/CGNAT_PL_%$year%.%$month%.%$day%.%$hour%"
> >
> > https://www.rsyslog.com/doc/v8-stable/configuration/properties.html
> >
> > Flo
> >
> > On Tue, Mar 5, 2019 at 2:04 PM sarjit yadav via rsyslog <
> > rsyslog@lists.adiscon.com> wrote:
> >
> >> Hello All,
> >>
> >> I have implemented syslog server on CentOS and using below *Template *to
> >> generate output file names -
> >>
> >>
> >> *$template TmplNationalIP_PL,
> >> "/var/log/NIPFW/MX480/CGNAT_PL_%$year%.%$month%.%$day%"*
> >> *if ($msg contains 'OR_NAT' and $msg contains '55.91.165.') then
> >> ?TmplNationalIP_PL*
> >> *& ~*
> >>
> >>
> >> Similar to above (*daily*), I am looking to have syslog splitted output
> >> file names *HOURLY *(instead of daily).
> >>
> >> Can you please suggest how I can split syslog messages on HOURLY basis
> as
> >> I couldn't find any variables to be used to have HOURLY in output file
> >> names ?
> >>
> >> Many thanks in advance.
> >>
> >>
> >> --
> >>
> >> Regards
> >>
> >> Sarjit Singh
> >>
> >> *(**:  +91-8806664923*
> >> ___
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com/professional-services/
> >> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> >> DON'T LIKE THAT.
> >>
> >
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Having issues with discard rule

[Peter Viskup via rsyslog]

You can also use RSYSLOG_DebugFormat template [1] to log into a file. You
will be able to see what is the value of all properties.

[1] https://www.rsyslog.com/doc/v8-stable/configuration/templates.html


On Mon, Mar 11, 2019 at 10:00 PM Adam Chalkley  wrote:

> I'll defer to others more knowledgable than I, but the 'Oracle' portion at
> first glance appears to be the hostname.
>
> In case it is helpful, here is an example entry that we're using:
>
> if ($programname == 'vmsvc' ) then {
> if ($msg contains '[ warning] [guestinfo] Failed to get vmstats.')
> then {
>
> # Discard the message instead of logging locally or allowing it to
> # continue on to be forwarded remotely by subsequent rules.
> stop
>
> }
> }
>
> This is an example of an exact match and of using 'contains' to match part
> of the message.
>
> This rule is used for the open-vm-tools package provided by Ubuntu 14.04.
> That package has a bug which results in a message being logged every 30
> seconds, so we match and drop it.
>
> -Original Message-
> From: rsyslog  On Behalf Of Bryan
> Arenal via rsyslog
> Sent: Monday, March 11, 2019 3:39 PM
> To: rsyslog@lists.adiscon.com
> Cc: Bryan Arenal 
> Subject: [rsyslog] Having issues with discard rule
>
> Hi,
>
> I'm trying to set up a rule to discard some messages on a CentOS 7 box
> (8.24.0) but can't seem to get the syntax right.  It seems that either
> it's not working at all out or it discards EVERYTHING.  Here's a
> sample of something I'm trying to discard:
>
> Mar 11 16:58:04 Oracle Audit[14958]: LENGTH: "225" SESSIONID:[8]
> "25480410" ENTRYID:[1] "1" USERID:[6] "DBSNMP" ACTION:[3] "101"
> RETURNCODE:[1] "0" LOGOFF$PREAD:[1] "0" LOGOFF$LREAD:[2] "16"
> LOGOFF$LWRITE:[1] "0" LOGOFF$DEAD:[1] "0" DBID:[10] "1221313690"
> SESSIONCPU:[1] "1"
>
> I've used 'if $programname == 'Oracle Audit' then stop' but that
> doesn't seem to do the trick.  I've also tried using 'msg' but that
> also doesn't do anything.  What am I doing wrong?
>
> Thanks!
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] DA queue mode without message variables

[Peter Viskup via rsyslog]

Within the debugging of the issues with DA queues not dequeuing, caused by
already fixed bug [1], realized the DA queue consists of - standard syslog
and input properties and also of localvars json array.

[1] https://github.com/rsyslog/rsyslog/issues/1404

At first it is causing old versions of rsyslog (below 8.24) not able to
dequeue messages from these files. At second it is increasing the size of
queue.

Seems the same is valid for in-memory part of the queue. Does it make sense
or would it be possible to implement some special mode in which the message
will be "queued" before the variables/templates will be applied (in similar
way the main queue is storing the messages)? Those templates with variables
processing might be applied as a part of dequeueing.

Peter
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] DA queue not dequeuing

[Peter Viskup via rsyslog]

; 1103.467775663:FWD queue[DA]:Reg/w0: FWD queue[DA]: error -2308
> dequeueing element - ignoring, but strange things may happen
> >> > 1103.467786682:FWD queue[DA]:Reg/w0: regular consumer finished,
> iret=-2308, szlog 295422 sz phys 295760
> >> > 1103.467793132:FWD queue[DA]:Reg/w0: DeleteProcessedBatch: we deleted
> 0 objects and enqueued 0 objects
> >> > 1103.467799188:FWD queue[DA]:Reg/w0: doDeleteBatch: delete batch from
> store, new sizes: log 295422, phys 295760
> >> > 1103.467805425:FWD queue[DA]:Reg/w0: objDeserialize error -2029
> during header processing - trying to recover
> >> > 1103.467812104:FWD queue[DA]:Reg/w0: deserializer has possibly been
> able to re-sync and recover, state 0
> >> > 1103.467842910:FWD queue[DA]:Reg/w0: strm 0x561bc0075640: file 9 read
> 4096 bytes
> >> > 1103.467897344:FWD queue[DA]:Reg/w0: FWD queue[DA]: error -2308
> dequeueing element - ignoring, but strange things may happen
> >> > 1103.467913962:FWD queue[DA]:Reg/w0: regular consumer finished,
> iret=-2308, szlog 295421 sz phys 295760
> >> > 1103.467957133:FWD queue[DA]:Reg/w0: DeleteProcessedBatch: we deleted
> 0 objects and enqueued 0 objects
> >> > 1103.467964731:FWD queue[DA]:Reg/w0: doDeleteBatch: delete batch from
> store, new sizes: log 295421, phys 295760
> >> > 1103.467976624:FWD queue[DA]:Reg/w0: objDeserialize error -2029
> during header processing - trying to recover
> >> > 1103.467983341:FWD queue[DA]:Reg/w0: deserializer has possibly been
> able to re-sync and recover, state 0
> >> >
> >> > Seems strange. Any thoughts?
> >> >
> >> > Peter
> >> >
> >> >
> >> > On Wed, Mar 6, 2019 at 12:10 PM Rainer Gerhards <
> rgerha...@hq.adiscon.com> wrote:
> >> >>
> >> >> looks like two different queue. I need the full log - or at least a
> >> >> couple of thousand lines around the place of where you think the
> error
> >> >> is. The 2040 seems to be pretty ok, but the .1 file open above is
> >> >> not.
> >> >>
> >> >> Rainer
> >> >>
> >> >> El mié., 6 mar. 2019 a las 10:26, Peter Viskup ()
> escribió:
> >> >> >
> >> >> > Following is complete log entry with 3 lines up and down:
> >> >> >
> >> >> > 3292.107997776:main thread: file
> >> >> > '/var/spool/rsyslog/fq_fwd.0001' opened as #-1 with mode 384
> >> >> > 3292.108007930:main thread: strm 0x5557282b2370: open error 2,
> >> >> > file '/var/spool/rsyslog/fq_fwd.0001': No such file or
> directory
> >> >> > 3292.108014657:main thread: strm 0x5557282afb10: file
> >> >> > 6(/var/spool/rsyslog/fq_fwd.qi) closing
> >> >> > 3292.108023902:main thread: FWD queue[DA]: state -2040 reading
> .qi
> >> >> > file - can not read persisted info (if any)
> >> >> > 3292.108030552:main thread: file stream N/A params: flush
> interval
> >> >> > 0, async write 0
> >> >> > 3292.108043010:main thread: file stream N/A params: flush
> interval
> >> >> > 0, async write 0
> >> >> > 3292.108052306:main thread: file stream N/A params: flush
> interval
> >> >> > 0, async write 0
> >> >> >
> >> >> > Peter
> >> >> >
> >> >> > On Tue, Mar 5, 2019 at 3:05 PM Rainer Gerhards <
> rgerha...@hq.adiscon.com> wrote:
> >> >> > >
> >> >> > > El mar., 5 mar. 2019 a las 15:00, Peter Viskup via rsyslog
> >> >> > > () escribió:
> >> >> > > >
> >> >> > > > After rsyslog crash and recover.qi.pl run the DA queue is not
> dequeued.
> >> >> > > > Rsyslog debug prints the message from queue.c file [1].
> >> >> > > > What is could be the reason for this? Only some servers are
> affected
> >> >> > > > by this issue. Others dequeue just fine.
> >> >> > >
> >> >> > > What's the state reported in the message?
> >> >> > >
> >> >> > > Rainer
> >> >> > > >
> >> >> > > > [1]
> https://github.com/rsyslog/rsyslog/blob/master/runtime/queue.c#L863
> >> >> > > >
> >> >> > > > Peter
> >> >> > > > ___
> >> >> > > > rsyslog mailing list
> >> >> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> >> > > > http://www.rsyslog.com/professional-services/
> >> >> > > > What's up with rsyslog? Follow https://twitter.com/rgerhards
> >> >> > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED
> by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
> if you DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] DA queue not dequeuing

[Peter Viskup via rsyslog]

/w0: doDeleteBatch: delete batch from
> store, new sizes: log 295421, phys 295760
> > 1103.467976624:FWD queue[DA]:Reg/w0: objDeserialize error -2029 during
> header processing - trying to recover
> > 1103.467983341:FWD queue[DA]:Reg/w0: deserializer has possibly been able
> to re-sync and recover, state 0
> >
> > Seems strange. Any thoughts?
> >
> > Peter
> >
> >
> > On Wed, Mar 6, 2019 at 12:10 PM Rainer Gerhards <
> rgerha...@hq.adiscon.com> wrote:
> >>
> >> looks like two different queue. I need the full log - or at least a
> >> couple of thousand lines around the place of where you think the error
> >> is. The 2040 seems to be pretty ok, but the .1 file open above is
> >> not.
> >>
> >> Rainer
> >>
> >> El mié., 6 mar. 2019 a las 10:26, Peter Viskup ()
> escribió:
> >> >
> >> > Following is complete log entry with 3 lines up and down:
> >> >
> >> > 3292.107997776:main thread    : file
> >> > '/var/spool/rsyslog/fq_fwd.0001' opened as #-1 with mode 384
> >> > 3292.108007930:main thread: strm 0x5557282b2370: open error 2,
> >> > file '/var/spool/rsyslog/fq_fwd.0001': No such file or directory
> >> > 3292.108014657:main thread: strm 0x5557282afb10: file
> >> > 6(/var/spool/rsyslog/fq_fwd.qi) closing
> >> > 3292.108023902:main thread: FWD queue[DA]: state -2040 reading .qi
> >> > file - can not read persisted info (if any)
> >> > 3292.108030552:main thread: file stream N/A params: flush interval
> >> > 0, async write 0
> >> > 3292.108043010:main thread: file stream N/A params: flush interval
> >> > 0, async write 0
> >> > 3292.108052306:main thread: file stream N/A params: flush interval
> >> > 0, async write 0
> >> >
> >> > Peter
> >> >
> >> > On Tue, Mar 5, 2019 at 3:05 PM Rainer Gerhards <
> rgerha...@hq.adiscon.com> wrote:
> >> > >
> >> > > El mar., 5 mar. 2019 a las 15:00, Peter Viskup via rsyslog
> >> > > () escribió:
> >> > > >
> >> > > > After rsyslog crash and recover.qi.pl run the DA queue is not
> dequeued.
> >> > > > Rsyslog debug prints the message from queue.c file [1].
> >> > > > What is could be the reason for this? Only some servers are
> affected
> >> > > > by this issue. Others dequeue just fine.
> >> > >
> >> > > What's the state reported in the message?
> >> > >
> >> > > Rainer
> >> > > >
> >> > > > [1]
> https://github.com/rsyslog/rsyslog/blob/master/runtime/queue.c#L863
> >> > > >
> >> > > > Peter
> >> > > > ___
> >> > > > rsyslog mailing list
> >> > > > http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> > > > http://www.rsyslog.com/professional-services/
> >> > > > What's up with rsyslog? Follow https://twitter.com/rgerhards
> >> > > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
> you DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] DA queue not dequeuing

[Peter Viskup via rsyslog]

:FWD queue[DA]:Reg/w0: FWD queue[DA]: error -2308 dequeueing
element - ignoring, but strange things may happen
1103.44393:FWD queue[DA]:Reg/w0: regular consumer finished, iret=-2308,
szlog 295424 sz phys 295760
1103.466670503:FWD queue[DA]:Reg/w0: DeleteProcessedBatch: we deleted 0
objects and enqueued 0 objects
1103.466680360:FWD queue[DA]:Reg/w0: doDeleteBatch: delete batch from
store, new sizes: log 295424, phys 295760
1103.466686560:FWD queue[DA]:Reg/w0: objDeserialize error -2029 during
header processing - trying to recover
1103.466726572:FWD queue[DA]:Reg/w0: deserializer has possibly been able to
re-sync and recover, state 0
1103.466765643:FWD queue[DA]:Reg/w0: FWD queue[DA]: error -2308 dequeueing
element - ignoring, but strange things may happen
1103.466772781:FWD queue[DA]:Reg/w0: regular consumer finished, iret=-2308,
szlog 295423 sz phys 295760
1103.466779061:FWD queue[DA]:Reg/w0: DeleteProcessedBatch: we deleted 0
objects and enqueued 0 objects
1103.466785131:FWD queue[DA]:Reg/w0: doDeleteBatch: delete batch from
store, new sizes: log 295423, phys 295760
1103.466791068:FWD queue[DA]:Reg/w0: objDeserialize error -2029 during
header processing - trying to recover
1103.466805481:FWD queue[DA]:Reg/w0: strm 0x561bc0075640: file 9 read 46
bytes
1103.466813286:FWD queue[DA]:Reg/w0: strm 0x561bc0075640: file 9 read 0
bytes
1103.466819304:FWD queue[DA]:Reg/w0: strm 0x561bc0075640: file 9 EOF
1103.466830741:FWD queue[DA]:Reg/w0: strm 0x561bc0075640: file 9(fq_fwd)
closing
*1103.466855189:FWD queue[DA]:Reg/w0: file
'/var/spool/rsyslog/fq_fwd.0004' opened as #9 with mode 384*
*1103.466865694:FWD queue[DA]:Reg/w0: strm 0x561bc0075640: opened file
'/var/spool/rsyslog/fq_fwd.0004' for READ as 9*
1103.467635746:FWD queue[DA]:Reg/w0: strm 0x561bc0075640: file 9 read 4096
bytes
1103.467646862:FWD queue[DA]:Reg/w0: deserializer has possibly been able to
re-sync and recover, state 0
1103.467775663:FWD queue[DA]:Reg/w0: FWD queue[DA]: error -2308 dequeueing
element - ignoring, but strange things may happen
1103.467786682:FWD queue[DA]:Reg/w0: regular consumer finished, iret=-2308,
szlog 295422 sz phys 295760
1103.467793132:FWD queue[DA]:Reg/w0: DeleteProcessedBatch: we deleted 0
objects and enqueued 0 objects
1103.467799188:FWD queue[DA]:Reg/w0: doDeleteBatch: delete batch from
store, new sizes: log 295422, phys 295760
1103.467805425:FWD queue[DA]:Reg/w0: objDeserialize error -2029 during
header processing - trying to recover
1103.467812104:FWD queue[DA]:Reg/w0: deserializer has possibly been able to
re-sync and recover, state 0
1103.467842910:FWD queue[DA]:Reg/w0: strm 0x561bc0075640: file 9 read 4096
bytes
1103.467897344:FWD queue[DA]:Reg/w0: FWD queue[DA]: error -2308 dequeueing
element - ignoring, but strange things may happen
1103.467913962:FWD queue[DA]:Reg/w0: regular consumer finished, iret=-2308,
szlog 295421 sz phys 295760
1103.467957133:FWD queue[DA]:Reg/w0: DeleteProcessedBatch: we deleted 0
objects and enqueued 0 objects
1103.467964731:FWD queue[DA]:Reg/w0: doDeleteBatch: delete batch from
store, new sizes: log 295421, phys 295760
1103.467976624:FWD queue[DA]:Reg/w0: objDeserialize error -2029 during
header processing - trying to recover
1103.467983341:FWD queue[DA]:Reg/w0: deserializer has possibly been able to
re-sync and recover, state 0

Seems strange. Any thoughts?

Peter


On Wed, Mar 6, 2019 at 12:10 PM Rainer Gerhards 
wrote:

> looks like two different queue. I need the full log - or at least a
> couple of thousand lines around the place of where you think the error
> is. The 2040 seems to be pretty ok, but the .1 file open above is
> not.
>
> Rainer
>
> El mié., 6 mar. 2019 a las 10:26, Peter Viskup ()
> escribió:
> >
> > Following is complete log entry with 3 lines up and down:
> >
> > 3292.107997776:main thread: file
> > '/var/spool/rsyslog/fq_fwd.0001' opened as #-1 with mode 384
> > 3292.108007930:main thread: strm 0x5557282b2370: open error 2,
> > file '/var/spool/rsyslog/fq_fwd.0001': No such file or directory
> > 3292.108014657:main thread: strm 0x5557282afb10: file
> > 6(/var/spool/rsyslog/fq_fwd.qi) closing
> > 3292.108023902:main thread: FWD queue[DA]: state -2040 reading .qi
> > file - can not read persisted info (if any)
> > 3292.108030552:main thread: file stream N/A params: flush interval
> > 0, async write 0
> > 3292.108043010:main thread: file stream N/A params: flush interval
> > 0, async write 0
> > 3292.108052306:main thread: file stream N/A params: flush interval
> > 0, async write 0
> >
> > Peter
> >
> > On Tue, Mar 5, 2019 at 3:05 PM Rainer Gerhards 
> wrote:
> > >
> > > El mar., 5 mar. 2019 a las 15:00, Peter Viskup via rsyslog
> > > () escribió:
> > > >
> > > > After rsyslog crash and recover.qi.pl run the DA queue 

Re: [rsyslog] DA queue not dequeuing

[Peter Viskup via rsyslog]

Following is complete log entry with 3 lines up and down:

3292.107997776:main thread: file
'/var/spool/rsyslog/fq_fwd.0001' opened as #-1 with mode 384
3292.108007930:main thread: strm 0x5557282b2370: open error 2,
file '/var/spool/rsyslog/fq_fwd.0001': No such file or directory
3292.108014657:main thread: strm 0x5557282afb10: file
6(/var/spool/rsyslog/fq_fwd.qi) closing
3292.108023902:main thread: FWD queue[DA]: state -2040 reading .qi
file - can not read persisted info (if any)
3292.108030552:main thread: file stream N/A params: flush interval
0, async write 0
3292.108043010:main thread: file stream N/A params: flush interval
0, async write 0
3292.108052306:main thread: file stream N/A params: flush interval
0, async write 0

Peter

On Tue, Mar 5, 2019 at 3:05 PM Rainer Gerhards  wrote:
>
> El mar., 5 mar. 2019 a las 15:00, Peter Viskup via rsyslog
> () escribió:
> >
> > After rsyslog crash and recover.qi.pl run the DA queue is not dequeued.
> > Rsyslog debug prints the message from queue.c file [1].
> > What is could be the reason for this? Only some servers are affected
> > by this issue. Others dequeue just fine.
>
> What's the state reported in the message?
>
> Rainer
> >
> > [1] https://github.com/rsyslog/rsyslog/blob/master/runtime/queue.c#L863
> >
> > Peter
> > ___
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> > LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Syslog Output File Generation Frequency (HOURLY) at Syslog Server

[Peter Viskup via rsyslog]

Hello Sarjit,
give it a try to have a look on time-related properties documented [1].

[1] https://www.rsyslog.com/doc/v8-stable/configuration/properties.html

Peter

On Tue, Mar 5, 2019 at 2:16 PM sarjit yadav via rsyslog
 wrote:
>
> Hi Experts,
>
> Any suggestion below query.
>
> On Thu, Feb 21, 2019 at 2:52 PM sarjit yadav  wrote:
>
> > Hello All,
> >
> > I have implemented syslog server on CentOS and using below *Template *to
> > generate output file names -
> >
> >
> > *$template TmplNationalIP_PL,
> > "/var/log/NIPFW/MX480/CGNAT_PL_%$year%.%$month%.%$day%"*
> > *if ($msg contains 'OR_NAT' and $msg contains '55.91.165.') then
> > ?TmplNationalIP_PL*
> > *& ~*
> >
> >
> > Similar to above (*daily*), I am looking to have syslog splitted output
> > file names *HOURLY *(instead of daily).
> >
> > Can you please suggest how I can split syslog messages on HOURLY basis  as
> > I couldn't find any variables to be used to have HOURLY in output file
> > names ?
> >
> > Many thanks in advance.
> >
> >
> > --
> >
> > Regards
> >
> > Sarjit Singh
> >
> >
> >
> --
>
> Regards
>
> Sarjit Singh
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] DA queue not dequeuing

[Peter Viskup via rsyslog]

After rsyslog crash and recover.qi.pl run the DA queue is not dequeued.
Rsyslog debug prints the message from queue.c file [1].
What is could be the reason for this? Only some servers are affected
by this issue. Others dequeue just fine.

[1] https://github.com/rsyslog/rsyslog/blob/master/runtime/queue.c#L863

Peter
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] DA queue not dequeuing

[Peter Viskup via rsyslog]

After rsyslog crash and recover.qi.pl run the DA queue is not dequeued.
Rsyslog debug prints the message from queue.c file [1].
What is could be the reason for this? Only some servers are affected
by this issue. Others dequeue just fine.

[1] https://github.com/rsyslog/rsyslog/blob/master/runtime/queue.c#L863

Peter
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog impstats disk-assisted queue size/enqueued counters

[Peter Viskup via rsyslog]

Thank you, David.

The stat interval and severity are set accordingly, the "issue" is
with the counters not changed as it was expected. Just realized it
were my wrong thought.

The line "size=22946 enqueued=3270967" means the enqueued counter
already contains the amount of messages in "size". Therefore in the
next line the enqueued was not changed.

Sorry, disregard.

Peter

On Tue, Feb 5, 2019 at 11:12 PM David Lang  wrote:
>
> I believe that with the old syntax, you have to set all the values before you
> load the module, not after.
>
> This is one of the reasons why you should really use the new syntax. It makes 
> it
> much clearer what you are doing.
>
> David Lang
>
> On Tue, 5 Feb 2019, Peter Viskup via rsyslog wrote:
>
> > The load and configuration is done like this:
> >
> > $ModLoad impstats
> > $PStatInterval 15
> > $PStatSeverity 7
> >
> > Peter
> >
> > On Sun, Jan 20, 2019 at 5:09 PM Emmanuel Seyman  wrote:
> >>
> >> * Alberto [20/01/2019 14:27] :
> >>>
> >>> How do you load the module?
> >>
> >> I use:
> >>
> >> module(load="impstats"
> >>interval="86400"
> >>severity="7"
> >>log.syslog="off"
> >># need to turn log stream logging off!
> >>log.file="/var/log/rsyslog-stats.log")
> >>
> >>> Do you load with "ResetCounters" parameter?
> >>
> >> Nope.
> >>
> >> Emmanuel
> >> ___
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com/professional-services/
> >> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad 
> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you 
> >> DON'T LIKE THAT.
> > ___
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> > http://www.rsyslog.com/professional-services/
> > What's up with rsyslog? Follow https://twitter.com/rgerhards
> > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> > sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> > LIKE THAT.
> >
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog impstats disk-assisted queue size/enqueued counters

[Peter Viskup via rsyslog]

The load and configuration is done like this:

$ModLoad impstats
$PStatInterval 15
$PStatSeverity 7

Peter

On Sun, Jan 20, 2019 at 5:09 PM Emmanuel Seyman  wrote:
>
> * Alberto [20/01/2019 14:27] :
> >
> > How do you load the module?
>
> I use:
>
> module(load="impstats"
>interval="86400"
>severity="7"
>log.syslog="off"
># need to turn log stream logging off!
>log.file="/var/log/rsyslog-stats.log")
>
> > Do you load with "ResetCounters" parameter?
>
> Nope.
>
> Emmanuel
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Rsyslog vs syslog-ng

[Peter Viskup via rsyslog]

To be honest,
the main reason Debian chosen rsyslog as primary syslog daemon was
that it does work with "standard syslog" configuration (more
information can be read on https://wiki.debian.org/Rsyslog ).
Nevertheless in newest versions of rsyslog you are always recommended
to move to "rainer-script" configuration.

Latest open-sourced versions of syslog-ng provide the TLS encryption
for message forwarding.

Have a look on comparison of syslog-ng releases to have some quick reference.
https://www.balasys.hu/en/network-security/syslog-ng/opensource-logging-system/features/comparison

Both provide "reliable" syslog forwarding. Rsyslog open-sourced,
syslog-ng closed-sourced within enterprise support.

If your budget is large enough, you can pay for enterprise support.
https://www.rsyslog.com/professional-services/enterprise-support/
https://support.oneidentity.com/syslog-ng-premium-edition

>From my feeling on rsyslog it seems that this project has serious
issues with project management. More regression occur last year, even
last stable versions were released with serious bugs. But that's fruit
of our today's "agile development" mania. There are long standing
issues with TLS encryption still waiting for fix. Even when not having
experience with syslog-ng in large environments, it seems to me like
more mature project. Last year the Balabit company (originated in
Hungary), responsible for syslog-ng development, was bought by One
Identity.
https://www.oneidentity.com/balabit-acquisition/

To have a better feeling, you can check the list of issues for both projects
https://github.com/balabit/syslog-ng/issues
https://github.com/rsyslog/rsyslog/issues

After that you might be able to do serious decision.

Peter

On Mon, Feb 4, 2019 at 7:46 AM vishal via rsyslog
 wrote:
>
> Hi,
> I am evaluating rsyslog and syslogng for our project.
> Though aware of some of the differences and pros and cons, but still
> would like to know the differences which users have faced and evaluated
> in terms of ease of use, robustness, handling huge volumes of logs and
> deployment scenarios (single host to multi host cluster) , and if there
> are any other important areas to be considered.
>
> The general deployment would be,
>
> Log sources -> rsyslog/syslogng -> elasticsearch
>
>
> Thanks.
>
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] rsyslog impstats disk-assisted queue size/enqueued counters

[Peter Viskup via rsyslog]

Just discovered not expected behavior.
The DA queue size counter was changed, without change in enqueued counter.

~$ grep "Jan 15 12:23" /var/log/remotelogs/lin/rsyslog-lin.stats|grep main
Jan 15 12:23:07 127.0.0.1 syslog.debug rsyslogd-pstats:main Q[DA]:
origin=core.queue size=0 enqueued=3244357 full=0 discarded.full=0
discarded.nf=0 maxqsize=69184
Jan 15 12:23:07 127.0.0.1 syslog.debug rsyslogd-pstats:main Q:
origin=core.queue size=20 enqueued=1918753024 full=0 discarded.full=0
discarded.nf=0 maxqsize=7000
Jan 15 12:23:22 127.0.0.1 syslog.debug rsyslogd-pstats:main Q[DA]:
origin=core.queue size=22946 enqueued=3270967 full=0 discarded.full=0
discarded.nf=0 maxqsize=69184
Jan 15 12:23:22 127.0.0.1 syslog.debug rsyslogd-pstats:main Q:
origin=core.queue size=19 enqueued=1918815373 full=0 discarded.full=0
discarded.nf=0 maxqsize=7000
Jan 15 12:23:37 127.0.0.1 syslog.debug rsyslogd-pstats:main Q[DA]:
origin=core.queue size=0 enqueued=3270967 full=0 discarded.full=0
discarded.nf=0 maxqsize=69184
Jan 15 12:23:37 127.0.0.1 syslog.debug rsyslogd-pstats:main Q:
origin=core.queue size=19 enqueued=1918847027 full=0 discarded.full=0
discarded.nf=0 maxqsize=7000
Jan 15 12:23:52 127.0.0.1 syslog.debug rsyslogd-pstats:main Q[DA]:
origin=core.queue size=0 enqueued=3270967 full=0 discarded.full=0
discarded.nf=0 maxqsize=69184
Jan 15 12:23:52 127.0.0.1 syslog.debug rsyslogd-pstats:main Q:
origin=core.queue size=19 enqueued=1918852425 full=0 discarded.full=0
discarded.nf=0 maxqsize=7000

When the DA enqueued counter gets updated? Why it was not updated in
this situation?

Peter
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Problem with substring function

[Peter Viskup via rsyslog]

Hello Oliver,
try change line
set $!user_name = substring(exec_template("username"),2,4);
to lines:
set $!user_name_tmp = exec_template("username");
set $!user_name= substring($!user_name_tmp,2,4);

-- 
Peter
On Thu, Nov 22, 2018 at 3:49 PM Neumann, Oliver
 wrote:
>
> Hi there,
>
> I’m in trouble with generating user-based log files.
>
> template (name="user_file_name" type="string"
>   string="/srv/log/%$!user_name%.log"
> )
>
> template (name="username" type="string"
>   string="%msg:R,ERE,0,DFLT:\\[[a-z][a-z][0-9][0-9]\\]--end%"
> )
>
> set $!user_name = substring(exec_template("username"),2,4);
> action(type="omfile" name="cloud-audit" dynaFile="user_file_name" )
>
> Previous config should generate per user log-files with the path 
> /srv/log/ab12.log for example. But it generates only one log with 
> /srv/log/0.log.
>
> Using the set $!user_name without the substring function generates files like 
> /srv/log/[ab12].log what is nearly what i want, but without brackets.
>
> Any ideas how to remove those brackets?
>
> Sheers,
>
> Oli
>
>
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] How to filter remote log on specific directory with rsyslog centralized server

[Peter Viskup via rsyslog]

Hello Jean-Marie,
you can try to use exec_template [1] which was developed for such purposes.
This can be a base for your configuration

template(name="getFromhostip" type="string"
string="%fromhost-ip:R,ERE,0,DFLT:([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})--end%")
# do not forget the ';' character on the end of following line
set $.subnet=exec_template("getFromhostipprefix");
$template FILENAME,"/var/log/rsyslog_remote/%$.subnet%/%fromhost-ip%.log"

You can test your regexes on rsyslog page[2]. And read more on setting
variables[3].

Use of regexes affects performance and thus lookup tables might help
here[4] or use setting $.subnet variable value based on simple if-else
with match on '$fromhost-ip startswith "10.10.4."'.

[1] https://www.rsyslog.com/how-to-use-set-variable-and-exec_template/
[2] https://www.rsyslog.com/regex/
[3] https://www.rsyslog.com/how-to-set-variables-in-rsyslog-v7/
[4] https://www.rsyslog.com/doc/master/configuration/lookup_tables.html

-- 
Peter
On Wed, Nov 28, 2018 at 5:07 PM External Jean marie MAGNIER -CAMPUS-
via rsyslog  wrote:
>
> Hello All,
>
> I'am trying to deploy a provisionning solution for more than 10.000 CentOS
> linux client from PXE server. One of my goal is to consolidate build log on
> centralized remote server.
>
> Each client is able to send syslog to remote server. And my dificulties is
> to log by subnet, example :
>
> Client 1  10.10.4.xx/24
>
> Client 2 10.10.4.xy/24
>
> Client 3 10.10.5.xx/24
>
>
> *I try to log in centralized rsyslog server :*
> /var/log/rsyslog_remote/10.10.4//syslog.log
> /syslog.log
>  /10.10.5//syslog.log
>
>
> *But I found only solution to log in*
> /var/log/rsyslog_remote//syslog.log
>  //syslog.log
>  //syslog.log
>
> To do that I have a config file /etc/rsyslog.d/10-remote.conf
>
> * # Define customized target*
> * $template FILENAME,"/var/log/rsyslog_remote/%fromhost-ip%/syslog.log"*
> * $template LOCALFILENAME,"/var/log/rsyslog_local/%fromhost%.log"*
>
> * # write remote og in previous defined file*
> * :fromhost-ip, !isequal, "127.0.0.1" -?FILENAME*
> * & ~*
>
> * # write local log*
> * *.* -?LOCALFILENAME*
>
>
>
> *Request Help :*
> Maybe you have an idea  to define something like
>
>  $template FILENAME,"/var/log/rsyslog_remote/**
> /%fromhost-ip%/syslog.log"
>
>
>
> Thanks for your help
>
> --
> *Cordialement Jean-Marie*
>
>
> Magnier Jean-Marie  -  IS System Engineer - Prestataire
> *IT Department | IT Retail Workstation* Business Unit
> *CE-INFRARETAIL Team*
> @Mail  | Tel : +33(0)6.08.75.52.68
>
> We need your help to improve our services and your satisfaction !
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] lognorm1 rules with optional message part

[Peter Viskup via rsyslog]

On Mon, Nov 19, 2018 at 9:29 PM David Lang  wrote:
>
> On Mon, 19 Nov 2018, Peter Viskup via rsyslog wrote:
>
> > Special SD-ELEMENT [syslogTimes@123456 relay-ip="timestamp-rfc3339"
> > ...] added to the end of structured-data. Every relay add it's own
> > relay-ip with timestamp to this element.
>
> I would suggest not trying to parse this structured data with mmnormalize, let
> the rfc5424 parser parse it.

The rfc5424 parser provide this string:
STRUCTURED-DATA: '[syslogTimes@123456
10.x.y.z="2018-11-20T07:55:03.832066+01:00"]'
but I would like to cover the case with other SD-ELEMENTS and removal
only the syslogTimes.
e.g.
STRUCTURED-DATA: '[element1@123456 key1="value1"
key2="value2"][syslogTimes@123456
10.x.y.z="2018-11-20T07:55:03.832066+01:00"]'

>
> > On some relay's this SD-ELEMENT needs to be removed. Will this rule
> > work as expected?
> >
> > prefix=<%-:number%>%-:number% %-:date-rfc5424% %-:word% %-:word%
> > %-:number% %-:word%
> > rule=%orig-sd:string-to:[syslogTimes@123456 % %time-sd:string-to: ]% 
> > %-:rest%
> >
> > How to reference the 'orig-sd' value in template afterwards?
>
> log the message with the template RSYSLOG_DebugFormat and you will see the $!
> variable tree, with orig-sd under it, you would access it with $!orig-sd

The rule is not parsing the message as expected. In debug there are
only values in $!:
$!:{ "originalmsg": "pam_unix(sudo:session): session closed for user
root", "unparsed-data": "pam_unix(sudo:session): session closed for
user root" }

rawmsg looks like this (all message parts are parsed by rfc5424
without any issue):
inputname: imtcp rawmsg: '<86>1 2018-11-20T07:55:03.832066+01:00
hostname_10.x.y.z sudo - - [syslogTimes@123456
10.x.y.z="2018-11-20T07:55:03.832066+01:00"] pam_unix(sudo:session):
session closed for user root'

Peter
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] lognorm1 rules with optional message part

[Peter Viskup via rsyslog]

It is for the first time I am working with liblognorm.
Read the documentation for lognorm1, but still not sure how to write
mmnormalize rules for optional parts of syslog message.
The base is RFC5424 message with modified structured-data.

Special SD-ELEMENT [syslogTimes@123456 relay-ip="timestamp-rfc3339"
...] added to the end of structured-data. Every relay add it's own
relay-ip with timestamp to this element.

On some relay's this SD-ELEMENT needs to be removed. Will this rule
work as expected?

prefix=<%-:number%>%-:number% %-:date-rfc5424% %-:word% %-:word%
%-:number% %-:word%
rule=%orig-sd:string-to:[syslogTimes@123456 % %time-sd:string-to: ]% %-:rest%

How to reference the 'orig-sd' value in template afterwards?

Is lognorm2 making this easier to implement?

Peter
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] nanoseconds

[Peter Viskup via rsyslog]

On modern systems the quality of time on all CPU cores should be
synced and "guaranteed".

Some readings related to this topic:
http://jijithchandran.blogspot.com/2014/06/linux-kernel-time-calculation.html
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_for_real_time/7/html/reference_guide/sect-posix_clocks
https://aufather.wordpress.com/2010/09/08/high-performance-time-measuremen-in-linux/
https://en.wikipedia.org/wiki/Time_Stamp_Counter
https://en.wikipedia.org/wiki/High_Precision_Event_Timer
http://man7.org/linux/man-pages/man7/time.7.html

Peter

On Tue, Oct 30, 2018 at 2:33 AM David Lang  wrote:
>
> I question how real the time is, even across cores in a single machine, when 
> you
> get down to that sort of timekeeping.
>
> David Lang
>
> On Mon, 29 Oct 2018, Jason Nordwick wrote:
>
> > Date: Mon, 29 Oct 2018 16:19:23 -0700
> > From: Jason Nordwick 
> > Reply-To: rsyslog-users 
> > To: rsyslog@lists.adiscon.com
> > Subject: Re: [rsyslog] nanoseconds
> >
> > It's most useful when you get multiple log messages per microseond as a way
> > of ordering them as you can do with multiple cpu cores generating log
> > traffic simultaneously.
> >
> > On Mon, Oct 29, 2018 at 12:47 PM David Lang  wrote:
> >
> >> I think it would be a resonable extension  to rfc5424 to allow it's parser
> >> to
> >> accept more digits in the timestamp.
> >>
> >> I'm not sure that digits beyond microseconds really represent valid time,
> >> but I
> >> don't think it's a big deal to support it.
> >>
> >> David Lang
> >>
> >> On Mon, 29 Oct 2018, Peter Viskup via rsyslog wrote:
> >>
> >>> It might be possible to extend the rfc3339 time format to rfc3339nano,
> >>> but that will break rfc5424 which allow up to microseconds precision
> >>> only. Similar already in use when rfc3164 syslog messages used with
> >>> rfc3339 timestamps.
> >>>
> >>>
> >> https://github.com/rsyslog/rsyslog/search?p=1=date-rfc3339_q=date-rfc3339
> >>>
> >> https://github.com/rsyslog/rsyslog/search?q=formatTimestamp3339_q=formatTimestamp3339
> >>>
> >> https://github.com/rsyslog/rsyslog/search?q=tplFmtRFC3339Date_q=tplFmtRFC3339Date
> >>> https://tools.ietf.org/html/rfc5424
> >>>
> >>> Consider opening github request for the implementation.
> >>>
> >>> Peter
> >>> On Fri, Oct 26, 2018 at 11:43 PM Jason Nordwick 
> >> wrote:
> >>>>
> >>>> Is there a way to get nanoseconds in and out of rsyslog? I'm using
> >>>> dateformat rfc3339, and it works find for micros, but at nanos, the
> >>>> timestamp gets correupted:
> >>>>
> >>>> property(name="timereported" dateFormat="rfc3339")
> >>>>
> >>>> Also, when writing out json formatted fields, is there a way to print
> >> out a
> >>>> numeric property? I would like to make syslogseverity a number, but I
> >> have
> >>>> been unsuccessful in creating a numeric json field. My workaround is to
> >>>> handcraft the entire json payload instead of using the jsonf option  on
> >> the
> >>>> template.
> >>>>
> >>>> Thanks,
> >>>> Jason
> >>>> ___
> >>>> rsyslog mailing list
> >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>>> http://www.rsyslog.com/professional-services/
> >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
> >> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if
> >> you DON'T LIKE THAT.
> >>> ___
> >>> rsyslog mailing list
> >>> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >>> http://www.rsyslog.com/professional-services/
> >>> What's up with rsyslog? Follow https://twitter.com/rgerhards
> >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> >> DON'T LIKE THAT.
> >>>
> >> ___
> >> rsyslog mailing list
> >> http://lists.adiscon.net/mailman/listinfo/rsyslog
> >> http://www.rsyslog.com/professional-services/
> &

Re: [rsyslog] nanoseconds

[Peter Viskup via rsyslog]

It might be possible to extend the rfc3339 time format to rfc3339nano,
but that will break rfc5424 which allow up to microseconds precision
only. Similar already in use when rfc3164 syslog messages used with
rfc3339 timestamps.

https://github.com/rsyslog/rsyslog/search?p=1=date-rfc3339_q=date-rfc3339
https://github.com/rsyslog/rsyslog/search?q=formatTimestamp3339_q=formatTimestamp3339
https://github.com/rsyslog/rsyslog/search?q=tplFmtRFC3339Date_q=tplFmtRFC3339Date
https://tools.ietf.org/html/rfc5424

Consider opening github request for the implementation.

Peter
On Fri, Oct 26, 2018 at 11:43 PM Jason Nordwick  wrote:
>
> Is there a way to get nanoseconds in and out of rsyslog? I'm using
> dateformat rfc3339, and it works find for micros, but at nanos, the
> timestamp gets correupted:
>
> property(name="timereported" dateFormat="rfc3339")
>
> Also, when writing out json formatted fields, is there a way to print out a
> numeric property? I would like to make syslogseverity a number, but I have
> been unsuccessful in creating a numeric json field. My workaround is to
> handcraft the entire json payload instead of using the jsonf option  on the
> template.
>
> Thanks,
> Jason
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Combining two working rsyslog.conf files

[Peter Viskup via rsyslog]

Show the final config you are trying to run.

It could be related to $DefaultNetstreamDriver* options which should
be mentioned only once.
https://www.rsyslog.com/doc/v8-stable/rainerscript/global.html?highlight=defaultnetstreamdriver

In case it is needed, you can copy systemd rsyslog.service file and
create new for second instance (both running different certs).
http://rsyslog-users.1305293.n2.nabble.com/Mix-of-GTLS-and-PTCP-listeners-running-same-instance-td7591434.html#a7591445

Peter
On Thu, Oct 25, 2018 at 11:22 PM Rory Toma via rsyslog
 wrote:
>
> I have two separate files that work just fine. I have not been able to
> successfully combine them. No matter what I try, I keep getting tls
> errors, because one or the other is using wrong certs. Can anyone help here?
>
> file1:
> $DefaultNetstreamDriver gtls
>
> # certificate files
> $DefaultNetstreamDriverCAFile /opt/rsyslog/certs/ca.pem
> $DefaultNetstreamDriverCertFile /opt/rsyslog/certs/cert.pem
> $DefaultNetstreamDriverKeyFile /opt/rsyslog/certs/key.pem
>
> $MaxOpenFiles 10
>
> module(load="imtcp" MaxSessions="65534" StreamDriver.Mode="1"
> StreamDriver.AuthMode="anon") # load TCP listener
>
> $WorkDirectory /export/rsyslog
> $ActionQueueType LinkedList
> $ActionQueueFileName srvrfwd
> $ActionResumeRetryCount -1
> $ActionQueueSaveOnShutdown on
>
> ruleset(name="remote"){
>  *.* @@10.66.13.148:8514
> }
>
> $InputTCPServerBindRuleset remote
> $InputTCPServerRun 110
>
>
> file2:
> $DefaultNetstreamDriver gtls
> $DefaultNetStreamDriverCAFile /opt/rsyslog/certs/relp/ca.pem
> $DefaultNetStreamDriverCertFile /opt/rsyslog/certs/relp/cert.pem
> $DefaultNetStreamDriverKeyFile /opt/rsyslog/certs/relp/key.pem
>
> $WorkDirectory /export/rsyslog
> $ActionQueueType LinkedList
> $ActionQueueFileName srvrfws
> $ActionResumeRetryCount -1
> $ActionQueueSaveOnShutdown on
>
> module(load="imrelp" ruleset="relp")
>
> input(type="imrelp" port="114" tls="on" tls.compression="on"
> tls.authmode="fingerprint" )
>
> ruleset(name="relp") {
> *.* @@10.66.13.148:8514
> }
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] Monitoring message delay

[Peter Viskup via rsyslog]

Interested in monitoring delay of message retrieval in syslog infrastructure.
We have syslog infrastructure with more rsyslog relays in chain and
would like to monitor the diff in times between timegenerated and
timereported.
Requirement is to be alerted when the messages will be delayed
reaching defined threshold.
What would be the best way to implement this?

-- 
Peter
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] how to get the original IP address in a relay chain

[Peter Viskup via rsyslog]

syslog-ng has special chain-hostname option for that.

You can simulate it with exec_template with use of standard syslog format:
http://rsyslog-users.1305293.n2.nabble.com/template/NamlServlet.jtp?macro=print_post=7594015

HTH
-- 
Peter
On Wed, Oct 17, 2018 at 1:38 AM wuhe  wrote:
>
>
>
> Thanks David for your reply,
> Actually i do the similar thing , i store the msg to "Relay Server" and use 
> template to append the "fromhost-ip" , then use "imfile " to resend the msg 
> with fromhost-ip to "log server"
> but this cause disk IO on relay server which worried about the disk 
> performance when client are huge
>  i never use JSON in rsyslog before, i will try you method later
> Another thing want to conform is this solution involve disk IO on "relay 
> server" ?
> Thanks!
>
>
> Regards!
> /Wu
> At 2018-10-17 06:28:19, "David Lang"  wrote:
> >On Wed, 17 Oct 2018, wuhe wrote:
> >
> >> Hi :
> >> want to check how to get the original IP address after forward twice 
> >> in a relay chain?
> >> like in this chain, how can the "Log Server" get the IP address (not 
> >> hostname ) of "Client A/B/C"
> >> {Client A, Client B, Client C} > {Relay Server} > {Log Server} 
> >>  (udp is used for forward)
> >
> >using the standard syslog format you cannot do so because the relays do no 
> >pass
> >that information on.
> >
> >What I do is I repackage the message as JSON so that I can add additional
> >metadata
> >
> >on the relay:
> >
> >set $!msg=$msg;
> >set $!trusted!orig!ip = $fromhost-ip;
> >
> >etc (I also have the relays store the timestamp of when they processed the 
> >log,
> >which relay processed it, parse the message, and do other cleanups)
> >
> >then define a forwarding format that has %$!% instead of $msg in it.
> >
> >then on the log server, you can extract the metadata or the original message
> >
> >David Lang
> >___
> >rsyslog mailing list
> >http://lists.adiscon.net/mailman/listinfo/rsyslog
> >http://www.rsyslog.com/professional-services/
> >What's up with rsyslog? Follow https://twitter.com/rgerhards
> >NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> >sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> >LIKE THAT.
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T 
> LIKE THAT.
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] TCP Keepalive on client and server side

[Peter Viskup via rsyslog]

Ack,
will check after upgrade.

As an workaround the tcp_retries2 kernel option was lowered according
https://pracucci.com/linux-tcp-rto-min-max-and-tcp-retries2.html
This make us sure the TCP forward session will be recognized as broken
sooner than default 924 seconds. We are loosing messages waiting in
TCP buffer on client side, but accepted.

-- 
Peter
On Tue, Oct 16, 2018 at 9:17 AM Rainer Gerhards
 wrote:
>
> You need to update to the current verson (8.38.0) first before I can
> look at this.
>
> Rainer
> El jue., 11 oct. 2018 a las 15:32, Peter Viskup
> () escribió:
> >
> > Thank you Rainer,
> > the Changelog answered why client is not answering keepalive packets
> > (bug fixed in 8.18).
> >
> > What about the TCP session open on client side?
> > This happen every 16 seconds in parallel with other TCP session opened
> > and used for data transfer.
> >
> > Following is session export from pcap:
> > "5606","316.433579","client","server","TCP","74","0x4da5
> > (19877)","51112 → 1514 [SYN] Seq=0 Win=29200 Len=0 MSS=1460
> > SACK_PERM=1 TSval=2136435442 TSecr=0 WS=128"
> > "5612","316.702389","server","client","TCP","74","0x (0)","1514 →
> > 51112 [SYN, ACK] Seq=0 Ack=1 Win=28960 Len=0 MSS=1350 SACK_PERM=1
> > TSval=339748485 TSecr=2136435442 WS=128"
> > "5613","316.702472","client","server","TCP","66","0x4da6
> > (19878)","51112 → 1514 [ACK] Seq=1 Ack=1 Win=29312 Len=0
> > TSval=2136435509 TSecr=339748485"
> > "5614","316.702562","client","server","TCP","66","0x4da7
> > (19879)","51112 → 1514 [FIN, ACK] Seq=1 Ack=1 Win=29312 Len=0
> > TSval=2136435509 TSecr=339748485"
> > "5615","316.970442","server","client","TCP","66","0x39d7
> > (14807)","1514 → 51112 [FIN, ACK] Seq=1 Ack=2 Win=29056 Len=0
> > TSval=339748552 TSecr=2136435509"
> > "5616","316.970503","client","server","TCP","66","0x4da8
> > (19880)","51112 → 1514 [ACK] Seq=2 Ack=2 Win=29312 Len=0
> > TSval=2136435576 TSecr=339748552"
> >
> > There are no data transferred.
> >
> > Peter
> >
> > On Thu, Oct 11, 2018 at 12:58 PM Rainer Gerhards
> >  wrote:
> > >
> > > I did a quick search through the ChangeLog
> > >
> > > https://github.com/rsyslog/rsyslog/blob/master/ChangeLog
> > >
> > > and it shows ample changes since Dec 2015 (8.15). Have a look yourself.
> > >
> > > HTH
> > > Rainer
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] TCP Keepalive on client and server side

[Peter Viskup via rsyslog]

Thank you Rainer,
the Changelog answered why client is not answering keepalive packets
(bug fixed in 8.18).

What about the TCP session open on client side?
This happen every 16 seconds in parallel with other TCP session opened
and used for data transfer.

Following is session export from pcap:
"5606","316.433579","client","server","TCP","74","0x4da5
(19877)","51112 → 1514 [SYN] Seq=0 Win=29200 Len=0 MSS=1460
SACK_PERM=1 TSval=2136435442 TSecr=0 WS=128"
"5612","316.702389","server","client","TCP","74","0x (0)","1514 →
51112 [SYN, ACK] Seq=0 Ack=1 Win=28960 Len=0 MSS=1350 SACK_PERM=1
TSval=339748485 TSecr=2136435442 WS=128"
"5613","316.702472","client","server","TCP","66","0x4da6
(19878)","51112 → 1514 [ACK] Seq=1 Ack=1 Win=29312 Len=0
TSval=2136435509 TSecr=339748485"
"5614","316.702562","client","server","TCP","66","0x4da7
(19879)","51112 → 1514 [FIN, ACK] Seq=1 Ack=1 Win=29312 Len=0
TSval=2136435509 TSecr=339748485"
"5615","316.970442","server","client","TCP","66","0x39d7
(14807)","1514 → 51112 [FIN, ACK] Seq=1 Ack=2 Win=29056 Len=0
TSval=339748552 TSecr=2136435509"
"5616","316.970503","client","server","TCP","66","0x4da8
(19880)","51112 → 1514 [ACK] Seq=2 Ack=2 Win=29312 Len=0
TSval=2136435576 TSecr=339748552"

There are no data transferred.

Peter

On Thu, Oct 11, 2018 at 12:58 PM Rainer Gerhards
 wrote:
>
> I did a quick search through the ChangeLog
>
> https://github.com/rsyslog/rsyslog/blob/master/ChangeLog
>
> and it shows ample changes since Dec 2015 (8.15). Have a look yourself.
>
> HTH
> Rainer
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] TCP Keepalive on client and server side

[Peter Viskup via rsyslog]

>From my latest observation it seems the TCP Keepalive is not working as
expected in our environment. We do run rsyslog 8.15, which I know is old,
but cannot update.

Want to make sure how the TCP Keepalive is developed in rsyslog and whether
there were some changes since 8.15 release. At the moment not interested in
deployment of RELP.

My understanding from reading TCP-Keepalive-HOWTO
 is both sides can
send empty ACK packet and the other side should response with empty ACK.
That way the TCP session will remain open and all the network devices will
keep track of it.

In our environment the rsyslog server send ACK with no data, but client
does not reply. After some time the rsyslog server closes the connection
sending TCP RST packet.
>From the client side the rsyslog tries to open new TCP connection. Once
opened, the client just closes it. It seems to me like client's TCP
Keepalive.

Why the client does not sent ACK reply to server's keepalive?
Is the client's TCP session opening the proper way to implement keepaliving
on client side?

-- 
Peter
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Forward template name based on variable

[Peter Viskup via rsyslog]

Want to use different forward formatting according the source message
format. Just discovered some devices use RFC5424, others BSD syslog.
Want to forward them with modified hostname, but with all other
properties untouched.
Use this config, but always getting the "newSyslogMessage" (message
never pass the standardSyslogMessage format).
It seems the 'if ( $structured-data != "" )' condition is not proper.
The 'if ( $structured-data != "-" )' didn't work either.
Is the $structured-data property reference valid? Debug format show
value of "-" for messages received in BSD syslog format.
Or is there other way to do the same change in %hostname%?

template(name="testing" type="string"
string="%$.message%")

template(name="standardSyslogMessage" type="string"
string="<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME%%$.ip% %syslogtag%
%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n")
template(name="newSyslogMessage" type="string" string="<%PRI%>1
%TIMESTAMP:::date-rfc3339% %HOSTNAME%%$.ip% %APP-NAME% %PROCID%
%MSGID% %structured-data% %msg%\n")


template(name="getFromhostip" type="string" string="_%fromhost-ip%")

if ( $hostname == $fromhost-ip or $fromhost-ip == "127.0.0.1" ) then {
set $.ip="";
}
else {
set $.ip=exec_template("getFromhostip");
}

if ( $structured-data != "" ) then {
set $.message=exec_template("newSyslogMessage");
}
else {
set $.message=exec_template("standardSyslogMessage");
}
On Wed, Sep 26, 2018 at 3:16 PM Peter Viskup  wrote:
>
> Thank you, David!
> Got it working with following configuration.
>
> template(name="FileFormatDyn" type="string"
> string="%TIMESTAMP:::date-rfc3339% %HOSTNAME%%$.ip%
> %syslogfacility-text%.%syslogseverity-text%
> %syslogtag%%msg:::sp-if-no-1st-sp% %msg:::drop-last-lf%\n")
>
> template(name="getFromhostip" type="string" string="_%fromhost-ip%")
>
> if ( $hostname == $fromhost-ip or $fromhost-ip == "127.0.0.1" ) then {
> set $.ip="";
> }
> else {
> set $.ip=exec_template("getFromhostip");
> }
>
> action(type="omfile" file="/var/log/lin/lin-dyna.log" 
> template="FileFormatDyn")
> On Wed, Sep 26, 2018 at 2:56 AM David Lang  wrote:
> >
> > On Tue, 25 Sep 2018, Peter Viskup via rsyslog wrote:
> >
> > > Is it possible to configure omfwd action with template name chosen by 
> > > variable?
> > >
> > > Want to use different template according the hostname value
> > > (simplified example):
> > >
> > > $template fwdrelay1,"<%PRI%>%TIMESTAMP:::date-rfc3339%
> > > %fromhost-ip%-%hostname% %syslogtag%%msg:::drop-last-lf%\n"
> > > $template fwdrelay2,"<%PRI%>%TIMESTAMP:::date-rfc3339% %fromhost-ip%
> > > %syslogtag%%msg:::drop-last-lf%\n"
> > >
> > > if ( $hostname == '127.0.0.1' ) then $.tmpl="fwdrelay1"
> > > else $.tmpl="fwdrelay2"
> > > *.* action (type="omfwd" template=tmpl target= protocol=tcp)
> >
> > no, template can only refer to a constant, but you could have the template 
> > be
> >
> > $template foo,"$.bar"
> >
> > and then do
> > if ( $hostname == '127.0.0.1' ) then set $.bar=exec_template("fwdrelay1");
> > else set $.bar=exec_template("fwdrelay2");
> >
> > action(template=foo)
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] Forward template name based on variable

[Peter Viskup via rsyslog]

Thank you, David!
Got it working with following configuration.

template(name="FileFormatDyn" type="string"
string="%TIMESTAMP:::date-rfc3339% %HOSTNAME%%$.ip%
%syslogfacility-text%.%syslogseverity-text%
%syslogtag%%msg:::sp-if-no-1st-sp% %msg:::drop-last-lf%\n")

template(name="getFromhostip" type="string" string="_%fromhost-ip%")

if ( $hostname == $fromhost-ip or $fromhost-ip == "127.0.0.1" ) then {
set $.ip="";
}
else {
set $.ip=exec_template("getFromhostip");
}

action(type="omfile" file="/var/log/lin/lin-dyna.log" template="FileFormatDyn")
On Wed, Sep 26, 2018 at 2:56 AM David Lang  wrote:
>
> On Tue, 25 Sep 2018, Peter Viskup via rsyslog wrote:
>
> > Is it possible to configure omfwd action with template name chosen by 
> > variable?
> >
> > Want to use different template according the hostname value
> > (simplified example):
> >
> > $template fwdrelay1,"<%PRI%>%TIMESTAMP:::date-rfc3339%
> > %fromhost-ip%-%hostname% %syslogtag%%msg:::drop-last-lf%\n"
> > $template fwdrelay2,"<%PRI%>%TIMESTAMP:::date-rfc3339% %fromhost-ip%
> > %syslogtag%%msg:::drop-last-lf%\n"
> >
> > if ( $hostname == '127.0.0.1' ) then $.tmpl="fwdrelay1"
> > else $.tmpl="fwdrelay2"
> > *.* action (type="omfwd" template=tmpl target= protocol=tcp)
>
> no, template can only refer to a constant, but you could have the template be
>
> $template foo,"$.bar"
>
> and then do
> if ( $hostname == '127.0.0.1' ) then set $.bar=exec_template("fwdrelay1");
> else set $.bar=exec_template("fwdrelay2");
>
> action(template=foo)
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] Forward template name based on variable

[Peter Viskup via rsyslog]

Is it possible to configure omfwd action with template name chosen by variable?

Want to use different template according the hostname value
(simplified example):

$template fwdrelay1,"<%PRI%>%TIMESTAMP:::date-rfc3339%
%fromhost-ip%-%hostname% %syslogtag%%msg:::drop-last-lf%\n"
$template fwdrelay2,"<%PRI%>%TIMESTAMP:::date-rfc3339% %fromhost-ip%
%syslogtag%%msg:::drop-last-lf%\n"

if ( $hostname == '127.0.0.1' ) then $.tmpl="fwdrelay1"
else $.tmpl="fwdrelay2"
*.* action (type="omfwd" template=tmpl target= protocol=tcp)

Peter
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.