Re: [rsyslog] rsyslog fails to start due to high queue

[David Lang]

I will point out that no matter what software you run, you will eventually run 
into a case like this where the distro packages break for you, and you are faced 
with the need to run non-standard packages to work around a bug.


As Rainer points out, if you aren't willing to compile your own version, even if 
he were to find that the bug was not fixed in the latest version, and create a 
fix for it, you still wouldn't be able to use the fix.


David Lang


On Fri, 25 Nov 2016, Kosta Psimoulis wrote:


Thank you for much guys for your support, right now I have a much better
picture of what is going on. I am aware how to build from source but this
would be something difficult to maintain, I would probably need to create a
custom repo. I think I have enough information right now to reconsider and
evaluate a business decision whether to use rsyslog or look for another
solution. I have some queue files that I can recover and rebuild with the
information you have given but I am still not sure of how they got
corrupted and what happened to the ones that were in memory, was everything
saved on the hard drive or was there information that was lost.

Kind Regards,
Kosta

On Fri, Nov 25, 2016 at 3:42 AM, David Lang  wrote:


On Fri, 25 Nov 2016, mosto...@gmail.com wrote:

TBH, it depends if you prefer building from source or instability

Kidding away, I know there are problems in older versions, they are


Isn't adiscon repo valid?



It looks like we don't have a repo for Jessie, at least it's not included
in the scripts/config.sh for rsyslog-pkg-debian

This probably also means we don't have a Travis test box running Jessie.

Packages for Wheezy will probably work (the only thing I can think of that
would be likely to break is gnutls related dependencies)

David Lang

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog fails to start due to high queue

[mosto...@gmail.com]

Agains't Rainer's advice, we are using adiscon repos and we're quite 
happy with them...



El 25/11/16 a las 09:59, Kosta Psimoulis escribió:

Thank you for much guys for your support, right now I have a much better
picture of what is going on. I am aware how to build from source but this
would be something difficult to maintain, I would probably need to create a
custom repo. I think I have enough information right now to reconsider and
evaluate a business decision whether to use rsyslog or look for another
solution. I have some queue files that I can recover and rebuild with the
information you have given but I am still not sure of how they got
corrupted and what happened to the ones that were in memory, was everything
saved on the hard drive or was there information that was lost.

Kind Regards,
Kosta

On Fri, Nov 25, 2016 at 3:42 AM, David Lang  wrote:


On Fri, 25 Nov 2016, mosto...@gmail.com wrote:

TBH, it depends if you prefer building from source or instability

Kidding away, I know there are problems in older versions, they are


Isn't adiscon repo valid?


It looks like we don't have a repo for Jessie, at least it's not included
in the scripts/config.sh for rsyslog-pkg-debian

This probably also means we don't have a Travis test box running Jessie.

Packages for Wheezy will probably work (the only thing I can think of that
would be likely to break is gnutls related dependencies)

David Lang

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog fails to start due to high queue

[Kosta Psimoulis]

Thank you for much guys for your support, right now I have a much better
picture of what is going on. I am aware how to build from source but this
would be something difficult to maintain, I would probably need to create a
custom repo. I think I have enough information right now to reconsider and
evaluate a business decision whether to use rsyslog or look for another
solution. I have some queue files that I can recover and rebuild with the
information you have given but I am still not sure of how they got
corrupted and what happened to the ones that were in memory, was everything
saved on the hard drive or was there information that was lost.

Kind Regards,
Kosta

On Fri, Nov 25, 2016 at 3:42 AM, David Lang  wrote:

> On Fri, 25 Nov 2016, mosto...@gmail.com wrote:
>
> TBH, it depends if you prefer building from source or instability
>>> Kidding away, I know there are problems in older versions, they are
>>>
>> Isn't adiscon repo valid?
>>
>
> It looks like we don't have a repo for Jessie, at least it's not included
> in the scripts/config.sh for rsyslog-pkg-debian
>
> This probably also means we don't have a Travis test box running Jessie.
>
> Packages for Wheezy will probably work (the only thing I can think of that
> would be likely to break is gnutls related dependencies)
>
> David Lang
>
> ___
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
> http://www.rsyslog.com/professional-services/
> What's up with rsyslog? Follow https://twitter.com/rgerhards
> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad
> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
> DON'T LIKE THAT.
>
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog fails to start due to high queue

[David Lang]

On Fri, 25 Nov 2016, mosto...@gmail.com wrote:


TBH, it depends if you prefer building from source or instability
Kidding away, I know there are problems in older versions, they are

Isn't adiscon repo valid?


It looks like we don't have a repo for Jessie, at least it's not included in the 
scripts/config.sh for rsyslog-pkg-debian


This probably also means we don't have a Travis test box running Jessie.

Packages for Wheezy will probably work (the only thing I can think of that would 
be likely to break is gnutls related dependencies)


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog fails to start due to high queue

[Rainer Gerhards]

2016-11-25 9:38 GMT+01:00 mosto...@gmail.com :
>
>> TBH, it depends if you prefer building from source or instability
>> Kidding away, I know there are problems in older versions, they are
>
> Isn't adiscon repo valid?

We don't have everything. Too few contributors :-(

Rainer
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog fails to start due to high queue

[mosto...@gmail.com]

TBH, it depends if you prefer building from source or instability
Kidding away, I know there are problems in older versions, they are

Isn't adiscon repo valid?
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog fails to start due to high queue

[David Lang]

On Fri, 25 Nov 2016, Kosta Psimoulis wrote:


Hello Rainer,

Thank you for your honest answer, so you are basically saying that the
queue will be lost ? I thought that my current version of rsyslog was
stable enough, I guess I was wrong, in that case I will try a different
queue system. I am sending a little bit extra debugging information, please
let me know if you see anything useful and if you believe that this issue
was fixed in 8.23


we do believe that it is fixed in the current version. Unfortunantly, you aren't 
running the current version. 8.4 is rather old, and even 8.16 is prior to some 
very substantial reliability fixes, including several that caused segfaults.


In the short term, move the queue files to another directory and see if rsyslog 
starts. If so, we can look at trying to fix the queue files (there is a command 
that will rebuild the .qi file from the raw queue files, that may solve your 
issues.


with the exception of encryption, the packages for wheezy should have no 
problems running on Jessie (although I agree that after a year we should be 
providing a Jessie repo, nobody called it to our attention, so we missed it)






Unfortunantly, due to manpower limits, we (the community) can only support the 
most recent release of rsyslog, we can't do backports of fixes (although that is 
an option for people who get paid support). We tried to do that sort of thing 
and the project almost died under the load.


When you rely on the distro provided versions instead of the upstream releases, 
it's on the basis that the distro is providing support for them. If the distro 
isn't providing support for a bug that you are running into, you need to look at 
shifting to the upstream version.


Sometimes this is going to require that you maintain your own version of the 
package, built from source (to apply a fix, or because you need a compile-time 
option that's not in the default packages, or to use a new feature)


The number of fixes since 8.16 (let alone 8.4) is huge, trying to backport them 
all, without backporting the other features added would probably result in a 
less reliable system than just running the latest 8.24 version.


IIRC, 8.16 was the last release before we started using libfastjson, and part of 
the reason why Debian hasn't picked up the newer versions, even into backports, 
is concerns over long-term maintinance of libfastjson. But it fixed so many 
errors that we killing people's systems that we really need to shift.




In theory, you should be able to clone the source git repos, and the 
rsyslog-pkg-debian repo and in the rsyslog-pkg-debian repo under 
scripts/config.sh add Jessie to the distro list and everything should work.


We just haven't tested it yet

(and yes, there is a bit more to getting this going, contact me off-list and 
I'll pass along the build scripts I put together for ubuntu, they should work 
almost the same on Debian)


David Lang



Kind Regards,
Kosta

1388.716086990:action 3 queue[DA]:Reg/w0: in destructor: sendbuf 0xada9030
1388.716159868:action 3 queue[DA]:Reg/w0: relp engine is dispatching frame
with command 'rsp'
1388.716530324:action 3 queue[DA]:Reg/w0: relpSessWaitState returns 10019
1388.718355599:action 3 queue[DA]:Reg/w0: in destructor: sendbuf 0xada9570
1388.719729919:action 3 queue[DA]:Reg/w0: relp engine created new client
0xadae250
1388.719865043:action 3 queue[DA]:Reg/w0: omrelp: endTransaction
==16816== Thread 2 rs:action 3 queue[DA]:R:
==16816== Invalid read of size 8
==16816==at 0x76FEE94: relpCltHintBurstEnd (in
/usr/lib/x86_64-linux-gnu/librelp.so.0.1.0)
==16816==by 0x74EFB7A: ??? (in /usr/lib/rsyslog/omrelp.so)
==16816==by 0x4529ED: ??? (in /usr/sbin/rsyslogd)
==16816==by 0x453517: ??? (in /usr/sbin/rsyslogd)
==16816==by 0x448F8A: ??? (in /usr/sbin/rsyslogd)
==16816==by 0x4457BD: wtiWorker (in /usr/sbin/rsyslogd)
==16816==by 0x4447CB: ??? (in /usr/sbin/rsyslogd)
==16816==by 0x50580A3: start_thread (pthread_create.c:309)
==16816==by 0x5F7A62C: clone (clone.S:111)
==16816==  Address 0x20 is not stack'd, malloc'd or (recently) free'd
==16816==
==16816==
==16816== Process terminating with default action of signal 11 (SIGSEGV)
==16816==  Access not within mapped region at address 0x20
==16816==at 0x76FEE94: relpCltHintBurstEnd (in
/usr/lib/x86_64-linux-gnu/librelp.so.0.1.0)
==16816==by 0x74EFB7A: ??? (in /usr/lib/rsyslog/omrelp.so)
==16816==by 0x4529ED: ??? (in /usr/sbin/rsyslogd)
==16816==by 0x453517: ??? (in /usr/sbin/rsyslogd)
==16816==by 0x448F8A: ??? (in /usr/sbin/rsyslogd)
==16816==by 0x4457BD: wtiWorker (in /usr/sbin/rsyslogd)
==16816==by 0x4447CB: ??? (in /usr/sbin/rsyslogd)
==16816==by 0x50580A3: start_thread (pthread_create.c:309)
==16816==by 0x5F7A62C: clone (clone.S:111)
==16816==  If you believe this happened as a result of a stack
==16816==  overflow in your program's main thread (unlikely but
==16816==  possible), you can try to increase the 

Re: [rsyslog] rsyslog fails to start due to high queue

[Rainer Gerhards]

2016-11-25 9:16 GMT+01:00 Kosta Psimoulis :
> Thank you for your honest answer, so you are basically saying that the
> queue will be lost ?

There is this little tool:

https://github.com/rsyslog/rsyslog/blob/master/tools/recover_qi.pl

It works for your version as well and, depending on what is wrong, it
can fix the queue files. Worth a try.

> I thought that my current version of rsyslog was
> stable enough, I guess I was wrong, in that case I will try a different
> queue system.

Well, as always in life: things are deemed stable unless someone finds
it isn't the case. Then those things are fixed and we go back to think
"we are stable" unless... That's a large part of why new releases
happen, and that's part of the frequent release philosophy in open
source.

> I am sending a little bit extra debugging information, please
> let me know if you see anything useful and if you believe that this issue
> was fixed in 8.23

It's indeed interesting, as it may point into a bug that was fixed in
librelp (I barely remember there was something). Unfortunately, I have
made it my policy to NEVER AGAN look at old versions for fixing bugs.
I have done so in the past, and wasted month of my life, because in
95% of the cases it turned out I hunt for a bug that was long fixed.
So I only do this if I am actually paid for the pain, via an Adiscon
support contract.

I am NOT trying to get money out of you, and I understand that you do
not like to upgrade or are even unable to do so. But again, time is
very limited, I have a large TODO list and year-long experience has
prooven that hunting bugs in old code bases is fruitless. Even more so
if after hours I discover it's already fixed AND you need to apply a
code patch, what you don't want to do/ are not permitted to do in the
first place. Which means the effort was fruitless and wasteful for
everyone involved (yes, this *is* *practical* *experience*, more than
once).

So the first filter in open source development for me is: if the user
(or his org) is unwilling to update to current, there is no point in
looking at something that might even remotely be a bug.

I hope for your understanding.

Rainer

>
> Kind Regards,
> Kosta
>
> 1388.716086990:action 3 queue[DA]:Reg/w0: in destructor: sendbuf 0xada9030
> 1388.716159868:action 3 queue[DA]:Reg/w0: relp engine is dispatching frame
> with command 'rsp'
> 1388.716530324:action 3 queue[DA]:Reg/w0: relpSessWaitState returns 10019
> 1388.718355599:action 3 queue[DA]:Reg/w0: in destructor: sendbuf 0xada9570
> 1388.719729919:action 3 queue[DA]:Reg/w0: relp engine created new client
> 0xadae250
> 1388.719865043:action 3 queue[DA]:Reg/w0: omrelp: endTransaction
> ==16816== Thread 2 rs:action 3 queue[DA]:R:
> ==16816== Invalid read of size 8
> ==16816==at 0x76FEE94: relpCltHintBurstEnd (in
> /usr/lib/x86_64-linux-gnu/librelp.so.0.1.0)
> ==16816==by 0x74EFB7A: ??? (in /usr/lib/rsyslog/omrelp.so)
> ==16816==by 0x4529ED: ??? (in /usr/sbin/rsyslogd)
> ==16816==by 0x453517: ??? (in /usr/sbin/rsyslogd)
> ==16816==by 0x448F8A: ??? (in /usr/sbin/rsyslogd)
> ==16816==by 0x4457BD: wtiWorker (in /usr/sbin/rsyslogd)
> ==16816==by 0x4447CB: ??? (in /usr/sbin/rsyslogd)
> ==16816==by 0x50580A3: start_thread (pthread_create.c:309)
> ==16816==by 0x5F7A62C: clone (clone.S:111)
> ==16816==  Address 0x20 is not stack'd, malloc'd or (recently) free'd
> ==16816==
> ==16816==
> ==16816== Process terminating with default action of signal 11 (SIGSEGV)
> ==16816==  Access not within mapped region at address 0x20
> ==16816==at 0x76FEE94: relpCltHintBurstEnd (in
> /usr/lib/x86_64-linux-gnu/librelp.so.0.1.0)
> ==16816==by 0x74EFB7A: ??? (in /usr/lib/rsyslog/omrelp.so)
> ==16816==by 0x4529ED: ??? (in /usr/sbin/rsyslogd)
> ==16816==by 0x453517: ??? (in /usr/sbin/rsyslogd)
> ==16816==by 0x448F8A: ??? (in /usr/sbin/rsyslogd)
> ==16816==by 0x4457BD: wtiWorker (in /usr/sbin/rsyslogd)
> ==16816==by 0x4447CB: ??? (in /usr/sbin/rsyslogd)
> ==16816==by 0x50580A3: start_thread (pthread_create.c:309)
> ==16816==by 0x5F7A62C: clone (clone.S:111)
> ==16816==  If you believe this happened as a result of a stack
> ==16816==  overflow in your program's main thread (unlikely but
> ==16816==  possible), you can try to increase the size of the
> ==16816==  main thread stack using the --main-stacksize= flag.
> ==16816==  The main thread stack size used in this run was 8388608.
> ==16816==
> ==16816== HEAP SUMMARY:
> ==16816== in use at exit: 1,136,880 bytes in 2,685 blocks
> ==16816==   total heap usage: 15,434 allocs, 12,749 frees, 3,325,304 bytes
> allocated
> ==16816==
> ==16816== LEAK SUMMARY:
> ==16816==definitely lost: 0 bytes in 0 blocks
> ==16816==indirectly lost: 0 bytes in 0 blocks
> ==16816==  possibly lost: 1,152 bytes in 4 blocks
> ==16816==still reachable: 1,135,728 bytes in 2,681 blocks
> ==16816== suppressed: 0 bytes in 0 blocks
> 

Re: [rsyslog] rsyslog fails to start due to high queue

[Kosta Psimoulis]

Hello Rainer,

Thank you for your honest answer, so you are basically saying that the
queue will be lost ? I thought that my current version of rsyslog was
stable enough, I guess I was wrong, in that case I will try a different
queue system. I am sending a little bit extra debugging information, please
let me know if you see anything useful and if you believe that this issue
was fixed in 8.23

Kind Regards,
Kosta

1388.716086990:action 3 queue[DA]:Reg/w0: in destructor: sendbuf 0xada9030
1388.716159868:action 3 queue[DA]:Reg/w0: relp engine is dispatching frame
with command 'rsp'
1388.716530324:action 3 queue[DA]:Reg/w0: relpSessWaitState returns 10019
1388.718355599:action 3 queue[DA]:Reg/w0: in destructor: sendbuf 0xada9570
1388.719729919:action 3 queue[DA]:Reg/w0: relp engine created new client
0xadae250
1388.719865043:action 3 queue[DA]:Reg/w0: omrelp: endTransaction
==16816== Thread 2 rs:action 3 queue[DA]:R:
==16816== Invalid read of size 8
==16816==at 0x76FEE94: relpCltHintBurstEnd (in
/usr/lib/x86_64-linux-gnu/librelp.so.0.1.0)
==16816==by 0x74EFB7A: ??? (in /usr/lib/rsyslog/omrelp.so)
==16816==by 0x4529ED: ??? (in /usr/sbin/rsyslogd)
==16816==by 0x453517: ??? (in /usr/sbin/rsyslogd)
==16816==by 0x448F8A: ??? (in /usr/sbin/rsyslogd)
==16816==by 0x4457BD: wtiWorker (in /usr/sbin/rsyslogd)
==16816==by 0x4447CB: ??? (in /usr/sbin/rsyslogd)
==16816==by 0x50580A3: start_thread (pthread_create.c:309)
==16816==by 0x5F7A62C: clone (clone.S:111)
==16816==  Address 0x20 is not stack'd, malloc'd or (recently) free'd
==16816==
==16816==
==16816== Process terminating with default action of signal 11 (SIGSEGV)
==16816==  Access not within mapped region at address 0x20
==16816==at 0x76FEE94: relpCltHintBurstEnd (in
/usr/lib/x86_64-linux-gnu/librelp.so.0.1.0)
==16816==by 0x74EFB7A: ??? (in /usr/lib/rsyslog/omrelp.so)
==16816==by 0x4529ED: ??? (in /usr/sbin/rsyslogd)
==16816==by 0x453517: ??? (in /usr/sbin/rsyslogd)
==16816==by 0x448F8A: ??? (in /usr/sbin/rsyslogd)
==16816==by 0x4457BD: wtiWorker (in /usr/sbin/rsyslogd)
==16816==by 0x4447CB: ??? (in /usr/sbin/rsyslogd)
==16816==by 0x50580A3: start_thread (pthread_create.c:309)
==16816==by 0x5F7A62C: clone (clone.S:111)
==16816==  If you believe this happened as a result of a stack
==16816==  overflow in your program's main thread (unlikely but
==16816==  possible), you can try to increase the size of the
==16816==  main thread stack using the --main-stacksize= flag.
==16816==  The main thread stack size used in this run was 8388608.
==16816==
==16816== HEAP SUMMARY:
==16816== in use at exit: 1,136,880 bytes in 2,685 blocks
==16816==   total heap usage: 15,434 allocs, 12,749 frees, 3,325,304 bytes
allocated
==16816==
==16816== LEAK SUMMARY:
==16816==definitely lost: 0 bytes in 0 blocks
==16816==indirectly lost: 0 bytes in 0 blocks
==16816==  possibly lost: 1,152 bytes in 4 blocks
==16816==still reachable: 1,135,728 bytes in 2,681 blocks
==16816== suppressed: 0 bytes in 0 blocks
==16816== Rerun with --leak-check=full to see details of leaked memory
==16816==
==16816== For counts of detected and suppressed errors, rerun with: -v
==16816== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
Killed


On Fri, Nov 25, 2016 at 3:03 AM, Rainer Gerhards 
wrote:

> 2016-11-25 8:59 GMT+01:00 Kosta Psimoulis  sourceknowledge.com>:
> > Hello,
> >
> > Perhaps I wasn't clear on the first message but I am getting Segmentation
> > fault.
> >
> > this error was on syslog:
> > Nov 24 20:18:25 ip-172-16-1-152 systemd[1]: rsyslog.service: main process
> > exited, code=killed, status=11/SEGV
> >
> > and when I run it debug mode (-dn), it is trying to process the queue and
> > gives the following error:
> > .
> > 9771.929711621:action 3 queue[DA]:Reg/w0: relp engine is dispatching
> frame
> > with command 'rsp'
> > 9771.929715606:action 3 queue[DA]:Reg/w0: in rsp command handler, txnr
> 201,
> > code 200, text 'OK'
> > 9771.929723831:action 3 queue[DA]:Reg/w0: DEL sess 0x7f21a40023b0 unacked
> > 1, sessState 6
> > 9771.929727757:action 3 queue[DA]:Reg/w0: in destructor: sendbuf
> > 0x7f21a401de00
> > 9771.929733344:action 3 queue[DA]:Reg/w0: relp engine is dispatching
> frame
> > with command 'rsp'
> > 9771.929739980:action 3 queue[DA]:Reg/w0: relpSessWaitState returns 10019
> > 9771.929754387:action 3 queue[DA]:Reg/w0: in destructor: sendbuf
> > 0x7f21a4025730
> > 9771.929771799:action 3 queue[DA]:Reg/w0: relp engine created new client
> > 0x7f21a40023b0
> > 9771.929780641:action 3 queue[DA]:Reg/w0: omrelp: endTransaction
> > Segmentation fault
> >
> > 8.4.2 is the latest version available as a package on Debian Jessie 8.6
> and
> > 8.16 if I use the backports. There are no repositories for Debian Jessie,
> > only for Wheezy, are you suggesting to install 8.23 from source on a
> > production server ?
>
> TBH, it depends if you 

Re: [rsyslog] rsyslog fails to start due to high queue

[Rainer Gerhards]

2016-11-25 8:59 GMT+01:00 Kosta Psimoulis :
> Hello,
>
> Perhaps I wasn't clear on the first message but I am getting Segmentation
> fault.
>
> this error was on syslog:
> Nov 24 20:18:25 ip-172-16-1-152 systemd[1]: rsyslog.service: main process
> exited, code=killed, status=11/SEGV
>
> and when I run it debug mode (-dn), it is trying to process the queue and
> gives the following error:
> .
> 9771.929711621:action 3 queue[DA]:Reg/w0: relp engine is dispatching frame
> with command 'rsp'
> 9771.929715606:action 3 queue[DA]:Reg/w0: in rsp command handler, txnr 201,
> code 200, text 'OK'
> 9771.929723831:action 3 queue[DA]:Reg/w0: DEL sess 0x7f21a40023b0 unacked
> 1, sessState 6
> 9771.929727757:action 3 queue[DA]:Reg/w0: in destructor: sendbuf
> 0x7f21a401de00
> 9771.929733344:action 3 queue[DA]:Reg/w0: relp engine is dispatching frame
> with command 'rsp'
> 9771.929739980:action 3 queue[DA]:Reg/w0: relpSessWaitState returns 10019
> 9771.929754387:action 3 queue[DA]:Reg/w0: in destructor: sendbuf
> 0x7f21a4025730
> 9771.929771799:action 3 queue[DA]:Reg/w0: relp engine created new client
> 0x7f21a40023b0
> 9771.929780641:action 3 queue[DA]:Reg/w0: omrelp: endTransaction
> Segmentation fault
>
> 8.4.2 is the latest version available as a package on Debian Jessie 8.6 and
> 8.16 if I use the backports. There are no repositories for Debian Jessie,
> only for Wheezy, are you suggesting to install 8.23 from source on a
> production server ?

TBH, it depends if you prefer building from source or instability ;-)
Kidding away, I know there are problems in older versions, they are
fixed, and so I do not know how *I* can help you without you making
sure you run the fixes. If you don't like to apply them yourself (I
can understand that), I think you should coordinate with the Debian
folks (Michael who packages rsyslog for deb is also on the list) and
ask them for update or to apply the relevant patches.

For the version you have, I think the solution is to remove the queue
files, restart rsyslog and hope for the best. Might work.

Sorry I have no better answer,
Rainer
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] rsyslog fails to start due to high queue

[Kosta Psimoulis]

Hello,

Perhaps I wasn't clear on the first message but I am getting Segmentation
fault.

this error was on syslog:
Nov 24 20:18:25 ip-172-16-1-152 systemd[1]: rsyslog.service: main process
exited, code=killed, status=11/SEGV

and when I run it debug mode (-dn), it is trying to process the queue and
gives the following error:
.
9771.929711621:action 3 queue[DA]:Reg/w0: relp engine is dispatching frame
with command 'rsp'
9771.929715606:action 3 queue[DA]:Reg/w0: in rsp command handler, txnr 201,
code 200, text 'OK'
9771.929723831:action 3 queue[DA]:Reg/w0: DEL sess 0x7f21a40023b0 unacked
1, sessState 6
9771.929727757:action 3 queue[DA]:Reg/w0: in destructor: sendbuf
0x7f21a401de00
9771.929733344:action 3 queue[DA]:Reg/w0: relp engine is dispatching frame
with command 'rsp'
9771.929739980:action 3 queue[DA]:Reg/w0: relpSessWaitState returns 10019
9771.929754387:action 3 queue[DA]:Reg/w0: in destructor: sendbuf
0x7f21a4025730
9771.929771799:action 3 queue[DA]:Reg/w0: relp engine created new client
0x7f21a40023b0
9771.929780641:action 3 queue[DA]:Reg/w0: omrelp: endTransaction
Segmentation fault

8.4.2 is the latest version available as a package on Debian Jessie 8.6 and
8.16 if I use the backports. There are no repositories for Debian Jessie,
only for Wheezy, are you suggesting to install 8.23 from source on a
production server ?

Kind Regards,
Kosta

On Fri, Nov 25, 2016 at 1:15 AM, Rainer Gerhards <rgerha...@hq.adiscon.com>
wrote:

> I don't see anything conrete either, but I would guess from context
> that the queue got corrupted and on startup rsyslog may run into a
> segfault. But no evidence for that. It is known that versions prior to
> 8.22.0 sometimes had problems with corrupted disk queues.
>
> My suggestion is to delete the queue files and update to the curretn
> 8.23.0 version.
>
> Rainer
>
> 2016-11-25 4:06 GMT+01:00 David Lang <da...@lang.hm>:
> > it's no clear to me that rsyslog actually failed. It looks like systemd
> saw
> > the warning about ~ and decided to shut it down
> >
> > David Lang
> >
> > On Thu, 24 Nov 2016, Kosta Psimoulis wrote:
> >
> >> Date: Thu, 24 Nov 2016 20:36:47 -0500
> >> From: Kosta Psimoulis <kosta.psimou...@sourceknowledge.com>
> >> Reply-To: rsyslog-users <rsyslog@lists.adiscon.com>
> >> To: rsyslog@lists.adiscon.com
> >> Subject: [rsyslog] rsyslog fails to start due to high queue
> >>
> >>
> >> Hello,
> >>
> >> There was a misconfiguration for a remote forwarding server, the issue
> was
> >> fixed and the host was corrected by rsyslog is refusing to start with
> the
> >> following error:
> >>
> >> Rsyslog version: 8.4.2
> >> Distribution: Latest Debian Jessie Linux 8.6
> >>
> >> -- Unit rsyslog.service has begun starting up.
> >> Nov 24 20:18:32 ip-172-16-1-152 systemd[1]: rsyslog.service start
> request
> >> repeated too quickly, refusing to start.
> >> Nov 24 20:18:32 ip-172-16-1-152 systemd[1]: Failed to start System
> Logging
> >> Service.
> >> -- Subject: Unit rsyslog.service has failed
> >> -- Defined-By: systemd
> >> -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
> >>
> >> Nov 24 20:18:26 ip-172-16-1-152 rsyslogd: [origin software="rsyslogd"
> >> swVersion="8.4.2" x-pid="16534" x-info="http://www.rsyslog.com;] start
> >> Nov 24 20:18:26 ip-172-16-1-152 rsyslogd-2307: warning: ~ action is
> >> deprecated, consider using the 'stop' statement instead [try
> >> http://www.rsyslog.com/e/2307 ]
> >> Nov 24 20:18:25 ip-172-16-1-152 systemd[1]: rsyslog.service: main
> process
> >> exited, code=killed, status=11/SEGV
> >> Nov 24 20:18:25 ip-172-16-1-152 systemd[1]: Unit rsyslog.service entered
> >> failed state.
> >> Nov 24 20:18:26 ip-172-16-1-152 systemd[1]: rsyslog.service holdoff time
> >> over, scheduling restart.
> >> Nov 24 20:18:26 ip-172-16-1-152 systemd[1]: Stopping System Logging
> >> Service...
> >> Nov 24 20:18:26 ip-172-16-1-152 systemd[1]: Starting System Logging
> >> Service...
> >> Nov 24 20:18:26 ip-172-16-1-152 systemd[1]: Started System Logging
> >> Service.
> >> Nov 24 20:18:26 ip-172-16-1-152 rsyslogd: [origin software="rsyslogd"
> >> swVersion="8.4.2" x-pid="16540" x-info="http://www.rsyslog.com;] start
> >> Nov 24 20:18:26 ip-172-16-1-152 rsyslogd-2307: warning: ~ action is
> >> deprecated, consider using the 'stop' statement instead [try
> >> http://www.rsyslog.com/e/2307 ]
> >>
> >>

Re: [rsyslog] rsyslog fails to start due to high queue

[Rainer Gerhards]

I don't see anything conrete either, but I would guess from context
that the queue got corrupted and on startup rsyslog may run into a
segfault. But no evidence for that. It is known that versions prior to
8.22.0 sometimes had problems with corrupted disk queues.

My suggestion is to delete the queue files and update to the curretn
8.23.0 version.

Rainer

2016-11-25 4:06 GMT+01:00 David Lang <da...@lang.hm>:
> it's no clear to me that rsyslog actually failed. It looks like systemd saw
> the warning about ~ and decided to shut it down
>
> David Lang
>
> On Thu, 24 Nov 2016, Kosta Psimoulis wrote:
>
>> Date: Thu, 24 Nov 2016 20:36:47 -0500
>> From: Kosta Psimoulis <kosta.psimou...@sourceknowledge.com>
>> Reply-To: rsyslog-users <rsyslog@lists.adiscon.com>
>> To: rsyslog@lists.adiscon.com
>> Subject: [rsyslog] rsyslog fails to start due to high queue
>>
>>
>> Hello,
>>
>> There was a misconfiguration for a remote forwarding server, the issue was
>> fixed and the host was corrected by rsyslog is refusing to start with the
>> following error:
>>
>> Rsyslog version: 8.4.2
>> Distribution: Latest Debian Jessie Linux 8.6
>>
>> -- Unit rsyslog.service has begun starting up.
>> Nov 24 20:18:32 ip-172-16-1-152 systemd[1]: rsyslog.service start request
>> repeated too quickly, refusing to start.
>> Nov 24 20:18:32 ip-172-16-1-152 systemd[1]: Failed to start System Logging
>> Service.
>> -- Subject: Unit rsyslog.service has failed
>> -- Defined-By: systemd
>> -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
>>
>> Nov 24 20:18:26 ip-172-16-1-152 rsyslogd: [origin software="rsyslogd"
>> swVersion="8.4.2" x-pid="16534" x-info="http://www.rsyslog.com;] start
>> Nov 24 20:18:26 ip-172-16-1-152 rsyslogd-2307: warning: ~ action is
>> deprecated, consider using the 'stop' statement instead [try
>> http://www.rsyslog.com/e/2307 ]
>> Nov 24 20:18:25 ip-172-16-1-152 systemd[1]: rsyslog.service: main process
>> exited, code=killed, status=11/SEGV
>> Nov 24 20:18:25 ip-172-16-1-152 systemd[1]: Unit rsyslog.service entered
>> failed state.
>> Nov 24 20:18:26 ip-172-16-1-152 systemd[1]: rsyslog.service holdoff time
>> over, scheduling restart.
>> Nov 24 20:18:26 ip-172-16-1-152 systemd[1]: Stopping System Logging
>> Service...
>> Nov 24 20:18:26 ip-172-16-1-152 systemd[1]: Starting System Logging
>> Service...
>> Nov 24 20:18:26 ip-172-16-1-152 systemd[1]: Started System Logging
>> Service.
>> Nov 24 20:18:26 ip-172-16-1-152 rsyslogd: [origin software="rsyslogd"
>> swVersion="8.4.2" x-pid="16540" x-info="http://www.rsyslog.com;] start
>> Nov 24 20:18:26 ip-172-16-1-152 rsyslogd-2307: warning: ~ action is
>> deprecated, consider using the 'stop' statement instead [try
>> http://www.rsyslog.com/e/2307 ]
>>
>> The warnings are fine, they are from aws cloudinit:
>>
>> My extra configuration file is:
>> $EscapeControlCharactersOnReceive off
>> $MaxMessageSize 64k
>> local6.info /efs/backup/i-XX/access-backup.log
>> local6.err /efs/backup/i-X/error-backup.log
>> module(load="omrelp")
>> local6.* action(
>>type="omrelp"
>>RebindInterval="200"
>>queue.FileName="apacheRule1"
>>queue.SaveOnShutdown="on"
>>queue.Type="LinkedList"
>>action.resumeRetryCount="-1"
>>target="XX.us-east-1.elb.amazonaws.com" port="20514"
>> )
>> & stop
>>
>>
>> There are 13 files in /var/spool/rsyslog that won't process, here are some
>> info:
>>
>> root@ip-172-16-1-152:/var/spool/rsyslog# cat apacheRule1.qi
>> > +iQueueSize:2:4:9959:
>> +tVars.disk.sizeOnDisk:2:8:13356962:
>>>
>>> End
>>
>> .
>> > +iCurrFNum:2:2:13:
>> +pszFName:1:11:apacheRule1:
>> +iMaxFiles:2:8:1000:
>> +bDeleteOnClose:2:1:0:
>> +sType:2:1:1:
>> +tOperationsMode:2:1:2:
>> +tOpenMode:2:3:384:
>> +iCurrOffs:2:6:762769:
>> +inode:2:1:0:
>>>
>>> End
>>
>> .
>> > +iCurrFNum:2:1:1:
>> +pszFName:1:11:apacheRule1:
>> +iMaxFiles:2:8:1000:
>> +bDeleteOnClose:2:1:1:
>> +sType:2:1:1:
>> +tOperationsMode:2:1:1:
>> +tOpenMode:2:3:384:
>> +iCurrOffs:2:5:20591:
>> +inode:2:6:414600:
>>>
>>> End
>>
>> .
>> Any hints 

Re: [rsyslog] rsyslog fails to start due to high queue

[David Lang]

it's no clear to me that rsyslog actually failed. It looks like systemd saw the 
warning about ~ and decided to shut it down


David Lang

On Thu, 24 Nov 2016, Kosta Psimoulis wrote:


Date: Thu, 24 Nov 2016 20:36:47 -0500
From: Kosta Psimoulis <kosta.psimou...@sourceknowledge.com>
Reply-To: rsyslog-users <rsyslog@lists.adiscon.com>
To: rsyslog@lists.adiscon.com
Subject: [rsyslog] rsyslog fails to start due to high queue

Hello,

There was a misconfiguration for a remote forwarding server, the issue was
fixed and the host was corrected by rsyslog is refusing to start with the
following error:

Rsyslog version: 8.4.2
Distribution: Latest Debian Jessie Linux 8.6

-- Unit rsyslog.service has begun starting up.
Nov 24 20:18:32 ip-172-16-1-152 systemd[1]: rsyslog.service start request
repeated too quickly, refusing to start.
Nov 24 20:18:32 ip-172-16-1-152 systemd[1]: Failed to start System Logging
Service.
-- Subject: Unit rsyslog.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Nov 24 20:18:26 ip-172-16-1-152 rsyslogd: [origin software="rsyslogd"
swVersion="8.4.2" x-pid="16534" x-info="http://www.rsyslog.com;] start
Nov 24 20:18:26 ip-172-16-1-152 rsyslogd-2307: warning: ~ action is
deprecated, consider using the 'stop' statement instead [try
http://www.rsyslog.com/e/2307 ]
Nov 24 20:18:25 ip-172-16-1-152 systemd[1]: rsyslog.service: main process
exited, code=killed, status=11/SEGV
Nov 24 20:18:25 ip-172-16-1-152 systemd[1]: Unit rsyslog.service entered
failed state.
Nov 24 20:18:26 ip-172-16-1-152 systemd[1]: rsyslog.service holdoff time
over, scheduling restart.
Nov 24 20:18:26 ip-172-16-1-152 systemd[1]: Stopping System Logging
Service...
Nov 24 20:18:26 ip-172-16-1-152 systemd[1]: Starting System Logging
Service...
Nov 24 20:18:26 ip-172-16-1-152 systemd[1]: Started System Logging Service.
Nov 24 20:18:26 ip-172-16-1-152 rsyslogd: [origin software="rsyslogd"
swVersion="8.4.2" x-pid="16540" x-info="http://www.rsyslog.com;] start
Nov 24 20:18:26 ip-172-16-1-152 rsyslogd-2307: warning: ~ action is
deprecated, consider using the 'stop' statement instead [try
http://www.rsyslog.com/e/2307 ]

The warnings are fine, they are from aws cloudinit:

My extra configuration file is:
$EscapeControlCharactersOnReceive off
$MaxMessageSize 64k
local6.info /efs/backup/i-XX/access-backup.log
local6.err /efs/backup/i-X/error-backup.log
module(load="omrelp")
local6.* action(
   type="omrelp"
   RebindInterval="200"
   queue.FileName="apacheRule1"
   queue.SaveOnShutdown="on"
   queue.Type="LinkedList"
   action.resumeRetryCount="-1"
   target="XX.us-east-1.elb.amazonaws.com" port="20514"
)
& stop


There are 13 files in /var/spool/rsyslog that won't process, here are some
info:

root@ip-172-16-1-152:/var/spool/rsyslog# cat apacheRule1.qi

End

.

End

.

End

.
Any hints of how I can fix this issue and prevent it from happening in the
future ?

Kind Regards,
Kosta
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

[rsyslog] rsyslog fails to start due to high queue

[Kosta Psimoulis]

Hello,

There was a misconfiguration for a remote forwarding server, the issue was
fixed and the host was corrected by rsyslog is refusing to start with the
following error:

Rsyslog version: 8.4.2
Distribution: Latest Debian Jessie Linux 8.6

-- Unit rsyslog.service has begun starting up.
Nov 24 20:18:32 ip-172-16-1-152 systemd[1]: rsyslog.service start request
repeated too quickly, refusing to start.
Nov 24 20:18:32 ip-172-16-1-152 systemd[1]: Failed to start System Logging
Service.
-- Subject: Unit rsyslog.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

Nov 24 20:18:26 ip-172-16-1-152 rsyslogd: [origin software="rsyslogd"
swVersion="8.4.2" x-pid="16534" x-info="http://www.rsyslog.com;] start
Nov 24 20:18:26 ip-172-16-1-152 rsyslogd-2307: warning: ~ action is
deprecated, consider using the 'stop' statement instead [try
http://www.rsyslog.com/e/2307 ]
Nov 24 20:18:25 ip-172-16-1-152 systemd[1]: rsyslog.service: main process
exited, code=killed, status=11/SEGV
Nov 24 20:18:25 ip-172-16-1-152 systemd[1]: Unit rsyslog.service entered
failed state.
Nov 24 20:18:26 ip-172-16-1-152 systemd[1]: rsyslog.service holdoff time
over, scheduling restart.
Nov 24 20:18:26 ip-172-16-1-152 systemd[1]: Stopping System Logging
Service...
Nov 24 20:18:26 ip-172-16-1-152 systemd[1]: Starting System Logging
Service...
Nov 24 20:18:26 ip-172-16-1-152 systemd[1]: Started System Logging Service.
Nov 24 20:18:26 ip-172-16-1-152 rsyslogd: [origin software="rsyslogd"
swVersion="8.4.2" x-pid="16540" x-info="http://www.rsyslog.com;] start
Nov 24 20:18:26 ip-172-16-1-152 rsyslogd-2307: warning: ~ action is
deprecated, consider using the 'stop' statement instead [try
http://www.rsyslog.com/e/2307 ]

The warnings are fine, they are from aws cloudinit:

My extra configuration file is:
$EscapeControlCharactersOnReceive off
$MaxMessageSize 64k
local6.info /efs/backup/i-XX/access-backup.log
local6.err /efs/backup/i-X/error-backup.log
module(load="omrelp")
local6.* action(
type="omrelp"
RebindInterval="200"
queue.FileName="apacheRule1"
queue.SaveOnShutdown="on"
queue.Type="LinkedList"
action.resumeRetryCount="-1"
target="XX.us-east-1.elb.amazonaws.com" port="20514"
)
& stop


There are 13 files in /var/spool/rsyslog that won't process, here are some
info:

root@ip-172-16-1-152:/var/spool/rsyslog# cat apacheRule1.qi
End
.
End
.
End
.
Any hints of how I can fix this issue and prevent it from happening in the
future ?

Kind Regards,
Kosta
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.