Hi guys,
We keep tracking this issue been awhile. The latest rsyslog still run into
this issue. I tried to find out more clues from the unix socket.
I notice that all delay logs doesn't have futex() system call in the unix
socket. For example, the d1 and d31 logs line doesn't show up in the log
files. It'll appear in the future but not the moment the log been sent to
unix socket.
recvmsg(0, {msg_name(0)=NULL, msg_iov(1)=[{"<134>object-updater: Begin
object update sweep\0", 8096}], msg_controllen=64, {cmsg_len=32,
cmsg_level=SOL_SOCKET, cmsg_type=0x1d /* SCM_??? */, ...},
msg_flags=0}, MSG_DONTWAIT) = 47
futex(0x190257c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x1902578,
{FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x19024e0, FUTEX_WAKE_PRIVATE, 1) = 1
select(1, [0], NULL, NULL, NULL) = 1 (in [0])
recvmsg(0, {msg_name(0)=NULL, msg_iov(1)=[{"<134>object-updater:
Object update sweep of d1 completed: 0.00s, 0 successes, 0
failures\0", 8096}], msg_controllen=64, {cmsg_len=32,
cmsg_level=SOL_SOCKET, cmsg_type=0x1d /* SCM_??? */, ...},
msg_flags=0}, MSG_DONTWAIT) = 89
select(1, [0], NULL, NULL, NULL) = 1 (in [0])
recvmsg(0, {msg_name(0)=NULL, msg_iov(1)=[{"<134>object-updater:
Object update sweep of d31 completed: 0.00s, 0 successes, 0
failures\0", 8096}], msg_controllen=64, {cmsg_len=32,
cmsg_level=SOL_SOCKET, cmsg_type=0x1d /* SCM_??? */, ...},
msg_flags=0}, MSG_DONTWAIT) = 90
select(1, [0], NULL, NULL, NULL) = 1 (in [0])
recvmsg(0, {msg_name(0)=NULL, msg_iov(1)=[{"<134>object-updater:
Object update sweep of d32 completed: 0.00s, 0 successes, 0
failures\0", 8096}], msg_controllen=64, {cmsg_len=32,
cmsg_level=SOL_SOCKET, cmsg_type=0x1d /* SCM_??? */, ...},
msg_flags=0}, MSG_DONTWAIT) = 90
futex(0x190257c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0x1902578,
{FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0x19024e0, FUTEX_WAKE_PRIVATE, 1) = 1
select(1, [0], NULL, NULL, NULL) = 1 (in [0])
More details.
https://gist.github.com/HugoKuo/c16e1f76217b136b2435d72e63a9949c
Thanks // Hugo
2016-05-05 11:13 GMT+08:00 Kuo Hugo <[email protected]>:
> Hi David,
>
> * Here's the full configs of our rsyslog.
> https://gist.github.com/HugoKuo/1bc7a61c87dafb2c9c7bd9bf792148fc
> * All logs happens on local only.
>
> Regards // Hugo
>
>
>
> 2016-05-05 1:28 GMT+08:00 Kuo Hugo <[email protected]>:
>
>> Hi Folks,
>>
>> I observed a weird behaviour with rsyslog. In the following logs, you can
>> find a line is the log of 8 hours ago. This issue appears occasionally.
>> Around 10 times per-day. And some old logs are more days ago.
>>
>> # grep from /var/log/swift/all.log #
>> May 4 09:59:59 cloud-paco02 object-updater: Object update sweep of d32
>> completed: 0.03s, 0 successes, 0 failures
>> May 4 09:59:59 cloud-paco02 object-updater: Object update sweep of d30
>> completed: 0.02s, 0 successes, 0 failures
>> May 4 09:59:59 cloud-paco02 object-updater: Object update sweep completed:
>> 0.46sMay 4 02:02:08 cloud-paco02 object-updater: Object update sweep of d31
>> completed: 0.00s, 0 successes, 0 failures
>> May 4 10:04:59 cloud-paco02 object-updater: Begin object update sweep
>> May 4 10:04:59 cloud-paco02 object-updater: Object update sweep of d31
>> completed: 0.00s, 0 successes, 0 failures
>> May 4 10:04:59 cloud-paco02 object-updater: Object update sweep of d32
>> completed: 0.00s, 0 successes, 0 failures
>>
>>
>> - Our daemons send log to the unix domain socket (/dev/log)
>> - Rsyslog version - 7.4.4-1ubuntu2.6
>>
>> [ Confs ]
>>
>> The log facilities are local0 and local2 from our daemons.
>>
>> local2.* /var/log/swift/proxy_access.log;RSYSLOG_FileFormat
>>
>> local0.*;local2.* /var/log/swift/all.log;RSYSLOG_TraditionalFileFormat
>> & ~
>>
>> Here’s gist link of the /etc/rsyslog.conf
>> https://gist.github.com/HugoKuo/1bc7a61c87dafb2c9c7bd9bf792148fc
>>
>> - Any recommendation would be great.
>> - I suspect the old logs are buffered somewhere
>> - Is there a way to dump all those buffered log?
>> - I believe it’s nothing about out daemon since I restarted our
>> processes but the old log is still showing up.
>> - I have four servers in this situation.
>>
>> Thanks // Hugo
>>
>
>
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
