Re: [rsyslog] segfault with mmkubernetes

[David Lang]

On Tue, 10 Apr 2018, Rich Megginson wrote:

I think this PR will address most of your concerns: 
https://github.com/rsyslog/rsyslog/pull/2635


Thanks, that looks like it does a lot!

I've got a problem that our test cluster doesn't yet have anything running on it 
other than kubernetes itself, so it seems I'm getting no metadata back (I 
haven't tried grabbign annotations yet, will be doing that tomorrow)


David Lang

Is there a way to change the log-driver for just one pod/container as 

opposed to the entire cluster?

Not sure.

On 04/05/2018 08:12 PM, David Lang wrote:

On Thu, 5 Apr 2018, Rich Megginson via rsyslog wrote:

this seems to work (I am new to kubernetes and docker, so I don't know 
what's correct, just that I'm getting data I wasn't before :-)


An interesting point, when it wasn't matching, it was adding metadata:{} 
(empty) to the variables, but now it's adding kubernetes:{} and docker:{} 
to the variables.


{
  "PRIORITY": "6",
  "_UID": "0",
  "_GID": "0",
  "_CAP_EFFECTIVE": "3f",
  "_SYSTEMD_SLICE": "system.slice",
  "_BOOT_ID": "4a6e7b4239b34a8f8fa8e8558e681f9c",
  "_MACHINE_ID": "4ae701c7ca9f40a695d0eaf0af2be9b7",
  "_HOSTNAME": "xvbb-c21.xv.dc.openx.org",
  "CONTAINER_ID": "f164986f5b75",
  "CONTAINER_ID_FULL": 
"f164986f5b75cee5e30933431f455dbf850f19293853fb142d33ba86b70fca8e",

  "CONTAINER_NAME":
"k8s_calico-node_calico-node-g9ptv_kube-system_1548bee2-339f-11e8-a1cb-90e2ba989580_0", 

  "CONTAINER_TAG": "\"docker-xvbb-c21\"",
  "SYSLOG_IDENTIFIER": "\"docker-xvbb-c21\"",
  "_TRANSPORT": "journal",
  "_PID": "12000",
  "_COMM": "dockerd",
  "_EXE": "/usr/bin/dockerd",
  "_CMDLINE": "docker --selinux-enabled --log-driver=journald --log-opt 
tag=\"docker-xvbb-c21\" -s overlay2 --insecure-registry docker.openx.org 
--bip=192.168.96.1/24 --ip-masq=true --mtu=1450 --bip=192.168.96.1/24 
--ip-masq=true --mtu=1450",

  "_SYSTEMD_CGROUP": "/system.slice/docker.service",
  "_SYSTEMD_UNIT": "docker.service",
  "MESSAGE": "2018-04-04 09:09:46.372 [INFO][129] int_dataplane.go 704: 
Finished applying updates to dataplane. msecToApply=1.301672",

  "_SOURCE_REALTIME_TIMESTAMP": "1522832986372607",
  "kubernetes": {
    "pod_name": "calico-node-g9ptv",
    "namespace_name": "kube-system",
    "container_name": "calico-node",
    "master_url": "https://192.168.128.1:443/";
  },
  "docker": {
    "container_id": 
"f164986f5b75cee5e30933431f455dbf850f19293853fb142d33ba86b70fca8e"

  }
}


now that I have it working, what do we want to do in this release to make 
it better :-) (thanks for your help by the way)


It still doesn't look like it is working.  There should be fields for 
kubernetes namespace_id, pod_id, labels, and usually annotations.
If you're token really isn't "foobar", it is likely that the problem is 
that it is authenticating to kubernetes as a bogus user, and for some 
reason, we aren't getting or processing a 401 or 403 error, so there is no 
metadata returned.


Ok, I'll work on that tomorrow. Would it be a good idea to put something in 
the log saying that the lookup failed (and why)?


fixing the defaults is a big one, monitoring stats would be another, but 
also some better error detection so that it generates an error rather 
than segfaulting if these variables are bogus


I think the sample should be simplified.

leave out the cacert bundle, assume that the system bundle will work

no need for a ruleset for output to a file, that just adds complexity.

leave out the guard rules since the worst that happens is an empty 
metadata tag (can we fix the code to do nothing if it can't get 
anything?)


we should not load the imfile module if we don't use it (possibly have 
two sample configs, one that is a good default for working with the 
docker json-file log-driver, and one that's a good default for working 
with the logs arriving via journald (and let's see if we can get a 
log-driver that sends to syslog with the metadata we need)


a test that uses the sample config, even if it doesn't do anything other 
than load things would probably be a good thing (making sure we don't 
break if we use the latest liblognorm ;-)



Another very interesting thing I ran into trying to debug this was that 
the kubernetes experts defaulted to running this in a container, and when 
debugging, that meant that rsyslog would read a log message from 
journald, output a few dozen new messages, that journald would then read 
and queue up to feed to rsyslog...  can you say blow through any 
ratelimits instantly :-0


Yes, there is a sort of feedback loop when the log collector is collecting 
its own logs . . .


Is there a way to change the log-driver for just one pod/container as 
opposed to the entire cluster?


David Lang





___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing 

Re: [rsyslog] segfault with mmkubernetes

[Rich Megginson via rsyslog]

I think this PR will address most of your concerns: 
https://github.com/rsyslog/rsyslog/pull/2635


> Is there a way to change the log-driver for just one pod/container as 
opposed to the entire cluster?


Not sure.

On 04/05/2018 08:12 PM, David Lang wrote:

On Thu, 5 Apr 2018, Rich Megginson via rsyslog wrote:

this seems to work (I am new to kubernetes and docker, so I don't 
know what's correct, just that I'm getting data I wasn't before :-)


An interesting point, when it wasn't matching, it was adding 
metadata:{} (empty) to the variables, but now it's adding 
kubernetes:{} and docker:{} to the variables.


{
  "PRIORITY": "6",
  "_UID": "0",
  "_GID": "0",
  "_CAP_EFFECTIVE": "3f",
  "_SYSTEMD_SLICE": "system.slice",
  "_BOOT_ID": "4a6e7b4239b34a8f8fa8e8558e681f9c",
  "_MACHINE_ID": "4ae701c7ca9f40a695d0eaf0af2be9b7",
  "_HOSTNAME": "xvbb-c21.xv.dc.openx.org",
  "CONTAINER_ID": "f164986f5b75",
  "CONTAINER_ID_FULL": 
"f164986f5b75cee5e30933431f455dbf850f19293853fb142d33ba86b70fca8e",

  "CONTAINER_NAME":
"k8s_calico-node_calico-node-g9ptv_kube-system_1548bee2-339f-11e8-a1cb-90e2ba989580_0", 


  "CONTAINER_TAG": "\"docker-xvbb-c21\"",
  "SYSLOG_IDENTIFIER": "\"docker-xvbb-c21\"",
  "_TRANSPORT": "journal",
  "_PID": "12000",
  "_COMM": "dockerd",
  "_EXE": "/usr/bin/dockerd",
  "_CMDLINE": "docker --selinux-enabled --log-driver=journald 
--log-opt tag=\"docker-xvbb-c21\" -s overlay2 --insecure-registry 
docker.openx.org --bip=192.168.96.1/24 --ip-masq=true --mtu=1450 
--bip=192.168.96.1/24 --ip-masq=true --mtu=1450",

  "_SYSTEMD_CGROUP": "/system.slice/docker.service",
  "_SYSTEMD_UNIT": "docker.service",
  "MESSAGE": "2018-04-04 09:09:46.372 [INFO][129] int_dataplane.go 
704: Finished applying updates to dataplane. msecToApply=1.301672",

  "_SOURCE_REALTIME_TIMESTAMP": "1522832986372607",
  "kubernetes": {
    "pod_name": "calico-node-g9ptv",
    "namespace_name": "kube-system",
    "container_name": "calico-node",
    "master_url": "https://192.168.128.1:443/";
  },
  "docker": {
    "container_id": 
"f164986f5b75cee5e30933431f455dbf850f19293853fb142d33ba86b70fca8e"

  }
}


now that I have it working, what do we want to do in this release to 
make it better :-) (thanks for your help by the way)


It still doesn't look like it is working.  There should be fields for 
kubernetes namespace_id, pod_id, labels, and usually annotations.
If you're token really isn't "foobar", it is likely that the problem 
is that it is authenticating to kubernetes as a bogus user, and for 
some reason, we aren't getting or processing a 401 or 403 error, so 
there is no metadata returned.


Ok, I'll work on that tomorrow. Would it be a good idea to put 
something in the log saying that the lookup failed (and why)?


fixing the defaults is a big one, monitoring stats would be another, 
but also some better error detection so that it generates an error 
rather than segfaulting if these variables are bogus


I think the sample should be simplified.

leave out the cacert bundle, assume that the system bundle will work

no need for a ruleset for output to a file, that just adds complexity.

leave out the guard rules since the worst that happens is an empty 
metadata tag (can we fix the code to do nothing if it can't get 
anything?)


we should not load the imfile module if we don't use it (possibly 
have two sample configs, one that is a good default for working with 
the docker json-file log-driver, and one that's a good default for 
working with the logs arriving via journald (and let's see if we can 
get a log-driver that sends to syslog with the metadata we need)


a test that uses the sample config, even if it doesn't do anything 
other than load things would probably be a good thing (making sure 
we don't break if we use the latest liblognorm ;-)



Another very interesting thing I ran into trying to debug this was 
that the kubernetes experts defaulted to running this in a 
container, and when debugging, that meant that rsyslog would read a 
log message from journald, output a few dozen new messages, that 
journald would then read and queue up to feed to rsyslog...  can you 
say blow through any ratelimits instantly :-0


Yes, there is a sort of feedback loop when the log collector is 
collecting its own logs . . .


Is there a way to change the log-driver for just one pod/container as 
opposed to the entire cluster?


David Lang



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] segfault with mmkubernetes

[Rich Megginson via rsyslog]

On 04/06/2018 10:36 PM, David Lang wrote:

On Fri, 6 Apr 2018, Rich Megginson wrote:


On 04/06/2018 09:08 PM, David Lang wrote:
In addition to the authentication, it looks like there is some other 
issue. where some info isn't there and it logs "filename: ''" and it 
looks like nothing is attempted


9992.862530379:main Q:Reg/w0  : ruleset.c: processBATCH: next msg 
43: E0329 22:17:55.816218   13181 dns.go:131] Nameserver limits were 
exceeded, some nameservers have been omitted, the applied names
9992.862626453:main Q:Reg/w0  : rainerscript.c: ACTION 0 
[mmkubernetes:action(type="mmkubernetes" ...)]

9992.862632518:main Q:Reg/w0  : ruleset.c: executing action 0
9992.862637582:main Q:Reg/w0  : ../action.c: action 'testtest': 
called, logging to mmkubernetes (susp 0/0, direct q 1)
9992.862641778:main Q:Reg/w0  : ../action.c: action[testtest] 
transitioned to state: itx
9992.862646131:main Q:Reg/w0  : ../action.c: entering 
actionCalldoAction(), state: itx, actionNbr 0
9992.862659596:main Q:Reg/w0  : mmkubernetes.c: mmkubernetes: 
CONTAINER_NAME: 'kubelet-node'  CONTAINER_ID_FULL: 
'b19af21d5383732b374f3707b09daf362f96d5c5e7651715d0fd3893e3a657e9'.


CONTAINER_NAME: 'kubelet-node' does not match the pattern in 
k8s_container_name:


rule=:%k8s_prefix:char-to:_%_%container_name:char-to:.%.%container_hash:char-to:_%_%pod_name:char-to:_%_%namespace_name:char-to:_%_%not_used_1:char-to:_%_%not_used_2:rest% 

rule=:%k8s_prefix:char-to:_%_%container_name:char-to:_%_%pod_name:char-to:_%_%namespace_name:char-to:_%_%not_used_1:char-to:_%_%not_used_2:rest% 



so it isn't a kubernetes container - probably just a plain docker 
container - and therefore isn't processed by mmkubernetes


hmm, I think it's part of the kubernetes infrastructure (there 
shouldn't be anything else on this system)



Not sure how plain kubernetes works, but this may be a "system 
container" which isn't directly managed by kubernetes per se as a 
container running in a pod.  At any rate, CONTAINER_NAME isn't in the 
right form to be able to get kubernetes metadata from it.


___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] segfault with mmkubernetes

[David Lang]

On Fri, 6 Apr 2018, Rich Megginson wrote:


On 04/06/2018 09:08 PM, David Lang wrote:
In addition to the authentication, it looks like there is some other issue. 
where some info isn't there and it logs "filename: ''" and it looks like 
nothing is attempted


9992.862530379:main Q:Reg/w0  : ruleset.c: processBATCH: next msg 43: E0329 
22:17:55.816218   13181 dns.go:131] Nameserver limits were exceeded, some 
nameservers have been omitted, the applied names
9992.862626453:main Q:Reg/w0  : rainerscript.c: ACTION 0 
[mmkubernetes:action(type="mmkubernetes" ...)]

9992.862632518:main Q:Reg/w0  : ruleset.c: executing action 0
9992.862637582:main Q:Reg/w0  : ../action.c: action 'testtest': called, 
logging to mmkubernetes (susp 0/0, direct q 1)
9992.862641778:main Q:Reg/w0  : ../action.c: action[testtest] transitioned 
to state: itx
9992.862646131:main Q:Reg/w0  : ../action.c: entering actionCalldoAction(), 
state: itx, actionNbr 0
9992.862659596:main Q:Reg/w0  : mmkubernetes.c: mmkubernetes: 
CONTAINER_NAME: 'kubelet-node'  CONTAINER_ID_FULL: 
'b19af21d5383732b374f3707b09daf362f96d5c5e7651715d0fd3893e3a657e9'.


CONTAINER_NAME: 'kubelet-node' does not match the pattern in 
k8s_container_name:


rule=:%k8s_prefix:char-to:_%_%container_name:char-to:.%.%container_hash:char-to:_%_%pod_name:char-to:_%_%namespace_name:char-to:_%_%not_used_1:char-to:_%_%not_used_2:rest%
rule=:%k8s_prefix:char-to:_%_%container_name:char-to:_%_%pod_name:char-to:_%_%namespace_name:char-to:_%_%not_used_1:char-to:_%_%not_used_2:rest%

so it isn't a kubernetes container - probably just a plain docker container - 
and therefore isn't processed by mmkubernetes


hmm, I think it's part of the kubernetes infrastructure (there shouldn't be 
anything else on this system)

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] segfault with mmkubernetes

[Rich Megginson via rsyslog]

On 04/06/2018 09:08 PM, David Lang wrote:
In addition to the authentication, it looks like there is some other 
issue. where some info isn't there and it logs "filename: ''" and it 
looks like nothing is attempted


9992.862530379:main Q:Reg/w0  : ruleset.c: processBATCH: next msg 43: 
E0329 22:17:55.816218   13181 dns.go:131] Nameserver limits were 
exceeded, some nameservers have been omitted, the applied names
9992.862626453:main Q:Reg/w0  : rainerscript.c: ACTION 0 
[mmkubernetes:action(type="mmkubernetes" ...)]

9992.862632518:main Q:Reg/w0  : ruleset.c: executing action 0
9992.862637582:main Q:Reg/w0  : ../action.c: action 'testtest': 
called, logging to mmkubernetes (susp 0/0, direct q 1)
9992.862641778:main Q:Reg/w0  : ../action.c: action[testtest] 
transitioned to state: itx
9992.862646131:main Q:Reg/w0  : ../action.c: entering 
actionCalldoAction(), state: itx, actionNbr 0
9992.862659596:main Q:Reg/w0  : mmkubernetes.c: mmkubernetes: 
CONTAINER_NAME: 'kubelet-node'  CONTAINER_ID_FULL: 
'b19af21d5383732b374f3707b09daf362f96d5c5e7651715d0fd3893e3a657e9'.


CONTAINER_NAME: 'kubelet-node' does not match the pattern in 
k8s_container_name:


rule=:%k8s_prefix:char-to:_%_%container_name:char-to:.%.%container_hash:char-to:_%_%pod_name:char-to:_%_%namespace_name:char-to:_%_%not_used_1:char-to:_%_%not_used_2:rest%
rule=:%k8s_prefix:char-to:_%_%container_name:char-to:_%_%pod_name:char-to:_%_%namespace_name:char-to:_%_%not_used_1:char-to:_%_%not_used_2:rest%

so it isn't a kubernetes container - probably just a plain docker 
container - and therefore isn't processed by mmkubernetes


9992.862666446:main Q:Reg/w0  : mmkubernetes.c: mmkubernetes: 
filename: ''.


David Lang



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] segfault with mmkubernetes

[David Lang]

by comparison

9992.875013025:main Q:Reg/w0  : mmkubernetes.c: mmkubernetes: CONTAINER_NAME: 
'k8s_tiller_tiller-deploy-68544fcd8b-g6l9k_kube-system_10decbb3-339f-11e8-a1cb-90e2ba989580_0'
  CONTAINER_ID_FULL: 
'1afa1dd3043a9ea4fb8155c05ae074b55247a2f4f8df53cc2f90162c744f5f7f'.
9992.875023070:main Q:Reg/w0  : mmkubernetes.c: mmkubernetes:
  podName: 'tiller-deploy-68544fcd8b-g6l9k'
  namespace: 'kube-system'
  containerName: 'tiller'
  containerID: 
'1afa1dd3043a9ea4fb8155c05ae074b55247a2f4f8df53cc2f90162c744f5f7f'
9992.875439034:main Q:Reg/w0  : mmkubernetes.c: mmkubernetes: queryKB reply:
{
  "kind":"Status",
  "apiVersion":"v1",
  "metadata":{
  },
  "status":"Failure",
  "message":"pods \"tiller-deploy-68544fcd8b-g6l9k\" is forbidden: User \"system:anonymous\" 
cannot get pods in the namespace \"kube-system\"",
  "reason":"Forbidden",
  "details":{
"name":"tiller-deploy-68544fcd8b-g6l9k",
"kind":"pods"
  },
  "code":403
}

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] segfault with mmkubernetes

[David Lang]

In addition to the authentication, it looks like there is some other issue. 
where some info isn't there and it logs "filename: ''" and it looks like nothing 
is attempted


9992.862530379:main Q:Reg/w0  : ruleset.c: processBATCH: next msg 43: E0329 
22:17:55.816218   13181 dns.go:131] Nameserver limits were exceeded, some 
nameservers have been omitted, the applied names
9992.862626453:main Q:Reg/w0  : rainerscript.c: ACTION 0 
[mmkubernetes:action(type="mmkubernetes" ...)]
9992.862632518:main Q:Reg/w0  : ruleset.c: executing action 0
9992.862637582:main Q:Reg/w0  : ../action.c: action 'testtest': called, logging 
to mmkubernetes (susp 0/0, direct q 1)
9992.862641778:main Q:Reg/w0  : ../action.c: action[testtest] transitioned to 
state: itx
9992.862646131:main Q:Reg/w0  : ../action.c: entering actionCalldoAction(), 
state: itx, actionNbr 0
9992.862659596:main Q:Reg/w0  : mmkubernetes.c: mmkubernetes: CONTAINER_NAME: 
'kubelet-node'  CONTAINER_ID_FULL: 
'b19af21d5383732b374f3707b09daf362f96d5c5e7651715d0fd3893e3a657e9'.
9992.862666446:main Q:Reg/w0  : mmkubernetes.c: mmkubernetes: filename: ''.

David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] segfault with mmkubernetes

[Rich Megginson via rsyslog]

On 04/06/2018 04:07 PM, David Lang wrote:

On Fri, 6 Apr 2018, Rich Megginson via rsyslog wrote:



It is establishing a connection with the kubernetes server and 
passing the SSL handshake, I can't see what's happening over the 
connection, so I don't know what messages are being passed.


Is there a way to get more detailed info from mmkubernetes?


Not without gdb . . .


I do have gdb on this machine (but not the source)

one thing I just learned is that we do not have RBAC configured on 
kubernetes yet, so no auth is needed on kubernetes (and token is 
meaningless), could that be what's messing things up?



Likely.  My (incorrect) assumption was that authentication is required.  
Looks like I need another change to allow operation with no token and no 
tokenfile.



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] segfault with mmkubernetes

[David Lang]

On Fri, 6 Apr 2018, Rich Megginson via rsyslog wrote:



It is establishing a connection with the kubernetes server and passing 
the SSL handshake, I can't see what's happening over the connection, 
so I don't know what messages are being passed.


Is there a way to get more detailed info from mmkubernetes?


Not without gdb . . .


I do have gdb on this machine (but not the source)

one thing I just learned is that we do not have RBAC configured on kubernetes 
yet, so no auth is needed on kubernetes (and token is meaningless), could that 
be what's messing things up?

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] segfault with mmkubernetes

[Rich Megginson via rsyslog]

On 04/06/2018 02:48 PM, David Lang wrote:

On Thu, 5 Apr 2018, David Lang wrote:



now that I have it working, what do we want to do in this release 
to make it better :-) (thanks for your help by the way)


It still doesn't look like it is working.  There should be fields 
for kubernetes namespace_id, pod_id, labels, and usually annotations.
If you're token really isn't "foobar", it is likely that the problem 
is that it is authenticating to kubernetes as a bogus user, and for 
some reason, we aren't getting or processing a 401 or 403 error, so 
there is no metadata returned.


Ok, I'll work on that tomorrow. Would it be a good idea to put 
something in the log saying that the lookup failed (and why)?


nothing useful is showing up in the debug log

# rsyslogd -dn |grep mmkubernetes.c |head -50
7503.650649084:main thread    : mmkubernetes.c: mmkubernetes: module 
compiled with rsyslog version 8.34.0.
7503.650714517:main thread    : modules.c: source file mmkubernetes.c 
requested reference for module 'lmregexp', reference count now 1
7503.650796305:main thread    : mmkubernetes.c: entry point 
'activateCnfPrePrivDrop' not present in module
7503.650800235:main thread    : mmkubernetes.c: entry point 'doHUP' 
not present in module
7503.650802825:main thread    : mmkubernetes.c: entry point 
'doHUPWrkr' not present in module
7503.650804368:main thread    : mmkubernetes.c: entry point 
'SetShutdownImmdtPtr' not present in module
7503.650806816:main thread    : mmkubernetes.c: entry point 
'beginTransaction' not present in module
7503.650808726:main thread    : mmkubernetes.c: entry point 
'commitTransaction' not present in module
7503.650811100:main thread    : mmkubernetes.c: entry point 
'endTransaction' not present in module
7503.650826674:main thread    : mmkubernetes.c: module (global) param 
blk for mmkubernetes:

7503.651251925:main thread    : mmkubernetes.c: newActInst (mmkubernetes)
7503.651254941:main thread    : mmkubernetes.c: action param blk in 
mmkubernetes:
7503.654439159:main Q:Reg/w0  : mmkubernetes.c: mmkubernetes: 
filename: ''.
7503.654639544:main Q:Reg/w0  : mmkubernetes.c: mmkubernetes: 
filename: ''.
7503.654859942:main Q:Reg/w0  : mmkubernetes.c: mmkubernetes: 
filename: ''.

(this then repeats for every action)

It is establishing a connection with the kubernetes server and passing 
the SSL handshake, I can't see what's happening over the connection, 
so I don't know what messages are being passed.


Is there a way to get more detailed info from mmkubernetes?


Not without gdb . . .



David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a 
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST 
if you DON'T LIKE THAT.



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] segfault with mmkubernetes

[David Lang]

On Thu, 5 Apr 2018, David Lang wrote:



now that I have it working, what do we want to do in this release to 
make it better :-) (thanks for your help by the way)


It still doesn't look like it is working.  There should be fields for 
kubernetes namespace_id, pod_id, labels, and usually annotations.
If you're token really isn't "foobar", it is likely that the problem is 
that it is authenticating to kubernetes as a bogus user, and for some 
reason, we aren't getting or processing a 401 or 403 error, so there is 
no metadata returned.


Ok, I'll work on that tomorrow. Would it be a good idea to put something in 
the 
log saying that the lookup failed (and why)?


nothing useful is showing up in the debug log

# rsyslogd -dn |grep mmkubernetes.c |head -50
7503.650649084:main thread: mmkubernetes.c: mmkubernetes: module compiled 
with rsyslog version 8.34.0.
7503.650714517:main thread: modules.c: source file mmkubernetes.c requested 
reference for module 'lmregexp', reference count now 1
7503.650796305:main thread: mmkubernetes.c: entry point 
'activateCnfPrePrivDrop' not present in module
7503.650800235:main thread: mmkubernetes.c: entry point 'doHUP' not present 
in module
7503.650802825:main thread: mmkubernetes.c: entry point 'doHUPWrkr' not 
present in module
7503.650804368:main thread: mmkubernetes.c: entry point 
'SetShutdownImmdtPtr' not present in module
7503.650806816:main thread: mmkubernetes.c: entry point 'beginTransaction' 
not present in module
7503.650808726:main thread: mmkubernetes.c: entry point 'commitTransaction' 
not present in module
7503.650811100:main thread: mmkubernetes.c: entry point 'endTransaction' not 
present in module
7503.650826674:main thread: mmkubernetes.c: module (global) param blk for 
mmkubernetes:

7503.651251925:main thread: mmkubernetes.c: newActInst (mmkubernetes)
7503.651254941:main thread: mmkubernetes.c: action param blk in 
mmkubernetes:

7503.654439159:main Q:Reg/w0  : mmkubernetes.c: mmkubernetes: filename: ''.
7503.654639544:main Q:Reg/w0  : mmkubernetes.c: mmkubernetes: filename: ''.
7503.654859942:main Q:Reg/w0  : mmkubernetes.c: mmkubernetes: filename: ''.
(this then repeats for every action)

It is establishing a connection with the kubernetes server and passing the SSL 
handshake, I can't see what's happening over the connection, so I don't know 
what messages are being passed.


Is there a way to get more detailed info from mmkubernetes?

David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] segfault with mmkubernetes

[David Lang]

On Thu, 5 Apr 2018, Rich Megginson via rsyslog wrote:

this seems to work (I am new to kubernetes and docker, so I don't know 
what's correct, just that I'm getting data I wasn't before :-)


An interesting point, when it wasn't matching, it was adding 
metadata:{} (empty) to the variables, but now it's adding 
kubernetes:{} and docker:{} to the variables.


{
  "PRIORITY": "6",
  "_UID": "0",
  "_GID": "0",
  "_CAP_EFFECTIVE": "3f",
  "_SYSTEMD_SLICE": "system.slice",
  "_BOOT_ID": "4a6e7b4239b34a8f8fa8e8558e681f9c",
  "_MACHINE_ID": "4ae701c7ca9f40a695d0eaf0af2be9b7",
  "_HOSTNAME": "xvbb-c21.xv.dc.openx.org",
  "CONTAINER_ID": "f164986f5b75",
  "CONTAINER_ID_FULL": 
"f164986f5b75cee5e30933431f455dbf850f19293853fb142d33ba86b70fca8e",
  "CONTAINER_NAME": 


"k8s_calico-node_calico-node-g9ptv_kube-system_1548bee2-339f-11e8-a1cb-90e2ba989580_0",

  "CONTAINER_TAG": "\"docker-xvbb-c21\"",
  "SYSLOG_IDENTIFIER": "\"docker-xvbb-c21\"",
  "_TRANSPORT": "journal",
  "_PID": "12000",
  "_COMM": "dockerd",
  "_EXE": "/usr/bin/dockerd",
  "_CMDLINE": "docker --selinux-enabled --log-driver=journald 
--log-opt tag=\"docker-xvbb-c21\" -s overlay2 --insecure-registry 
docker.openx.org --bip=192.168.96.1/24 --ip-masq=true --mtu=1450 
--bip=192.168.96.1/24 --ip-masq=true --mtu=1450",

  "_SYSTEMD_CGROUP": "/system.slice/docker.service",
  "_SYSTEMD_UNIT": "docker.service",
  "MESSAGE": "2018-04-04 09:09:46.372 [INFO][129] int_dataplane.go 
704: Finished applying updates to dataplane. msecToApply=1.301672",

  "_SOURCE_REALTIME_TIMESTAMP": "1522832986372607",
  "kubernetes": {
    "pod_name": "calico-node-g9ptv",
    "namespace_name": "kube-system",
    "container_name": "calico-node",
    "master_url": "https://192.168.128.1:443/";
  },
  "docker": {
    "container_id": 
"f164986f5b75cee5e30933431f455dbf850f19293853fb142d33ba86b70fca8e"

  }
}


now that I have it working, what do we want to do in this release to 
make it better :-) (thanks for your help by the way)


It still doesn't look like it is working.  There should be fields for 
kubernetes namespace_id, pod_id, labels, and usually annotations.
If you're token really isn't "foobar", it is likely that the problem is 
that it is authenticating to kubernetes as a bogus user, and for some 
reason, we aren't getting or processing a 401 or 403 error, so there is 
no metadata returned.


Ok, I'll work on that tomorrow. Would it be a good idea to put something in the 
log saying that the lookup failed (and why)?


fixing the defaults is a big one, monitoring stats would be another, 
but also some better error detection so that it generates an error 
rather than segfaulting if these variables are bogus


I think the sample should be simplified.

leave out the cacert bundle, assume that the system bundle will work

no need for a ruleset for output to a file, that just adds complexity.

leave out the guard rules since the worst that happens is an empty 
metadata tag (can we fix the code to do nothing if it can't get 
anything?)


we should not load the imfile module if we don't use it (possibly have 
two sample configs, one that is a good default for working with the 
docker json-file log-driver, and one that's a good default for working 
with the logs arriving via journald (and let's see if we can get a 
log-driver that sends to syslog with the metadata we need)


a test that uses the sample config, even if it doesn't do anything 
other than load things would probably be a good thing (making sure we 
don't break if we use the latest liblognorm ;-)



Another very interesting thing I ran into trying to debug this was 
that the kubernetes experts defaulted to running this in a container, 
and when debugging, that meant that rsyslog would read a log message 
from journald, output a few dozen new messages, that journald would 
then read and queue up to feed to rsyslog...  can you say blow through 
any ratelimits instantly :-0


Yes, there is a sort of feedback loop when the log collector is 
collecting its own logs . . .


Is there a way to change the log-driver for just one pod/container as opposed to 
the entire cluster?


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] segfault with mmkubernetes

[Rich Megginson via rsyslog]

On 04/05/2018 07:33 PM, David Lang wrote:

On Thu, 5 Apr 2018, David Lang wrote:


On Thu, 5 Apr 2018, Rich Megginson via rsyslog wrote:


On 04/05/2018 01:59 PM, David Lang wrote:
I had the wrong URL, but even fixing the URL doesn't seem to be 
working, I see no attempts to establish a connection to the 
kubernetes server


module(load="imuxsock") # provides support for local system logging 
(e.g. via logger command)
module(load="imklog")   # provides kernel logging support 
(previously done by rklogd)

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
module(load="imjournal"  PersistStateInterval="100" 
Ratelimit.Interval="0")

module(load="mmkubernetes"
  kubernetesurl="https://192.168.128.1:443/";
  allowunsignedcerts="on"
  token="foobar"
  filenamerulebase="/etc/rsyslog.d/k8s_filename.rulebase"
  containerrulebase="/etc/rsyslog.d/k8s_filename.rulebase"


This should be /etc/rsyslog.d/k8s_container_name.rulebase, so it 
never matches $!CONTAINER_NAME, so it never connects to the URL.


Ok, I figured that since there are no files involved here, these 
would never end up being used. Let me give that a try.


this seems to work (I am new to kubernetes and docker, so I don't know 
what's correct, just that I'm getting data I wasn't before :-)


An interesting point, when it wasn't matching, it was adding 
metadata:{} (empty) to the variables, but now it's adding 
kubernetes:{} and docker:{} to the variables.


{
  "PRIORITY": "6",
  "_UID": "0",
  "_GID": "0",
  "_CAP_EFFECTIVE": "3f",
  "_SYSTEMD_SLICE": "system.slice",
  "_BOOT_ID": "4a6e7b4239b34a8f8fa8e8558e681f9c",
  "_MACHINE_ID": "4ae701c7ca9f40a695d0eaf0af2be9b7",
  "_HOSTNAME": "xvbb-c21.xv.dc.openx.org",
  "CONTAINER_ID": "f164986f5b75",
  "CONTAINER_ID_FULL": 
"f164986f5b75cee5e30933431f455dbf850f19293853fb142d33ba86b70fca8e",
  "CONTAINER_NAME": 
"k8s_calico-node_calico-node-g9ptv_kube-system_1548bee2-339f-11e8-a1cb-90e2ba989580_0",

  "CONTAINER_TAG": "\"docker-xvbb-c21\"",
  "SYSLOG_IDENTIFIER": "\"docker-xvbb-c21\"",
  "_TRANSPORT": "journal",
  "_PID": "12000",
  "_COMM": "dockerd",
  "_EXE": "/usr/bin/dockerd",
  "_CMDLINE": "docker --selinux-enabled --log-driver=journald 
--log-opt tag=\"docker-xvbb-c21\" -s overlay2 --insecure-registry 
docker.openx.org --bip=192.168.96.1/24 --ip-masq=true --mtu=1450 
--bip=192.168.96.1/24 --ip-masq=true --mtu=1450",

  "_SYSTEMD_CGROUP": "/system.slice/docker.service",
  "_SYSTEMD_UNIT": "docker.service",
  "MESSAGE": "2018-04-04 09:09:46.372 [INFO][129] int_dataplane.go 
704: Finished applying updates to dataplane. msecToApply=1.301672",

  "_SOURCE_REALTIME_TIMESTAMP": "1522832986372607",
  "kubernetes": {
    "pod_name": "calico-node-g9ptv",
    "namespace_name": "kube-system",
    "container_name": "calico-node",
    "master_url": "https://192.168.128.1:443/";
  },
  "docker": {
    "container_id": 
"f164986f5b75cee5e30933431f455dbf850f19293853fb142d33ba86b70fca8e"

  }
}


now that I have it working, what do we want to do in this release to 
make it better :-) (thanks for your help by the way)


It still doesn't look like it is working.  There should be fields for 
kubernetes namespace_id, pod_id, labels, and usually annotations.
If you're token really isn't "foobar", it is likely that the problem is 
that it is authenticating to kubernetes as a bogus user, and for some 
reason, we aren't getting or processing a 401 or 403 error, so there is 
no metadata returned.




fixing the defaults is a big one, monitoring stats would be another, 
but also some better error detection so that it generates an error 
rather than segfaulting if these variables are bogus


I think the sample should be simplified.

leave out the cacert bundle, assume that the system bundle will work

no need for a ruleset for output to a file, that just adds complexity.

leave out the guard rules since the worst that happens is an empty 
metadata tag (can we fix the code to do nothing if it can't get 
anything?)


we should not load the imfile module if we don't use it (possibly have 
two sample configs, one that is a good default for working with the 
docker json-file log-driver, and one that's a good default for working 
with the logs arriving via journald (and let's see if we can get a 
log-driver that sends to syslog with the metadata we need)


a test that uses the sample config, even if it doesn't do anything 
other than load things would probably be a good thing (making sure we 
don't break if we use the latest liblognorm ;-)



Another very interesting thing I ran into trying to debug this was 
that the kubernetes experts defaulted to running this in a container, 
and when debugging, that meant that rsyslog would read a log message 
from journald, output a few dozen new messages, that journald would 
then read and queue up to feed to rsyslog...  can you say blow through 
any ratelimits instantly :-0


Yes, there is a sort of feedback loop when the log collector is 
collecting its own logs . . .




David La

Re: [rsyslog] segfault with mmkubernetes

[David Lang]

On Thu, 5 Apr 2018, David Lang wrote:


On Thu, 5 Apr 2018, Rich Megginson via rsyslog wrote:


On 04/05/2018 01:59 PM, David Lang wrote:
I had the wrong URL, but even fixing the URL doesn't seem to be 
working, I see no attempts to establish a connection to the kubernetes 
server


module(load="imuxsock") # provides support for local system logging 
(e.g. via logger command)
module(load="imklog")   # provides kernel logging support (previously 
done by rklogd)

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
module(load="imjournal"  PersistStateInterval="100" 
Ratelimit.Interval="0")

module(load="mmkubernetes"
  kubernetesurl="https://192.168.128.1:443/";
  allowunsignedcerts="on"
  token="foobar"
  filenamerulebase="/etc/rsyslog.d/k8s_filename.rulebase"
  containerrulebase="/etc/rsyslog.d/k8s_filename.rulebase"


This should be /etc/rsyslog.d/k8s_container_name.rulebase, so it never 
matches $!CONTAINER_NAME, so it never connects to the URL.


Ok, I figured that since there are no files involved here, these would never 
end 
up being used. Let me give that a try.


this seems to work (I am new to kubernetes and docker, so I don't know what's 
correct, just that I'm getting data I wasn't before :-)


An interesting point, when it wasn't matching, it was adding metadata:{} (empty) 
to the variables, but now it's adding kubernetes:{} and docker:{} to the 
variables.


{
  "PRIORITY": "6",
  "_UID": "0",
  "_GID": "0",
  "_CAP_EFFECTIVE": "3f",
  "_SYSTEMD_SLICE": "system.slice",
  "_BOOT_ID": "4a6e7b4239b34a8f8fa8e8558e681f9c",
  "_MACHINE_ID": "4ae701c7ca9f40a695d0eaf0af2be9b7",
  "_HOSTNAME": "xvbb-c21.xv.dc.openx.org",
  "CONTAINER_ID": "f164986f5b75",
  "CONTAINER_ID_FULL": 
"f164986f5b75cee5e30933431f455dbf850f19293853fb142d33ba86b70fca8e",
  "CONTAINER_NAME": 
"k8s_calico-node_calico-node-g9ptv_kube-system_1548bee2-339f-11e8-a1cb-90e2ba989580_0",
  "CONTAINER_TAG": "\"docker-xvbb-c21\"",
  "SYSLOG_IDENTIFIER": "\"docker-xvbb-c21\"",
  "_TRANSPORT": "journal",
  "_PID": "12000",
  "_COMM": "dockerd",
  "_EXE": "/usr/bin/dockerd",
  "_CMDLINE": "docker --selinux-enabled --log-driver=journald --log-opt 
tag=\"docker-xvbb-c21\" -s overlay2 --insecure-registry docker.openx.org --bip=192.168.96.1/24 
--ip-masq=true --mtu=1450 --bip=192.168.96.1/24 --ip-masq=true --mtu=1450",
  "_SYSTEMD_CGROUP": "/system.slice/docker.service",
  "_SYSTEMD_UNIT": "docker.service",
  "MESSAGE": "2018-04-04 09:09:46.372 [INFO][129] int_dataplane.go 704: Finished 
applying updates to dataplane. msecToApply=1.301672",
  "_SOURCE_REALTIME_TIMESTAMP": "1522832986372607",
  "kubernetes": {
"pod_name": "calico-node-g9ptv",
"namespace_name": "kube-system",
"container_name": "calico-node",
"master_url": "https://192.168.128.1:443/";
  },
  "docker": {
"container_id": 
"f164986f5b75cee5e30933431f455dbf850f19293853fb142d33ba86b70fca8e"
  }
}


now that I have it working, what do we want to do in this release to make it 
better :-) (thanks for your help by the way)


fixing the defaults is a big one, monitoring stats would be another, but also 
some better error detection so that it generates an error rather than 
segfaulting if these variables are bogus


I think the sample should be simplified.

leave out the cacert bundle, assume that the system bundle will work

no need for a ruleset for output to a file, that just adds complexity.

leave out the guard rules since the worst that happens is an empty metadata tag 
(can we fix the code to do nothing if it can't get anything?)


we should not load the imfile module if we don't use it (possibly have two 
sample configs, one that is a good default for working with the docker json-file 
log-driver, and one that's a good default for working with the logs arriving via 
journald (and let's see if we can get a log-driver that sends to syslog with the 
metadata we need)


a test that uses the sample config, even if it doesn't do anything other than 
load things would probably be a good thing (making sure we don't break if we use 
the latest liblognorm ;-)



Another very interesting thing I ran into trying to debug this was that the 
kubernetes experts defaulted to running this in a container, and when debugging, 
that meant that rsyslog would read a log message from journald, output a few 
dozen new messages, that journald would then read and queue up to feed to 
rsyslog...  can you say blow through any ratelimits instantly :-0


David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] segfault with mmkubernetes

[David Lang]

On Thu, 5 Apr 2018, Rich Megginson via rsyslog wrote:


On 04/05/2018 01:59 PM, David Lang wrote:
I had the wrong URL, but even fixing the URL doesn't seem to be 
working, I see no attempts to establish a connection to the kubernetes 
server


module(load="imuxsock") # provides support for local system logging 
(e.g. via logger command)
module(load="imklog")   # provides kernel logging support (previously 
done by rklogd)

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
module(load="imjournal"  PersistStateInterval="100" 
Ratelimit.Interval="0")

module(load="mmkubernetes"
  kubernetesurl="https://192.168.128.1:443/";
  allowunsignedcerts="on"
  token="foobar"
  filenamerulebase="/etc/rsyslog.d/k8s_filename.rulebase"
  containerrulebase="/etc/rsyslog.d/k8s_filename.rulebase"


This should be /etc/rsyslog.d/k8s_container_name.rulebase, so it never 
matches $!CONTAINER_NAME, so it never connects to the URL.


Ok, I figured that since there are no files involved here, these would never end 
up being used. Let me give that a try.


This brings up a good point about monitoring i.e. what sorts of data 
should we be providing?  For example, in this case, if we provided 
something like 'number-of-containerrulebase-matches' it would always be 
0, which would be a clue that the match isn't working.


at the very least, a number-of-logs-we-didnt-try-to-lookup, 
number-of-logs-processed, and number-of-logs-no-metadata-available would be good 
ones (with better names than I come up with :-)

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] segfault with mmkubernetes

[Rich Megginson via rsyslog]

On 04/05/2018 01:59 PM, David Lang wrote:
I had the wrong URL, but even fixing the URL doesn't seem to be 
working, I see no attempts to establish a connection to the kubernetes 
server


module(load="imuxsock") # provides support for local system logging 
(e.g. via logger command)
module(load="imklog")   # provides kernel logging support (previously 
done by rklogd)

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
module(load="imjournal"  PersistStateInterval="100" 
Ratelimit.Interval="0")

module(load="mmkubernetes"
  kubernetesurl="https://192.168.128.1:443/";
  allowunsignedcerts="on"
  token="foobar"
  filenamerulebase="/etc/rsyslog.d/k8s_filename.rulebase"
  containerrulebase="/etc/rsyslog.d/k8s_filename.rulebase"


This should be /etc/rsyslog.d/k8s_container_name.rulebase, so it never 
matches $!CONTAINER_NAME, so it never connects to the URL.


This brings up a good point about monitoring i.e. what sorts of data 
should we be providing?  For example, in this case, if we provided 
something like 'number-of-containerrulebase-matches' it would always be 
0, which would be a clue that the match isn't working.



)
action(type="mmkubernetes" name="testtest")
if ($!_SYSTEMD_UNIT == "docker.service") and (strlen($!CONTAINER_NAME) 
> 0) then /var/log/k8s;RSYSLOG_DebugFormat


# tail  k8s |grep '\$\!'|grep -i docker |cut -f 2- -d : |jq
{
  "PRIORITY": "6",
  "_UID": "0",
  "_GID": "0",
  "_CAP_EFFECTIVE": "3f",
  "_SYSTEMD_SLICE": "system.slice",
  "_BOOT_ID": "4a6e7b4239b34a8f8fa8e8558e681f9c",
  "_MACHINE_ID": "4ae701c7ca9f40a695d0eaf0af2be9b7",
  "_HOSTNAME": "xvbb-c21.xv.dc.openx.org",
  "CONTAINER_TAG": "\"docker-xvbb-c21\"",
  "SYSLOG_IDENTIFIER": "\"docker-xvbb-c21\"",
  "CONTAINER_ID": "f164986f5b75",
  "CONTAINER_ID_FULL": 
"f164986f5b75cee5e30933431f455dbf850f19293853fb142d33ba86b70fca8e",
  "CONTAINER_NAME": 
"k8s_calico-node_calico-node-g9ptv_kube-system_1548bee2-339f-11e8-a1cb-90e2ba989580_0",

  "_TRANSPORT": "journal",
  "_PID": "12000",
  "_COMM": "dockerd",
  "_EXE": "/usr/bin/dockerd",
  "_CMDLINE": "docker --selinux-enabled --log-driver=journald 
--log-opt tag=\"docker-xvbb-c21\" -s overlay2 --insecure-registry 
docker.openx.org --bip=192.168.96.1/24 --ip-masq=true --mtu=1450 
--bip=192.168.96.1/24 --ip-masq=true --mtu=1450",

  "_SYSTEMD_CGROUP": "/system.slice/docker.service",
  "_SYSTEMD_UNIT": "docker.service",
  "MESSAGE": "2018-04-05 19:59:24.025 [INFO][129] int_dataplane.go 
704: Finished applying updates to dataplane. msecToApply=3.010066",

  "_SOURCE_REALTIME_TIMESTAMP": "1522958364025225",
  "metadata": {}
}
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a 
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST 
if you DON'T LIKE THAT.



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] segfault with mmkubernetes

[David Lang]

I had the wrong URL, but even fixing the URL doesn't seem to be working, I see 
no attempts to establish a connection to the kubernetes server


module(load="imuxsock") # provides support for local system logging (e.g. via 
logger command)
module(load="imklog")   # provides kernel logging support (previously done by 
rklogd)

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
module(load="imjournal"  PersistStateInterval="100" Ratelimit.Interval="0")
module(load="mmkubernetes"
  kubernetesurl="https://192.168.128.1:443/";
  allowunsignedcerts="on"
  token="foobar"
  filenamerulebase="/etc/rsyslog.d/k8s_filename.rulebase"
  containerrulebase="/etc/rsyslog.d/k8s_filename.rulebase"
)
action(type="mmkubernetes" name="testtest")
if ($!_SYSTEMD_UNIT == "docker.service") and (strlen($!CONTAINER_NAME) > 0) then 
/var/log/k8s;RSYSLOG_DebugFormat


# tail  k8s |grep '\$\!'|grep -i docker |cut -f 2- -d : |jq
{
  "PRIORITY": "6",
  "_UID": "0",
  "_GID": "0",
  "_CAP_EFFECTIVE": "3f",
  "_SYSTEMD_SLICE": "system.slice",
  "_BOOT_ID": "4a6e7b4239b34a8f8fa8e8558e681f9c",
  "_MACHINE_ID": "4ae701c7ca9f40a695d0eaf0af2be9b7",
  "_HOSTNAME": "xvbb-c21.xv.dc.openx.org",
  "CONTAINER_TAG": "\"docker-xvbb-c21\"",
  "SYSLOG_IDENTIFIER": "\"docker-xvbb-c21\"",
  "CONTAINER_ID": "f164986f5b75",
  "CONTAINER_ID_FULL": 
"f164986f5b75cee5e30933431f455dbf850f19293853fb142d33ba86b70fca8e",
  "CONTAINER_NAME": 
"k8s_calico-node_calico-node-g9ptv_kube-system_1548bee2-339f-11e8-a1cb-90e2ba989580_0",

  "_TRANSPORT": "journal",
  "_PID": "12000",
  "_COMM": "dockerd",
  "_EXE": "/usr/bin/dockerd",
  "_CMDLINE": "docker --selinux-enabled --log-driver=journald --log-opt 
tag=\"docker-xvbb-c21\" -s overlay2 --insecure-registry docker.openx.org 
--bip=192.168.96.1/24 --ip-masq=true --mtu=1450 --bip=192.168.96.1/24 
--ip-masq=true --mtu=1450",

  "_SYSTEMD_CGROUP": "/system.slice/docker.service",
  "_SYSTEMD_UNIT": "docker.service",
  "MESSAGE": "2018-04-05 19:59:24.025 [INFO][129] int_dataplane.go 704: Finished 
applying updates to dataplane. msecToApply=3.010066",

  "_SOURCE_REALTIME_TIMESTAMP": "1522958364025225",
  "metadata": {}
}
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] segfault with mmkubernetes

[David Lang]

On Wed, 4 Apr 2018, Rich Megginson via rsyslog wrote:


Is there a way to not use them with later versions?


Yes, but apparently not without crashing . . .


Try defining both filenamerulebase and containerrulebase:

module(load="mmkubernetes" 
kubernetesurl="http://k8s-master.lc.dc.openx.org:8080";

  token="foobar"
  filenamerulebase="/etc/rsyslog.d/k8s_filename.rulebase"
  containerrulebase="/etc/rsyslog.d/k8s_container_name.rulebase"
)

Using the rulebase files from 
https://github.com/rsyslog/rsyslog/tree/master/contrib/mmkubernetes


this no longer segfaults, but it also doesn't seem to be getting the metadata

a sample message

[root@xvbb-c21 log]# tail -20 k8s |grep '\$\!' |cut -f 2- -d : |jq
{
  "PRIORITY": "6",
  "_UID": "0",
  "_GID": "0",
  "_SYSTEMD_SLICE": "system.slice",
  "_BOOT_ID": "4a6e7b4239b34a8f8fa8e8558e681f9c",
  "_MACHINE_ID": "4ae701c7ca9f40a695d0eaf0af2be9b7",
  "_TRANSPORT": "journal",
  "_CAP_EFFECTIVE": "3f",
  "_HOSTNAME": "xvbb-c21.xv.dc.openx.org",
  "_PID": "12000",
  "_COMM": "dockerd",
  "_EXE": "/usr/bin/dockerd",
  "_CMDLINE": "docker --selinux-enabled --log-driver=journald --log-opt 
tag=\"docker-xvbb-c21\" -s overlay2 --insecure-registry docker.openx.org 
--bip=192.168.96.1/24 --ip-masq=true --mtu=1450 --bip=192.168.96.1/24 
--ip-masq=true --mtu=1450",

  "_SYSTEMD_CGROUP": "/system.slice/docker.service",
  "_SYSTEMD_UNIT": "docker.service",
  "CONTAINER_TAG": "\"docker-xvbb-c21\"",
  "SYSLOG_IDENTIFIER": "\"docker-xvbb-c21\"",
  "CONTAINER_NAME": 
"k8s_calico-node_calico-node-g9ptv_kube-system_1548bee2-339f-11e8-a1cb-90e2ba989580_0",

  "CONTAINER_ID": "f164986f5b75",
  "CONTAINER_ID_FULL": 
"f164986f5b75cee5e30933431f455dbf850f19293853fb142d33ba86b70fca8e",
  "MESSAGE": "2018-03-30 00:16:32.013 [INFO][129] health.go 150: Overall health 
summary=&health.HealthReport{Live:true, Ready:true}",

  "_SOURCE_REALTIME_TIMESTAMP": "1522368992013740",
  "metadata": {}
}

note that this test k8s instance is not running https, is this potentially the 
problem?


# cat /etc/rsyslog.conf
module(load="imuxsock") # provides support for local system logging (e.g. via 
logger command)
module(load="imklog")   # provides kernel logging support (previously done by 
rklogd)

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
module(load="imjournal"  PersistStateInterval="100")
module(load="mmkubernetes" 
kubernetesurl="http://k8s-master.lc.dc.openx.org:8080";

  token="foobar"
  filenamerulebase="/etc/rsyslog.d/k8s_filename.rulebase"
  containerrulebase="/etc/rsyslog.d/k8s_filename.rulebase"
)
action(type="mmkubernetes")
/var/log/k8s;RSYSLOG_DebugFormat

___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] segfault with mmkubernetes

[Rich Megginson via rsyslog]

On 04/04/2018 08:05 PM, Rich Megginson via rsyslog wrote:

On 04/04/2018 07:51 PM, David Lang wrote:

On Wed, 4 Apr 2018, Rich Megginson wrote:


On 04/04/2018 07:18 PM, David Lang wrote:

On Wed, 4 Apr 2018, Rich Megginson wrote:


On 04/04/2018 06:42 PM, David Lang wrote:

On Wed, 4 Apr 2018, Rich Megginson wrote:



Stacktrace?


a bit hard to get (things are being run on a production machine 
after umpteen layers of automation)


here's what I can get so far


ok - so definitely something wrong with the filenamerules parsing.


possibly my error, I had the filenamerules split into two lines and 
didn't catch it. does this look any better?


looks better (at least, looks like what I would expect an Array 
configuration parameter to look like) but still crashing.


oh well - I guess I should have tested with rules - I did all of my 
testing with liblognorm 2.0.2 which makes mmkubernetes not use the 
rules parameters .


Is there a way to not use them with later versions?


Yes, but apparently not without crashing . . .


Try defining both filenamerulebase and containerrulebase:

module(load="mmkubernetes" 
kubernetesurl="http://k8s-master.lc.dc.openx.org:8080";

  token="foobar"
  filenamerulebase="/etc/rsyslog.d/k8s_filename.rulebase"
  containerrulebase="/etc/rsyslog.d/k8s_container_name.rulebase"
)

Using the rulebase files from 
https://github.com/rsyslog/rsyslog/tree/master/contrib/mmkubernetes







I'm (initially) trying to use this module with data arriving via 
imjournal, so whatever is in them won't be used.


module(load="imfile" PollingInterval="10") # provided support for 
monitoring alternate log files
module(load="imuxsock") # provides support for local system logging 
(e.g. via logger command)
module(load="imklog")   # provides kernel logging support (previously 
done by rklogd)

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
module(load="imjournal"  PersistStateInterval="100")
module(load="mmkubernetes" 
kubernetesurl="http://k8s-master.lc.dc.openx.org:8080";

  token="foobar"
  filenamerules=["rule=:/just/some/junk"]
)
action(type="mmkubernetes")

here is the tail end of an strace, why is it loading libregex?


It uses regex to parse the annotations/labels, but you aren't using 
them because you didn't define a pattern.  So there is this code:


DEFobjCurrIf(regexp)
...
BEGINmodInit()
CODESTARTmodInit
    *ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the 
current interface specification */

CODEmodInit_QueryRegCFSLineHdlr
    DBGPRINTF("mmkubernetes: module compiled with rsyslog version 
%s.\n", VERSION);

    CHKiRet(objUse(errmsg, CORE_COMPONENT));
    CHKiRet(objUse(regexp, LM_REGEXP_FILENAME));

Does that cause lmregexp.so to be loaded?



open("/usr/lib64/rsyslog/mmkubernetes.so", O_RDONLY|O_CLOEXEC) = 4
read(4, 
"\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300#\0\0\0\0\0\0"..., 
832) = 832

fstat(4, {st_mode=S_IFREG|0755, st_size=36712, ...}) = 0
mmap(NULL, 2130696, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 
4, 0) = 0x7f5cc759b000

mprotect(0x7f5cc75a3000, 2093056, PROT_NONE) = 0
mmap(0x7f5cc77a2000, 8192, PROT_READ|PROT_WRITE, 
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 4, 0x7000) = 0x7f5cc77a2000

close(4)    = 0
open("/usr/lib64/tls/liblognorm.so.5", O_RDONLY|O_CLOEXEC) = -1 
ENOENT (No such file or directory)

open("/usr/lib64/liblognorm.so.5", O_RDONLY|O_CLOEXEC) = 4
read(4, 
"\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20X\0\0\0\0\0\0"..., 
832) = 832

fstat(4, {st_mode=S_IFREG|0755, st_size=144072, ...}) = 0
mmap(NULL, 2237600, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 
4, 0) = 0x7f5cc7378000

mprotect(0x7f5cc7399000, 2097152, PROT_NONE) = 0
mmap(0x7f5cc7599000, 8192, PROT_READ|PROT_WRITE, 
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 4, 0x21000) = 0x7f5cc7599000

close(4)    = 0
mprotect(0x7f5cc7599000, 4096, PROT_READ) = 0
mprotect(0x7f5cc77a2000, 4096, PROT_READ) = 0
open("/usr/lib64/rsyslog/lmregexp.so", O_RDONLY|O_CLOEXEC) = 4
read(4, 
"\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\10\0\0\0\0\0\0"..., 
832) = 832

fstat(4, {st_mode=S_IFREG|0755, st_size=7064, ...}) = 0
mmap(NULL, 2101392, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 
4, 0) = 0x7f5cc7176000

mprotect(0x7f5cc7177000, 2093056, PROT_NONE) = 0
mmap(0x7f5cc7376000, 8192, PROT_READ|PROT_WRITE, 
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 4, 0) = 0x7f5cc7376000

close(4)    = 0
mprotect(0x7f5cc7376000, 4096, PROT_READ) = 0
gettid()    = 14269
rt_sigaction(SIGPIPE, {SIG_IGN, [], SA_RESTORER, 0x7f5cce0155e0}, 
NULL, 8) = 0

brk(NULL)   = 0x558c751d4000
brk(0x558c751f5000) = 0x558c751f5000
brk(NULL)   = 0x558c751f5000
brk(NULL)   = 0x558c751f5000
brk(0x558c751f4000) = 0x558c751f4000
brk(NULL)   = 0x558c751f4000
--- SIGSEGV {s

Re: [rsyslog] segfault with mmkubernetes

[Rich Megginson via rsyslog]

On 04/04/2018 07:51 PM, David Lang wrote:

On Wed, 4 Apr 2018, Rich Megginson wrote:


On 04/04/2018 07:18 PM, David Lang wrote:

On Wed, 4 Apr 2018, Rich Megginson wrote:


On 04/04/2018 06:42 PM, David Lang wrote:

On Wed, 4 Apr 2018, Rich Megginson wrote:



Stacktrace?


a bit hard to get (things are being run on a production machine 
after umpteen layers of automation)


here's what I can get so far


ok - so definitely something wrong with the filenamerules parsing.


possibly my error, I had the filenamerules split into two lines and 
didn't catch it. does this look any better?


looks better (at least, looks like what I would expect an Array 
configuration parameter to look like) but still crashing.


oh well - I guess I should have tested with rules - I did all of my 
testing with liblognorm 2.0.2 which makes mmkubernetes not use the 
rules parameters .


Is there a way to not use them with later versions?


Yes, but apparently not without crashing . . .



I'm (initially) trying to use this module with data arriving via 
imjournal, so whatever is in them won't be used.


module(load="imfile" PollingInterval="10") # provided support for 
monitoring alternate log files
module(load="imuxsock") # provides support for local system logging 
(e.g. via logger command)
module(load="imklog")   # provides kernel logging support (previously 
done by rklogd)

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
module(load="imjournal"  PersistStateInterval="100")
module(load="mmkubernetes" 
kubernetesurl="http://k8s-master.lc.dc.openx.org:8080";

  token="foobar"
  filenamerules=["rule=:/just/some/junk"]
)
action(type="mmkubernetes")

here is the tail end of an strace, why is it loading libregex?


It uses regex to parse the annotations/labels, but you aren't using them 
because you didn't define a pattern.  So there is this code:


DEFobjCurrIf(regexp)
...
BEGINmodInit()
CODESTARTmodInit
    *ipIFVersProvided = CURR_MOD_IF_VERSION; /* we only support the 
current interface specification */

CODEmodInit_QueryRegCFSLineHdlr
    DBGPRINTF("mmkubernetes: module compiled with rsyslog version 
%s.\n", VERSION);

    CHKiRet(objUse(errmsg, CORE_COMPONENT));
    CHKiRet(objUse(regexp, LM_REGEXP_FILENAME));

Does that cause lmregexp.so to be loaded?



open("/usr/lib64/rsyslog/mmkubernetes.so", O_RDONLY|O_CLOEXEC) = 4
read(4, 
"\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300#\0\0\0\0\0\0"..., 
832) = 832

fstat(4, {st_mode=S_IFREG|0755, st_size=36712, ...}) = 0
mmap(NULL, 2130696, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 4, 
0) = 0x7f5cc759b000

mprotect(0x7f5cc75a3000, 2093056, PROT_NONE) = 0
mmap(0x7f5cc77a2000, 8192, PROT_READ|PROT_WRITE, 
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 4, 0x7000) = 0x7f5cc77a2000

close(4)    = 0
open("/usr/lib64/tls/liblognorm.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT 
(No such file or directory)

open("/usr/lib64/liblognorm.so.5", O_RDONLY|O_CLOEXEC) = 4
read(4, 
"\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20X\0\0\0\0\0\0"..., 
832) = 832

fstat(4, {st_mode=S_IFREG|0755, st_size=144072, ...}) = 0
mmap(NULL, 2237600, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 4, 
0) = 0x7f5cc7378000

mprotect(0x7f5cc7399000, 2097152, PROT_NONE) = 0
mmap(0x7f5cc7599000, 8192, PROT_READ|PROT_WRITE, 
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 4, 0x21000) = 0x7f5cc7599000

close(4)    = 0
mprotect(0x7f5cc7599000, 4096, PROT_READ) = 0
mprotect(0x7f5cc77a2000, 4096, PROT_READ) = 0
open("/usr/lib64/rsyslog/lmregexp.so", O_RDONLY|O_CLOEXEC) = 4
read(4, 
"\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\10\0\0\0\0\0\0"..., 
832) = 832

fstat(4, {st_mode=S_IFREG|0755, st_size=7064, ...}) = 0
mmap(NULL, 2101392, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 4, 
0) = 0x7f5cc7176000

mprotect(0x7f5cc7177000, 2093056, PROT_NONE) = 0
mmap(0x7f5cc7376000, 8192, PROT_READ|PROT_WRITE, 
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 4, 0) = 0x7f5cc7376000

close(4)    = 0
mprotect(0x7f5cc7376000, 4096, PROT_READ) = 0
gettid()    = 14269
rt_sigaction(SIGPIPE, {SIG_IGN, [], SA_RESTORER, 0x7f5cce0155e0}, 
NULL, 8) = 0

brk(NULL)   = 0x558c751d4000
brk(0x558c751f5000) = 0x558c751f5000
brk(NULL)   = 0x558c751f5000
brk(NULL)   = 0x558c751f5000
brk(0x558c751f4000) = 0x558c751f4000
brk(NULL)   = 0x558c751f4000
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0} ---
+++ killed by SIGSEGV (core dumped) +++
Segmentation fault


I'm going to be offline for an hour or so while I drive home. I'll 
check my e-mail after I get home and can VPN to the office to continue 
troubleshooting.


Thanks for your help.

David Lang



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com

Re: [rsyslog] segfault with mmkubernetes

[David Lang]

On Wed, 4 Apr 2018, Rich Megginson wrote:


On 04/04/2018 07:18 PM, David Lang wrote:

On Wed, 4 Apr 2018, Rich Megginson wrote:


On 04/04/2018 06:42 PM, David Lang wrote:

On Wed, 4 Apr 2018, Rich Megginson wrote:



Stacktrace?


a bit hard to get (things are being run on a production machine after 
umpteen layers of automation)


here's what I can get so far


ok - so definitely something wrong with the filenamerules parsing.


possibly my error, I had the filenamerules split into two lines and didn't 
catch it. does this look any better?


looks better (at least, looks like what I would expect an Array configuration 
parameter to look like) but still crashing.


oh well - I guess I should have tested with rules - I did all of my testing 
with liblognorm 2.0.2 which makes mmkubernetes not use the rules parameters .


Is there a way to not use them with later versions?

I'm (initially) trying to use this module with data arriving via imjournal, so 
whatever is in them won't be used.


module(load="imfile" PollingInterval="10") # provided support for monitoring 
alternate log files
module(load="imuxsock") # provides support for local system logging (e.g. via 
logger command)
module(load="imklog")   # provides kernel logging support (previously done by 
rklogd)
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
module(load="imjournal"  PersistStateInterval="100")
module(load="mmkubernetes" 
kubernetesurl="http://k8s-master.lc.dc.openx.org:8080";

  token="foobar"
  filenamerules=["rule=:/just/some/junk"]
)
action(type="mmkubernetes")

here is the tail end of an strace, why is it loading libregex?

open("/usr/lib64/rsyslog/mmkubernetes.so", O_RDONLY|O_CLOEXEC) = 4
read(4, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300#\0\0\0\0\0\0"..., 
832) = 832

fstat(4, {st_mode=S_IFREG|0755, st_size=36712, ...}) = 0
mmap(NULL, 2130696, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 4, 0) = 
0x7f5cc759b000

mprotect(0x7f5cc75a3000, 2093056, PROT_NONE) = 0
mmap(0x7f5cc77a2000, 8192, PROT_READ|PROT_WRITE, 
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 4, 0x7000) = 0x7f5cc77a2000

close(4)= 0
open("/usr/lib64/tls/liblognorm.so.5", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such 
file or directory)

open("/usr/lib64/liblognorm.so.5", O_RDONLY|O_CLOEXEC) = 4
read(4, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20X\0\0\0\0\0\0"..., 
832) = 832

fstat(4, {st_mode=S_IFREG|0755, st_size=144072, ...}) = 0
mmap(NULL, 2237600, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 4, 0) = 
0x7f5cc7378000

mprotect(0x7f5cc7399000, 2097152, PROT_NONE) = 0
mmap(0x7f5cc7599000, 8192, PROT_READ|PROT_WRITE, 
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 4, 0x21000) = 0x7f5cc7599000

close(4)= 0
mprotect(0x7f5cc7599000, 4096, PROT_READ) = 0
mprotect(0x7f5cc77a2000, 4096, PROT_READ) = 0
open("/usr/lib64/rsyslog/lmregexp.so", O_RDONLY|O_CLOEXEC) = 4
read(4, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\0\10\0\0\0\0\0\0"..., 
832) = 832

fstat(4, {st_mode=S_IFREG|0755, st_size=7064, ...}) = 0
mmap(NULL, 2101392, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 4, 0) = 
0x7f5cc7176000

mprotect(0x7f5cc7177000, 2093056, PROT_NONE) = 0
mmap(0x7f5cc7376000, 8192, PROT_READ|PROT_WRITE, 
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 4, 0) = 0x7f5cc7376000

close(4)= 0
mprotect(0x7f5cc7376000, 4096, PROT_READ) = 0
gettid()= 14269
rt_sigaction(SIGPIPE, {SIG_IGN, [], SA_RESTORER, 0x7f5cce0155e0}, NULL, 8) = 0
brk(NULL)   = 0x558c751d4000
brk(0x558c751f5000) = 0x558c751f5000
brk(NULL)   = 0x558c751f5000
brk(NULL)   = 0x558c751f5000
brk(0x558c751f4000) = 0x558c751f4000
brk(NULL)   = 0x558c751f4000
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0} ---
+++ killed by SIGSEGV (core dumped) +++
Segmentation fault


I'm going to be offline for an hour or so while I drive home. I'll check my 
e-mail after I get home and can VPN to the office to continue troubleshooting.


Thanks for your help.

David Lang
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] segfault with mmkubernetes

[Rich Megginson via rsyslog]

On 04/04/2018 07:18 PM, David Lang wrote:

On Wed, 4 Apr 2018, Rich Megginson wrote:


On 04/04/2018 06:42 PM, David Lang wrote:

On Wed, 4 Apr 2018, Rich Megginson wrote:



Stacktrace?


a bit hard to get (things are being run on a production machine 
after umpteen layers of automation)


here's what I can get so far


ok - so definitely something wrong with the filenamerules parsing.


possibly my error, I had the filenamerules split into two lines and 
didn't catch it. does this look any better?


looks better (at least, looks like what I would expect an Array 
configuration parameter to look like) but still crashing.


oh well - I guess I should have tested with rules - I did all of my 
testing with liblognorm 2.0.2 which makes mmkubernetes not use the rules 
parameters . . .




note: I had to add rule= in each of the rules, just having them start 
with : generated the earlier error message


[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `rsyslogd -dn'.
Program terminated with signal 11, Segmentation fault.
#0  0x7f6c47e3c18c in fgetpos@@GLIBC_2.2.5 () from 
/usr/lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install 
rsyslog-8.34.0-1.el7.centos.x86_64

(gdb) bt
#0  0x7f6c47e3c18c in fgetpos@@GLIBC_2.2.5 () from 
/usr/lib64/libc.so.6
#1  0x7f6c42145993 in ln_sampChkRunawayRule () from 
/usr/lib64/liblognorm.so.5

#2  0x7f6c42145cae in ln_sampRead () from /usr/lib64/liblognorm.so.5
#3  0x7f6c42146920 in ln_sampLoadFromString () from 
/usr/lib64/liblognorm.so.5
#4  0x7f6c42141b6f in ln_loadSamplesFromString () from 
/usr/lib64/liblognorm.so.5
#5  0x7f6c42361e88 in set_lnctx () from 
/usr/lib64/rsyslog/mmkubernetes.so
#6  0x7f6c42363482 in newActInst () from 
/usr/lib64/rsyslog/mmkubernetes.so

#7  0x55b45d49bb51 in actionNewInst ()
#8  0x55b45d4663bf in cnfstmtNewAct ()
#9  0x55b45d45b0e2 in yyparse ()
#10 0x55b45d46f234 in load ()
#11 0x55b45d44dff6 in initAll ()
#12 0x55b45d44bd8f in main ()
(gdb) quit
[root@xvbb-c21 rsyslog.d]# grep -v '^#' ../rsyslog.conf|grep [a-z\)]
module(load="imfile" PollingInterval="10") # provided support for 
monitoring alternate log files
module(load="imuxsock") # provides support for local system logging 
(e.g. via logger command)
module(load="imklog")   # provides kernel logging support (previously 
done by rklogd)

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
module(load="imjournal"  PersistStateInterval="100")
module(load="mmkubernetes" 
kubernetesurl="http://k8s-master.lc.dc.openx.org:8080";

  token="foobar"
  annotation_match=[".*"]
filenamerules=["rule=:/var/log/containers/%pod_name:char-to:.%.%container_hash:char-to:_%_%namespace_name:char-to:_%_%container_name:char-to:-%-%container_id:char-to:.%.log","rule=:/var/log/containers/%pod_name:char-:_%_%namespace_name:char-to:_%_%container_name:char-to:-%-%container_id:char-to:.%.log"]
)
action(type="mmkubernetes")
stop



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] segfault with mmkubernetes

[David Lang]

On Wed, 4 Apr 2018, Rich Megginson wrote:


On 04/04/2018 06:42 PM, David Lang wrote:

On Wed, 4 Apr 2018, Rich Megginson wrote:



Stacktrace?


a bit hard to get (things are being run on a production machine after 
umpteen layers of automation)


here's what I can get so far


ok - so definitely something wrong with the filenamerules parsing.


possibly my error, I had the filenamerules split into two lines and didn't catch 
it. does this look any better?


note: I had to add rule= in each of the rules, just having them start with : 
generated the earlier error message


[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `rsyslogd -dn'.
Program terminated with signal 11, Segmentation fault.
#0  0x7f6c47e3c18c in fgetpos@@GLIBC_2.2.5 () from /usr/lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install 
rsyslog-8.34.0-1.el7.centos.x86_64

(gdb) bt
#0  0x7f6c47e3c18c in fgetpos@@GLIBC_2.2.5 () from /usr/lib64/libc.so.6
#1  0x7f6c42145993 in ln_sampChkRunawayRule () from 
/usr/lib64/liblognorm.so.5

#2  0x7f6c42145cae in ln_sampRead () from /usr/lib64/liblognorm.so.5
#3  0x7f6c42146920 in ln_sampLoadFromString () from 
/usr/lib64/liblognorm.so.5
#4  0x7f6c42141b6f in ln_loadSamplesFromString () from 
/usr/lib64/liblognorm.so.5

#5  0x7f6c42361e88 in set_lnctx () from /usr/lib64/rsyslog/mmkubernetes.so
#6  0x7f6c42363482 in newActInst () from /usr/lib64/rsyslog/mmkubernetes.so
#7  0x55b45d49bb51 in actionNewInst ()
#8  0x55b45d4663bf in cnfstmtNewAct ()
#9  0x55b45d45b0e2 in yyparse ()
#10 0x55b45d46f234 in load ()
#11 0x55b45d44dff6 in initAll ()
#12 0x55b45d44bd8f in main ()
(gdb) quit
[root@xvbb-c21 rsyslog.d]# grep -v '^#' ../rsyslog.conf|grep [a-z\)]
module(load="imfile" PollingInterval="10") # provided support for monitoring 
alternate log files
module(load="imuxsock") # provides support for local system logging (e.g. via 
logger command)
module(load="imklog")   # provides kernel logging support (previously done by 
rklogd)

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
module(load="imjournal"  PersistStateInterval="100")
module(load="mmkubernetes" 
kubernetesurl="http://k8s-master.lc.dc.openx.org:8080";

  token="foobar"
  annotation_match=[".*"]
  
filenamerules=["rule=:/var/log/containers/%pod_name:char-to:.%.%container_hash:char-to:_%_%namespace_name:char-to:_%_%container_name:char-to:-%-%container_id:char-to:.%.log","rule=:/var/log/containers/%pod_name:char-:_%_%namespace_name:char-to:_%_%container_name:char-to:-%-%container_id:char-to:.%.log"]

)
action(type="mmkubernetes")
stop
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] segfault with mmkubernetes

[David Lang]

On Wed, 4 Apr 2018, Rich Megginson wrote:


On 04/04/2018 06:42 PM, David Lang wrote:

On Wed, 4 Apr 2018, Rich Megginson wrote:



Stacktrace?


a bit hard to get (things are being run on a production machine after 
umpteen layers of automation)


here's what I can get so far


ok - so definitely something wrong with the filenamerules parsing.

Does it work if you use

   filenamerulebase="/etc/rsyslog.d/k8s_filename.rulebase"


nope

Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `rsyslogd -dn'.
Program terminated with signal 11, Segmentation fault.
#0  0x7f90c7a4b18c in fgetpos@@GLIBC_2.2.5 () from /usr/lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install 
rsyslog-8.34.0-1.el7.centos.x86_64

(gdb) bt
#0  0x7f90c7a4b18c in fgetpos@@GLIBC_2.2.5 () from /usr/lib64/libc.so.6
#1  0x7f90c1d54993 in ln_sampChkRunawayRule () from 
/usr/lib64/liblognorm.so.5

#2  0x7f90c1d54cae in ln_sampRead () from /usr/lib64/liblognorm.so.5
#3  0x7f90c1d55920 in ln_sampLoadFromString () from 
/usr/lib64/liblognorm.so.5
#4  0x7f90c1d50b6f in ln_loadSamplesFromString () from 
/usr/lib64/liblognorm.so.5

#5  0x7f90c1f70e88 in set_lnctx () from /usr/lib64/rsyslog/mmkubernetes.so
#6  0x7f90c1f72482 in newActInst () from /usr/lib64/rsyslog/mmkubernetes.so
#7  0x563432580b51 in actionNewInst ()
#8  0x56343254b3bf in cnfstmtNewAct ()
#9  0x5634325400e2 in yyparse ()
#10 0x563432554234 in load ()
#11 0x563432532ff6 in initAll ()
#12 0x563432530d8f in main ()
(gdb) quit
[root@xvbb-c21 rsyslog.d]# grep -v '^#' ../rsyslog.conf|grep [a-z\)]
module(load="imfile" PollingInterval="10") # provided support for monitoring 
alternate log files
module(load="imuxsock") # provides support for local system logging (e.g. via 
logger command)
module(load="imklog")   # provides kernel logging support (previously done by 
rklogd)

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
module(load="imjournal"  PersistStateInterval="100")
module(load="mmkubernetes" 
kubernetesurl="http://k8s-master.lc.dc.openx.org:8080";

  token="foobar"
  annotation_match=[".*"]
  filenamerulebase="/etc/rsyslog.d/k8s_filename.rulebase"
)
action(type="mmkubernetes")
stop




?



# gdb /usr/sbin/rsyslogd core.3364
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7_4.1
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 


This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
...
Reading symbols from /usr/sbin/rsyslogd...Reading symbols from 
/usr/sbin/rsyslogd...(no debugging symbols found)...done.

(no debugging symbols found)...done.
[New LWP 3364]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `rsyslogd -dn'.
Program terminated with signal 11, Segmentation fault.
#0  0x7fe0419d518c in fgetpos@@GLIBC_2.2.5 () from /usr/lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install 
rsyslog-8.34.0-1.el7.centos.x86_64

(gdb) bt
#0  0x7fe0419d518c in fgetpos@@GLIBC_2.2.5 () from /usr/lib64/libc.so.6
#1  0x7fe03bcde993 in ln_sampChkRunawayRule () from 
/usr/lib64/liblognorm.so.5

#2  0x7fe03bcdecae in ln_sampRead () from /usr/lib64/liblognorm.so.5
#3  0x7fe03bcdf920 in ln_sampLoadFromString () from 
/usr/lib64/liblognorm.so.5
#4  0x7fe03bcdab6f in ln_loadSamplesFromString () from 
/usr/lib64/liblognorm.so.5
#5  0x7fe03befae88 in set_lnctx () from 
/usr/lib64/rsyslog/mmkubernetes.so
#6  0x7fe03befc448 in newActInst () from 
/usr/lib64/rsyslog/mmkubernetes.so

#7  0x560790b9bb51 in actionNewInst ()
#8  0x560790b663bf in cnfstmtNewAct ()
#9  0x560790b5b0e2 in yyparse ()
#10 0x560790b6f234 in load ()
#11 0x560790b4dff6 in initAll ()
#12 0x560790b4bd8f in main ()
(gdb)





___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] segfault with mmkubernetes

[Rich Megginson via rsyslog]

On 04/04/2018 06:42 PM, David Lang wrote:

On Wed, 4 Apr 2018, Rich Megginson wrote:



Stacktrace?


a bit hard to get (things are being run on a production machine after 
umpteen layers of automation)


here's what I can get so far


ok - so definitely something wrong with the filenamerules parsing.

Does it work if you use

   filenamerulebase="/etc/rsyslog.d/k8s_filename.rulebase"

?



# gdb /usr/sbin/rsyslogd core.3364
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7_4.1
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 


This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show 
copying"

and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
...
Reading symbols from /usr/sbin/rsyslogd...Reading symbols from 
/usr/sbin/rsyslogd...(no debugging symbols found)...done.

(no debugging symbols found)...done.
[New LWP 3364]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `rsyslogd -dn'.
Program terminated with signal 11, Segmentation fault.
#0  0x7fe0419d518c in fgetpos@@GLIBC_2.2.5 () from 
/usr/lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install 
rsyslog-8.34.0-1.el7.centos.x86_64

(gdb) bt
#0  0x7fe0419d518c in fgetpos@@GLIBC_2.2.5 () from 
/usr/lib64/libc.so.6
#1  0x7fe03bcde993 in ln_sampChkRunawayRule () from 
/usr/lib64/liblognorm.so.5

#2  0x7fe03bcdecae in ln_sampRead () from /usr/lib64/liblognorm.so.5
#3  0x7fe03bcdf920 in ln_sampLoadFromString () from 
/usr/lib64/liblognorm.so.5
#4  0x7fe03bcdab6f in ln_loadSamplesFromString () from 
/usr/lib64/liblognorm.so.5
#5  0x7fe03befae88 in set_lnctx () from 
/usr/lib64/rsyslog/mmkubernetes.so
#6  0x7fe03befc448 in newActInst () from 
/usr/lib64/rsyslog/mmkubernetes.so

#7  0x560790b9bb51 in actionNewInst ()
#8  0x560790b663bf in cnfstmtNewAct ()
#9  0x560790b5b0e2 in yyparse ()
#10 0x560790b6f234 in load ()
#11 0x560790b4dff6 in initAll ()
#12 0x560790b4bd8f in main ()
(gdb)



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] segfault with mmkubernetes

[David Lang]

On Wed, 4 Apr 2018, Rich Megginson wrote:



Stacktrace?


a bit hard to get (things are being run on a production machine after umpteen 
layers of automation)


here's what I can get so far

# gdb /usr/sbin/rsyslogd core.3364
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7_4.1
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
...
Reading symbols from /usr/sbin/rsyslogd...Reading symbols from 
/usr/sbin/rsyslogd...(no debugging symbols found)...done.

(no debugging symbols found)...done.
[New LWP 3364]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `rsyslogd -dn'.
Program terminated with signal 11, Segmentation fault.
#0  0x7fe0419d518c in fgetpos@@GLIBC_2.2.5 () from /usr/lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install 
rsyslog-8.34.0-1.el7.centos.x86_64

(gdb) bt
#0  0x7fe0419d518c in fgetpos@@GLIBC_2.2.5 () from /usr/lib64/libc.so.6
#1  0x7fe03bcde993 in ln_sampChkRunawayRule () from 
/usr/lib64/liblognorm.so.5

#2  0x7fe03bcdecae in ln_sampRead () from /usr/lib64/liblognorm.so.5
#3  0x7fe03bcdf920 in ln_sampLoadFromString () from 
/usr/lib64/liblognorm.so.5
#4  0x7fe03bcdab6f in ln_loadSamplesFromString () from 
/usr/lib64/liblognorm.so.5

#5  0x7fe03befae88 in set_lnctx () from /usr/lib64/rsyslog/mmkubernetes.so
#6  0x7fe03befc448 in newActInst () from /usr/lib64/rsyslog/mmkubernetes.so
#7  0x560790b9bb51 in actionNewInst ()
#8  0x560790b663bf in cnfstmtNewAct ()
#9  0x560790b5b0e2 in yyparse ()
#10 0x560790b6f234 in load ()
#11 0x560790b4dff6 in initAll ()
#12 0x560790b4bd8f in main ()
(gdb)
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] segfault with mmkubernetes

[Rich Megginson via rsyslog]

On 04/04/2018 06:28 PM, David Lang wrote:

On Wed, 4 Apr 2018, David Lang wrote:



On Wed, 4 Apr 2018, Rich Megginson wrote:


On 04/04/2018 05:57 PM, David Lang wrote:

On Wed, 4 Apr 2018, Rich Megginson via rsyslog wrote:


On 04/04/2018 05:22 PM, David Lang wrote:
I'm trying to use mmkubernetes where the docker containers are 
logging via journald.


my config is:

module(load="imfile" PollingInterval="10") # provided support for 
monitoring alternate log files
module(load="imuxsock") # provides support for local system logging 

(e.g.

via logger command)
module(load="imklog")   # provides kernel logging support 
(previously done by rklogd)

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat


I don't see imjournal here?

module(load="mmkubernetes" 
kubernetesurl="http://k8s-master.lc.dc.openx.org:8080";

  tls.cacert="/etc/ssl/certs/ca-bundle.crt"
  allowunsignedcerts="on"
  token="foobar"
  annotation_match=[".*"]
  filenamerules="rule=:/var/log/containers/"


filenamerules is an array value:
https://github.com/rsyslog/rsyslog/blob/master/contrib/mmkubernetes/mmkubernetes.c#L183 



That may be causing the crash.

so there is a doc bug:
http://www.rsyslog.com/doc/v8-stable/configuration/modules/mmkubernetes.html?highlight=kubernetes#filenamerules 






Does it work if you use a filenamerules like this:
https://github.com/rsyslog/rsyslog/blob/master/contrib/mmkubernetes/mmkubernetes.c#L78 




filenamerules=[":/var/log/containers/%pod_name:char-to:.%.%container_hash:char-to:_%_%namespace_name:char-to:_%_%container_name:char-to:-%-%container_id:char-to:.%.log",":/var/log/containers/%pod_name:char-to:_%_%namespace_name:char-to:_%_%container_name:char-to:-%-%container_id:char-to:.%.log"] 



still segfaults

module(load="imfile" PollingInterval="10") # provided support for 
monitoring alternate log files
module(load="imuxsock") # provides support for local system logging 
(e.g. via logger command)
module(load="imklog")   # provides kernel logging support (previously 
done by rklogd)

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
module(load="mmkubernetes" 
kubernetesurl="http://k8s-master.lc.dc.openx.org:8080";

  token="foobar"
  annotation_match=[".*"]
filenamerules=[":/var/log/containers/%pod_name:char-to:.%.%container_hash:char-to:_%_%namespace_name:char-to:_%_%container_name:char-to:-%-%container_id:c
har-to:.%.log",":/var/log/containers/%pod_name:char-to:_%_%namespace_name:char-to:_%_%container_name:char-to:-%-%container_id:char-to:.%.log"] 


)
action(type="mmkubernetes")
stop



but the following doesn't (it's not processing logs, so I need to add 
the imjournal module)


no it doesn't, the error messages were lost in the debug output :-)



Stacktrace?



the debug output for the config above starting with the module load is:

7960.680695995:main thread    : rsconf.c: cnf:global:obj: obj: 'module'
7960.680700067:main thread    : rainerscript.c: nvlst 0x55ca470e76a0:
7960.680704148:main thread    : rainerscript.c: name: 
'filenamerules':

7960.680707930:main thread    : rainerscript.c:   ARRAY:
7960.680720787:main thread    : rainerscript.c: string 
':/var/log/containers/%pod_name:char-to:.%.%container_hash:char-to:_%_%namespace_name:char-to:_%_%container_name:char-to:-%-%container_id:c 


har-to:.%.log'
7960.680737798:main thread    : rainerscript.c: string 
':/var/log/containers/%pod_name:char-to:_%_%namespace_name:char-to:_%_%container_name:char-to:-%-%container_id:char-to:.%.log' 

7960.680750651:main thread    : rainerscript.c: name: 
'annotation_match':

7960.680754059:main thread    : rainerscript.c:   ARRAY:
7960.680764436:main thread    : rainerscript.c: string '.*'
7960.680778540:main thread    : rainerscript.c: name: 'token', 
value 'foobar'
7960.680782221:main thread    : rainerscript.c: name: 
'kubernetesurl', value 'http://k8s-master.lc.dc.openx.org:8080'
7960.680786044:main thread    : rainerscript.c: name: 'load', 
value 'mmkubernetes'
7960.680789819:main thread    : rainerscript.c: nvlstGetParam: name 
'load', type 13, valnode->bUsed 0

7960.680793260:main thread    : modules.c: modulesProcessCnf params:
7960.680796751:main thread    : rainerscript.c: load: 'mmkubernetes'
7960.680803622:main thread    : modules.c: Requested to load module 
'mmkubernetes'
7960.680808070:main thread    : modules.c: loading module 
'/usr/lib64/rsyslog/mmkubernetes.so'
7960.681174148:main thread    : mmkubernetes.c: mmkubernetes: module 
compiled with rsyslog version 8.34.0.
7960.681181581:main thread    : obj.c: caller requested object 
'regexp', not found (iRet -3003)
7960.681190342:main thread    : modules.c: Requested to load module 
'lmregexp'
7960.681195495:main thread    : modules.c: loading module 
'/usr/lib64/rsyslog/lmregexp.so'
7960.681249645:main thread    : modules.c: module lmregexp of type 2 
being loaded (keepType=0).
7960.681254287:main thread    : regexp.c: entry point 
'i

Re: [rsyslog] segfault with mmkubernetes

[David Lang]

On Wed, 4 Apr 2018, David Lang wrote:



On Wed, 4 Apr 2018, Rich Megginson wrote:


On 04/04/2018 05:57 PM, David Lang wrote:

On Wed, 4 Apr 2018, Rich Megginson via rsyslog wrote:


On 04/04/2018 05:22 PM, David Lang wrote:
I'm trying to use mmkubernetes where the docker containers are logging 
via journald.


my config is:

module(load="imfile" PollingInterval="10") # provided support for 
monitoring alternate log files
module(load="imuxsock") # provides support for local system logging 
(e.g. 

via logger command)
module(load="imklog")   # provides kernel logging support (previously 
done by rklogd)

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat


I don't see imjournal here?

module(load="mmkubernetes" 
kubernetesurl="http://k8s-master.lc.dc.openx.org:8080";

  tls.cacert="/etc/ssl/certs/ca-bundle.crt"
  allowunsignedcerts="on"
  token="foobar"
  annotation_match=[".*"]
  filenamerules="rule=:/var/log/containers/"


filenamerules is an array value: 


https://github.com/rsyslog/rsyslog/blob/master/contrib/mmkubernetes/mmkubernetes.c#L183


That may be causing the crash.

so there is a doc bug: 


http://www.rsyslog.com/doc/v8-stable/configuration/modules/mmkubernetes.html?highlight=kubernetes#filenamerules





Does it work if you use a filenamerules like this: 


https://github.com/rsyslog/rsyslog/blob/master/contrib/mmkubernetes/mmkubernetes.c#L78




filenamerules=[":/var/log/containers/%pod_name:char-to:.%.%container_hash:char-to:_%_%namespace_name:char-to:_%_%container_name:char-to:-%-%container_id:char-to:.%.log",":/var/log/containers/%pod_name:char-to:_%_%namespace_name:char-to:_%_%container_name:char-to:-%-%container_id:char-to:.%.log"]

still segfaults

module(load="imfile" PollingInterval="10") # provided support for monitoring 
alternate log files
module(load="imuxsock") # provides support for local system logging (e.g. via 
logger command)
module(load="imklog")   # provides kernel logging support (previously done by 
rklogd)

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
module(load="mmkubernetes" 
kubernetesurl="http://k8s-master.lc.dc.openx.org:8080";

  token="foobar"
  annotation_match=[".*"]
  
filenamerules=[":/var/log/containers/%pod_name:char-to:.%.%container_hash:char-to:_%_%namespace_name:char-to:_%_%container_name:char-to:-%-%container_id:c
har-to:.%.log",":/var/log/containers/%pod_name:char-to:_%_%namespace_name:char-to:_%_%container_name:char-to:-%-%container_id:char-to:.%.log"]
)
action(type="mmkubernetes")
stop



but the following doesn't (it's not processing logs, so I need to add the 
imjournal module)


no it doesn't, the error messages were lost in the debug output :-)


the debug output for the config above starting with the module load is:

7960.680695995:main thread: rsconf.c: cnf:global:obj: obj: 'module'
7960.680700067:main thread: rainerscript.c: nvlst 0x55ca470e76a0:
7960.680704148:main thread: rainerscript.c: name: 'filenamerules':
7960.680707930:main thread: rainerscript.c:   ARRAY:
7960.680720787:main thread: rainerscript.c: string 
':/var/log/containers/%pod_name:char-to:.%.%container_hash:char-to:_%_%namespace_name:char-to:_%_%container_name:char-to:-%-%container_id:c

har-to:.%.log'
7960.680737798:main thread: rainerscript.c: string 
':/var/log/containers/%pod_name:char-to:_%_%namespace_name:char-to:_%_%container_name:char-to:-%-%container_id:char-to:.%.log'
7960.680750651:main thread: rainerscript.c: name: 
'annotation_match':

7960.680754059:main thread: rainerscript.c:   ARRAY:
7960.680764436:main thread: rainerscript.c: string '.*'
7960.680778540:main thread: rainerscript.c: name: 'token', value 
'foobar'
7960.680782221:main thread: rainerscript.c: name: 'kubernetesurl', 
value 'http://k8s-master.lc.dc.openx.org:8080'
7960.680786044:main thread: rainerscript.c: name: 'load', value 
'mmkubernetes'
7960.680789819:main thread: rainerscript.c: nvlstGetParam: name 'load', type 
13, valnode->bUsed 0

7960.680793260:main thread: modules.c: modulesProcessCnf params:
7960.680796751:main thread: rainerscript.c: load:  'mmkubernetes'
7960.680803622:main thread: modules.c: Requested to load module 
'mmkubernetes'
7960.680808070:main thread: modules.c: loading module 
'/usr/lib64/rsyslog/mmkubernetes.so'
7960.681174148:main thread: mmkubernetes.c: mmkubernetes: module compiled 
with rsyslog version 8.34.0.
7960.681181581:main thread: obj.c: caller requested object 'regexp', not 
found (iRet -3003)

7960.681190342:main thread: modules.c: Requested to load module 'lmregexp'
7960.681195495:main thread: modules.c: loading module 
'/usr/lib64/rsyslog/lmregexp.so'
7960.681249645:main thread: modules.c: module lmregexp of type 2 being 
loaded (keepType=0).
7960.681254287:main thread: regexp.c: entry point 'isCompatibleWithFeature' 
not present in module
7960.681257540:ma

Re: [rsyslog] segfault with mmkubernetes

[David Lang]

On Wed, 4 Apr 2018, Rich Megginson wrote:


On 04/04/2018 05:57 PM, David Lang wrote:

On Wed, 4 Apr 2018, Rich Megginson via rsyslog wrote:


On 04/04/2018 05:22 PM, David Lang wrote:
I'm trying to use mmkubernetes where the docker containers are logging 
via journald.


my config is:

module(load="imfile" PollingInterval="10") # provided support for 
monitoring alternate log files
module(load="imuxsock") # provides support for local system logging (e.g. 
via logger command)
module(load="imklog")   # provides kernel logging support (previously 
done by rklogd)

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat


I don't see imjournal here?

module(load="mmkubernetes" 
kubernetesurl="http://k8s-master.lc.dc.openx.org:8080";

  tls.cacert="/etc/ssl/certs/ca-bundle.crt"
  allowunsignedcerts="on"
  token="foobar"
  annotation_match=[".*"]
  filenamerules="rule=:/var/log/containers/"


filenamerules is an array value: 
https://github.com/rsyslog/rsyslog/blob/master/contrib/mmkubernetes/mmkubernetes.c#L183


That may be causing the crash.

so there is a doc bug: 
http://www.rsyslog.com/doc/v8-stable/configuration/modules/mmkubernetes.html?highlight=kubernetes#filenamerules





Does it work if you use a filenamerules like this: 
https://github.com/rsyslog/rsyslog/blob/master/contrib/mmkubernetes/mmkubernetes.c#L78


filenamerules=[":/var/log/containers/%pod_name:char-to:.%.%container_hash:char-to:_%_%namespace_name:char-to:_%_%container_name:char-to:-%-%container_id:char-to:.%.log",":/var/log/containers/%pod_name:char-to:_%_%namespace_name:char-to:_%_%container_name:char-to:-%-%container_id:char-to:.%.log"]


still segfaults

module(load="imfile" PollingInterval="10") # provided support for monitoring 
alternate log files
module(load="imuxsock") # provides support for local system logging (e.g. via 
logger command)
module(load="imklog")   # provides kernel logging support (previously done by 
rklogd)

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
module(load="mmkubernetes" 
kubernetesurl="http://k8s-master.lc.dc.openx.org:8080";

  token="foobar"
  annotation_match=[".*"]
  
filenamerules=[":/var/log/containers/%pod_name:char-to:.%.%container_hash:char-to:_%_%namespace_name:char-to:_%_%container_name:char-to:-%-%container_id:c
har-to:.%.log",":/var/log/containers/%pod_name:char-to:_%_%namespace_name:char-to:_%_%container_name:char-to:-%-%container_id:char-to:.%.log"]
)
action(type="mmkubernetes")
stop

but the following doesn't (it's not processing logs, so I need to add the 
imjournal module)


module(load="imfile" PollingInterval="10") # provided support for monitoring 
alternate log files
module(load="imuxsock") # provides support for local system logging (e.g. via 
logger command)
module(load="imklog")   # provides kernel logging support (previously done by 
rklogd)

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
module(load="mmkubernetes" 
kubernetesurl="http://k8s-master.lc.dc.openx.org:8080";

  token="foobar"
  annotation_match=[".*"]

filenamerules=":/var/log/containers/%pod_name:char-to:.%.%container_hash:char-to:_%_%namespace_name:char-to:_%_%container_name:char-to:-%-%container_id:c
har-to:.%.log",":/var/log/containers/%pod_name:char-to:_%_%namespace_name:char-to:_%_%container_name:char-to:-%-%container_id:char-to:.%.log"
)
action(type="mmkubernetes")
stop
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] segfault with mmkubernetes

[Rich Megginson via rsyslog]

On 04/04/2018 05:57 PM, David Lang wrote:

On Wed, 4 Apr 2018, Rich Megginson via rsyslog wrote:


On 04/04/2018 05:22 PM, David Lang wrote:
I'm trying to use mmkubernetes where the docker containers are 
logging via journald.


my config is:

module(load="imfile" PollingInterval="10") # provided support for 
monitoring alternate log files
module(load="imuxsock") # provides support for local system logging 
(e.g. via logger command)
module(load="imklog")   # provides kernel logging support 
(previously done by rklogd)

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat


I don't see imjournal here?

module(load="mmkubernetes" 
kubernetesurl="http://k8s-master.lc.dc.openx.org:8080";

  tls.cacert="/etc/ssl/certs/ca-bundle.crt"
  allowunsignedcerts="on"
  token="foobar"
  annotation_match=[".*"]
  filenamerules="rule=:/var/log/containers/"


filenamerules is an array value: 
https://github.com/rsyslog/rsyslog/blob/master/contrib/mmkubernetes/mmkubernetes.c#L183


That may be causing the crash.

so there is a doc bug: 
http://www.rsyslog.com/doc/v8-stable/configuration/modules/mmkubernetes.html?highlight=kubernetes#filenamerules





Does it work if you use a filenamerules like this: 
https://github.com/rsyslog/rsyslog/blob/master/contrib/mmkubernetes/mmkubernetes.c#L78


filenamerules=[":/var/log/containers/%pod_name:char-to:.%.%container_hash:char-to:_%_%namespace_name:char-to:_%_%container_name:char-to:-%-%container_id:char-to:.%.log",":/var/log/containers/%pod_name:char-to:_%_%namespace_name:char-to:_%_%container_name:char-to:-%-%container_id:char-to:.%.log"]

?



If I remove the filenamerules line, I end up with

6127.976782909:main thread    : rainerscript.c: containerrules: (unset)
6127.976814716:main thread    : errmsg.c: Called LogMsg, msg: 
liblognorm error: rulebase file --NO-FILE--[1]: invalid record type 
detected: 
':/var/log/containers/%pod_name:char-to:.%.%container_hash:char-to:_%_%namespace_name:char-to:_%_%container_name:char-to:-%-%container_id:char-to:.%.log'
rsyslogd: liblognorm error: rulebase file --NO-FILE--[1]: invalid 
record type detected: 
':/var/log/containers/%pod_name:char-to:.%.%container_hash:char-to:_%_%namespace_name:char-to:_%_%container_name:char-to:-%-%container_id:char-to:.%.log' 
[v8.34.0 try http://www.rsyslog.com/e/2427 ]
6127.976840359:main thread    : errmsg.c: Called LogMsg, msg: 
liblognorm error: rulebase file --NO-FILE--[1]: invalid record type 
detected: 
':/var/log/containers/%pod_name:char-to:_%_%namespace_name:char-to:_%_%container_name:char-to:-%-%container_id:char-to:.%.log'
rsyslogd: liblognorm error: rulebase file --NO-FILE--[1]: invalid 
record type detected: 
':/var/log/containers/%pod_name:char-to:_%_%namespace_name:char-to:_%_%container_name:char-to:-%-%container_id:char-to:.%.log' 
[v8.34.0 try http://www.rsyslog.com/e/2427 ]

Segmentation fault

the config is:

# grep -v '^#' ../rsyslog.conf|grep [a-z]
module(load="imfile" PollingInterval="10") # provided support for 
monitoring alternate log files
module(load="imuxsock") # provides support for local system logging 
(e.g. via logger command)
module(load="imklog")   # provides kernel logging support (previously 
done by rklogd)

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
module(load="mmkubernetes" 
kubernetesurl="http://k8s-master.lc.dc.openx.org:8080";

  tls.cacert="/etc/ssl/certs/ca-bundle.crt"
  allowunsignedcerts="on"
  token="foobar"
  annotation_match=[".*"]
)
action(type="mmkubernetes")
stop



___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] segfault with mmkubernetes

[David Lang]

On Wed, 4 Apr 2018, Rich Megginson via rsyslog wrote:


On 04/04/2018 05:22 PM, David Lang wrote:
I'm trying to use mmkubernetes where the docker containers are logging 
via journald.


my config is:

module(load="imfile" PollingInterval="10") # provided support for 
monitoring alternate log files
module(load="imuxsock") # provides support for local system logging 
(e.g. via logger command)
module(load="imklog")   # provides kernel logging support (previously 
done by rklogd)

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat


I don't see imjournal here?

module(load="mmkubernetes" 
kubernetesurl="http://k8s-master.lc.dc.openx.org:8080";

  tls.cacert="/etc/ssl/certs/ca-bundle.crt"
  allowunsignedcerts="on"
  token="foobar"
  annotation_match=[".*"]
  filenamerules="rule=:/var/log/containers/"


filenamerules is an array value: 
https://github.com/rsyslog/rsyslog/blob/master/contrib/mmkubernetes/mmkubernetes.c#L183


That may be causing the crash.

so there is a doc bug: 
http://www.rsyslog.com/doc/v8-stable/configuration/modules/mmkubernetes.html?highlight=kubernetes#filenamerules



If I remove the filenamerules line, I end up with

6127.976782909:main thread: rainerscript.c: containerrules: (unset)
6127.976814716:main thread: errmsg.c: Called LogMsg, msg: liblognorm error: 
rulebase file --NO-FILE--[1]: invalid record type detected: 
':/var/log/containers/%pod_name:char-to:.%.%container_hash:char-to:_%_%namespace_name:char-to:_%_%container_name:char-to:-%-%container_id:char-to:.%.log'
rsyslogd: liblognorm error: rulebase file --NO-FILE--[1]: invalid record type 
detected: 
':/var/log/containers/%pod_name:char-to:.%.%container_hash:char-to:_%_%namespace_name:char-to:_%_%container_name:char-to:-%-%container_id:char-to:.%.log' 
[v8.34.0 try http://www.rsyslog.com/e/2427 ]
6127.976840359:main thread: errmsg.c: Called LogMsg, msg: liblognorm error: 
rulebase file --NO-FILE--[1]: invalid record type detected: 
':/var/log/containers/%pod_name:char-to:_%_%namespace_name:char-to:_%_%container_name:char-to:-%-%container_id:char-to:.%.log'
rsyslogd: liblognorm error: rulebase file --NO-FILE--[1]: invalid record type 
detected: 
':/var/log/containers/%pod_name:char-to:_%_%namespace_name:char-to:_%_%container_name:char-to:-%-%container_id:char-to:.%.log' 
[v8.34.0 try http://www.rsyslog.com/e/2427 ]

Segmentation fault

the config is:

# grep -v '^#' ../rsyslog.conf|grep [a-z]
module(load="imfile" PollingInterval="10") # provided support for monitoring 
alternate log files
module(load="imuxsock") # provides support for local system logging (e.g. via 
logger command)
module(load="imklog")   # provides kernel logging support (previously done by 
rklogd)
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
module(load="mmkubernetes" 
kubernetesurl="http://k8s-master.lc.dc.openx.org:8080";

  tls.cacert="/etc/ssl/certs/ca-bundle.crt"
  allowunsignedcerts="on"
  token="foobar"
  annotation_match=[".*"]
)
action(type="mmkubernetes")
stop
___
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of 
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE 
THAT.

Re: [rsyslog] segfault with mmkubernetes

[Rich Megginson via rsyslog]

On 04/04/2018 05:43 PM, Rich Megginson via rsyslog wrote:

On 04/04/2018 05:22 PM, David Lang wrote:
I'm trying to use mmkubernetes where the docker containers are 
logging via journald.


my config is:

module(load="imfile" PollingInterval="10") # provided support for 
monitoring alternate log files
module(load="imuxsock") # provides support for local system logging 
(e.g. via logger command)
module(load="imklog")   # provides kernel logging support (previously 
done by rklogd)

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat


I don't see imjournal here?

module(load="mmkubernetes" 
kubernetesurl="http://k8s-master.lc.dc.openx.org:8080";

  tls.cacert="/etc/ssl/certs/ca-bundle.crt"


I don't think you need to specify tls.cacert if using the system ca 
bundle - if you do, that's a bug.



  allowunsignedcerts="on"


This means - do not check the cert - is that what you want?  Is there a 
problem with the cert for k8s-master.lc.dc.openx.org?



  token="foobar"
  annotation_match=[".*"]
  filenamerules="rule=:/var/log/containers/"


filenamerules is an array value: 
https://github.com/rsyslog/rsyslog/blob/master/contrib/mmkubernetes/mmkubernetes.c#L183


That may be causing the crash.

so there is a doc bug: 
http://www.rsyslog.com/doc/v8-stable/configuration/modules/mmkubernetes.html?highlight=kubernetes#filenamerules




action(type="mmkubernetes")
stop

one note, filenamerules is only supposed to be needed if you are 
using the file log-driver in docker, but you cannot start up without it.


That's another bug - you should not have to specify any filename or 
container rules or rulebases - it should just work, no matter what the 
log driver is.




rsyslogd 8.34.0, compiled with:
    PLATFORM: x86_64-redhat-linux-gnu
    PLATFORM (lsb_release -d):
    FEATURE_REGEXP: Yes
    GSSAPI Kerberos 5 support:  Yes
    FEATURE_DEBUG (debug build, slow code): No
    32bit Atomic operations supported:  Yes
    64bit Atomic operations supported:  Yes
    memory allocator:   system default
    Runtime Instrumentation (slow code):    No
    uuid support:   Yes
    systemd support:    Yes
    Number of Bits in RainerScript integers: 64

# rpm -qa |grep -e liblognorm -e rsyslog
liblognorm5-2.0.4-3.el7.x86_64
rsyslog-8.34.0-1.el7.centos.x86_64
rsyslog-mmkubernetes-8.34.0-1.el7.centos.x86_64

I get a segfault during startup of rsyslog -dn as soon as it gets to 
the config line for the mmkubernetes action.


That sounds like the array value vs. word value problem.  A stack 
trace would likely confirm.




...
3657.410240277:main thread    : rsconf.c: cnf:global:obj: obj: 'module'
3657.410244521:main thread    : rainerscript.c: nvlst 0x5621d2bf3650:
3657.410247414:main thread    : rainerscript.c: name: 
'filenamerules', value 'rule=:/var/log/containers/'
3657.410250422:main thread    : rainerscript.c: name: 
'annotation_match':

3657.410252983:main thread    : rainerscript.c:   ARRAY:
3657.410262441:main thread    : rainerscript.c: string '.*'
3657.410274927:main thread    : rainerscript.c: name: 
'token', value 'foobar'
3657.410277832:main thread    : rainerscript.c: name: 
'allowunsignedcerts', value 'on'
3657.410280948:main thread    : rainerscript.c: name: 
'tls.cacert', value '/etc/ssl/certs/ca-bundle.crt'
3657.410284269:main thread    : rainerscript.c: name: 
'kubernetesurl', value 'http://k8s-master.lc.dc.openx.org:8080'
3657.410287235:main thread    : rainerscript.c: name: 'load', 
value 'mmkubernetes'
3657.410291003:main thread    : rainerscript.c: nvlstGetParam: name 
'load', type 13, valnode->bUsed 0

3657.410293938:main thread    : modules.c: modulesProcessCnf params:
3657.410296334:main thread    : rainerscript.c: load: 'mmkubernetes'
3657.410301781:main thread    : modules.c: Requested to load module 
'mmkubernetes'
3657.410305008:main thread    : modules.c: loading module 
'/usr/lib64/rsyslog/mmkubernetes.so'
3657.410638835:main thread    : mmkubernetes.c: mmkubernetes: module 
compiled with rsyslog version 8.34.0.
3657.410644530:main thread    : obj.c: caller requested object 
'regexp', not found (iRet -3003)
3657.410650711:main thread    : modules.c: Requested to load module 
'lmregexp'
3657.410654508:main thread    : modules.c: loading module 
'/usr/lib64/rsyslog/lmregexp.so'
3657.410697814:main thread    : modules.c: module lmregexp of type 2 
being loaded (keepType=0).
3657.410701260:main thread    : regexp.c: entry point 
'isCompatibleWithFeature' not present in module
3657.410704285:main thread    : regexp.c: entry point 'setModCnf' not 
present in module
3657.410707099:main thread    : regexp.c: entry point 'getModCnfName' 
not present in module
3657.410709863:main thread    : regexp.c: entry point 'beginCnfLoad' 
not present in module
3657.410713726:main thread    : modules

Re: [rsyslog] segfault with mmkubernetes

[Rich Megginson via rsyslog]

On 04/04/2018 05:22 PM, David Lang wrote:
I'm trying to use mmkubernetes where the docker containers are logging 
via journald.


my config is:

module(load="imfile" PollingInterval="10") # provided support for 
monitoring alternate log files
module(load="imuxsock") # provides support for local system logging 
(e.g. via logger command)
module(load="imklog")   # provides kernel logging support (previously 
done by rklogd)

$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat


I don't see imjournal here?

module(load="mmkubernetes" 
kubernetesurl="http://k8s-master.lc.dc.openx.org:8080";

  tls.cacert="/etc/ssl/certs/ca-bundle.crt"
  allowunsignedcerts="on"
  token="foobar"
  annotation_match=[".*"]
  filenamerules="rule=:/var/log/containers/"


filenamerules is an array value: 
https://github.com/rsyslog/rsyslog/blob/master/contrib/mmkubernetes/mmkubernetes.c#L183


That may be causing the crash.

so there is a doc bug: 
http://www.rsyslog.com/doc/v8-stable/configuration/modules/mmkubernetes.html?highlight=kubernetes#filenamerules




action(type="mmkubernetes")
stop

one note, filenamerules is only supposed to be needed if you are using 
the file log-driver in docker, but you cannot start up without it.


That's another bug - you should not have to specify any filename or 
container rules or rulebases - it should just work, no matter what the 
log driver is.




rsyslogd 8.34.0, compiled with:
    PLATFORM: x86_64-redhat-linux-gnu
    PLATFORM (lsb_release -d):
    FEATURE_REGEXP: Yes
    GSSAPI Kerberos 5 support:  Yes
    FEATURE_DEBUG (debug build, slow code): No
    32bit Atomic operations supported:  Yes
    64bit Atomic operations supported:  Yes
    memory allocator:   system default
    Runtime Instrumentation (slow code):    No
    uuid support:   Yes
    systemd support:    Yes
    Number of Bits in RainerScript integers: 64

# rpm -qa |grep -e liblognorm -e rsyslog
liblognorm5-2.0.4-3.el7.x86_64
rsyslog-8.34.0-1.el7.centos.x86_64
rsyslog-mmkubernetes-8.34.0-1.el7.centos.x86_64

I get a segfault during startup of rsyslog -dn as soon as it gets to 
the config line for the mmkubernetes action.


That sounds like the array value vs. word value problem.  A stack trace 
would likely confirm.




...
3657.410240277:main thread    : rsconf.c: cnf:global:obj: obj: 'module'
3657.410244521:main thread    : rainerscript.c: nvlst 0x5621d2bf3650:
3657.410247414:main thread    : rainerscript.c: name: 
'filenamerules', value 'rule=:/var/log/containers/'
3657.410250422:main thread    : rainerscript.c: name: 
'annotation_match':

3657.410252983:main thread    : rainerscript.c:   ARRAY:
3657.410262441:main thread    : rainerscript.c: string '.*'
3657.410274927:main thread    : rainerscript.c: name: 'token', 
value 'foobar'
3657.410277832:main thread    : rainerscript.c: name: 
'allowunsignedcerts', value 'on'
3657.410280948:main thread    : rainerscript.c: name: 
'tls.cacert', value '/etc/ssl/certs/ca-bundle.crt'
3657.410284269:main thread    : rainerscript.c: name: 
'kubernetesurl', value 'http://k8s-master.lc.dc.openx.org:8080'
3657.410287235:main thread    : rainerscript.c: name: 'load', 
value 'mmkubernetes'
3657.410291003:main thread    : rainerscript.c: nvlstGetParam: name 
'load', type 13, valnode->bUsed 0

3657.410293938:main thread    : modules.c: modulesProcessCnf params:
3657.410296334:main thread    : rainerscript.c: load: 'mmkubernetes'
3657.410301781:main thread    : modules.c: Requested to load module 
'mmkubernetes'
3657.410305008:main thread    : modules.c: loading module 
'/usr/lib64/rsyslog/mmkubernetes.so'
3657.410638835:main thread    : mmkubernetes.c: mmkubernetes: module 
compiled with rsyslog version 8.34.0.
3657.410644530:main thread    : obj.c: caller requested object 
'regexp', not found (iRet -3003)
3657.410650711:main thread    : modules.c: Requested to load module 
'lmregexp'
3657.410654508:main thread    : modules.c: loading module 
'/usr/lib64/rsyslog/lmregexp.so'
3657.410697814:main thread    : modules.c: module lmregexp of type 2 
being loaded (keepType=0).
3657.410701260:main thread    : regexp.c: entry point 
'isCompatibleWithFeature' not present in module
3657.410704285:main thread    : regexp.c: entry point 'setModCnf' not 
present in module
3657.410707099:main thread    : regexp.c: entry point 'getModCnfName' 
not present in module
3657.410709863:main thread    : regexp.c: entry point 'beginCnfLoad' 
not present in module
3657.410713726:main thread    : modules.c: source file mmkubernetes.c 
requested reference for module 'lmregexp', reference count now 1
3657.410785276:main thread    : modules.c: module mmkubernetes of type 
1 being loaded (keepType=1).
3657.410790205:main thread    : modules.c: module config name is 
'mmkubernetes'
3657.410