Hi all,
I've finally discovered the source of my excess CPU load and high load
averages on my Graylog nodes!
I've got a bunch of extractors that I use to pull information from my
vSphere platform's VMKernel logs.
The catch with these is that a lot of items in the message string vary
quite a bit, so finding a regex to match is quite difficult... read pretty
much impossible for my limited regex skills :)
The way I've worked around this is to use wildcards in the regex strings
and that seems to be causing my load average to go from ~0.4 to ~2 or even
more and the CPU's regularly peak at 100%.
Is this expected behaviour?
I recall an issue with earlier versions of Graylog where wildcards in
stream rules would cause this but I believe that was much improved in the
1.0 release and I have noticed that difference. I'm running 1.0.2 at
present.
Is there a similar improvement with extractors in 1.1 or is it being worked
on perhaps?
I intend to put 1.1 into my test lab early next week but it doesn't see
anywhere near as many messages/sec as Production so I won't really see any
indications until I get it into Production.
I've attached my current extractors.
Any feedback on this would be great, and in the meantime I'll start trying
to optimise my extractors a bit more to see if I can remove some wildcards.
Cheers, Pete
--
You received this message because you are subscribed to the Google Groups
"graylog2" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to graylog2+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
{
"extractors": [
{
"condition_type": "regex",
"condition_value": "(?i).*NMP: nmp_ThrottleLogForDevice.*",
"converters": [],
"cursor_strategy": "copy",
"extractor_config": {
"regex_value": "(?i).*NMP:.*Cmd (0x..).*"
},
"extractor_type": "regex",
"order": 0,
"source_field": "message",
"target_field": "Cmd",
"title": "ESXi: Extract SCSI Command"
},
{
"condition_type": "regex",
"condition_value": "(?i).*NMP: nmp_ThrottleLogForDevice:.*",
"converters": [],
"cursor_strategy": "copy",
"extractor_config": {
"regex_value": "(?i).*NMP: nmp_ThrottleLogForDevice:.*dev \"(.*?)\".*"
},
"extractor_type": "regex",
"order": 0,
"source_field": "message",
"target_field": "Device",
"title": "ESXi: Extract Device"
},
{
"condition_type": "regex",
"condition_value": "(?i).*NMP: nmp_ThrottleLogForDevice:.*",
"converters": [],
"cursor_strategy": "copy",
"extractor_config": {
"regex_value": "(?i).*NMP: nmp_ThrottleLogForDevice:.*path \"(.*?)\".*"
},
"extractor_type": "regex",
"order": 0,
"source_field": "message",
"target_field": "Path",
"title": "ESXi: Extract Path"
},
{
"condition_type": "regex",
"condition_value": "(?i).*NMP: nmp_ThrottleLogForDevice.*",
"converters": [],
"cursor_strategy": "copy",
"extractor_config": {
"regex_value": "(?i).*NMP: nmp_ThrottleLogForDevice.*Failed:
(H:0x.{1,2} D:0x.{1,2} P:0x.{1,2}).*"
},
"extractor_type": "regex",
"order": 0,
"source_field": "message",
"target_field": "Status",
"title": "ESXi: Extract Host, Device, or NMP Plugin Status"
},
{
"condition_type": "regex",
"condition_value": "(?i).*NMP: nmp_ThrottleLogForDevice",
"converters": [],
"cursor_strategy": "copy",
"extractor_config": {
"regex_value": "(?i).*NMP: nmp_ThrottleLogForDevice.*sense data:
(0x.{1,2} 0x.{1,2} 0x.{1,2}).*"
},
"extractor_type": "regex",
"order": 0,
"source_field": "message",
"target_field": "SenseData",
"title": "ESXi: Extract Sense Data"
},
{
"condition_type": "regex",
"condition_value": "(?i).*NMP: nmp_ThrottleLogForDevice",
"converters": [],
"cursor_strategy": "copy",
"extractor_config": {
"regex_value": "(?i).*NMP: nmp_ThrottleLogForDevice.*Act:(.*)$"
},
"extractor_type": "regex",
"order": 0,
"source_field": "message",
"target_field": "Action",
"title": "ESXi: Extract pathing action"
},
{
"condition_type": "regex",
"condition_value": "(?i).*Lost access to volume (.{8}-.{8}-.{4}-.{12})
\\(.+?\\)*",
"converters": [],
"cursor_strategy": "copy",
"extractor_config": {
"regex_value": "(?i).*Lost access to volume (.{8}-.{8}-.{4}-.{12})
\\(.+?\\)*"
},
"extractor_type": "regex",
"order": 0,
"source_field": "message",
"target_field": "VolumeID",
"title": "ESXi: Lost Access to Volume - Volume ID"
},
{
"condition_type": "regex",
"condition_value": "(?i).*Lost access to volume (.{8}-.{8}-.{4}-.{12})
\\(.+?\\)*",
"converters": [],
"cursor_strategy": "copy",
"extractor_config": {
"regex_value": "(?i).*Lost access to volume .{8}-.{8}-.{4}-.{12}
\\((.*)\\) *"
},
"extractor_type": "regex",
"order": 0,
"source_field": "message",
"target_field": "Datastore",
"title": "ESXi: Lost Access to Volume - Datastore Name"
},
{
"condition_type": "regex",
"condition_value": "(?i).*'Hostsvc.FSVolumeProvider'] SetAPDStatus: Added
.* to apd start list",
"converters": [],
"cursor_strategy": "copy",
"extractor_config": {
"regex_value": "(?i).*'Hostsvc.FSVolumeProvider'] SetAPDStatus: Added
(.*) to apd start list"
},
"extractor_type": "regex",
"order": 0,
"source_field": "message",
"target_field": "Datastore",
"title": "ESXI: APD Added - Datastore Name"
},
{
"condition_type": "regex",
"condition_value": "(?i).*'Hostsvc.FSVolumeProvider'] DeleteAPDStarted:
Clearing volume .* from APD Started list",
"converters": [],
"cursor_strategy": "copy",
"extractor_config": {
"regex_value": "(?i).*'Hostsvc.FSVolumeProvider'] DeleteAPDStarted:
Clearing volume (.*) from APD Started list"
},
"extractor_type": "regex",
"order": 0,
"source_field": "message",
"target_field": "Datastore",
"title": "ESXI: APD Cleared - Datastore Name"
},
{
"condition_type": "regex",
"condition_value": "(?i).*WARNING\\: NMP\\:
nmp_DeviceRequestFastDeviceProbe\\:237\\: NMP device \".+?\" state in doubt;
requested fast path state update...",
"converters": [],
"cursor_strategy": "copy",
"extractor_config": {
"regex_value": "(?i).*WARNING\\: NMP\\:
nmp_DeviceRequestFastDeviceProbe\\:237\\: NMP device \"(.+?)\" state in doubt;
requested fast path state update..."
},
"extractor_type": "regex",
"order": 0,
"source_field": "message",
"target_field": "NAAID",
"title": "ESXi Path State In Doubt NAA ID"
}
],
"version": "1.0.2 (e5432f1)"
}
