Re: [graylog2] Re: Geo-Location Processor doesn't create _geolocation fields for custom fields created by pipeline rules

[Jason Haar]

On Thu, Sep 1, 2016 at 2:12 AM, Jan  wrote:

> Found the error. In my original pipeline-rule I used the "to_ip" function
> to convert the pattern match to an IP. With this setting resolving the IP
> to a geo location fails.
> I changed the rule now to convert the pattern match to a string by using
> the "to_string" function. Voila... geo location works for all custom fields
> now.
>

(to Graylog devs). That's a bug isn't it? I mean, what's wrong with
assuming an IP address is an IP address? Shouldn't the GeoIP processor
support both string and "ip" fields types?


-- 
Cheers

Jason Haar
Information Security Manager, Trimble Navigation Ltd.
Phone: +1 408 481 8171
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/CAFChrgJFzC5HFDnX2c1soixC_7LH5n%3D2-MEiymEp88GQeUHhuw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Geo-Location Processor doesn't create _geolocation fields for custom fields created by pipeline rules

[Jochen Schalanda]

Thanks for the feedback!

On Wednesday, 31 August 2016 16:12:11 UTC+2, Jan wrote:
>
> Found the error. In my original pipeline-rule I used the "to_ip" function 
> to convert the pattern match to an IP. With this setting resolving the IP 
> to a geo location fails.
> I changed the rule now to convert the pattern match to a string by using 
> the "to_string" function. Voila... geo location works for all custom fields 
> now.
>
> This is what my rule looks like now:
>
> let matcherSrcIp = regex(".*srcip=((? ?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,
> 2}|2[0-4][0-9]|25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|
> 25[0-5])[.](?:[0-1]?[0-9]{1,2}|2[0-4][0-9]|25[0-5]))(?![0-9])).*", 
> to_string($message.message));
> set_field("FW_SourceIP", to_string(matcherSrcIp["0"]));
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/b117a1a1-405d-4f29-b4a7-576eba99f0a8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Geo-Location Processor doesn't create _geolocation fields for custom fields created by pipeline rules

[Jan]

Found the error. In my original pipeline-rule I used the "to_ip" function 
to convert the pattern match to an IP. With this setting resolving the IP 
to a geo location fails.
I changed the rule now to convert the pattern match to a string by using 
the "to_string" function. Voila... geo location works for all custom fields 
now.

This is what my rule looks like now:

let matcherSrcIp = regex(".*srcip=((?https://groups.google.com/d/msgid/graylog2/b1cc9b66-8250-4df1-b647-ec9b576102a5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Geo-Location Processor doesn't create _geolocation fields for custom fields created by pipeline rules

[Jan]

I checked the message processing setting thru the API-Browser to make sure 
all Graylog nodes have the correct settings.
They all report back

{

  "processor_order": [
"org.graylog.plugins.pipelineprocessor.processors.PipelineInterpreter",
"org.graylog2.messageprocessors.MessageFilterChainProcessor",
"org.graylog.plugins.map.geoip.processor.GeoIpProcessor"
  ],
  "disabled_processors": []
}


So still the open question... what is going on with the GeoIpProcessor and 
why is it ignoring custom fields.

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/4aae168f-2119-4d69-bab7-0621de7abfe1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Geo-Location Processor doesn't create _geolocation fields for custom fields created by pipeline rules

[Jochen Schalanda]

Hi Jan,

from your description and the order of message processors you've described 
(please check again according to 
http://docs.graylog.org/en/2.0/pages/geolocation.html#configure-the-message-processor)
 
it should work.

You can set the logger org.graylog.plugins.map.geoip to DEBUG for more 
information what's happening inside the GeoIP resolver (see /system/loggers 
in the Graylog REST API or the log4j2.xml configuration file).

Also keep in mind that 192.168.100.95 is an IP address from a private IP 
range (see RFC 1918) and will naturally not yield any geo location 
information.


Cheers,
Jochen

On Tuesday, 30 August 2016 10:39:59 UTC+2, Jan wrote:
>
>
> 
>
> Not sure... I thought I posted some examples. So here is a screenshot:
>
>
>
>
>
> Am Dienstag, 30. August 2016 10:16:01 UTC+2 schrieb Jochen Schalanda:
>>
>> Hi Jan,
>>
>> On Tuesday, 30 August 2016 10:03:24 UTC+2, Jan wrote:
>>>
>>> An Example message can look like this […]
>>>
>>
>> Okay, and how does it look like after you've extracted those IP addresses?
>>
>>
>> Cheers,
>> Jochen
>>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/eb7bb75d-4a2a-4196-b9a4-8e5878fc05b3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Geo-Location Processor doesn't create _geolocation fields for custom fields created by pipeline rules

[Jan]

Not sure... I thought I posted some examples. So here is a screenshot:





Am Dienstag, 30. August 2016 10:16:01 UTC+2 schrieb Jochen Schalanda:
>
> Hi Jan,
>
> On Tuesday, 30 August 2016 10:03:24 UTC+2, Jan wrote:
>>
>> An Example message can look like this […]
>>
>
> Okay, and how does it look like after you've extracted those IP addresses?
>
>
> Cheers,
> Jochen
>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/e03d325a-b65e-4665-bfc2-ba740fd05bf2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Geo-Location Processor doesn't create _geolocation fields for custom fields created by pipeline rules

[Jochen Schalanda]

Hi Jan,

On Tuesday, 30 August 2016 10:03:24 UTC+2, Jan wrote:
>
> An Example message can look like this […]
>

Okay, and how does it look like after you've extracted those IP addresses?


Cheers,
Jochen

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/5d306584-1866-4389-aac4-273f96d0ca65%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[graylog2] Re: Geo-Location Processor doesn't create _geolocation fields for custom fields created by pipeline rules

[Jan]

Hi Jochen,

An Example message can look like this

<189>date=2016-08-30 time=08:34:23 devname=fw-cluster1 
devid=FGT3HD4895600243 logid=000114 type=traffic subtype=local 
level=notice vd=mgmt-domain srcip=80.0.0.1 srcport=12345 srcintf="agg1.208" 
dstip=80.0.0.2 dstport=162 dstintf="mgmt-1" sessionid=1582245083 proto=17 
action=deny policyid=0 dstcountry="Germany" srccountry="Germany" 
trandisp=noop service="snmptrap-udp" app="snmptrap-udp" duration=0 
sentbyte=0 rcvdbyte=0 sentpkt=0

With help of the pipeline-rule I extract values like


   - 10.208.1.1
   - 80.0.0.1
   - 212.6.1.1

Some of the addresses are private others are public. I'm not sure how the 
plugin handles private IP-ranges. Should I see at least an empty 
_geolocation field?

Regards,
Jan

Am Montag, 29. August 2016 16:57:55 UTC+2 schrieb Jochen Schalanda:
>
> Hi Jan,
>
> please post some examples of the content of the "FW_SourceIP" field of the 
> messages in your Graylog instance.
>
> Cheers,
> Jochen
>
> On Monday, 29 August 2016 14:30:51 UTC+2, Jan wrote:
>>
>> Hi all,
>>
>> I've just activated the Geo-Location processor within my Graylog 
>> environment and noticed that it does not create _geolocation fields for any 
>> of my custom fields containing an IP-address.
>> Other fields like "source" work fine so I think this is not a general 
>> issue with the plugin. I changed the order for message processing to 1. 
>> Pipeline Processor, 2. Message Filter Chain and 3. GeoIP Resolver
>> cause I extract a lot of fields within pipeline rules.
>>
>> As an example I create a field called "FW_SourceIP":
>>
>> let matcherSrcIp = regex(
>> ".*srcip=((?> , to_string($message.message));
>> set_field("FW_SourceIP", to_ip(matcherSrcIp["0"]));
>>
>> I'm able to use the created field and use it without any problems but I 
>> never get a field "FW_SourceIP_geolocation".
>> The field is stored as a string within the ES index.
>>
>> Has anyone used this combinition of fields, pipeline rules and the GeoIP 
>> plugin?
>>
>> Regards,
>> Jan
>>
>>

-- 
You received this message because you are subscribed to the Google Groups 
"Graylog Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to graylog2+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/graylog2/471db105-f10e-4ad9-84c6-c56262910b93%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.