Summary: If geolocation data is mapped to geo_points in ElasticSearch (to enable bounding box/polygon/pin and distance searches), Graylog maps no longer function with the new data.
System: Graylog 2.1 OVA, ES and everything else as shipped in the OVA. Background: Enabled the geolocation processor on 27th September 2016 (over 4 months ago). The maps worked since that time without issue (I have another problem with the maps - revolving around the apparent inability to graph/map anything other than the message count - but that is a separate discussion and I'll start a new thread for that). Decided to try and run some geo bounding box searches on ElasticSearch using the geolocation info acquired (from 4 months ago to current). Only to discover that ES requires a mapping of the geolocation points to be actual geo_points rather than strings (as the geolocation plugin creates them now). Result: that 4 months worth of data is unusable. So the geolocation processor/plugin is loading geolocation data into a geolocation-aware database, but *not* in a format that the geolocation-aware database can use to do a native geo-based search. This in itself appears counterintuitive. So I added the mapping to ES (after a lot of digging - eventually found the 1st reference below). And the bounding box search works beautifully (using a python script to query ES directly). However, all the maps are now broken and return status 500 errors when clicking the "World Map" option in a geolocation enabled field (message shown in the red background at lower part of screen): Could not load map information Loading map information failed: cannot POST http://localhost/api/plugins/org.graylog.plugins.map/mapdata (500) Maps of the data before the change to the ES mapping still work as expected - so can still map the old data. So it seems that I can have mapping ability in Graylog, _or_ I can do a (python) bounding box/polygon/pin and distance search - but not simultaneously have both? It would seem to me to be a bug - when I enabled the geolocation processor I expected: 1) to be able to do a bounding box/polygon/pin-distance style search at a later time (ie:that the location data is put in the db in a format the db can _natively_ use in a geo query - a standard function of said db) 2) to have maps in the graylog interface 3) for 1 and 2 not to be mutually exclusive "1)" is currently not the case unless the geo_point mapping is added. "2)" only works while "1)" is not enabled. So the question is: How to fix maps in graylog without removing the geo_point mapping from elasticsearch? Am confident this was a minor oversight when the map processor/plugin was built. Note: The format of the data loaded into ES is unchanged - it is still a string as such in query results eg: "srcip_geolocation" : "-37.7204,145.0698" - it is only the mapping in ES that has been changed. References: Graylog plugin - elastic field types: Last post describes the way to enable correct ES mapping of the geo_points. https://groups.google.com/forum/?hl=en#!searchin/graylog2/geopoint%7Csort:relevance/graylog2/G1hOxGMFls4/FDAsN3h3BQAJ Not sure if relevant, but this page indicated additional requirement for the mapping, namely "lat_lon" : true - unsure if that is required/necessary but it didn't cause any errors - I obviously changed the URL to suit my graylog instance: ie: curl -XPUT 'localhost:9200/listing/listing/_mapping?pretty' -d '{"listing": {"properties": {"rent": {"type": "intger"}, "location": {"type": "geo_point", *"lat_lon": true*}}}}' https://gist.github.com/mastermunj/a58298bc56615b54d643 -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/d631139d-77bd-451b-a819-5c5f8e96f0fc%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
