We're using ELK stack for log management of a distributed application. In
our log entries we have a field called "data" that can be of arbitrary
type. We use this field to allow the developer to log any custom metadata
releated to the incident that occurred. For example the following three
entries could be samples of items we'd like to be able to log and store to
Elasticsearch.
{"message" : "Fire Alarm", "level" : "emergency", "data" : { "Location" : 1,
"Temperature" : 76.3 } }
{"message" : "Remote query exceeded warning timeout", "level" : "warning",
"data" : 500}
{"message" : "UPS not available", "level" : "warning", "data" : { "Location"
: "San Francisco", "Rack" : 1 } }
All other fields map nicely to Elasticsearch built-in field types but we
don't know how to treat the "data" field as the value can be of any
Elasticsearch type and different for each entry. When using the built-in
Elasticsearch mapping shipping with Logstash, only the mapping of the first
entry will be used. In the above example only the first message would be
stored to the Elasticsearch index.
It's okay if the data field is not indexed. But we'd still like to be able
to store and retrieve it from elasticsearch. What kind of mapping document
would work? I was trying different options but didn't seem to find one that
would do what we needed. The logstash 1.5 default Elasticsearch mapping is
below.
{
"template" : "logstash-*",
"settings" : {
"index.refresh_interval" : "5s"
},
"mappings" : {
"_default_" : {
"_all" : {"enabled" : true},
"dynamic_templates" : [ {
"message_field" : {
"match" : "message",
"match_mapping_type" : "string",
"mapping" : {
"type" : "string", "index" : "analyzed", "omit_norms" : true
}
}
}, {
"string_fields" : {
"match" : "*",
"match_mapping_type" : "string",
"mapping" : {
"type" : "string", "index" : "analyzed", "omit_norms" : true,
"fields" : {
"raw" : {"type": "string", "index" : "not_analyzed",
"ignore_above" : 256}
}
}
}
} ],
"properties" : {
"@version": { "type": "string", "index": "not_analyzed" },
"geoip" : {
"type" : "object",
"dynamic": true,
"path": "full",
"properties" : {
"location" : { "type" : "geo_point" }
}
}
}
}
}
}
Thanks,
Tomi
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/cbf47107-8061-429f-a024-0006a47420f3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
