yes, you are using string properties on a date mapping field --
Itamar Syn-Hershko http://code972.com | @synhershko <https://twitter.com/synhershko> Freelance Developer & Consultant Lucene.NET committer and PMC member On Mon, Feb 9, 2015 at 12:23 PM, Paul Kavanagh <[email protected]> wrote: > I think you have something there. I have come up with this: > > curl -XPUT localhost:9200/_template/template_1 -d ' > { > "template" : "logstash-*", > "order" : 0, > "settings" : { > "number_of_shards" : 15 > }, > "mappings" : { > "dynamic_templates":[ > {"apiservice_logstash":{ > "match":"[email protected].*", > "match_mapping_type":"dateOptionalTime", > "mapping":{ > "type":"string", > "analyzer":"english" > } > } > } > ] > } > } > ' > > However... When I try to post it, Elasticsearch throws: > {"error":"ElasticsearchIllegalArgumentException[Malformed mappings section > for type [dynamic_templates], should include an inner object describing the > mapping]","status":400} > > i've tried a few things, but it doesn't seem to like my mappings block for > some reason. > > Any idea why? > > > On Friday, February 6, 2015 at 11:41:49 AM UTC, Itamar Syn-Hershko wrote: >> >> You mean something like dynamic templates? http://code972.com/ >> blog/2015/02/81-elasticsearch-one-tip-a-day-using-dynamic- >> templates-to-avoid-rigorous-mappings >> >> -- >> >> Itamar Syn-Hershko >> http://code972.com | @synhershko <https://twitter.com/synhershko> >> Freelance Developer & Consultant >> Lucene.NET committer and PMC member >> >> On Fri, Feb 6, 2015 at 1:39 PM, Paul Kavanagh <[email protected]> >> wrote: >> >>> Hi all, >>> We're having a MapperParsingException problem with some field values >>> when we get when we use the JSON Filter for Logstash to explode out a JSON >>> document to Elasticsearch fields. >>> >>> In 99.9% of cases, certain of these fields are either blank, or contain >>> dates in the format of yyyy-mm-dd. This allows ES to dynamically map this >>> field to type dateOptionalTime. >>> >>> However, we occasionally see non-standard date formats in these fields, >>> which our main service can handle fine, but which throws a >>> MapperParsingException in Elasticsearch - such are here: >>> >>> >>> >>> [2015-02-06 10:46:50,679][WARN ][cluster.action.shard ] [logging- >>> production-elasticsearch-ip-xxx-xxx-xxx-148] [logstash-2015.02.06][2] >>> received shard failed for [logstash-2015.02.06][2], node[ >>> GZpltBjAQUqGyp2B1SLz_g], [R], s[INITIALIZING], indexUUID [BEdTwj- >>> QRuOZB713YAQwvA], reason [Failed to start shard, message [ >>> RecoveryFailedException[[logstash-2015.02.06][2]: Recovery failed from [ >>> logging-production-elasticsearch-ip-xxx-xxx-xxx-82][IALW-92RReiLffQjSL3I >>> -g][logging-production-elasticsearch-ip-xxx-xxx-xxx-82][inet[ip-xxx-xxx- >>> xxx-82.ec2.internal/xxx.xxx.xxx.82:9300]]{max_local_storage_nodes=1, >>> aws_availability_zone=us-east-1e, aws_az=us-east-1e} into [logging- >>> production-elasticsearch-ip-xxx-xxx-xxx-148][GZpltBjAQUqGyp2B1SLz_g][ >>> logging-production-elasticsearch-ip-xxx-xxx-xxx-148][inet[ip-xxx.xxx.xxx >>> .148.ec2.internal/xxx.xxx.xxx.148:9300]]{max_local_storage_nodes=1, >>> aws_availability_zone=us-east-1c, aws_az=us-east-1c}]; nested: >>> RemoteTransportException[[logging-production-elasticsearch-ip-xxx-xxx- >>> xxx-82][inet[/xxx.xxx.xxx.82:9300]][internal:index/shard/recovery/ >>> start_recovery]]; nested: RecoveryEngineException[[logstash-2015.02.06][ >>> 2] Phase[2] Execution failed]; nested: RemoteTransportException[[logging >>> -production-elasticsearch-ip-xxx-xxx-xxx-148][inet[/xxx.xxx.xxx.148:9300 >>> ]][internal:index/shard/recovery/translog_ops]]; nested: >>> MapperParsingException[failed to parse [[email protected] >>> arameters.start_time]]; nested: MapperParsingException[failed to parse >>> date field [Feb 5 2015 12:00 AM], tried both date format [ >>> dateOptionalTime], and timestamp number with locale []]; nested: >>> IllegalArgumentException[Invalid format: "Feb 5 2015 12:00 AM"]; ]] >>> >>> 2015-02-06 10:46:53,685][WARN ][cluster.action.shard ] [logging- >>> production-elasticsearch-ip-xxx-xxx-xxx-148] [logstash-2015.02.06][2] >>> received shard failed for [logstash-2015.02.06][2], node[ >>> GZpltBjAQUqGyp2B1SLz_g], [R], s[INITIALIZING], indexUUID [BEdTwj- >>> QRuOZB713YAQwvA], reason [master [logging-production-elasticsearch-ip- >>> xxx-xxx-xxx-148][GZpltBjAQUqGyp2B1SLz_g][logging-production- >>> elasticsearch-ip-xxx-xxx-xxx-148][inet[ip-xxx-xxx-xxx-148.ec2.internal/ >>> xxx.xxx.xxx.148:9300]]{max_local_storage_nodes=1, aws_availability_zone= >>> us-east-1c, aws_az=us-east-1c} marked shard as initializing, but shard >>> is marked as failed, resend shard failure] >>> >>> >>> Our planned solution was to create a template for Logstash indices that >>> will set these fields to string. But as the field above isn't the only >>> culprit, and more may be added overtime, it makes more sense to create a >>> template to map all fields under [email protected].* >>> to be string. (We never need to query on user entered data, but it's great >>> to have logged for debugging) >>> >>> Is it possible to do this with a template? I could not find a way to do >>> this via the template documentation on the ES site. >>> >>> Any guidance would be great! >>> >>> Thanks, >>> -Paul >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit https://groups.google.com/d/ >>> msgid/elasticsearch/6ca4030f-b6bb-4907-b2fc-e3166fa2a6af% >>> 40googlegroups.com >>> <https://groups.google.com/d/msgid/elasticsearch/6ca4030f-b6bb-4907-b2fc-e3166fa2a6af%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/6ab11e08-873b-4610-9573-d8781327aeb2%40googlegroups.com > <https://groups.google.com/d/msgid/elasticsearch/6ab11e08-873b-4610-9573-d8781327aeb2%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsrLZV2JMEAdyQx5GOzbac5QmzG%2Bni-CJXP8Brky4aYtA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
