Alex,
Thank-you very much for your reply. I've fixed my issue with your help.
For future reference I added the following to my logstash indexer for
apache;
date {
locale => "en"
timezone => "Europe/Zurich"
match => [ "apachetime", "dd/MMM/yyyy:HH:mm:ss Z" ]
target => "@timestamp"
}
Thanks again for your reply.
Cheers,
Ben
On Thu, Dec 12, 2013 at 11:01 AM, Alexander Reelsen <[email protected]>wrote:
> Hey,
>
> judging from a quick peek at the problem: The value you are trying to
> parse as the timestamp field (note: not the logstash @timestamp field) is
> an unknown date format. This means it will only happen with apache access
> logs. Somehow the timestamp field has already been configured to be of a
> different format (from a past log entry most likely).
>
> Two possibilites (note: maybe there are others, I am not a logstash wizard)
>
> * Make the timestamp field a string and do not analyze it (which also
> means, you cannot search for it, but usually you are using the @timestamp
> field for that anyway)
> * Have an own type for the apache access logs in your logstash config,
> which uses a special timestamp based mapping format, that fits to CLF
> timestamps
>
> See
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping.htmland
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-put-mapping.htmland
> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-templates.html
>
>
> --Alex
>
>
>
> On Thu, Dec 12, 2013 at 10:29 AM, Ben Morrice <[email protected]>wrote:
>
>> Hi Phil,
>>
>> Unfortunately i'm still seeing the same issue. Like your case this issue
>> comes and goes as well. I am running a 2 node cluster (ES 0.90.7) with
>> default configuration. I'm using logstash-1.2.2; however looking at the
>> debug logs within logstash the events are parsed correctly, they just die
>> when elasticsearch takes the floor.
>>
>> Sorry I can't be of any more help... If you have any theories, please
>> share your thoughts.
>>
>> Cheers,
>>
>> Ben
>>
>> On Wednesday, 11 December 2013 21:26:41 UTC+1, Phil Dougherty wrote:
>>>
>>> I'm having the same exact issue. Running on a 4 node cluster and using
>>> ES 0.90.3. Strangely this issue will randomly bite me, and it seems like it
>>> happens when the index is living on a certain server. Were you able to
>>> resolve this issue or could you provide some more information that may help
>>> me resolve this?
>>>
>>> On Monday, November 25, 2013 5:36:41 AM UTC-5, Ben Morrice wrote:
>>>>
>>>> Hello,
>>>>
>>>> I have a basic logstash -> elasticsearch instance running to parse
>>>> apache logs; or at least I did. I don't know what has changed but
>>>> elasticsearch is now bailing on my timestamp field and is not able to
>>>> correctly parse it.
>>>>
>>>> I'm using logstash standard grok patterns (COMBINEDAPACHELOG in this
>>>> case)
>>>>
>>>> Is anyone able to assist me? I'm sure this is a simple fix?!
>>>>
>>>> Please see below an example from my elasticsearch log
>>>>
>>>>
>>>> [2013-11-25 09:49:47,585][DEBUG][action.bulk ] [Slade,
>>>> Hamilton] [logstash-2013.11.25][3] failed to execute bulk item (index)
>>>> index {[logstash-2013.11.25][logs][CxGMkA9nSjej7jHdNfXh4g],
>>>> source[{"message":"10.80.5.30 - - [25/Nov/2013:09:49:45 +0100] \"GET
>>>> /examplerequest HTTP/1.1\" 200 11415 \"-\" \"Jakarta
>>>> Commons-HttpClient/3.1\"","@timestamp":"2013-11-25T08:49:
>>>> 47.548Z","@version":"1","type":"webfrontend","file":"/var/
>>>> loghttpdssl_access_log","host":"example.com","offset":"
>>>> 629386952","clientip":"10.80.5.30","ident":"-","auth":"-","
>>>> timestamp":"25/Nov/2013:09:49:45 +0100","verb":"GET","request":
>>>> "/examplerequest","httpversion":"1.1","response":
>>>> "200","bytes":"11415","referrer":"\"-\"","agent":"\"Jakarta
>>>> Commons-HttpClient/3.1\"","tags":["apache"]}]}
>>>> org.elasticsearch.index.mapper.MapperParsingException: failed to parse
>>>> [timestamp]
>>>> at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(
>>>> AbstractFieldMapper.java:398)
>>>> at org.elasticsearch.index.mapper.object.ObjectMapper.
>>>> serializeValue(ObjectMapper.java:618)
>>>> at org.elasticsearch.index.mapper.object.ObjectMapper.
>>>> parse(ObjectMapper.java:471)
>>>> at org.elasticsearch.index.mapper.DocumentMapper.parse(
>>>> DocumentMapper.java:513)
>>>> at org.elasticsearch.index.mapper.DocumentMapper.parse(
>>>> DocumentMapper.java:457)
>>>> at org.elasticsearch.index.shard.service.InternalIndexShard.
>>>> prepareCreate(InternalIndexShard.java:342)
>>>> at org.elasticsearch.action.bulk.TransportShardBulkAction.
>>>> shardIndexOperation(TransportShardBulkAction.java:401)
>>>> at org.elasticsearch.action.bulk.TransportShardBulkAction.
>>>> shardOperationOnPrimary(TransportShardBulkAction.java:155)
>>>> at org.elasticsearch.action.support.replication.
>>>> TransportShardReplicationOperationAction$AsyncShardOperationAction.
>>>> performOnPrimary(TransportShardReplicationOperationAction.java:556)
>>>> at org.elasticsearch.action.support.replication.
>>>> TransportShardReplicationOperationAction$AsyncShardOperationAction$1.
>>>> run(TransportShardReplicationOperationAction.java:426)
>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.
>>>> runTask(ThreadPoolExecutor.java:895)
>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>>>> ThreadPoolExecutor.java:918)
>>>> at java.lang.Thread.run(Thread.java:662)
>>>> Caused by: org.elasticsearch.index.mapper.MapperParsingException:
>>>> failed to parse date field [25/Nov/2013:09:49:45 +0100], tried both date
>>>> format [dateOptionalTime], and timestamp number with locale []
>>>> at org.elasticsearch.index.mapper.core.DateFieldMapper.
>>>> parseStringValue(DateFieldMapper.java:486)
>>>> at org.elasticsearch.index.mapper.core.DateFieldMapper.
>>>> innerParseCreateField(DateFieldMapper.java:424)
>>>> at org.elasticsearch.index.mapper.core.NumberFieldMapper.
>>>> parseCreateField(NumberFieldMapper.java:188)
>>>> at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(
>>>> AbstractFieldMapper.java:387)
>>>> ... 12 more
>>>> Caused by: java.lang.IllegalArgumentException: Invalid format:
>>>> "25/Nov/2013:09:49:45 +0100" is malformed at "/Nov/2013:09:49:45 +0100"
>>>> at org.elasticsearch.common.joda.time.format.DateTimeFormatter.
>>>> parseMillis(DateTimeFormatter.java:754)
>>>> at org.elasticsearch.index.mapper.core.DateFieldMapper.
>>>> parseStringValue(DateFieldMapper.java:480)
>>>> ... 15 more
>>>>
>>>> Thanks in advance!
>>>>
>>>> Cheers,
>>>>
>>>> Ben
>>>>
>>>> --
>> You received this message because you are subscribed to the Google Groups
>> "elasticsearch" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected].
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/elasticsearch/087f7e84-261c-4bde-b214-9be3580edae1%40googlegroups.com
>> .
>>
>> For more options, visit https://groups.google.com/groups/opt_out.
>>
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/elasticsearch/wUrtAVUE1o4/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> [email protected].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/CAGCwEM_eOC8rpT82U_4AQ3GNX93BWvRPZkP9oe1j1q0unRu_BA%40mail.gmail.com
> .
>
> For more options, visit https://groups.google.com/groups/opt_out.
>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAJAhm_oORRy6EJ8YA2r9FBLnjc9%3DEACbu23MNEgvH8BDug1pUw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.
