Hi Teo, Max,
>From the query, I can see that it that cluster_state is not shipped. The
marvel index template adds the cluster_type with some non-default fields
but all fields (like @timestamp) are added when the first document is
shipped.
Can you validate there are no documents of that type?
Further a couple of clarifying questions:
1) Did you install the marvel agent on all cluster nodes? i.e., also master
nodes? The later is the ones that ship that data..
2) Did you upgrade from a previous version of marvel or is it a fresh
install?
Cheers,
Boaz
On Wednesday, November 26, 2014 5:24:58 PM UTC+1, Teo Ruiz wrote:
>
> Hi guys,
>
> We're seeing exactly the same error. This is the error we see:
> org.elasticsearch.search.SearchParseException: [.marvel-2014.11.26][0]:
> from[-1],size[1]: Parse Failure [Failed to parse source [{"size":1,"sort":
> {"@timestamp":{"order":"desc"}}}]]
>
> Full trace here:
> https://gist.github.com/teoruiz/9cdb4e9cd7dc5e564346#file-es_error_log
> Mappings here:
> https://gist.github.com/teoruiz/9cdb4e9cd7dc5e564346#file-marvel_index_mapping-json
>
> ES version on monitoring and production clusters: 1.3.4
> Marvel version on monitoring and production clusters: 1.2.1
>
> I'm not sure how to proceed, since we removed and reinstalled marvel
> already. The problem started showing when we upgraded the production
> cluster from 0.90.x to 1.3.4.
>
> I'd appreciate some light shedding on this :)
>
> On Tuesday, November 25, 2014 10:29:45 AM UTC+1, Max Charas wrote:
>>
>> Same problem here.
>>
>> Log says:
>>
>> Parse Failure [Failed to parse source
>> [{"size":1,"sort":{"@timestamp":{"order":"desc"}}}]]
>> Verified that the index(es) have the timestamp field.
>>
>> Running Marvel 1.2.1 and Elasticsearch 1.3.2.
>>
>> Any help would be appreciated as we cant get our Marvel up and running.
>>
>> Den fredagen den 7:e november 2014 kl. 17:54:09 UTC+1 skrev Ryan
>> Hendrickson:
>>>
>>> I see the same error Were you able to resolve it?
>>>
>>> Thanks
>>>
>>> On Wednesday, September 17, 2014 12:52:14 PM UTC-4, Sean Kipling wrote:
>>>>
>>>> I am on Marvel 1.2.1 and had the same error (see below). The time on
>>>> one of my nodes was off a fair bit. Fixed that and the problem went away.
>>>>
>>>> [2014-09-17 11:55:14,615][DEBUG][action.search.type ] [Pip the
>>>> Troll] All shards failed for phase: [query_fetch]
>>>> [2014-09-17 12:16:00,532][DEBUG][action.search.type ] [Pip the
>>>> Troll] [.marvel-2014.09.17][0], node[2d8iMl4tRHSS8JFPJMN01A], [P],
>>>> s[STARTED]: Failed to execute
>>>> [org.elasticsearch.action.search.SearchRequest@769694e9]
>>>> org.elasticsearch.search.SearchParseException: [.marvel-2014.09.17][0]:
>>>> from[-1],size[1]: Parse Failure [Failed to parse source
>>>> [{"size":1,"sort":{"@timestamp":{"order":"desc"}}}]]
>>>> at
>>>> org.elasticsearch.search.SearchService.parseSource(SearchService.java:664)
>>>> at
>>>> org.elasticsearch.search.SearchService.createContext(SearchService.java:515)
>>>> at
>>>> org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:487)
>>>> at
>>>> org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:328)
>>>> at
>>>> org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:308)
>>>> at
>>>> org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:305)
>>>> at
>>>> org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>> at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>> at java.lang.Thread.run(Thread.java:745)
>>>> Caused by: org.elasticsearch.search.SearchParseException:
>>>> [.marvel-2014.09.17][0]: from[-1],size[1]: Parse Failure [No mapping found
>>>> for [@timestamp] in order to sort on]
>>>> at
>>>> org.elasticsearch.search.sort.SortParseElement.addSortField(SortParseElement.java:198)
>>>> at
>>>> org.elasticsearch.search.sort.SortParseElement.addCompoundSortField(SortParseElement.java:172)
>>>> at
>>>> org.elasticsearch.search.sort.SortParseElement.parse(SortParseElement.java:90)
>>>> at
>>>> org.elasticsearch.search.SearchService.parseSource(SearchService.java:648)
>>>> ... 9 more
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Monday, June 9, 2014 12:39:45 PM UTC-4, [email protected] wrote:
>>>>>
>>>>> Looks like this has already been found and resolved. I'll upgrade to
>>>>> Marvel 1.2.1.
>>>>>
>>>>> http://www.elasticsearch.org/blog/marvel-1-2-1-released/
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Monday, June 9, 2014 9:35:30 AM UTC-7, [email protected] wrote:
>>>>>>
>>>>>> Hi Boaz,
>>>>>>
>>>>>> The only other exception I see in the logs is this:
>>>>>>
>>>>>> [2014-06-09 15:38:11,340][ERROR][marvel.agent ] [xxxx]
>>>>>> exporter [es_exporter] has thrown an exception:
>>>>>> java.lang.IllegalStateException: array not available
>>>>>> at
>>>>>> org.elasticsearch.common.bytes.PagedBytesReference.array(PagedBytesReference.java:289)
>>>>>> at
>>>>>> org.elasticsearch.marvel.agent.exporter.ESExporter.addXContentRendererToConnection(ESExporter.java:209)
>>>>>> at
>>>>>> org.elasticsearch.marvel.agent.exporter.ESExporter.exportXContent(ESExporter.java:252)
>>>>>> at
>>>>>> org.elasticsearch.marvel.agent.exporter.ESExporter.exportEvents(ESExporter.java:161)
>>>>>> at
>>>>>> org.elasticsearch.marvel.agent.AgentService$ExportingWorker.exportEvents(AgentService.java:305)
>>>>>> at
>>>>>> org.elasticsearch.marvel.agent.AgentService$ExportingWorker.run(AgentService.java:240)
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Saturday, June 7, 2014 11:50:16 AM UTC-7, Boaz Leskes wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> Is there anything in the log indicating data shipping problems from
>>>>>>> the agent? Check the log of the current master node.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Boaz
>>>>>>>
>>>>>>> On Saturday, June 7, 2014 12:51:44 AM UTC+2, [email protected]
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> I'm running elasticsearch 1.2.1, and just upgraded to Marvel 1.2.
>>>>>>>>
>>>>>>>> When I go to the new "Shard Allocation" dashboard in Marvel, I see
>>>>>>>> the following error:
>>>>>>>>
>>>>>>>> SearchPhaseExecutionException[Failed to execute phase
>>>>>>>> [query_fetch], all shards failed; shardFailures
>>>>>>>> {[5FNjdIADRE2rRuOkDZdieA][.marvel-2014.06.04][0]:
>>>>>>>> RemoteTransportException[[xxxx][inet[/x.x.x.x:9300]][search/phase/query+fetch]];
>>>>>>>>
>>>>>>>> nested: SearchParseException[[.marvel-2014.06.04][0]:
>>>>>>>> from[0],size[300]:
>>>>>>>> Parse Failure [Failed to parse source
>>>>>>>> [{"size":300,"from":0,"fields":["@timestamp","message","status"],"sort":{"@timestamp":{"order":"desc"}},"query":{"filtered":{"filter":{"range":{"@timestamp":{"from":"2014-05-30T22:33:17.689Z","to":"2014-06-06T22:33:17.689Z"}}}}}}]]];
>>>>>>>>
>>>>>>>> nested: SearchParseException[[.marvel-2014.06.04][0]:
>>>>>>>> from[0],size[300]:
>>>>>>>> Parse Failure [No mapping found for [@timestamp] in order to sort
>>>>>>>> on]]; }]
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> I see the following in my elasticsearch.log
>>>>>>>>
>>>>>>>>
>>>>>>>> [2014-06-06 22:48:07,919][DEBUG][action.search.type ] [xxxx]
>>>>>>>> All shards failed for phase: [query_fetch]
>>>>>>>> [2014-06-06 22:48:17,937][DEBUG][action.search.type ] [xxxx]
>>>>>>>> [.marvel-2014.06.06][0], node[5FNjdIADRE2rRuOkDZdieA], [R],
>>>>>>>> s[STARTED]:
>>>>>>>> Failed to execute
>>>>>>>> [org.elasticsearch.action.search.SearchRequest@6af079e8]
>>>>>>>> org.elasticsearch.search.SearchParseException:
>>>>>>>> [.marvel-2014.06.06][0]: from[-1],size[1]: Parse Failure [Failed to
>>>>>>>> parse
>>>>>>>> source [{"size":1,"sort":{"@timestamp":{"order":"desc"}}}]]
>>>>>>>> at
>>>>>>>> org.elasticsearch.search.SearchService.parseSource(SearchService.java:649)
>>>>>>>> at
>>>>>>>> org.elasticsearch.search.SearchService.createContext(SearchService.java:511)
>>>>>>>> at
>>>>>>>> org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:483)
>>>>>>>> at
>>>>>>>> org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:324)
>>>>>>>> at
>>>>>>>> org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:308)
>>>>>>>> at
>>>>>>>> org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:305)
>>>>>>>> at
>>>>>>>> org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>>> at
>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>> Caused by: org.elasticsearch.search.SearchParseException:
>>>>>>>> [.marvel-2014.06.06][0]: from[-1],size[1]: Parse Failure [No mapping
>>>>>>>> found
>>>>>>>> for [@timestamp] in order to sort on]
>>>>>>>> at
>>>>>>>> org.elasticsearch.search.sort.SortParseElement.addSortField(SortParseElement.java:198)
>>>>>>>> at
>>>>>>>> org.elasticsearch.search.sort.SortParseElement.addCompoundSortField(SortParseElement.java:172)
>>>>>>>> at
>>>>>>>> org.elasticsearch.search.sort.SortParseElement.parse(SortParseElement.java:90)
>>>>>>>> at
>>>>>>>> org.elasticsearch.search.SearchService.parseSource(SearchService.java:633)
>>>>>>>> ... 9 more
>>>>>>>>
>>>>>>>>
>>>>>>>> I have confirmed that marvel is upgraded on every node in the
>>>>>>>> cluster.
>>>>>>>>
>>>>>>>> I tried deleting the .marvel* indexes. Didn't help
>>>>>>>> Shutdown the cluster, and restarted it. Didn't help.
>>>>>>>>
>>>>>>>> Anybody have ideas?
>>>>>>>>
>>>>>>>
>
>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/d0519c6d-0cba-4f55-9521-0ce6c945f284%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
