Hi Boaz,
I finally restarted the master node and it worked, indeed. It's anyway a
quite confusing error message / situation :)
Cheers,
On Sunday, November 30, 2014 11:45:15 PM UTC+1, Boaz Leskes wrote:
>
> Hi Teo, Max,
>
> From the query, I can see that it that cluster_state is not shipped. The
> marvel index template adds the cluster_type with some non-default fields
> but all fields (like @timestamp) are added when the first document is
> shipped.
>
> Can you validate there are no documents of that type?
>
> Further a couple of clarifying questions:
> 1) Did you install the marvel agent on all cluster nodes? i.e., also
> master nodes? The later is the ones that ship that data..
> 2) Did you upgrade from a previous version of marvel or is it a fresh
> install?
>
> Cheers,
> Boaz
>
>
> On Wednesday, November 26, 2014 5:24:58 PM UTC+1, Teo Ruiz wrote:
>>
>> Hi guys,
>>
>> We're seeing exactly the same error. This is the error we see:
>> org.elasticsearch.search.SearchParseException: [.marvel-2014.11.26][0]:
>> from[-1],size[1]: Parse Failure [Failed to parse source [{"size":1,"sort"
>> :{"@timestamp":{"order":"desc"}}}]]
>>
>> Full trace here:
>> https://gist.github.com/teoruiz/9cdb4e9cd7dc5e564346#file-es_error_log
>> Mappings here:
>> https://gist.github.com/teoruiz/9cdb4e9cd7dc5e564346#file-marvel_index_mapping-json
>>
>> ES version on monitoring and production clusters: 1.3.4
>> Marvel version on monitoring and production clusters: 1.2.1
>>
>> I'm not sure how to proceed, since we removed and reinstalled marvel
>> already. The problem started showing when we upgraded the production
>> cluster from 0.90.x to 1.3.4.
>>
>> I'd appreciate some light shedding on this :)
>>
>> On Tuesday, November 25, 2014 10:29:45 AM UTC+1, Max Charas wrote:
>>>
>>> Same problem here.
>>>
>>> Log says:
>>>
>>> Parse Failure [Failed to parse source
>>> [{"size":1,"sort":{"@timestamp":{"order":"desc"}}}]]
>>> Verified that the index(es) have the timestamp field.
>>>
>>> Running Marvel 1.2.1 and Elasticsearch 1.3.2.
>>>
>>> Any help would be appreciated as we cant get our Marvel up and running.
>>>
>>> Den fredagen den 7:e november 2014 kl. 17:54:09 UTC+1 skrev Ryan
>>> Hendrickson:
>>>>
>>>> I see the same error Were you able to resolve it?
>>>>
>>>> Thanks
>>>>
>>>> On Wednesday, September 17, 2014 12:52:14 PM UTC-4, Sean Kipling wrote:
>>>>>
>>>>> I am on Marvel 1.2.1 and had the same error (see below). The time on
>>>>> one of my nodes was off a fair bit. Fixed that and the problem went away.
>>>>>
>>>>> [2014-09-17 11:55:14,615][DEBUG][action.search.type ] [Pip the
>>>>> Troll] All shards failed for phase: [query_fetch]
>>>>> [2014-09-17 12:16:00,532][DEBUG][action.search.type ] [Pip the
>>>>> Troll] [.marvel-2014.09.17][0], node[2d8iMl4tRHSS8JFPJMN01A], [P],
>>>>> s[STARTED]: Failed to execute
>>>>> [org.elasticsearch.action.search.SearchRequest@769694e9]
>>>>> org.elasticsearch.search.SearchParseException:
>>>>> [.marvel-2014.09.17][0]: from[-1],size[1]: Parse Failure [Failed to parse
>>>>> source [{"size":1,"sort":{"@timestamp":{"order":"desc"}}}]]
>>>>> at
>>>>> org.elasticsearch.search.SearchService.parseSource(SearchService.java:664)
>>>>> at
>>>>> org.elasticsearch.search.SearchService.createContext(SearchService.java:515)
>>>>> at
>>>>> org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:487)
>>>>> at
>>>>> org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:328)
>>>>> at
>>>>> org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:308)
>>>>> at
>>>>> org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:305)
>>>>> at
>>>>> org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>> Caused by: org.elasticsearch.search.SearchParseException:
>>>>> [.marvel-2014.09.17][0]: from[-1],size[1]: Parse Failure [No mapping
>>>>> found
>>>>> for [@timestamp] in order to sort on]
>>>>> at
>>>>> org.elasticsearch.search.sort.SortParseElement.addSortField(SortParseElement.java:198)
>>>>> at
>>>>> org.elasticsearch.search.sort.SortParseElement.addCompoundSortField(SortParseElement.java:172)
>>>>> at
>>>>> org.elasticsearch.search.sort.SortParseElement.parse(SortParseElement.java:90)
>>>>> at
>>>>> org.elasticsearch.search.SearchService.parseSource(SearchService.java:648)
>>>>> ... 9 more
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Monday, June 9, 2014 12:39:45 PM UTC-4, [email protected] wrote:
>>>>>>
>>>>>> Looks like this has already been found and resolved. I'll upgrade to
>>>>>> Marvel 1.2.1.
>>>>>>
>>>>>> http://www.elasticsearch.org/blog/marvel-1-2-1-released/
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Monday, June 9, 2014 9:35:30 AM UTC-7, [email protected] wrote:
>>>>>>>
>>>>>>> Hi Boaz,
>>>>>>>
>>>>>>> The only other exception I see in the logs is this:
>>>>>>>
>>>>>>> [2014-06-09 15:38:11,340][ERROR][marvel.agent ] [xxxx]
>>>>>>> exporter [es_exporter] has thrown an exception:
>>>>>>> java.lang.IllegalStateException: array not available
>>>>>>> at
>>>>>>> org.elasticsearch.common.bytes.PagedBytesReference.array(PagedBytesReference.java:289)
>>>>>>> at
>>>>>>> org.elasticsearch.marvel.agent.exporter.ESExporter.addXContentRendererToConnection(ESExporter.java:209)
>>>>>>> at
>>>>>>> org.elasticsearch.marvel.agent.exporter.ESExporter.exportXContent(ESExporter.java:252)
>>>>>>> at
>>>>>>> org.elasticsearch.marvel.agent.exporter.ESExporter.exportEvents(ESExporter.java:161)
>>>>>>> at
>>>>>>> org.elasticsearch.marvel.agent.AgentService$ExportingWorker.exportEvents(AgentService.java:305)
>>>>>>> at
>>>>>>> org.elasticsearch.marvel.agent.AgentService$ExportingWorker.run(AgentService.java:240)
>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Saturday, June 7, 2014 11:50:16 AM UTC-7, Boaz Leskes wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Is there anything in the log indicating data shipping problems from
>>>>>>>> the agent? Check the log of the current master node.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Boaz
>>>>>>>>
>>>>>>>> On Saturday, June 7, 2014 12:51:44 AM UTC+2, [email protected]
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> I'm running elasticsearch 1.2.1, and just upgraded to Marvel 1.2.
>>>>>>>>>
>>>>>>>>> When I go to the new "Shard Allocation" dashboard in Marvel, I see
>>>>>>>>> the following error:
>>>>>>>>>
>>>>>>>>> SearchPhaseExecutionException[Failed to execute phase
>>>>>>>>> [query_fetch], all shards failed; shardFailures
>>>>>>>>> {[5FNjdIADRE2rRuOkDZdieA][.marvel-2014.06.04][0]:
>>>>>>>>> RemoteTransportException[[xxxx][inet[/x.x.x.x:9300]][search/phase/query+fetch]];
>>>>>>>>>
>>>>>>>>> nested: SearchParseException[[.marvel-2014.06.04][0]:
>>>>>>>>> from[0],size[300]:
>>>>>>>>> Parse Failure [Failed to parse source
>>>>>>>>> [{"size":300,"from":0,"fields":["@timestamp","message","status"],"sort":{"@timestamp":{"order":"desc"}},"query":{"filtered":{"filter":{"range":{"@timestamp":{"from":"2014-05-30T22:33:17.689Z","to":"2014-06-06T22:33:17.689Z"}}}}}}]]];
>>>>>>>>>
>>>>>>>>> nested: SearchParseException[[.marvel-2014.06.04][0]:
>>>>>>>>> from[0],size[300]:
>>>>>>>>> Parse Failure [No mapping found for [@timestamp] in order to sort
>>>>>>>>> on]]; }]
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I see the following in my elasticsearch.log
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [2014-06-06 22:48:07,919][DEBUG][action.search.type ] [xxxx]
>>>>>>>>> All shards failed for phase: [query_fetch]
>>>>>>>>> [2014-06-06 22:48:17,937][DEBUG][action.search.type ] [xxxx]
>>>>>>>>> [.marvel-2014.06.06][0], node[5FNjdIADRE2rRuOkDZdieA], [R],
>>>>>>>>> s[STARTED]:
>>>>>>>>> Failed to execute
>>>>>>>>> [org.elasticsearch.action.search.SearchRequest@6af079e8]
>>>>>>>>> org.elasticsearch.search.SearchParseException:
>>>>>>>>> [.marvel-2014.06.06][0]: from[-1],size[1]: Parse Failure [Failed to
>>>>>>>>> parse
>>>>>>>>> source [{"size":1,"sort":{"@timestamp":{"order":"desc"}}}]]
>>>>>>>>> at
>>>>>>>>> org.elasticsearch.search.SearchService.parseSource(SearchService.java:649)
>>>>>>>>> at
>>>>>>>>> org.elasticsearch.search.SearchService.createContext(SearchService.java:511)
>>>>>>>>> at
>>>>>>>>> org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:483)
>>>>>>>>> at
>>>>>>>>> org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:324)
>>>>>>>>> at
>>>>>>>>> org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:308)
>>>>>>>>> at
>>>>>>>>> org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:305)
>>>>>>>>> at
>>>>>>>>> org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>>> Caused by: org.elasticsearch.search.SearchParseException:
>>>>>>>>> [.marvel-2014.06.06][0]: from[-1],size[1]: Parse Failure [No mapping
>>>>>>>>> found
>>>>>>>>> for [@timestamp] in order to sort on]
>>>>>>>>> at
>>>>>>>>> org.elasticsearch.search.sort.SortParseElement.addSortField(SortParseElement.java:198)
>>>>>>>>> at
>>>>>>>>> org.elasticsearch.search.sort.SortParseElement.addCompoundSortField(SortParseElement.java:172)
>>>>>>>>> at
>>>>>>>>> org.elasticsearch.search.sort.SortParseElement.parse(SortParseElement.java:90)
>>>>>>>>> at
>>>>>>>>> org.elasticsearch.search.SearchService.parseSource(SearchService.java:633)
>>>>>>>>> ... 9 more
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I have confirmed that marvel is upgraded on every node in the
>>>>>>>>> cluster.
>>>>>>>>>
>>>>>>>>> I tried deleting the .marvel* indexes. Didn't help
>>>>>>>>> Shutdown the cluster, and restarted it. Didn't help.
>>>>>>>>>
>>>>>>>>> Anybody have ideas?
>>>>>>>>>
>>>>>>>>
>>
>>
>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/14c54640-a555-4b6e-be4a-5a52d1fb2f7f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
