Agreed. Happy things work now.
On Tue, Dec 2, 2014 at 11:07 AM, Teo Ruiz <[email protected]> wrote:
> Hi Boaz,
> I finally restarted the master node and it worked, indeed. It's anyway a
> quite confusing error message / situation :)
> Cheers,
> On Sunday, November 30, 2014 11:45:15 PM UTC+1, Boaz Leskes wrote:
>>
>> Hi Teo, Max,
>>
>> From the query, I can see that it that cluster_state is not shipped. The
>> marvel index template adds the cluster_type with some non-default fields
>> but all fields (like @timestamp) are added when the first document is
>> shipped.
>>
>> Can you validate there are no documents of that type?
>>
>> Further a couple of clarifying questions:
>> 1) Did you install the marvel agent on all cluster nodes? i.e., also
>> master nodes? The later is the ones that ship that data..
>> 2) Did you upgrade from a previous version of marvel or is it a fresh
>> install?
>>
>> Cheers,
>> Boaz
>>
>>
>> On Wednesday, November 26, 2014 5:24:58 PM UTC+1, Teo Ruiz wrote:
>>>
>>> Hi guys,
>>>
>>> We're seeing exactly the same error. This is the error we see:
>>> org.elasticsearch.search.SearchParseException: [.marvel-2014.11.26][0]:
>>> from[-1],size[1]: Parse Failure [Failed to parse source [{"size":1,"sort"
>>> :{"@timestamp":{"order":"desc"}}}]]
>>>
>>> Full trace here:
>>> https://gist.github.com/teoruiz/9cdb4e9cd7dc5e564346#file-es_error_log
>>> Mappings here:
>>> https://gist.github.com/teoruiz/9cdb4e9cd7dc5e564346#file-marvel_index_mapping-json
>>>
>>> ES version on monitoring and production clusters: 1.3.4
>>> Marvel version on monitoring and production clusters: 1.2.1
>>>
>>> I'm not sure how to proceed, since we removed and reinstalled marvel
>>> already. The problem started showing when we upgraded the production
>>> cluster from 0.90.x to 1.3.4.
>>>
>>> I'd appreciate some light shedding on this :)
>>>
>>> On Tuesday, November 25, 2014 10:29:45 AM UTC+1, Max Charas wrote:
>>>>
>>>> Same problem here.
>>>>
>>>> Log says:
>>>>
>>>> Parse Failure [Failed to parse source
>>>> [{"size":1,"sort":{"@timestamp":{"order":"desc"}}}]]
>>>> Verified that the index(es) have the timestamp field.
>>>>
>>>> Running Marvel 1.2.1 and Elasticsearch 1.3.2.
>>>>
>>>> Any help would be appreciated as we cant get our Marvel up and running.
>>>>
>>>> Den fredagen den 7:e november 2014 kl. 17:54:09 UTC+1 skrev Ryan
>>>> Hendrickson:
>>>>>
>>>>> I see the same error Were you able to resolve it?
>>>>>
>>>>> Thanks
>>>>>
>>>>> On Wednesday, September 17, 2014 12:52:14 PM UTC-4, Sean Kipling wrote:
>>>>>>
>>>>>> I am on Marvel 1.2.1 and had the same error (see below). The time on
>>>>>> one of my nodes was off a fair bit. Fixed that and the problem went away.
>>>>>>
>>>>>> [2014-09-17 11:55:14,615][DEBUG][action.search.type ] [Pip the
>>>>>> Troll] All shards failed for phase: [query_fetch]
>>>>>> [2014-09-17 12:16:00,532][DEBUG][action.search.type ] [Pip the
>>>>>> Troll] [.marvel-2014.09.17][0], node[2d8iMl4tRHSS8JFPJMN01A], [P],
>>>>>> s[STARTED]: Failed to execute
>>>>>> [org.elasticsearch.action.search.SearchRequest@769694e9]
>>>>>> org.elasticsearch.search.SearchParseException:
>>>>>> [.marvel-2014.09.17][0]: from[-1],size[1]: Parse Failure [Failed to
>>>>>> parse
>>>>>> source [{"size":1,"sort":{"@timestamp":{"order":"desc"}}}]]
>>>>>> at
>>>>>> org.elasticsearch.search.SearchService.parseSource(SearchService.java:664)
>>>>>> at
>>>>>> org.elasticsearch.search.SearchService.createContext(SearchService.java:515)
>>>>>> at
>>>>>> org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:487)
>>>>>> at
>>>>>> org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:328)
>>>>>> at
>>>>>> org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:308)
>>>>>> at
>>>>>> org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:305)
>>>>>> at
>>>>>> org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>> Caused by: org.elasticsearch.search.SearchParseException:
>>>>>> [.marvel-2014.09.17][0]: from[-1],size[1]: Parse Failure [No mapping
>>>>>> found
>>>>>> for [@timestamp] in order to sort on]
>>>>>> at
>>>>>> org.elasticsearch.search.sort.SortParseElement.addSortField(SortParseElement.java:198)
>>>>>> at
>>>>>> org.elasticsearch.search.sort.SortParseElement.addCompoundSortField(SortParseElement.java:172)
>>>>>> at
>>>>>> org.elasticsearch.search.sort.SortParseElement.parse(SortParseElement.java:90)
>>>>>> at
>>>>>> org.elasticsearch.search.SearchService.parseSource(SearchService.java:648)
>>>>>> ... 9 more
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Monday, June 9, 2014 12:39:45 PM UTC-4, [email protected] wrote:
>>>>>>>
>>>>>>> Looks like this has already been found and resolved. I'll upgrade to
>>>>>>> Marvel 1.2.1.
>>>>>>>
>>>>>>> http://www.elasticsearch.org/blog/marvel-1-2-1-released/
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Monday, June 9, 2014 9:35:30 AM UTC-7, [email protected] wrote:
>>>>>>>>
>>>>>>>> Hi Boaz,
>>>>>>>>
>>>>>>>> The only other exception I see in the logs is this:
>>>>>>>>
>>>>>>>> [2014-06-09 15:38:11,340][ERROR][marvel.agent ] [xxxx]
>>>>>>>> exporter [es_exporter] has thrown an exception:
>>>>>>>> java.lang.IllegalStateException: array not available
>>>>>>>> at
>>>>>>>> org.elasticsearch.common.bytes.PagedBytesReference.array(PagedBytesReference.java:289)
>>>>>>>> at
>>>>>>>> org.elasticsearch.marvel.agent.exporter.ESExporter.addXContentRendererToConnection(ESExporter.java:209)
>>>>>>>> at
>>>>>>>> org.elasticsearch.marvel.agent.exporter.ESExporter.exportXContent(ESExporter.java:252)
>>>>>>>> at
>>>>>>>> org.elasticsearch.marvel.agent.exporter.ESExporter.exportEvents(ESExporter.java:161)
>>>>>>>> at
>>>>>>>> org.elasticsearch.marvel.agent.AgentService$ExportingWorker.exportEvents(AgentService.java:305)
>>>>>>>> at
>>>>>>>> org.elasticsearch.marvel.agent.AgentService$ExportingWorker.run(AgentService.java:240)
>>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Saturday, June 7, 2014 11:50:16 AM UTC-7, Boaz Leskes wrote:
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> Is there anything in the log indicating data shipping problems from
>>>>>>>>> the agent? Check the log of the current master node.
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Boaz
>>>>>>>>>
>>>>>>>>> On Saturday, June 7, 2014 12:51:44 AM UTC+2, [email protected]
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> I'm running elasticsearch 1.2.1, and just upgraded to Marvel 1.2.
>>>>>>>>>>
>>>>>>>>>> When I go to the new "Shard Allocation" dashboard in Marvel, I see
>>>>>>>>>> the following error:
>>>>>>>>>>
>>>>>>>>>> SearchPhaseExecutionException[Failed to execute phase
>>>>>>>>>> [query_fetch], all shards failed; shardFailures
>>>>>>>>>> {[5FNjdIADRE2rRuOkDZdieA][.marvel-2014.06.04][0]:
>>>>>>>>>> RemoteTransportException[[xxxx][inet[/x.x.x.x:9300]][search/phase/query+fetch]];
>>>>>>>>>>
>>>>>>>>>> nested: SearchParseException[[.marvel-2014.06.04][0]:
>>>>>>>>>> from[0],size[300]:
>>>>>>>>>> Parse Failure [Failed to parse source
>>>>>>>>>> [{"size":300,"from":0,"fields":["@timestamp","message","status"],"sort":{"@timestamp":{"order":"desc"}},"query":{"filtered":{"filter":{"range":{"@timestamp":{"from":"2014-05-30T22:33:17.689Z","to":"2014-06-06T22:33:17.689Z"}}}}}}]]];
>>>>>>>>>>
>>>>>>>>>> nested: SearchParseException[[.marvel-2014.06.04][0]:
>>>>>>>>>> from[0],size[300]:
>>>>>>>>>> Parse Failure [No mapping found for [@timestamp] in order to sort
>>>>>>>>>> on]]; }]
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I see the following in my elasticsearch.log
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> [2014-06-06 22:48:07,919][DEBUG][action.search.type ] [xxxx]
>>>>>>>>>> All shards failed for phase: [query_fetch]
>>>>>>>>>> [2014-06-06 22:48:17,937][DEBUG][action.search.type ] [xxxx]
>>>>>>>>>> [.marvel-2014.06.06][0], node[5FNjdIADRE2rRuOkDZdieA], [R],
>>>>>>>>>> s[STARTED]:
>>>>>>>>>> Failed to execute
>>>>>>>>>> [org.elasticsearch.action.search.SearchRequest@6af079e8]
>>>>>>>>>> org.elasticsearch.search.SearchParseException:
>>>>>>>>>> [.marvel-2014.06.06][0]: from[-1],size[1]: Parse Failure [Failed to
>>>>>>>>>> parse
>>>>>>>>>> source [{"size":1,"sort":{"@timestamp":{"order":"desc"}}}]]
>>>>>>>>>> at
>>>>>>>>>> org.elasticsearch.search.SearchService.parseSource(SearchService.java:649)
>>>>>>>>>> at
>>>>>>>>>> org.elasticsearch.search.SearchService.createContext(SearchService.java:511)
>>>>>>>>>> at
>>>>>>>>>> org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:483)
>>>>>>>>>> at
>>>>>>>>>> org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:324)
>>>>>>>>>> at
>>>>>>>>>> org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:308)
>>>>>>>>>> at
>>>>>>>>>> org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:305)
>>>>>>>>>> at
>>>>>>>>>> org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>>>>> at
>>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>>>> Caused by: org.elasticsearch.search.SearchParseException:
>>>>>>>>>> [.marvel-2014.06.06][0]: from[-1],size[1]: Parse Failure [No mapping
>>>>>>>>>> found
>>>>>>>>>> for [@timestamp] in order to sort on]
>>>>>>>>>> at
>>>>>>>>>> org.elasticsearch.search.sort.SortParseElement.addSortField(SortParseElement.java:198)
>>>>>>>>>> at
>>>>>>>>>> org.elasticsearch.search.sort.SortParseElement.addCompoundSortField(SortParseElement.java:172)
>>>>>>>>>> at
>>>>>>>>>> org.elasticsearch.search.sort.SortParseElement.parse(SortParseElement.java:90)
>>>>>>>>>> at
>>>>>>>>>> org.elasticsearch.search.SearchService.parseSource(SearchService.java:633)
>>>>>>>>>> ... 9 more
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I have confirmed that marvel is upgraded on every node in the
>>>>>>>>>> cluster.
>>>>>>>>>>
>>>>>>>>>> I tried deleting the .marvel* indexes. Didn't help
>>>>>>>>>> Shutdown the cluster, and restarted it. Didn't help.
>>>>>>>>>>
>>>>>>>>>> Anybody have ideas?
>>>>>>>>>>
>>>>>>>>>
>>>
>>>
>>
> --
> You received this message because you are subscribed to a topic in the Google
> Groups "elasticsearch" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/elasticsearch/YET1SifrKy4/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> [email protected].
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/elasticsearch/14c54640-a555-4b6e-be4a-5a52d1fb2f7f%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1417517208059.f25f0b45%40Nodemailer.
For more options, visit https://groups.google.com/d/optout.
