Hello Boaz,
Thanks for helping!
I read through your post a few times and realized that it was probably the
master node that wasn't sending any date as you indicated. A restart of the
node fixed the problem.
I have no idea why it wasn't shipping any data to start with, most likely
we hadn't restarted it since we installed the plugin but I'm not completely
sure.
In any case its working now.
Many thanks,
Max
Den söndagen den 30:e november 2014 kl. 23:45:15 UTC+1 skrev Boaz Leskes:
>
> Hi Teo, Max,
>
> From the query, I can see that it that cluster_state is not shipped. The
> marvel index template adds the cluster_type with some non-default fields
> but all fields (like @timestamp) are added when the first document is
> shipped.
>
> Can you validate there are no documents of that type?
>
> Further a couple of clarifying questions:
> 1) Did you install the marvel agent on all cluster nodes? i.e., also
> master nodes? The later is the ones that ship that data..
> 2) Did you upgrade from a previous version of marvel or is it a fresh
> install?
>
> Cheers,
> Boaz
>
>
> On Wednesday, November 26, 2014 5:24:58 PM UTC+1, Teo Ruiz wrote:
>>
>> Hi guys,
>>
>> We're seeing exactly the same error. This is the error we see:
>> org.elasticsearch.search.SearchParseException: [.marvel-2014.11.26][0]:
>> from[-1],size[1]: Parse Failure [Failed to parse source [{"size":1,"sort"
>> :{"@timestamp":{"order":"desc"}}}]]
>>
>> Full trace here:
>> https://gist.github.com/teoruiz/9cdb4e9cd7dc5e564346#file-es_error_log
>> Mappings here:
>> https://gist.github.com/teoruiz/9cdb4e9cd7dc5e564346#file-marvel_index_mapping-json
>>
>> ES version on monitoring and production clusters: 1.3.4
>> Marvel version on monitoring and production clusters: 1.2.1
>>
>> I'm not sure how to proceed, since we removed and reinstalled marvel
>> already. The problem started showing when we upgraded the production
>> cluster from 0.90.x to 1.3.4.
>>
>> I'd appreciate some light shedding on this :)
>>
>> On Tuesday, November 25, 2014 10:29:45 AM UTC+1, Max Charas wrote:
>>>
>>> Same problem here.
>>>
>>> Log says:
>>>
>>> Parse Failure [Failed to parse source
>>> [{"size":1,"sort":{"@timestamp":{"order":"desc"}}}]]
>>> Verified that the index(es) have the timestamp field.
>>>
>>> Running Marvel 1.2.1 and Elasticsearch 1.3.2.
>>>
>>> Any help would be appreciated as we cant get our Marvel up and running.
>>>
>>> Den fredagen den 7:e november 2014 kl. 17:54:09 UTC+1 skrev Ryan
>>> Hendrickson:
>>>>
>>>> I see the same error Were you able to resolve it?
>>>>
>>>> Thanks
>>>>
>>>> On Wednesday, September 17, 2014 12:52:14 PM UTC-4, Sean Kipling wrote:
>>>>>
>>>>> I am on Marvel 1.2.1 and had the same error (see below). The time on
>>>>> one of my nodes was off a fair bit. Fixed that and the problem went away.
>>>>>
>>>>> [2014-09-17 11:55:14,615][DEBUG][action.search.type ] [Pip the
>>>>> Troll] All shards failed for phase: [query_fetch]
>>>>> [2014-09-17 12:16:00,532][DEBUG][action.search.type ] [Pip the
>>>>> Troll] [.marvel-2014.09.17][0], node[2d8iMl4tRHSS8JFPJMN01A], [P],
>>>>> s[STARTED]: Failed to execute
>>>>> [org.elasticsearch.action.search.SearchRequest@769694e9]
>>>>> org.elasticsearch.search.SearchParseException:
>>>>> [.marvel-2014.09.17][0]: from[-1],size[1]: Parse Failure [Failed to parse
>>>>> source [{"size":1,"sort":{"@timestamp":{"order":"desc"}}}]]
>>>>> at
>>>>> org.elasticsearch.search.SearchService.parseSource(SearchService.java:664)
>>>>> at
>>>>> org.elasticsearch.search.SearchService.createContext(SearchService.java:515)
>>>>> at
>>>>> org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:487)
>>>>> at
>>>>> org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:328)
>>>>> at
>>>>> org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:308)
>>>>> at
>>>>> org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:305)
>>>>> at
>>>>> org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>> at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>> Caused by: org.elasticsearch.search.SearchParseException:
>>>>> [.marvel-2014.09.17][0]: from[-1],size[1]: Parse Failure [No mapping
>>>>> found
>>>>> for [@timestamp] in order to sort on]
>>>>> at
>>>>> org.elasticsearch.search.sort.SortParseElement.addSortField(SortParseElement.java:198)
>>>>> at
>>>>> org.elasticsearch.search.sort.SortParseElement.addCompoundSortField(SortParseElement.java:172)
>>>>> at
>>>>> org.elasticsearch.search.sort.SortParseElement.parse(SortParseElement.java:90)
>>>>> at
>>>>> org.elasticsearch.search.SearchService.parseSource(SearchService.java:648)
>>>>> ... 9 more
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Monday, June 9, 2014 12:39:45 PM UTC-4, [email protected] wrote:
>>>>>>
>>>>>> Looks like this has already been found and resolved. I'll upgrade to
>>>>>> Marvel 1.2.1.
>>>>>>
>>>>>> http://www.elasticsearch.org/blog/marvel-1-2-1-released/
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Monday, June 9, 2014 9:35:30 AM UTC-7, [email protected] wrote:
>>>>>>>
>>>>>>> Hi Boaz,
>>>>>>>
>>>>>>> The only other exception I see in the logs is this:
>>>>>>>
>>>>>>> [2014-06-09 15:38:11,340][ERROR][marvel.agent ] [xxxx]
>>>>>>> exporter [es_exporter] has thrown an exception:
>>>>>>> java.lang.IllegalStateException: array not available
>>>>>>> at
>>>>>>> org.elasticsearch.common.bytes.PagedBytesReference.array(PagedBytesReference.java:289)
>>>>>>> at
>>>>>>> org.elasticsearch.marvel.agent.exporter.ESExporter.addXContentRendererToConnection(ESExporter.java:209)
>>>>>>> at
>>>>>>> org.elasticsearch.marvel.agent.exporter.ESExporter.exportXContent(ESExporter.java:252)
>>>>>>> at
>>>>>>> org.elasticsearch.marvel.agent.exporter.ESExporter.exportEvents(ESExporter.java:161)
>>>>>>> at
>>>>>>> org.elasticsearch.marvel.agent.AgentService$ExportingWorker.exportEvents(AgentService.java:305)
>>>>>>> at
>>>>>>> org.elasticsearch.marvel.agent.AgentService$ExportingWorker.run(AgentService.java:240)
>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Saturday, June 7, 2014 11:50:16 AM UTC-7, Boaz Leskes wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Is there anything in the log indicating data shipping problems from
>>>>>>>> the agent? Check the log of the current master node.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Boaz
>>>>>>>>
>>>>>>>> On Saturday, June 7, 2014 12:51:44 AM UTC+2, [email protected]
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> I'm running elasticsearch 1.2.1, and just upgraded to Marvel 1.2.
>>>>>>>>>
>>>>>>>>> When I go to the new "Shard Allocation" dashboard in Marvel, I see
>>>>>>>>> the following error:
>>>>>>>>>
>>>>>>>>> SearchPhaseExecutionException[Failed to execute phase
>>>>>>>>> [query_fetch], all shards failed; shardFailures
>>>>>>>>> {[5FNjdIADRE2rRuOkDZdieA][.marvel-2014.06.04][0]:
>>>>>>>>> RemoteTransportException[[xxxx][inet[/x.x.x.x:9300]][search/phase/query+fetch]];
>>>>>>>>>
>>>>>>>>> nested: SearchParseException[[.marvel-2014.06.04][0]:
>>>>>>>>> from[0],size[300]:
>>>>>>>>> Parse Failure [Failed to parse source
>>>>>>>>> [{"size":300,"from":0,"fields":["@timestamp","message","status"],"sort":{"@timestamp":{"order":"desc"}},"query":{"filtered":{"filter":{"range":{"@timestamp":{"from":"2014-05-30T22:33:17.689Z","to":"2014-06-06T22:33:17.689Z"}}}}}}]]];
>>>>>>>>>
>>>>>>>>> nested: SearchParseException[[.marvel-2014.06.04][0]:
>>>>>>>>> from[0],size[300]:
>>>>>>>>> Parse Failure [No mapping found for [@timestamp] in order to sort
>>>>>>>>> on]]; }]
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I see the following in my elasticsearch.log
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> [2014-06-06 22:48:07,919][DEBUG][action.search.type ] [xxxx]
>>>>>>>>> All shards failed for phase: [query_fetch]
>>>>>>>>> [2014-06-06 22:48:17,937][DEBUG][action.search.type ] [xxxx]
>>>>>>>>> [.marvel-2014.06.06][0], node[5FNjdIADRE2rRuOkDZdieA], [R],
>>>>>>>>> s[STARTED]:
>>>>>>>>> Failed to execute
>>>>>>>>> [org.elasticsearch.action.search.SearchRequest@6af079e8]
>>>>>>>>> org.elasticsearch.search.SearchParseException:
>>>>>>>>> [.marvel-2014.06.06][0]: from[-1],size[1]: Parse Failure [Failed to
>>>>>>>>> parse
>>>>>>>>> source [{"size":1,"sort":{"@timestamp":{"order":"desc"}}}]]
>>>>>>>>> at
>>>>>>>>> org.elasticsearch.search.SearchService.parseSource(SearchService.java:649)
>>>>>>>>> at
>>>>>>>>> org.elasticsearch.search.SearchService.createContext(SearchService.java:511)
>>>>>>>>> at
>>>>>>>>> org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:483)
>>>>>>>>> at
>>>>>>>>> org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:324)
>>>>>>>>> at
>>>>>>>>> org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:308)
>>>>>>>>> at
>>>>>>>>> org.elasticsearch.search.action.SearchServiceTransportAction$11.call(SearchServiceTransportAction.java:305)
>>>>>>>>> at
>>>>>>>>> org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>>>> at
>>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>>>> Caused by: org.elasticsearch.search.SearchParseException:
>>>>>>>>> [.marvel-2014.06.06][0]: from[-1],size[1]: Parse Failure [No mapping
>>>>>>>>> found
>>>>>>>>> for [@timestamp] in order to sort on]
>>>>>>>>> at
>>>>>>>>> org.elasticsearch.search.sort.SortParseElement.addSortField(SortParseElement.java:198)
>>>>>>>>> at
>>>>>>>>> org.elasticsearch.search.sort.SortParseElement.addCompoundSortField(SortParseElement.java:172)
>>>>>>>>> at
>>>>>>>>> org.elasticsearch.search.sort.SortParseElement.parse(SortParseElement.java:90)
>>>>>>>>> at
>>>>>>>>> org.elasticsearch.search.SearchService.parseSource(SearchService.java:633)
>>>>>>>>> ... 9 more
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I have confirmed that marvel is upgraded on every node in the
>>>>>>>>> cluster.
>>>>>>>>>
>>>>>>>>> I tried deleting the .marvel* indexes. Didn't help
>>>>>>>>> Shutdown the cluster, and restarted it. Didn't help.
>>>>>>>>>
>>>>>>>>> Anybody have ideas?
>>>>>>>>>
>>>>>>>>
>>
>>
>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/aa6036bc-1979-4b0c-a7f2-e35c4a96d80f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
