The Elasticsearch log files can be found in the logs directory of your
node's Elasticsearch directory. If you re-create the error and have a look
at the end of the log file you should see the stacktrace
Colin
On Wednesday, 30 July 2014 10:53:05 UTC+1, Valentin wrote:
>
> Hi Colin,
>
> I try increasing it up to 40 but nothing changes. I would post the stack
> trace but I don't know how to find them.
>
> Thanks
> Valentin
>
> On Wednesday, July 30, 2014 10:24:09 AM UTC+2, Colin Goodheart-Smithe
> wrote:
>>
>> Also, your shard_size parameter should always be greater than the size
>> parameter. So if you are asking for size of 10 then I would try setting
>> shard_size to 20 or 30.
>>
>> On Wednesday, 30 July 2014 09:22:16 UTC+1, Colin Goodheart-Smithe wrote:
>>>
>>> Would you be able to re-run your query and post the stack trace from the
>>> Elasticsearch server logs. This might help to work out whats going on.
>>>
>>> Thanks
>>>
>>> Colin
>>>
>>> On Tuesday, 29 July 2014 12:29:00 UTC+1, Valentin wrote:
>>>>
>>>> Ok. I think I found the problem. As soon as I try to sort on the script
>>>> value it ceases to work
>>>>
>>>> works, but unsorted
>>>> {
>>>> "size": 0,
>>>> "aggs": {
>>>> "winners": {
>>>> "terms": {
>>>> "field": "tit",
>>>> "size": 10,
>>>> "shard_size": 4
>>>> },
>>>> "aggs": {
>>>> "articles_over_time": {
>>>> "date_histogram": {
>>>> "field": "datetime",
>>>> "interval": "1d"
>>>> }
>>>> },
>>>> "diff": {
>>>> "sum": {
>>>> "script": "(doc['datetime'].value < 1406412000000) ? -1 :
>>>> 1",
>>>> "lang": "groovy"
>>>> }
>>>> }
>>>> }
>>>> }
>>>> }
>>>> }
>>>>
>>>> does not work:
>>>> {
>>>> "size": 0,
>>>> "aggs": {
>>>> "winners": {
>>>> "terms": {
>>>> "field": "tit",
>>>> "size": 10,
>>>> "order": {
>>>> "diff": "desc"
>>>> },
>>>> "shard_size": 4
>>>> },
>>>> "aggs": {
>>>> "articles_over_time": {
>>>> "date_histogram": {
>>>> "field": "datetime",
>>>> "interval": "1d"
>>>> }
>>>> },
>>>> "diff": {
>>>> "sum": {
>>>> "script": "(doc['datetime'].value < 1406412000000) ? -1 :
>>>> 1",
>>>> "lang": "groovy"
>>>> }
>>>> }
>>>> }
>>>> }
>>>> }
>>>> }
>>>>
>>>>
>>>>
>>>>
>>>> On Tuesday, July 29, 2014 12:40:15 PM UTC+2, Valentin wrote:
>>>>>
>>>>> Hi Colin,
>>>>>
>>>>> I could figure out the shard_size problem thanks to your help.
>>>>>
>>>>> For the 'datetime' error: I checked and it exists in all the indices.
>>>>> It has the correct mappings and the therefor probably could not have
>>>>> wrong
>>>>> values I guess. And using the elasticsearch-head plugin I dont get the
>>>>> error but a wrong result which really seems strange.
>>>>>
>>>>> Thanks
>>>>> Valentin
>>>>>
>>>>> On Tuesday, July 29, 2014 11:54:08 AM UTC+2, Colin Goodheart-Smithe
>>>>> wrote:
>>>>>>
>>>>>> Firstly, I think the reason you are only getting results from one
>>>>>> index when you are asking for a size of 1 in your terms aggregation is
>>>>>> because you are asking for the top 1 bucket from each shard on each
>>>>>> index.
>>>>>> This will then be merged together and only the top bucket will be kept.
>>>>>> If the top bucket is not the same on all indexes then you will not get
>>>>>> results from all indices. Setting the shard_size parameter to something
>>>>>> like 10 can help with this (see
>>>>>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#_document_counts_are_approximate
>>>>>>
>>>>>> for more information on this)
>>>>>>
>>>>>> Second, I wonder if the reason you are getting the error from your
>>>>>> script is that you don't have a 'datetime' value for all of your
>>>>>> documents
>>>>>> in some of your indices?
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Colin
>>>>>>
>>>>>> On Monday, 28 July 2014 16:04:55 UTC+1, Valentin wrote:
>>>>>>>
>>>>>>> Hi Colin,
>>>>>>>
>>>>>>> now it gets really strange. First my alias
>>>>>>> curl 'http://localhost:9200/_alias?pretty'
>>>>>>> {
>>>>>>> "live-2014-07-27" : {
>>>>>>>
>>>>>>> "aliases" : {
>>>>>>>
>>>>>>> "aggtest" : { }
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> },
>>>>>>>
>>>>>>> "live-2014-07-26" : {
>>>>>>>
>>>>>>> "aliases" : {
>>>>>>>
>>>>>>> "aggtest" : { }
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>> I tried two different queries:
>>>>>>> curl -XPOST 'http://localhost:9200/aggtest/video/_search?pretty=true'
>>>>>>> -d '{
>>>>>>> "size": 0,
>>>>>>> "aggs": {
>>>>>>> "winners": {
>>>>>>> "terms": {
>>>>>>> "field": "tit",
>>>>>>> "order": {
>>>>>>> "diff": "desc"
>>>>>>> },
>>>>>>> "size": 1
>>>>>>> },
>>>>>>> "aggs": {
>>>>>>> "articles_over_time": {
>>>>>>> "date_histogram": {
>>>>>>> "field": "datetime",
>>>>>>> "interval": "1d"
>>>>>>> }
>>>>>>> },
>>>>>>> "diff": {
>>>>>>> "sum": {
>>>>>>> "script": "(doc['datetime'].value < 1406412000000) ? -1
>>>>>>> : 1",
>>>>>>> "lang": "groovy"
>>>>>>> }
>>>>>>> }
>>>>>>> }
>>>>>>> }
>>>>>>> }
>>>>>>> }'
>>>>>>>
>>>>>>> and
>>>>>>>
>>>>>>> curl -XPOST '
>>>>>>> http://localhost:9200/live-2014-07-26,live-2014-07-27/video/_search?pretty=true'
>>>>>>>
>>>>>>> .....
>>>>>>>
>>>>>>> both do give me a result (but a wrong one) when I do query using
>>>>>>> elasticsearch-head but result in an error if I use the commandline
>>>>>>>
>>>>>>> {
>>>>>>>
>>>>>>> "error" : "SearchPhaseExecutionException[Failed to execute phase
>>>>>>> [query], all shards failed; shardFailures
>>>>>>> {[_MxuihP3TfmZV4FYUQaRQQ][live-2014-07-26][1]:
>>>>>>> QueryPhaseExecutionException[[live-2014-07-26][1]:
>>>>>>> query[ConstantScore(cache(_type:video))],from[0],size[0]: Query Failed
>>>>>>> [Failed to execute main query]]; nested:
>>>>>>> GroovyScriptExecutionException[MissingPropertyException[No such
>>>>>>> property:
>>>>>>> datetime for class: Script126]];
>>>>>>> }{[FYhB58m7T1W3HjhzUmtzww][live-2014-07-27][0]:
>>>>>>> RemoteTransportException[[live02][inet[/10.XXX.XX.XX:9300]][search/phase/query]];
>>>>>>>
>>>>>>> nested: QueryPhaseExecutionException[[live-2014-07-27][0]:
>>>>>>> query[ConstantScore(cache(_type:video))],from[0],size[0]: Query Failed
>>>>>>> [Failed to execute main query]]; nested:
>>>>>>> GroovyScriptExecutionException[MissingPropertyException[No such
>>>>>>> property:
>>>>>>> datetime for class: Script119]];
>>>>>>> }{[_MxuihP3TfmZV4FYUQaRQQ][live-2014-07-27][1]:
>>>>>>> QueryPhaseExecutionException[[live-2014-07-27][1]:
>>>>>>> query[ConstantScore(cache(_type:video))],from[0],size[0]: Query Failed
>>>>>>> [Failed to execute main query]]; nested:
>>>>>>> GroovyScriptExecutionException[MissingPropertyException[No such
>>>>>>> property:
>>>>>>> datetime for class: Script126]];
>>>>>>> }{[FYhB58m7T1W3HjhzUmtzww][live-2014-07-26][0]:
>>>>>>> RemoteTransportException[[live02][inet[/10.XXX.XX.XX:9300]][search/phase/query]];
>>>>>>>
>>>>>>> nested: QueryPhaseExecutionException[[live-2014-07-26][0]:
>>>>>>> query[ConstantScore(cache(_type:video))],from[0],size[0]: Query Failed
>>>>>>> [Failed to execute main query]]; nested:
>>>>>>> GroovyScriptExecutionException[MissingPropertyException[No such
>>>>>>> property:
>>>>>>> datetime for class: Script119]]; }]",
>>>>>>>
>>>>>>> "status" : 500
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>> But I noticed something strange. This works:
>>>>>>> curl -XPOST 'http://localhost:9200/aggtest/video/_search?pretty=true'
>>>>>>> -d '{
>>>>>>> "size": 0,
>>>>>>> "aggs": {
>>>>>>> "winners": {
>>>>>>> "terms": {
>>>>>>> "field": "tit"
>>>>>>> },
>>>>>>> "aggs": {
>>>>>>> "articles_over_time": {
>>>>>>> "date_histogram": {
>>>>>>> "field": "datetime",
>>>>>>> "interval": "1d"
>>>>>>> }
>>>>>>> }
>>>>>>> }
>>>>>>> }
>>>>>>> }
>>>>>>> }'
>>>>>>> result:
>>>>>>>
>>>>>>> {
>>>>>>>
>>>>>>> "took" : 26,
>>>>>>>
>>>>>>> "timed_out" : false,
>>>>>>>
>>>>>>> "_shards" : {
>>>>>>>
>>>>>>> "total" : 4,
>>>>>>>
>>>>>>> "successful" : 4,
>>>>>>>
>>>>>>> "failed" : 0
>>>>>>>
>>>>>>> },
>>>>>>>
>>>>>>> "hits" : {
>>>>>>>
>>>>>>> "total" : 89419,
>>>>>>>
>>>>>>> "max_score" : 0.0,
>>>>>>>
>>>>>>> "hits" : [ ]
>>>>>>>
>>>>>>> },
>>>>>>>
>>>>>>> "aggregations" : {
>>>>>>>
>>>>>>> "winners" : {
>>>>>>>
>>>>>>> "buckets" : [ {
>>>>>>>
>>>>>>> "key" : "videotitle",
>>>>>>>
>>>>>>> "doc_count" : 3539,
>>>>>>>
>>>>>>> "articles_over_time" : {
>>>>>>>
>>>>>>> "buckets" : [ {
>>>>>>>
>>>>>>> "key_as_string" : "2014-07-26T00:00:00.000Z",
>>>>>>>
>>>>>>> "key" : 1406332800000,
>>>>>>>
>>>>>>> "doc_count" : 2820
>>>>>>>
>>>>>>> }, {
>>>>>>>
>>>>>>> "key_as_string" : "2014-07-27T00:00:00.000Z",
>>>>>>>
>>>>>>> "key" : 1406419200000,
>>>>>>>
>>>>>>> "doc_count" : 719
>>>>>>>
>>>>>>> } ]
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> }, {
>>>>>>>
>>>>>>> But this does not: (notice the size-limit to 1)
>>>>>>> curl -XPOST 'http://localhost:9200/aggtest/video/_search?pretty=true'
>>>>>>> -d '{
>>>>>>> "size": 0,
>>>>>>> "aggs": {
>>>>>>> "winners": {
>>>>>>> "terms": {
>>>>>>> "field": "tit",
>>>>>>> "size": 1
>>>>>>> },
>>>>>>> "aggs": {
>>>>>>> "articles_over_time": {
>>>>>>> "date_histogram": {
>>>>>>> "field": "datetime",
>>>>>>> "interval": "1d"
>>>>>>> }
>>>>>>> }
>>>>>>> }
>>>>>>> }
>>>>>>> }
>>>>>>> }'
>>>>>>> result:
>>>>>>>
>>>>>>> {
>>>>>>>
>>>>>>> "took" : 17,
>>>>>>>
>>>>>>> "timed_out" : false,
>>>>>>>
>>>>>>> "_shards" : {
>>>>>>>
>>>>>>> "total" : 4,
>>>>>>>
>>>>>>> "successful" : 4,
>>>>>>>
>>>>>>> "failed" : 0
>>>>>>>
>>>>>>> },
>>>>>>>
>>>>>>> "hits" : {
>>>>>>>
>>>>>>> "total" : 89419,
>>>>>>>
>>>>>>> "max_score" : 0.0,
>>>>>>>
>>>>>>> "hits" : [ ]
>>>>>>>
>>>>>>> },
>>>>>>>
>>>>>>> "aggregations" : {
>>>>>>>
>>>>>>> "winners" : {
>>>>>>>
>>>>>>> "buckets" : [ {
>>>>>>>
>>>>>>> "key" : "videotitle",
>>>>>>>
>>>>>>> "doc_count" : 2820,
>>>>>>>
>>>>>>> "articles_over_time" : {
>>>>>>>
>>>>>>> "buckets" : [ {
>>>>>>>
>>>>>>> "key_as_string" : "2014-07-26T00:00:00.000Z",
>>>>>>>
>>>>>>> "key" : 1406332800000,
>>>>>>>
>>>>>>> "doc_count" : 2820
>>>>>>>
>>>>>>> } ]
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> } ]
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> }
>>>>>>>
>>>>>>> Which seems to be a related problem to my original query, because it
>>>>>>> always seem to query one index but not the other
>>>>>>>
>>>>>>> my original query I used in elasticsearch-head:
>>>>>>> /aggtest/video/
>>>>>>>
>>>>>>> {
>>>>>>> "size": 0,
>>>>>>> "aggs": {
>>>>>>> "winners": {
>>>>>>> "terms": {
>>>>>>> "field": "tit",
>>>>>>> "order": {
>>>>>>> "diff": "desc"
>>>>>>> }
>>>>>>> },
>>>>>>> "aggs": {
>>>>>>> "articles_over_time": {
>>>>>>> "date_histogram": {
>>>>>>> "field": "datetime",
>>>>>>> "interval": "1d"
>>>>>>> }
>>>>>>> },
>>>>>>> "diff": {
>>>>>>> "sum": {
>>>>>>> "script": "(doc['datetime'].value < 1406412000000) ? -1
>>>>>>> : 1",
>>>>>>> "lang": "groovy"
>>>>>>> }
>>>>>>> }
>>>>>>> }
>>>>>>> }
>>>>>>> }
>>>>>>> }
>>>>>>> and the result:
>>>>>>>
>>>>>>> - {
>>>>>>> - key: videotitle
>>>>>>> - doc_count: 719
>>>>>>> - articles_over_time: {
>>>>>>> - buckets: [
>>>>>>> - {
>>>>>>> - key_as_string: 2014-07-27T00:00:00.000Z
>>>>>>> - key: 1406419200000
>>>>>>> - doc_count: 719
>>>>>>> }
>>>>>>> ]
>>>>>>> }
>>>>>>> - diff: {
>>>>>>> - value: 719
>>>>>>> }
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Valentin
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Monday, July 28, 2014 3:42:46 PM UTC+2, Colin Goodheart-Smithe
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> How are you searching over the multiple indexes? are you using
>>>>>>>> aliases? It would be helpful if you could post your alias
>>>>>>>> configuration
>>>>>>>> (see [1]) and an cURL example of a search request that fails
>>>>>>>>
>>>>>>>> [1]
>>>>>>>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-aliases.html#alias-retrieving
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>>
>>>>>>>> Colin
>>>>>>>>
>>>>>>>> On Monday, 28 July 2014 14:00:55 UTC+1, Valentin wrote:
>>>>>>>>>
>>>>>>>>> Hi Colin,
>>>>>>>>>
>>>>>>>>> thanks for checking. I could successfully reproduce your example
>>>>>>>>> and I even splitted it into 2 indeces and it worked (Elasticsearch
>>>>>>>>> 1.3.0).
>>>>>>>>> But as soon as I try it with my data it doesnt work. I ran some
>>>>>>>>> additional
>>>>>>>>> tests and it works if I only use the current index (day) and split it
>>>>>>>>> in
>>>>>>>>> half. But as soon as I try to compare yesterday and the day before it
>>>>>>>>> only
>>>>>>>>> seems to get the data from one day but not the other.
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Valentin
>>>>>>>>>
>>>>>>>>> On Monday, July 28, 2014 10:07:43 AM UTC+2, Colin Goodheart-Smithe
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I ran the commands in the following gist, on master, without
>>>>>>>>>> error. Would you be able to post the error you get and a similar
>>>>>>>>>> reproducible example to help diagnose the issue you are running
>>>>>>>>>> into? Also,
>>>>>>>>>> which version of Elasticsearch are you running?
>>>>>>>>>>
>>>>>>>>>> https://gist.github.com/colings86/46fbb0b22c2f3c4348ae
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>>
>>>>>>>>>> Colin
>>>>>>>>>>
>>>>>>>>>> On Sunday, 27 July 2014 17:53:29 UTC+1, Valentin wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> I am trying to use this aggregation which does not work:
>>>>>>>>>>> "aggs": {
>>>>>>>>>>> "winners": {
>>>>>>>>>>> "terms": {
>>>>>>>>>>> "field": "urls",
>>>>>>>>>>> "order": {
>>>>>>>>>>> "diff": "desc"
>>>>>>>>>>> }
>>>>>>>>>>> },
>>>>>>>>>>> "aggs": {
>>>>>>>>>>> "diff": {
>>>>>>>>>>> "sum": {
>>>>>>>>>>> "script": "(doc['datetime'].date.getMillis() <
>>>>>>>>>>> 1406332800000) ? -1 : 1",
>>>>>>>>>>> "lang": "groovy"
>>>>>>>>>>> }
>>>>>>>>>>> }
>>>>>>>>>>> }
>>>>>>>>>>> }
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> Can anyone help?
>>>>>>>>>>>
>>>>>>>>>>> Cheers,
>>>>>>>>>>> Valentin
>>>>>>>>>>>
>>>>>>>>>>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/525281c1-dce7-42ed-9926-8bf3605d89ba%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
