Would you be able to re-run your query and post the stack trace from the
Elasticsearch server logs. This might help to work out whats going on.
Thanks
Colin
On Tuesday, 29 July 2014 12:29:00 UTC+1, Valentin wrote:
>
> Ok. I think I found the problem. As soon as I try to sort on the script
> value it ceases to work
>
> works, but unsorted
> {
> "size": 0,
> "aggs": {
> "winners": {
> "terms": {
> "field": "tit",
> "size": 10,
> "shard_size": 4
> },
> "aggs": {
> "articles_over_time": {
> "date_histogram": {
> "field": "datetime",
> "interval": "1d"
> }
> },
> "diff": {
> "sum": {
> "script": "(doc['datetime'].value < 1406412000000) ? -1 : 1",
> "lang": "groovy"
> }
> }
> }
> }
> }
> }
>
> does not work:
> {
> "size": 0,
> "aggs": {
> "winners": {
> "terms": {
> "field": "tit",
> "size": 10,
> "order": {
> "diff": "desc"
> },
> "shard_size": 4
> },
> "aggs": {
> "articles_over_time": {
> "date_histogram": {
> "field": "datetime",
> "interval": "1d"
> }
> },
> "diff": {
> "sum": {
> "script": "(doc['datetime'].value < 1406412000000) ? -1 : 1",
> "lang": "groovy"
> }
> }
> }
> }
> }
> }
>
>
>
>
> On Tuesday, July 29, 2014 12:40:15 PM UTC+2, Valentin wrote:
>>
>> Hi Colin,
>>
>> I could figure out the shard_size problem thanks to your help.
>>
>> For the 'datetime' error: I checked and it exists in all the indices. It
>> has the correct mappings and the therefor probably could not have wrong
>> values I guess. And using the elasticsearch-head plugin I dont get the
>> error but a wrong result which really seems strange.
>>
>> Thanks
>> Valentin
>>
>> On Tuesday, July 29, 2014 11:54:08 AM UTC+2, Colin Goodheart-Smithe wrote:
>>>
>>> Firstly, I think the reason you are only getting results from one index
>>> when you are asking for a size of 1 in your terms aggregation is because
>>> you are asking for the top 1 bucket from each shard on each index. This
>>> will then be merged together and only the top bucket will be kept. If the
>>> top bucket is not the same on all indexes then you will not get results
>>> from all indices. Setting the shard_size parameter to something like 10
>>> can help with this (see
>>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#_document_counts_are_approximate
>>>
>>> for more information on this)
>>>
>>> Second, I wonder if the reason you are getting the error from your
>>> script is that you don't have a 'datetime' value for all of your documents
>>> in some of your indices?
>>>
>>> Regards,
>>>
>>> Colin
>>>
>>> On Monday, 28 July 2014 16:04:55 UTC+1, Valentin wrote:
>>>>
>>>> Hi Colin,
>>>>
>>>> now it gets really strange. First my alias
>>>> curl 'http://localhost:9200/_alias?pretty'
>>>> {
>>>> "live-2014-07-27" : {
>>>>
>>>> "aliases" : {
>>>>
>>>> "aggtest" : { }
>>>>
>>>> }
>>>>
>>>> },
>>>>
>>>> "live-2014-07-26" : {
>>>>
>>>> "aliases" : {
>>>>
>>>> "aggtest" : { }
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>>
>>>> I tried two different queries:
>>>> curl -XPOST 'http://localhost:9200/aggtest/video/_search?pretty=true'
>>>> -d '{
>>>> "size": 0,
>>>> "aggs": {
>>>> "winners": {
>>>> "terms": {
>>>> "field": "tit",
>>>> "order": {
>>>> "diff": "desc"
>>>> },
>>>> "size": 1
>>>> },
>>>> "aggs": {
>>>> "articles_over_time": {
>>>> "date_histogram": {
>>>> "field": "datetime",
>>>> "interval": "1d"
>>>> }
>>>> },
>>>> "diff": {
>>>> "sum": {
>>>> "script": "(doc['datetime'].value < 1406412000000) ? -1 :
>>>> 1",
>>>> "lang": "groovy"
>>>> }
>>>> }
>>>> }
>>>> }
>>>> }
>>>> }'
>>>>
>>>> and
>>>>
>>>> curl -XPOST '
>>>> http://localhost:9200/live-2014-07-26,live-2014-07-27/video/_search?pretty=true'
>>>>
>>>> .....
>>>>
>>>> both do give me a result (but a wrong one) when I do query using
>>>> elasticsearch-head but result in an error if I use the commandline
>>>>
>>>> {
>>>>
>>>> "error" : "SearchPhaseExecutionException[Failed to execute phase
>>>> [query], all shards failed; shardFailures
>>>> {[_MxuihP3TfmZV4FYUQaRQQ][live-2014-07-26][1]:
>>>> QueryPhaseExecutionException[[live-2014-07-26][1]:
>>>> query[ConstantScore(cache(_type:video))],from[0],size[0]: Query Failed
>>>> [Failed to execute main query]]; nested:
>>>> GroovyScriptExecutionException[MissingPropertyException[No such property:
>>>> datetime for class: Script126]];
>>>> }{[FYhB58m7T1W3HjhzUmtzww][live-2014-07-27][0]:
>>>> RemoteTransportException[[live02][inet[/10.XXX.XX.XX:9300]][search/phase/query]];
>>>>
>>>> nested: QueryPhaseExecutionException[[live-2014-07-27][0]:
>>>> query[ConstantScore(cache(_type:video))],from[0],size[0]: Query Failed
>>>> [Failed to execute main query]]; nested:
>>>> GroovyScriptExecutionException[MissingPropertyException[No such property:
>>>> datetime for class: Script119]];
>>>> }{[_MxuihP3TfmZV4FYUQaRQQ][live-2014-07-27][1]:
>>>> QueryPhaseExecutionException[[live-2014-07-27][1]:
>>>> query[ConstantScore(cache(_type:video))],from[0],size[0]: Query Failed
>>>> [Failed to execute main query]]; nested:
>>>> GroovyScriptExecutionException[MissingPropertyException[No such property:
>>>> datetime for class: Script126]];
>>>> }{[FYhB58m7T1W3HjhzUmtzww][live-2014-07-26][0]:
>>>> RemoteTransportException[[live02][inet[/10.XXX.XX.XX:9300]][search/phase/query]];
>>>>
>>>> nested: QueryPhaseExecutionException[[live-2014-07-26][0]:
>>>> query[ConstantScore(cache(_type:video))],from[0],size[0]: Query Failed
>>>> [Failed to execute main query]]; nested:
>>>> GroovyScriptExecutionException[MissingPropertyException[No such property:
>>>> datetime for class: Script119]]; }]",
>>>>
>>>> "status" : 500
>>>>
>>>> }
>>>>
>>>>
>>>> But I noticed something strange. This works:
>>>> curl -XPOST 'http://localhost:9200/aggtest/video/_search?pretty=true'
>>>> -d '{
>>>> "size": 0,
>>>> "aggs": {
>>>> "winners": {
>>>> "terms": {
>>>> "field": "tit"
>>>> },
>>>> "aggs": {
>>>> "articles_over_time": {
>>>> "date_histogram": {
>>>> "field": "datetime",
>>>> "interval": "1d"
>>>> }
>>>> }
>>>> }
>>>> }
>>>> }
>>>> }'
>>>> result:
>>>>
>>>> {
>>>>
>>>> "took" : 26,
>>>>
>>>> "timed_out" : false,
>>>>
>>>> "_shards" : {
>>>>
>>>> "total" : 4,
>>>>
>>>> "successful" : 4,
>>>>
>>>> "failed" : 0
>>>>
>>>> },
>>>>
>>>> "hits" : {
>>>>
>>>> "total" : 89419,
>>>>
>>>> "max_score" : 0.0,
>>>>
>>>> "hits" : [ ]
>>>>
>>>> },
>>>>
>>>> "aggregations" : {
>>>>
>>>> "winners" : {
>>>>
>>>> "buckets" : [ {
>>>>
>>>> "key" : "videotitle",
>>>>
>>>> "doc_count" : 3539,
>>>>
>>>> "articles_over_time" : {
>>>>
>>>> "buckets" : [ {
>>>>
>>>> "key_as_string" : "2014-07-26T00:00:00.000Z",
>>>>
>>>> "key" : 1406332800000,
>>>>
>>>> "doc_count" : 2820
>>>>
>>>> }, {
>>>>
>>>> "key_as_string" : "2014-07-27T00:00:00.000Z",
>>>>
>>>> "key" : 1406419200000,
>>>>
>>>> "doc_count" : 719
>>>>
>>>> } ]
>>>>
>>>> }
>>>>
>>>> }, {
>>>>
>>>> But this does not: (notice the size-limit to 1)
>>>> curl -XPOST 'http://localhost:9200/aggtest/video/_search?pretty=true'
>>>> -d '{
>>>> "size": 0,
>>>> "aggs": {
>>>> "winners": {
>>>> "terms": {
>>>> "field": "tit",
>>>> "size": 1
>>>> },
>>>> "aggs": {
>>>> "articles_over_time": {
>>>> "date_histogram": {
>>>> "field": "datetime",
>>>> "interval": "1d"
>>>> }
>>>> }
>>>> }
>>>> }
>>>> }
>>>> }'
>>>> result:
>>>>
>>>> {
>>>>
>>>> "took" : 17,
>>>>
>>>> "timed_out" : false,
>>>>
>>>> "_shards" : {
>>>>
>>>> "total" : 4,
>>>>
>>>> "successful" : 4,
>>>>
>>>> "failed" : 0
>>>>
>>>> },
>>>>
>>>> "hits" : {
>>>>
>>>> "total" : 89419,
>>>>
>>>> "max_score" : 0.0,
>>>>
>>>> "hits" : [ ]
>>>>
>>>> },
>>>>
>>>> "aggregations" : {
>>>>
>>>> "winners" : {
>>>>
>>>> "buckets" : [ {
>>>>
>>>> "key" : "videotitle",
>>>>
>>>> "doc_count" : 2820,
>>>>
>>>> "articles_over_time" : {
>>>>
>>>> "buckets" : [ {
>>>>
>>>> "key_as_string" : "2014-07-26T00:00:00.000Z",
>>>>
>>>> "key" : 1406332800000,
>>>>
>>>> "doc_count" : 2820
>>>>
>>>> } ]
>>>>
>>>> }
>>>>
>>>> } ]
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>> }
>>>>
>>>> Which seems to be a related problem to my original query, because it
>>>> always seem to query one index but not the other
>>>>
>>>> my original query I used in elasticsearch-head:
>>>> /aggtest/video/
>>>>
>>>> {
>>>> "size": 0,
>>>> "aggs": {
>>>> "winners": {
>>>> "terms": {
>>>> "field": "tit",
>>>> "order": {
>>>> "diff": "desc"
>>>> }
>>>> },
>>>> "aggs": {
>>>> "articles_over_time": {
>>>> "date_histogram": {
>>>> "field": "datetime",
>>>> "interval": "1d"
>>>> }
>>>> },
>>>> "diff": {
>>>> "sum": {
>>>> "script": "(doc['datetime'].value < 1406412000000) ? -1 :
>>>> 1",
>>>> "lang": "groovy"
>>>> }
>>>> }
>>>> }
>>>> }
>>>> }
>>>> }
>>>> and the result:
>>>>
>>>> - {
>>>> - key: videotitle
>>>> - doc_count: 719
>>>> - articles_over_time: {
>>>> - buckets: [
>>>> - {
>>>> - key_as_string: 2014-07-27T00:00:00.000Z
>>>> - key: 1406419200000
>>>> - doc_count: 719
>>>> }
>>>> ]
>>>> }
>>>> - diff: {
>>>> - value: 719
>>>> }
>>>> }
>>>>
>>>>
>>>>
>>>> Thanks,
>>>> Valentin
>>>>
>>>>
>>>>
>>>>
>>>> On Monday, July 28, 2014 3:42:46 PM UTC+2, Colin Goodheart-Smithe wrote:
>>>>>
>>>>> How are you searching over the multiple indexes? are you using
>>>>> aliases? It would be helpful if you could post your alias configuration
>>>>> (see [1]) and an cURL example of a search request that fails
>>>>>
>>>>> [1]
>>>>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-aliases.html#alias-retrieving
>>>>>
>>>>> Thanks
>>>>>
>>>>> Colin
>>>>>
>>>>> On Monday, 28 July 2014 14:00:55 UTC+1, Valentin wrote:
>>>>>>
>>>>>> Hi Colin,
>>>>>>
>>>>>> thanks for checking. I could successfully reproduce your example and
>>>>>> I even splitted it into 2 indeces and it worked (Elasticsearch 1.3.0).
>>>>>> But
>>>>>> as soon as I try it with my data it doesnt work. I ran some additional
>>>>>> tests and it works if I only use the current index (day) and split it in
>>>>>> half. But as soon as I try to compare yesterday and the day before it
>>>>>> only
>>>>>> seems to get the data from one day but not the other.
>>>>>>
>>>>>> Cheers,
>>>>>> Valentin
>>>>>>
>>>>>> On Monday, July 28, 2014 10:07:43 AM UTC+2, Colin Goodheart-Smithe
>>>>>> wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I ran the commands in the following gist, on master, without error.
>>>>>>> Would you be able to post the error you get and a similar reproducible
>>>>>>> example to help diagnose the issue you are running into? Also, which
>>>>>>> version of Elasticsearch are you running?
>>>>>>>
>>>>>>> https://gist.github.com/colings86/46fbb0b22c2f3c4348ae
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> Colin
>>>>>>>
>>>>>>> On Sunday, 27 July 2014 17:53:29 UTC+1, Valentin wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I am trying to use this aggregation which does not work:
>>>>>>>> "aggs": {
>>>>>>>> "winners": {
>>>>>>>> "terms": {
>>>>>>>> "field": "urls",
>>>>>>>> "order": {
>>>>>>>> "diff": "desc"
>>>>>>>> }
>>>>>>>> },
>>>>>>>> "aggs": {
>>>>>>>> "diff": {
>>>>>>>> "sum": {
>>>>>>>> "script": "(doc['datetime'].date.getMillis() <
>>>>>>>> 1406332800000) ? -1 : 1",
>>>>>>>> "lang": "groovy"
>>>>>>>> }
>>>>>>>> }
>>>>>>>> }
>>>>>>>> }
>>>>>>>> }
>>>>>>>>
>>>>>>>> Can anyone help?
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Valentin
>>>>>>>>
>>>>>>>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/95def0cd-952b-46ed-89c1-33c4cc4254a0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
