Ok. I think I found the problem. As soon as I try to sort on the script
value it ceases to work
works, but unsorted
{
"size": 0,
"aggs": {
"winners": {
"terms": {
"field": "tit",
"size": 10,
"shard_size": 4
},
"aggs": {
"articles_over_time": {
"date_histogram": {
"field": "datetime",
"interval": "1d"
}
},
"diff": {
"sum": {
"script": "(doc['datetime'].value < 1406412000000) ? -1 : 1",
"lang": "groovy"
}
}
}
}
}
}
does not work:
{
"size": 0,
"aggs": {
"winners": {
"terms": {
"field": "tit",
"size": 10,
"order": {
"diff": "desc"
},
"shard_size": 4
},
"aggs": {
"articles_over_time": {
"date_histogram": {
"field": "datetime",
"interval": "1d"
}
},
"diff": {
"sum": {
"script": "(doc['datetime'].value < 1406412000000) ? -1 : 1",
"lang": "groovy"
}
}
}
}
}
}
On Tuesday, July 29, 2014 12:40:15 PM UTC+2, Valentin wrote:
>
> Hi Colin,
>
> I could figure out the shard_size problem thanks to your help.
>
> For the 'datetime' error: I checked and it exists in all the indices. It
> has the correct mappings and the therefor probably could not have wrong
> values I guess. And using the elasticsearch-head plugin I dont get the
> error but a wrong result which really seems strange.
>
> Thanks
> Valentin
>
> On Tuesday, July 29, 2014 11:54:08 AM UTC+2, Colin Goodheart-Smithe wrote:
>>
>> Firstly, I think the reason you are only getting results from one index
>> when you are asking for a size of 1 in your terms aggregation is because
>> you are asking for the top 1 bucket from each shard on each index. This
>> will then be merged together and only the top bucket will be kept. If the
>> top bucket is not the same on all indexes then you will not get results
>> from all indices. Setting the shard_size parameter to something like 10
>> can help with this (see
>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#_document_counts_are_approximate
>>
>> for more information on this)
>>
>> Second, I wonder if the reason you are getting the error from your script
>> is that you don't have a 'datetime' value for all of your documents in some
>> of your indices?
>>
>> Regards,
>>
>> Colin
>>
>> On Monday, 28 July 2014 16:04:55 UTC+1, Valentin wrote:
>>>
>>> Hi Colin,
>>>
>>> now it gets really strange. First my alias
>>> curl 'http://localhost:9200/_alias?pretty'
>>> {
>>> "live-2014-07-27" : {
>>>
>>> "aliases" : {
>>>
>>> "aggtest" : { }
>>>
>>> }
>>>
>>> },
>>>
>>> "live-2014-07-26" : {
>>>
>>> "aliases" : {
>>>
>>> "aggtest" : { }
>>>
>>> }
>>>
>>> }
>>>
>>> }
>>>
>>>
>>> I tried two different queries:
>>> curl -XPOST 'http://localhost:9200/aggtest/video/_search?pretty=true'
>>> -d '{
>>> "size": 0,
>>> "aggs": {
>>> "winners": {
>>> "terms": {
>>> "field": "tit",
>>> "order": {
>>> "diff": "desc"
>>> },
>>> "size": 1
>>> },
>>> "aggs": {
>>> "articles_over_time": {
>>> "date_histogram": {
>>> "field": "datetime",
>>> "interval": "1d"
>>> }
>>> },
>>> "diff": {
>>> "sum": {
>>> "script": "(doc['datetime'].value < 1406412000000) ? -1 : 1",
>>> "lang": "groovy"
>>> }
>>> }
>>> }
>>> }
>>> }
>>> }'
>>>
>>> and
>>>
>>> curl -XPOST '
>>> http://localhost:9200/live-2014-07-26,live-2014-07-27/video/_search?pretty=true'
>>>
>>> .....
>>>
>>> both do give me a result (but a wrong one) when I do query using
>>> elasticsearch-head but result in an error if I use the commandline
>>>
>>> {
>>>
>>> "error" : "SearchPhaseExecutionException[Failed to execute phase
>>> [query], all shards failed; shardFailures
>>> {[_MxuihP3TfmZV4FYUQaRQQ][live-2014-07-26][1]:
>>> QueryPhaseExecutionException[[live-2014-07-26][1]:
>>> query[ConstantScore(cache(_type:video))],from[0],size[0]: Query Failed
>>> [Failed to execute main query]]; nested:
>>> GroovyScriptExecutionException[MissingPropertyException[No such property:
>>> datetime for class: Script126]];
>>> }{[FYhB58m7T1W3HjhzUmtzww][live-2014-07-27][0]:
>>> RemoteTransportException[[live02][inet[/10.XXX.XX.XX:9300]][search/phase/query]];
>>>
>>> nested: QueryPhaseExecutionException[[live-2014-07-27][0]:
>>> query[ConstantScore(cache(_type:video))],from[0],size[0]: Query Failed
>>> [Failed to execute main query]]; nested:
>>> GroovyScriptExecutionException[MissingPropertyException[No such property:
>>> datetime for class: Script119]];
>>> }{[_MxuihP3TfmZV4FYUQaRQQ][live-2014-07-27][1]:
>>> QueryPhaseExecutionException[[live-2014-07-27][1]:
>>> query[ConstantScore(cache(_type:video))],from[0],size[0]: Query Failed
>>> [Failed to execute main query]]; nested:
>>> GroovyScriptExecutionException[MissingPropertyException[No such property:
>>> datetime for class: Script126]];
>>> }{[FYhB58m7T1W3HjhzUmtzww][live-2014-07-26][0]:
>>> RemoteTransportException[[live02][inet[/10.XXX.XX.XX:9300]][search/phase/query]];
>>>
>>> nested: QueryPhaseExecutionException[[live-2014-07-26][0]:
>>> query[ConstantScore(cache(_type:video))],from[0],size[0]: Query Failed
>>> [Failed to execute main query]]; nested:
>>> GroovyScriptExecutionException[MissingPropertyException[No such property:
>>> datetime for class: Script119]]; }]",
>>>
>>> "status" : 500
>>>
>>> }
>>>
>>>
>>> But I noticed something strange. This works:
>>> curl -XPOST 'http://localhost:9200/aggtest/video/_search?pretty=true'
>>> -d '{
>>> "size": 0,
>>> "aggs": {
>>> "winners": {
>>> "terms": {
>>> "field": "tit"
>>> },
>>> "aggs": {
>>> "articles_over_time": {
>>> "date_histogram": {
>>> "field": "datetime",
>>> "interval": "1d"
>>> }
>>> }
>>> }
>>> }
>>> }
>>> }'
>>> result:
>>>
>>> {
>>>
>>> "took" : 26,
>>>
>>> "timed_out" : false,
>>>
>>> "_shards" : {
>>>
>>> "total" : 4,
>>>
>>> "successful" : 4,
>>>
>>> "failed" : 0
>>>
>>> },
>>>
>>> "hits" : {
>>>
>>> "total" : 89419,
>>>
>>> "max_score" : 0.0,
>>>
>>> "hits" : [ ]
>>>
>>> },
>>>
>>> "aggregations" : {
>>>
>>> "winners" : {
>>>
>>> "buckets" : [ {
>>>
>>> "key" : "videotitle",
>>>
>>> "doc_count" : 3539,
>>>
>>> "articles_over_time" : {
>>>
>>> "buckets" : [ {
>>>
>>> "key_as_string" : "2014-07-26T00:00:00.000Z",
>>>
>>> "key" : 1406332800000,
>>>
>>> "doc_count" : 2820
>>>
>>> }, {
>>>
>>> "key_as_string" : "2014-07-27T00:00:00.000Z",
>>>
>>> "key" : 1406419200000,
>>>
>>> "doc_count" : 719
>>>
>>> } ]
>>>
>>> }
>>>
>>> }, {
>>>
>>> But this does not: (notice the size-limit to 1)
>>> curl -XPOST 'http://localhost:9200/aggtest/video/_search?pretty=true'
>>> -d '{
>>> "size": 0,
>>> "aggs": {
>>> "winners": {
>>> "terms": {
>>> "field": "tit",
>>> "size": 1
>>> },
>>> "aggs": {
>>> "articles_over_time": {
>>> "date_histogram": {
>>> "field": "datetime",
>>> "interval": "1d"
>>> }
>>> }
>>> }
>>> }
>>> }
>>> }'
>>> result:
>>>
>>> {
>>>
>>> "took" : 17,
>>>
>>> "timed_out" : false,
>>>
>>> "_shards" : {
>>>
>>> "total" : 4,
>>>
>>> "successful" : 4,
>>>
>>> "failed" : 0
>>>
>>> },
>>>
>>> "hits" : {
>>>
>>> "total" : 89419,
>>>
>>> "max_score" : 0.0,
>>>
>>> "hits" : [ ]
>>>
>>> },
>>>
>>> "aggregations" : {
>>>
>>> "winners" : {
>>>
>>> "buckets" : [ {
>>>
>>> "key" : "videotitle",
>>>
>>> "doc_count" : 2820,
>>>
>>> "articles_over_time" : {
>>>
>>> "buckets" : [ {
>>>
>>> "key_as_string" : "2014-07-26T00:00:00.000Z",
>>>
>>> "key" : 1406332800000,
>>>
>>> "doc_count" : 2820
>>>
>>> } ]
>>>
>>> }
>>>
>>> } ]
>>>
>>> }
>>>
>>> }
>>>
>>> }
>>>
>>> Which seems to be a related problem to my original query, because it
>>> always seem to query one index but not the other
>>>
>>> my original query I used in elasticsearch-head:
>>> /aggtest/video/
>>>
>>> {
>>> "size": 0,
>>> "aggs": {
>>> "winners": {
>>> "terms": {
>>> "field": "tit",
>>> "order": {
>>> "diff": "desc"
>>> }
>>> },
>>> "aggs": {
>>> "articles_over_time": {
>>> "date_histogram": {
>>> "field": "datetime",
>>> "interval": "1d"
>>> }
>>> },
>>> "diff": {
>>> "sum": {
>>> "script": "(doc['datetime'].value < 1406412000000) ? -1 : 1",
>>> "lang": "groovy"
>>> }
>>> }
>>> }
>>> }
>>> }
>>> }
>>> and the result:
>>>
>>> - {
>>> - key: videotitle
>>> - doc_count: 719
>>> - articles_over_time: {
>>> - buckets: [
>>> - {
>>> - key_as_string: 2014-07-27T00:00:00.000Z
>>> - key: 1406419200000
>>> - doc_count: 719
>>> }
>>> ]
>>> }
>>> - diff: {
>>> - value: 719
>>> }
>>> }
>>>
>>>
>>>
>>> Thanks,
>>> Valentin
>>>
>>>
>>>
>>>
>>> On Monday, July 28, 2014 3:42:46 PM UTC+2, Colin Goodheart-Smithe wrote:
>>>>
>>>> How are you searching over the multiple indexes? are you using aliases?
>>>> It would be helpful if you could post your alias configuration (see [1])
>>>> and an cURL example of a search request that fails
>>>>
>>>> [1]
>>>> http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-aliases.html#alias-retrieving
>>>>
>>>> Thanks
>>>>
>>>> Colin
>>>>
>>>> On Monday, 28 July 2014 14:00:55 UTC+1, Valentin wrote:
>>>>>
>>>>> Hi Colin,
>>>>>
>>>>> thanks for checking. I could successfully reproduce your example and I
>>>>> even splitted it into 2 indeces and it worked (Elasticsearch 1.3.0). But
>>>>> as
>>>>> soon as I try it with my data it doesnt work. I ran some additional tests
>>>>> and it works if I only use the current index (day) and split it in half.
>>>>> But as soon as I try to compare yesterday and the day before it only
>>>>> seems
>>>>> to get the data from one day but not the other.
>>>>>
>>>>> Cheers,
>>>>> Valentin
>>>>>
>>>>> On Monday, July 28, 2014 10:07:43 AM UTC+2, Colin Goodheart-Smithe
>>>>> wrote:
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I ran the commands in the following gist, on master, without error.
>>>>>> Would you be able to post the error you get and a similar reproducible
>>>>>> example to help diagnose the issue you are running into? Also, which
>>>>>> version of Elasticsearch are you running?
>>>>>>
>>>>>> https://gist.github.com/colings86/46fbb0b22c2f3c4348ae
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> Colin
>>>>>>
>>>>>> On Sunday, 27 July 2014 17:53:29 UTC+1, Valentin wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I am trying to use this aggregation which does not work:
>>>>>>> "aggs": {
>>>>>>> "winners": {
>>>>>>> "terms": {
>>>>>>> "field": "urls",
>>>>>>> "order": {
>>>>>>> "diff": "desc"
>>>>>>> }
>>>>>>> },
>>>>>>> "aggs": {
>>>>>>> "diff": {
>>>>>>> "sum": {
>>>>>>> "script": "(doc['datetime'].date.getMillis() <
>>>>>>> 1406332800000) ? -1 : 1",
>>>>>>> "lang": "groovy"
>>>>>>> }
>>>>>>> }
>>>>>>> }
>>>>>>> }
>>>>>>> }
>>>>>>>
>>>>>>> Can anyone help?
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Valentin
>>>>>>>
>>>>>>
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1940d9b0-3a8e-4639-9b45-14230c009d21%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
