Hello All,
We are testing usergrid 2.x (master branch) for our application that was
previously being prototyped on usergrid 1.x. We are noticing some weird
anomalies which are causing errors in our application which otherwise works
fine against usergrid 1.x. Specifically, we are seeing empty responses when
querying custom collections for a particular entity record.
Following is an example of one such query:
http://server-name/b2perf1/default/userdata?client_id=
<...>&client_secret=<....>&ql=userproductid='4d543507-9839-11e5-ba08-0a75091e6d25~~5c856de9-9828-11e5-ba08-0a75091e6d25'
In the above scenario, we are querying a custom collection “userdata”. And
under high load conditions (performance tests), this query starts returning
an empty entities array (see below), even though this entity did exist at
one time and we have no code / logic to delete entities.
{
"action": "get",
"application": "0f7a2396-9826-11e5-ba08-0a75091e6d25",
"params": {
"ql": [
"userproductid='4d543507-9839-11e5-ba08-0a75091e6d25~~5c856de9-9828-11e5-ba08-0a75091e6d25'"
]
},
"path": "/userdata",
"uri": "http://localhost:8080/b2perf1/default/userdata";,
"entities": [],
"timestamp": 1449322746733,
"duration": 1053,
"organization": "b2perf1",
"applicationName": "default",
"count": 0
}
This has been happening quite randomly / intermittently and we have not
been able to isolate any replication steps besides, running load /
performance tests when this problem does eventually show up.
Note, the creation of the entities is prior to the load test and we can
confirm that they existed before running the load test.
We have never noticed this issue for ‘non’ query calls (ie calls that do
not directly provide a field to query on)
Our suspicion is that while these records do exist in Cassandra (because we
have never deleted them), but the ElasticSearch index is ‘not’ in sync or
is not functioning properly.
How do we go about debugging this problem? Is there any particular logging
or metric that we can check for us to confirm if all the elasticsearch
index is upto date with the changes in cassandra.
Any other suggestions will be greatly appreciated.
Thanks
Jaskaran
