ElasticSearch clusters has 12 nodes, 12 shards and number_of_replicas is 0.
A Storm topology sends MultiSearch requests to ElasticSearch cluster in
parallel using 10 Storm nodes.
The problem is the values of *response.getHits().getTotalHits()* and
*response.getHits().getHits().length* are different for some of responses.
(response.getHits().getTotalHits() > 0 and
response.getHits().getHits().length ==0)
So SearchResponse objects are in inconsistent state.
It is a serious issue for our architecture because
(response.getHits().getTotalHits() > 0) means that there is a matched
document but we cannot process it.
The queries are "filtered term queries".
Source Code:
public List<FooBean> queryBulk(List<String> matchQueries) {
MultiSearchRequestBuilder multiSearch =
_client.prepareMultiSearch();
for (String query : matchQueries) {
SearchRequestBuilder search = _client.prepareSearch(_index);
search.setTypes(_type);
search.setPreference("_primary");
search.setVersion(true);
search.setQuery(query);
search.setSize(1);
multiSearch.add(search);
}
List<FooBean> ret = new ArrayList<FooBean>(matchQueries.size());
MultiSearchResponse sr = multiSearch.execute().actionGet();
MultiSearchResponse.Item[] items = sr.getResponses();
int number_of_results = 0;
int number_of_success = 0;
int number_of_failure = 0;
for (int i = 0; i < items.length; i++)
{
SearchResponse response = null;
if (items[i] != null) response = items[i].getResponse();
SearchHits hits = null;
if (response != null) hits = response.getHits();
long noOfHits = 0;
if (hits != null) noOfHits = hits.getTotalHits();
number_of_results++;
if (noOfHits <= 0) {
ret.add(null);
} else {
if (hits.getHits().length > 0) {
SearchHit firstHit = hits.getAt(0);
VoterBean result = new FooBean(firstHit.getId(),
firstHit.getVersion(), firstHit.getSourceAsString());
number_of_success++;
ret.add(result);
} else {
LOG.info("Inconsistent State : No of Hits =" + noOfHits
+ " but content =" + hits.getHits().length + " . Query = " +
matchQueries.get(i));
number_of_failure++;
ret.add(null);
}
}
}
if (number_of_failure > 0)
{
LOG.info("MultiSearchResponse completed. Number_of_results: " +
number_of_results +
" / number_of_success: " + number_of_success + " /
number_of_failure: " + number_of_failure);
}
return ret;
}
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/086d140b-4221-421b-bbd8-72a965348cf4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
