Sailesh Mukil has posted comments on this change. Change subject: IMPALA-2626: In-flight queries fail when statestore comes back online. ......................................................................
Patch Set 1: (1 comment) > (1 comment) > > I think this only fixes a particular instance of the problem: if > the statestore hasn't yet got updates from all the subscribers, it > will send a partial update which will have roughly the same effect > (since most queries run on all machines). > > Doesn't the statestore give a list of deletions with an update? > Presumably if it restarts, it won't send deletions for any entries > because it never knew they existed. The subscriber could only > cancel queries on nodes for which there is an actual deletion (i.e. > the node was known to have failed), but not include the missing > nodes in any new scheduling decisions. Yes you're right, it does only fix the problem if the statestore's first callback after coming back up is empty, I've mentioned that as the last para of the commit message. If the statestore comes back up and gets updates only from a few subscribers, it sends a partial update. But it's hard to determine at that point if this callback is a partial update, or if it is the complete update which means all the hosts that are not in the update actually went down. Due to this ambiguity, we handle only the empty updates case. Before this patch when the statestore sends an empty update, the known_backend_ map gets cleared. So all queries get cancelled. The deletion picks out individual backends from the map, but it doesn't matter if the map is empty. In short, if a backend is not in the known_backends_ map, the queries running on that backend are cancelled. http://gerrit.cloudera.org:8080/#/c/1380/1/be/src/service/impala-server.cc File be/src/service/impala-server.cc: Line 1377: if (!delta.is_delta) { > prefer Done -- To view, visit http://gerrit.cloudera.org:8080/1380 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: comment Gerrit-Change-Id: I102391ab63270a9686cf45457b8384ffcd2abe8a Gerrit-PatchSet: 1 Gerrit-Project: Impala Gerrit-Branch: cdh5-trunk Gerrit-Owner: Sailesh Mukil <[email protected]> Gerrit-Reviewer: Henry Robinson <[email protected]> Gerrit-Reviewer: Sailesh Mukil <[email protected]> Gerrit-Reviewer: Tim Armstrong <[email protected]> Gerrit-HasComments: Yes
