Sailesh Mukil has uploaded a new change for review. http://gerrit.cloudera.org:8080/7425
Change subject: IMPALA-4875: Bound maximum statestore topic update size to 2GB. ...................................................................... IMPALA-4875: Bound maximum statestore topic update size to 2GB. This patch enforces a maximum topic update size for the statestore to be 2GB. If that limit is exceeded, the statestore will split topic updates over several RPCs to the subscriber. If a single topic key + value exceeds the limit, it will be dropped by the statestore when the subscriber tries to publish it. Doing so introduces the possibility that a catalog update (all updates pertaining to a single catalog version) might be split across two separate UpdateState() RPCs. To ensure that all updates are flushed to the catalog FE at the same time, we buffer updates until a new object of type CATALOG is received. Remove the special treatment of deletions in the UpdateState() RPC. Previously they were not sent with their version. New statestore tests added to ensure: * Update sizes don't exceed the configured max. * 'from' version is properly handled (not previously tested) so that subscribers can process only a prefix of the update log. TODO : Check min subscriber version Change-Id: Icbe42704d3c98d9619e3737786b75ee67fb2a264 --- M be/src/catalog/catalog-server.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/scheduling/scheduler-test-util.cc M be/src/scheduling/scheduler.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/statestore/statestore-test.cc M be/src/statestore/statestore.cc M be/src/statestore/statestore.h M common/thrift/StatestoreService.thrift 11 files changed, 308 insertions(+), 163 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/25/7425/1 -- To view, visit http://gerrit.cloudera.org:8080/7425 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-MessageType: newchange Gerrit-Change-Id: Icbe42704d3c98d9619e3737786b75ee67fb2a264 Gerrit-PatchSet: 1 Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-Owner: Sailesh Mukil <[email protected]> Gerrit-Reviewer: Henry Robinson <[email protected]> Gerrit-Reviewer: Michael Ho <[email protected]>
