Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21367 )
Change subject: IMPALA-13020 (part 1): Change thrift_rpc_max_message_size to int64_t ...................................................................... IMPALA-13020 (part 1): Change thrift_rpc_max_message_size to int64_t Thrift 0.16.0 introduced a max message size to protect receivers against a malicious message allocating large amounts of memory. That limit is a 32-bit signed integer, so the max value is 2GB. Impala introduced the thrift_rpc_max_message_size startup option to set that for Impala's thrift servers. There are times when Impala wants to send a message that is larger than 2GB. In particular, the catalog-update topic for the statestore can exceed 2GBs when there is a lot of metadata loaded using the old v1 catalog. When there is a 2GB max message size, the statestore can create and send a >2GB message, but the impalads will reject it. This can lead to impalads having stale metadata. This switches to a patched Thrift that uses an int64_t for the max message size for C++ code. It does not modify the limit. The MaxMessageSize error was being swallowed in TAcceptQueueServer.cpp, so this fixes that location to always print MaxMessageSize exceptions. This is only patching the Thrift C++ library. It does not patch the Thrift Java library. There are a few reasons for that: - This specific issue involves C++ to C++ communication and will be solved by patching the C++ library. - C++ is easy to patch as it is built via the native-toolchain. There is no corresponding mechanism for patching our Java dependencies (though one could be developed). - Java modifications have implications for other dependencies like Hive which use Thrift to communicate with HMS. For the Java code that uses max message size, this converts the 64-bit value to 32-bit value by capping the value at Integer.MAX_VALUE. Testing: - Added enough tables to produce a >2GB catalog-topic and restarted an impalad with a higher limit specific. Without the patch, the catalog-topic update would be rejected by the impalad. With the patch, it succeeds. Change-Id: I681b1849cc565dcb25de8c070c18776ce69cbb87 Reviewed-on: http://gerrit.cloudera.org:8080/21367 Reviewed-by: Michael Smith <[email protected]> Reviewed-by: Joe McDonnell <[email protected]> Tested-by: Joe McDonnell <[email protected]> --- M be/src/common/init.cc M be/src/rpc/TAcceptQueueServer.cpp M be/src/rpc/thrift-server-test.cc M be/src/rpc/thrift-util.cc M be/src/rpc/thrift-util.h M be/src/util/backend-gflag-util.cc M bin/impala-config.sh M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/service/BackendConfig.java 9 files changed, 28 insertions(+), 18 deletions(-) Approvals: Michael Smith: Looks good to me, but someone else must approve Joe McDonnell: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/21367 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I681b1849cc565dcb25de8c070c18776ce69cbb87 Gerrit-Change-Number: 21367 Gerrit-PatchSet: 6 Gerrit-Owner: Joe McDonnell <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]> Gerrit-Reviewer: Quanlong Huang <[email protected]>
