Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21420 )
Change subject: IMPALA-13020 (part 2): Split out external vs internal Thrift max message size ...................................................................... IMPALA-13020 (part 2): Split out external vs internal Thrift max message size The Thrift max message size is designed to protect against malicious messages that consume a lot of memory on the receiver. This is an important security measure for externally facing services, but it can interfere with internal communication within the cluster. Currently, the max message size is controlled by a single startup flag for both. This puts tensions between having a low value to protect against malicious messages versus having a high value to avoid issues with internal communication (e.g. large statestore updates). This introduces a new flag thrift_external_rpc_max_message_size to specify the limit for externally-facing services. The current thrift_rpc_max_message_size now applies only for internal services. Splitting them apart allows setting a much higher value for internal services (64GB) while leaving the externally facing services using the current 2GB limit. This modifies various code locations that wrap a Thrift transport to pass in the original transport's TConfiguration. This also adds DCHECKs to make sure that the new transport inherits the max message size. This limits the locations where we actually need to set max message size. ThriftServer/ThriftServerBuilder have a setting "is_external_facing" which can be specified on each ThriftServer. This modifies statestore and catalog to set is_external_facing to false. All other servers stay with the default of true. Testing: - This adds a test case to verify that is_external_facing uses the higher limit. - Ran through the steps in testdata/scale_test_metadata/README.md and updated the value in that doc. - Created many tables to push the catalog-update topic to be >2GB and verified that statestore successfully sends it when an impalad restarts. Change-Id: Ib9a649ef49a8a99c7bd9a1b73c37c4c621661311 Reviewed-on: http://gerrit.cloudera.org:8080/21420 Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Reviewed-by: Riza Suminto <riza.sumi...@cloudera.com> Reviewed-by: Michael Smith <michael.sm...@cloudera.com> --- M be/src/catalog/catalogd-main.cc M be/src/common/init.cc M be/src/rpc/TAcceptQueueServer.cpp M be/src/rpc/TAcceptQueueServer.h M be/src/rpc/authentication.cc M be/src/rpc/thrift-client.cc M be/src/rpc/thrift-server-test.cc M be/src/rpc/thrift-server.cc M be/src/rpc/thrift-server.h M be/src/rpc/thrift-util.cc M be/src/rpc/thrift-util.h M be/src/statestore/statestore-subscriber.cc M be/src/statestore/statestore.cc M be/src/transport/THttpTransport.cpp M be/src/transport/TSaslServerTransport.cpp M be/src/transport/TSaslTransport.cpp M be/src/util/backend-gflag-util.cc M be/src/util/parquet-reader.cc M testdata/scale_test_metadata/README.md M testdata/scale_test_metadata/create-wide-table.sql 20 files changed, 177 insertions(+), 56 deletions(-) Approvals: Impala Public Jenkins: Verified Riza Suminto: Looks good to me, but someone else must approve Michael Smith: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/21420 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ib9a649ef49a8a99c7bd9a1b73c37c4c621661311 Gerrit-Change-Number: 21420 Gerrit-PatchSet: 4 Gerrit-Owner: Joe McDonnell <joemcdonn...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Joe McDonnell <joemcdonn...@cloudera.com> Gerrit-Reviewer: Michael Smith <michael.sm...@cloudera.com> Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com> Gerrit-Reviewer: Riza Suminto <riza.sumi...@cloudera.com>