Hello Quanlong Huang, Riza Suminto, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21420
to look at the new patch set (#3).
Change subject: IMPALA-13020 (part 2): Split out external vs internal Thrift
max message size
......................................................................
IMPALA-13020 (part 2): Split out external vs internal Thrift max message size
The Thrift max message size is designed to protect against malicious
messages that consume a lot of memory on the receiver. This is an
important security measure for externally facing services, but it
can interfere with internal communication within the cluster.
Currently, the max message size is controlled by a single startup
flag for both. This puts tensions between having a low value to
protect against malicious messages versus having a high value to
avoid issues with internal communication (e.g. large statestore
updates).
This introduces a new flag thrift_external_rpc_max_message_size to
specify the limit for externally-facing services. The current
thrift_rpc_max_message_size now applies only for internal services.
Splitting them apart allows setting a much higher value for
internal services (64GB) while leaving the externally facing services
using the current 2GB limit.
This modifies various code locations that wrap a Thrift transport to
pass in the original transport's TConfiguration. This also adds DCHECKs
to make sure that the new transport inherits the max message size. This
limits the locations where we actually need to set max message size.
ThriftServer/ThriftServerBuilder have a setting "is_external_facing"
which can be specified on each ThriftServer. This modifies statestore
and catalog to set is_external_facing to false. All other servers stay
with the default of true.
Testing:
- This adds a test case to verify that is_external_facing uses the
higher limit.
- Ran through the steps in testdata/scale_test_metadata/README.md
and updated the value in that doc.
- Created many tables to push the catalog-update topic to be >2GB
and verified that statestore successfully sends it when an impalad
restarts.
Change-Id: Ib9a649ef49a8a99c7bd9a1b73c37c4c621661311
---
M be/src/catalog/catalogd-main.cc
M be/src/common/init.cc
M be/src/rpc/TAcceptQueueServer.cpp
M be/src/rpc/TAcceptQueueServer.h
M be/src/rpc/authentication.cc
M be/src/rpc/thrift-client.cc
M be/src/rpc/thrift-server-test.cc
M be/src/rpc/thrift-server.cc
M be/src/rpc/thrift-server.h
M be/src/rpc/thrift-util.cc
M be/src/rpc/thrift-util.h
M be/src/statestore/statestore-subscriber.cc
M be/src/statestore/statestore.cc
M be/src/transport/THttpTransport.cpp
M be/src/transport/TSaslServerTransport.cpp
M be/src/transport/TSaslTransport.cpp
M be/src/util/backend-gflag-util.cc
M be/src/util/parquet-reader.cc
M testdata/scale_test_metadata/README.md
M testdata/scale_test_metadata/create-wide-table.sql
20 files changed, 177 insertions(+), 56 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/21420/3
--
To view, visit http://gerrit.cloudera.org:8080/21420
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib9a649ef49a8a99c7bd9a1b73c37c4c621661311
Gerrit-Change-Number: 21420
Gerrit-PatchSet: 3
Gerrit-Owner: Joe McDonnell <joemcdonn...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <joemcdonn...@cloudera.com>
Gerrit-Reviewer: Michael Smith <michael.sm...@cloudera.com>
Gerrit-Reviewer: Quanlong Huang <huangquanl...@gmail.com>
Gerrit-Reviewer: Riza Suminto <riza.sumi...@cloudera.com>
