Alexey Serbin has uploaded this change for review. (
http://gerrit.cloudera.org:8080/19231
Change subject: WIP: KUDU-3357 endpoints for proxied RPCs
......................................................................
WIP: KUDU-3357 endpoints for proxied RPCs
This patch introduces a solution to the problem outlined in KUDU-3357.
The idea is to establish separate RPC endpoint(s) for Kudu servers to
handle traffic proxied from external network(s). So, when a Kudu server
receives an RPC request, it has enough information to decide whether
to handle the request as arriving from the internal or some external
network. All the communications of Kudu components in the cluster
should be routed through the standard RPC endpoints, but the requests
proxied from external networks should be routed through those dedicated
RPC endpoints. When a Kudu server receives an RPC through such an
endpoint, it can substitute internal RPC addresses of Kudu servers with
corresponding RPC addresses reachable to the client through a TCP proxy.
With that, the following new flags have been introduced, both accepting
comma-separated list of strings of form <dns_name|ip_addr>:<port>:
--rpc_proxy_advertised_addresses
That's to set the server's RPC endpoints exposed to the outer
network via a TCP proxy.
--rpc_proxied_addresses
That's to define RPC endpoints in the inner network to handle
RPC requests forwarded/proxied from outside networks. It's possible
to use a wildcard for IP address (i.e. 0.0.0.0)
and the port number (i.e. 0) for the elements of this address list.
The newly introduced --rpc_proxy_advertised_addresses is orthogonal
to already existing --rpc_advertised_addresses, so it's possible to use
both simultaneously if the network environment for Docker containers
in the private internal network is configured in a funny way.
This approach allows for separating the internal and the external
traffic, meanwhile providing the connectivity for Kudu clients running
in outer networks, where the internal traffic is never routed through
a proxy's or a loadbalancer's endpoint. The approach with having only
--rpc_advertised_addresses for public cloud deployments (referred
by KUDU-3357) routes _all_ the Kudu traffic through the endpoints
exposed by the proxy/loadbalancer, and that's the problem this
patch addresses.
I verified this patch works as expected in k8s environment running in
AWS/EC2 cloud where Kudu cluster was deployed in a containerized manner
using Kudu Docker images. In particular, RPC calls from a client
running in the outer network (I was running it from my laptop behind
a firewall) were forwarded/proxied via a TCP proxy (NGINX) to Kudu
servers running in a AWS cluster deployed behind a load balancer.
I used the "kudu perf loadgen" CLI tool to create tables and write
data, and "kudu perf scan" to read data. A test Kudu Java client
application worked as well.
NOTE: even if "kudu cluster ksck" tool also worked, it's not yet a goal
to be able to run "kudu cluster ksck" and other administrative
tools from the outside; those tasks are expected to be performed
from within Kudu cluster's internal network
WIP:
* collect initial feedback on this approach
* add test scenarios at least to verify that tablet server and master
RPC endpoints are substituted with the proxy advertised RPC
addresses when a request comes from an endpoint dedicated to serve
proxied RPCs
* address TODO in CatalogManager::BuildLocationsForTablet()
* address TODO in RpcServer::Bind()
(should be done in a separate changelist?)
* substitute webserver's addresses as well for proxied RPCs
(should be done in a separate changelist?)
* the multi-master setup should be handled properly as well
(should be done in a separate changelist?)
Change-Id: Ic300250556d3f6e522a71923bed6aa5cd45375ea
---
M src/kudu/client/client-test.cc
M src/kudu/common/wire_protocol.proto
M src/kudu/integration-tests/consistency-itest.cc
M src/kudu/integration-tests/create-table-stress-test.cc
M src/kudu/integration-tests/registration-test.cc
M src/kudu/integration-tests/table_locations-itest.cc
M src/kudu/master/auto_rebalancer.cc
M src/kudu/master/catalog_manager.cc
M src/kudu/master/catalog_manager.h
M src/kudu/master/master-test-util.h
M src/kudu/master/master.cc
M src/kudu/master/master.h
M src/kudu/master/master_service.cc
M src/kudu/master/ts_descriptor.cc
M src/kudu/master/ts_descriptor.h
M src/kudu/server/rpc_server.cc
M src/kudu/server/rpc_server.h
M src/kudu/tserver/heartbeater.cc
18 files changed, 281 insertions(+), 52 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/31/19231/1
--
To view, visit http://gerrit.cloudera.org:8080/19231
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ic300250556d3f6e522a71923bed6aa5cd45375ea
Gerrit-Change-Number: 19231
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <[email protected]>
