[kudu-CR] WIP: KUDU-3357 endpoints for proxied RPCs
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/19231 ) Change subject: WIP: KUDU-3357 endpoints for proxied RPCs .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/19231/1/src/kudu/server/rpc_server.cc File src/kudu/server/rpc_server.cc: http://gerrit.cloudera.org:8080/#/c/19231/1/src/kudu/server/rpc_server.cc@270 PS1, Line 270: already > nit: double 'already' Done http://gerrit.cloudera.org:8080/#/c/19231/1/src/kudu/server/rpc_server.cc@361 PS1, Line 361: && > shouldn't this be || ? Well, that actually should be checking just for server_state against UNINITIALIZED since rpc_proxied_addresses_ is populated in RpcServer::Init(). Done. -- To view, visit http://gerrit.cloudera.org:8080/19231 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic300250556d3f6e522a71923bed6aa5cd45375ea Gerrit-Change-Number: 19231 Gerrit-PatchSet: 1 Gerrit-Owner: Alexey Serbin Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Alexey Serbin Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Zoltan Chovan Gerrit-Comment-Date: Fri, 11 Nov 2022 23:19:39 + Gerrit-HasComments: Yes
[kudu-CR] WIP: KUDU-3357 endpoints for proxied RPCs
Hello Zoltan Chovan, Kurt Deschler, Attila Bukor, Kudu Jenkins, Abhishek Chennaka, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/19231 to look at the new patch set (#2). Change subject: WIP: KUDU-3357 endpoints for proxied RPCs .. WIP: KUDU-3357 endpoints for proxied RPCs This patch introduces a solution to the problem outlined in KUDU-3357. The idea is to establish separate RPC endpoint(s) for Kudu servers to handle traffic proxied from external network(s). So, when a Kudu server receives an RPC request, it has enough information to decide whether to handle the request as arriving from the internal or some external network. All the communications of Kudu components in the cluster should be routed through the standard RPC endpoints, but the requests proxied from external networks should be routed through those dedicated RPC endpoints. When a Kudu server receives an RPC through such an endpoint, it can substitute internal RPC addresses of Kudu servers with corresponding RPC addresses reachable to the client through a TCP proxy. With that, the following new flags have been introduced, both accepting comma-separated list of strings of form :: --rpc_proxy_advertised_addresses That's to set the server's RPC endpoints exposed to the outer network via a TCP proxy. --rpc_proxied_addresses That's to define RPC endpoints in the inner network to handle RPC requests forwarded/proxied from outside networks. It's possible to use a wildcard for IP address (i.e. 0.0.0.0) and the port number (i.e. 0) for the elements of this address list. The newly introduced --rpc_proxy_advertised_addresses is orthogonal to already existing --rpc_advertised_addresses, so it's possible to use both simultaneously if the network environment for Docker containers in the private internal network is configured in a funny way. This approach allows for separating the internal and the external traffic, meanwhile providing the connectivity for Kudu clients running in outer networks, where the internal traffic is never routed through a proxy's or a loadbalancer's endpoint. The approach with having only --rpc_advertised_addresses for public cloud deployments (referred by KUDU-3357) routes _all_ the Kudu traffic through the endpoints exposed by the proxy/loadbalancer, and that's the problem this patch addresses. I verified this patch works as expected in k8s environment running in AWS/EC2 cloud where Kudu cluster was deployed in a containerized manner using Kudu Docker images. In particular, RPC calls from a client running in the outer network (I was running it from my laptop behind a firewall) were forwarded/proxied via a TCP proxy (NGINX) to Kudu servers running in a AWS cluster deployed behind a load balancer. I used the "kudu perf loadgen" CLI tool to create tables and write data, and "kudu perf scan" to read data. A test Kudu Java client application worked as well. NOTE: even if "kudu cluster ksck" tool also worked, it's not yet a goal to be able to run "kudu cluster ksck" and other administrative tools from the outside; those tasks are expected to be performed from within Kudu cluster's internal network WIP: * collect initial feedback on this approach * add test scenarios at least to verify that tablet server and master RPC endpoints are substituted with the proxy advertised RPC addresses when a request comes from an endpoint dedicated to serve proxied RPCs * address TODO in RpcServer::Bind() (should be done in a separate changelist?) * substitute webserver's addresses as well for proxied RPCs (should be done in a separate changelist?) * the multi-master setup should be handled properly as well (should be done in a separate changelist?) Change-Id: Ic300250556d3f6e522a71923bed6aa5cd45375ea --- M src/kudu/client/client-test.cc M src/kudu/common/wire_protocol.proto M src/kudu/integration-tests/consistency-itest.cc M src/kudu/integration-tests/create-table-stress-test.cc M src/kudu/integration-tests/registration-test.cc M src/kudu/integration-tests/table_locations-itest.cc M src/kudu/master/auto_rebalancer.cc M src/kudu/master/catalog_manager.cc M src/kudu/master/catalog_manager.h M src/kudu/master/master-test-util.h M src/kudu/master/master.cc M src/kudu/master/master.h M src/kudu/master/master_service.cc M src/kudu/master/ts_descriptor.cc M src/kudu/master/ts_descriptor.h M src/kudu/server/rpc_server.cc M src/kudu/server/rpc_server.h M src/kudu/tserver/heartbeater.cc 18 files changed, 332 insertions(+), 94 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/31/19231/2 -- To view, visit http://gerrit.cloudera.org:8080/19231 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic300250556d3f6e522a71923bed6aa5cd45375ea Gerrit-Chang
[kudu-CR] WIP: KUDU-3357 endpoints for proxied RPCs
Zoltan Chovan has posted comments on this change. ( http://gerrit.cloudera.org:8080/19231 ) Change subject: WIP: KUDU-3357 endpoints for proxied RPCs .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/19231/1/src/kudu/server/rpc_server.cc File src/kudu/server/rpc_server.cc: http://gerrit.cloudera.org:8080/#/c/19231/1/src/kudu/server/rpc_server.cc@270 PS1, Line 270: already nit: double 'already' http://gerrit.cloudera.org:8080/#/c/19231/1/src/kudu/server/rpc_server.cc@361 PS1, Line 361: && shouldn't this be || ? -- To view, visit http://gerrit.cloudera.org:8080/19231 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic300250556d3f6e522a71923bed6aa5cd45375ea Gerrit-Change-Number: 19231 Gerrit-PatchSet: 1 Gerrit-Owner: Alexey Serbin Gerrit-Reviewer: Abhishek Chennaka Gerrit-Reviewer: Attila Bukor Gerrit-Reviewer: Kudu Jenkins (120) Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Zoltan Chovan Gerrit-Comment-Date: Fri, 11 Nov 2022 10:31:38 + Gerrit-HasComments: Yes
[kudu-CR] WIP: KUDU-3357 endpoints for proxied RPCs
Alexey Serbin has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19231 Change subject: WIP: KUDU-3357 endpoints for proxied RPCs .. WIP: KUDU-3357 endpoints for proxied RPCs This patch introduces a solution to the problem outlined in KUDU-3357. The idea is to establish separate RPC endpoint(s) for Kudu servers to handle traffic proxied from external network(s). So, when a Kudu server receives an RPC request, it has enough information to decide whether to handle the request as arriving from the internal or some external network. All the communications of Kudu components in the cluster should be routed through the standard RPC endpoints, but the requests proxied from external networks should be routed through those dedicated RPC endpoints. When a Kudu server receives an RPC through such an endpoint, it can substitute internal RPC addresses of Kudu servers with corresponding RPC addresses reachable to the client through a TCP proxy. With that, the following new flags have been introduced, both accepting comma-separated list of strings of form :: --rpc_proxy_advertised_addresses That's to set the server's RPC endpoints exposed to the outer network via a TCP proxy. --rpc_proxied_addresses That's to define RPC endpoints in the inner network to handle RPC requests forwarded/proxied from outside networks. It's possible to use a wildcard for IP address (i.e. 0.0.0.0) and the port number (i.e. 0) for the elements of this address list. The newly introduced --rpc_proxy_advertised_addresses is orthogonal to already existing --rpc_advertised_addresses, so it's possible to use both simultaneously if the network environment for Docker containers in the private internal network is configured in a funny way. This approach allows for separating the internal and the external traffic, meanwhile providing the connectivity for Kudu clients running in outer networks, where the internal traffic is never routed through a proxy's or a loadbalancer's endpoint. The approach with having only --rpc_advertised_addresses for public cloud deployments (referred by KUDU-3357) routes _all_ the Kudu traffic through the endpoints exposed by the proxy/loadbalancer, and that's the problem this patch addresses. I verified this patch works as expected in k8s environment running in AWS/EC2 cloud where Kudu cluster was deployed in a containerized manner using Kudu Docker images. In particular, RPC calls from a client running in the outer network (I was running it from my laptop behind a firewall) were forwarded/proxied via a TCP proxy (NGINX) to Kudu servers running in a AWS cluster deployed behind a load balancer. I used the "kudu perf loadgen" CLI tool to create tables and write data, and "kudu perf scan" to read data. A test Kudu Java client application worked as well. NOTE: even if "kudu cluster ksck" tool also worked, it's not yet a goal to be able to run "kudu cluster ksck" and other administrative tools from the outside; those tasks are expected to be performed from within Kudu cluster's internal network WIP: * collect initial feedback on this approach * add test scenarios at least to verify that tablet server and master RPC endpoints are substituted with the proxy advertised RPC addresses when a request comes from an endpoint dedicated to serve proxied RPCs * address TODO in CatalogManager::BuildLocationsForTablet() * address TODO in RpcServer::Bind() (should be done in a separate changelist?) * substitute webserver's addresses as well for proxied RPCs (should be done in a separate changelist?) * the multi-master setup should be handled properly as well (should be done in a separate changelist?) Change-Id: Ic300250556d3f6e522a71923bed6aa5cd45375ea --- M src/kudu/client/client-test.cc M src/kudu/common/wire_protocol.proto M src/kudu/integration-tests/consistency-itest.cc M src/kudu/integration-tests/create-table-stress-test.cc M src/kudu/integration-tests/registration-test.cc M src/kudu/integration-tests/table_locations-itest.cc M src/kudu/master/auto_rebalancer.cc M src/kudu/master/catalog_manager.cc M src/kudu/master/catalog_manager.h M src/kudu/master/master-test-util.h M src/kudu/master/master.cc M src/kudu/master/master.h M src/kudu/master/master_service.cc M src/kudu/master/ts_descriptor.cc M src/kudu/master/ts_descriptor.h M src/kudu/server/rpc_server.cc M src/kudu/server/rpc_server.h M src/kudu/tserver/heartbeater.cc 18 files changed, 281 insertions(+), 52 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/31/19231/1 -- To view, visit http://gerrit.cloudera.org:8080/19231 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ic300250556d3f6e522a71923bed6aa5cd45375ea Gerrit-Change-Number: 19231 Gerrit-PatchSet: 1 Gerrit-Owner: Alexey Ser
[kudu-CR] wip KUDU-3357
Andrew Wong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18478 Change subject: wip KUDU-3357 .. wip KUDU-3357 - RPC advertised addresses are now exposed through a different channel than the 'rpc_addresses' of the ServerRegistrationPB. WIP: - don't add a new field - instead, reuse rpc_addresses, and add a flag that dictates whether to resolve addresses - advertised_addresses should be sent to clients - need to distinguish when to use the advertised addresses - it's not necessarily the case that we want to use internal addresses for tserver traffic - maybe add flag where tserver is allowed to not expose internal address? Change-Id: I19025048dbc9bada3704c04140b131b7b15c5b96 --- M src/kudu/client/client.cc M src/kudu/client/meta_cache.cc M src/kudu/common/wire_protocol.proto M src/kudu/integration-tests/dns_alias-itest.cc M src/kudu/master/catalog_manager.cc M src/kudu/master/ts_descriptor.cc M src/kudu/mini-cluster/external_mini_cluster.cc M src/kudu/mini-cluster/external_mini_cluster.h M src/kudu/server/rpc_server.cc M src/kudu/server/rpc_server.h M src/kudu/server/server_base.cc M src/kudu/tserver/heartbeater.cc 12 files changed, 183 insertions(+), 32 deletions(-) git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/78/18478/1 -- To view, visit http://gerrit.cloudera.org:8080/18478 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: kudu Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I19025048dbc9bada3704c04140b131b7b15c5b96 Gerrit-Change-Number: 18478 Gerrit-PatchSet: 1 Gerrit-Owner: Andrew Wong
