Aihua Xu created HDFS-15905:
-------------------------------
Summary: Improve Router performance with router redirection
Key: HDFS-15905
URL: https://issues.apache.org/jira/browse/HDFS-15905
Project: Hadoop HDFS
Issue Type: New Feature
Components: rbf
Affects Versions: 3.1.0
Reporter: Aihua Xu
Assignee: Aihua Xu
Router implementation currently takes the proxy approach to handle the client
requests: the routers receive the requests from the clients and send the
requests to the target clusters on behalf of the clients.
This approach works well, while after moving more clusters on top of routers,
we are seeing that routers are becoming the bottleneck since e.g., without RBF,
the clients themselves manage the connections for themselves, while with RBF,
the limited routers manage much more connections for the clients; we also keep
idle connections to boost the connection performance. We have done some work to
tune connection management but it doesn't help much.
We are proposing to reduce the functionality on the router side and use them as
actual router instead of proxy: the clients talk to routers to resolve target
cluster info given a path and get router delegation token; the clients directly
send the requests to target cluster.
A big challenge here is the token authentication against target cluster with
router token only. One approach: we can ask router to return target cluster
token along with router token so the clients can authenticate against target
cluster. Second approach: similar to block token mechanism, the router
exchanges secret keys with target clusters through heart-beats so the clients
can authenticate with target cluster with that router token.
I would like to know your feedback.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
