[
https://issues.apache.org/jira/browse/HADOOP-16119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16785610#comment-16785610
]
He Xiaoqiao edited comment on HADOOP-16119 at 3/6/19 1:12 PM:
--------------------------------------------------------------
[~jojochuang] I would like to offer issues about current version of KMS that I
meet in practice.
# Scalability: now it is difficult to scale KMS instance friendly since
delegation token and all key data are isolated between different KMS instances
at all.
# Transparent: KMSClient has to upgrade the configuration even when add one
KMS instance.
# HA: it seems that KMS instances is peer-to-peer arch. but client has to try
one by one util success if some one fault. the cost is very high.
# Data Consistency: KMS instance manages key by Java KeyStore isolated, KMS
client request to create key to all KMS instances serially, if one of them
failed for some reason, create-request will throw exception and key in KeyStore
of different instances will be same completely, also no check background as far
as I know.
Some of them are also mentioned and as exit criteria in [^Design doc_ KMS
v2.pdf] via [~jojochuang].
I think the core issue is no share-storage between different instance in one
word.
I propose to create a plugin ShareStore as file/dbms/zookeeper behind KMS
instance, and let KMS stateless. It seems work well using the share storage
reference RBF. Another side, we can also retain cache mechanism to improve the
performance.
[~jojochuang] please do correct me if I am wrong.
was (Author: hexiaoqiao):
[~jojochuang] I would like to offer issues about current version of KMS that I
meet in practice.
# Scalability: now it is difficult to scale KMS instance friendly since
delegation token and all key data are isolated between different KMS instances
at all.
# Transparent: KMSClient has to upgrade the configuration even when add one
KMS instance.
# HA: it seems that KMS instances is peer-to-peer arch. but client has to try
one by one util success if some one fault. the cost is very high.
# Data Consistency: KMS instance manages key by Java KeyStore isolated, KMS
client request to create key to all KMS instances serially, if one of them
failed for some reason, create-request will throw exception and key in KeyStore
of different instances will be same completely, also no check background as far
as I know.
I think the core issue is no share-storage between different instance in one
word.
I propose to create a plugin ShareStore as file/dbms/zookeeper behind KMS
instance, and let KMS stateless. It seems work well using the share storage
reference RBF. Another side, we can also retain cache mechanism to improve the
performance.
[~jojochuang] please do correct me if I am wrong.
> KMS on Hadoop RPC Engine
> ------------------------
>
> Key: HADOOP-16119
> URL: https://issues.apache.org/jira/browse/HADOOP-16119
> Project: Hadoop Common
> Issue Type: New Feature
> Reporter: Jonathan Eagles
> Assignee: Wei-Chiu Chuang
> Priority: Major
> Attachments: Design doc_ KMS v2.pdf
>
>
> Per discussion on common-dev and text copied here for ease of reference.
> https://lists.apache.org/thread.html/0e2eeaf07b013f17fad6d362393f53d52041828feec53dcddff04808@%3Ccommon-dev.hadoop.apache.org%3E
> {noformat}
> Thanks all for the inputs,
> To offer additional information (while Daryn is working on his stuff),
> optimizing RPC encryption opens up another possibility: migrating KMS
> service to use Hadoop RPC.
> Today's KMS uses HTTPS + REST API, much like webhdfs. It has very
> undesirable performance (a few thousand ops per second) compared to
> NameNode. Unfortunately for each NameNode namespace operation you also need
> to access KMS too.
> Migrating KMS to Hadoop RPC greatly improves its performance (if
> implemented correctly), and RPC encryption would be a prerequisite. So
> please keep that in mind when discussing the Hadoop RPC encryption
> improvements. Cloudera is very interested to help with the Hadoop RPC
> encryption project because a lot of our customers are using at-rest
> encryption, and some of them are starting to hit KMS performance limit.
> This whole "migrating KMS to Hadoop RPC" was Daryn's idea. I heard this
> idea in the meetup and I am very thrilled to see this happening because it
> is a real issue bothering some of our customers, and I suspect it is the
> right solution to address this tech debt.
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
