[
https://issues.apache.org/jira/browse/STORM-2898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323160#comment-16323160
]
Robert Joseph Evans commented on STORM-2898:
--------------------------------------------
*Storm Delegation Token Design.*
The goal is to provide a clean and secure way for workers to be able to
authenticate to Nimbus, Supervisors, and DRPC servers without the need for
kerberos credentials.
First off rolling your own security is almost always a bad idea. Even in this
case I would argue that this is a far from ideal situation, but we don't have
an out of the box option that does what we want to do so we are going to base
our solution off of Hadoop that has a working, although not ideal,
implementation of something similar. It would be great if we could just rely
on the hadoop-common library, but their implementation relies on the Hadoop
UserGroupInformation and Configuration classes that we don't currently use and
could cause a lot of configuration issues for our users if we started to try
and use them. We also want to make some modifications/improvements that are
either storm specific or Hadoop could not make because of backwards
incompatibilities.
*Hadoop:*
Hadoop uses symmetric key signing for their tokens. To manage these secrets
they employ a SecretManager
https://github.com/apache/hadoop/blob/2e0a451a8dd64ccd7199b2b435e1b6c704a9d3ae/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/SecretManager.java
implementation, usually some subclass of an AbstractDelegationTokenSecretManager
https://github.com/apache/hadoop/blob/2e0a451a8dd64ccd7199b2b435e1b6c704a9d3ae/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java
The secret manager will create private keys periodically that are persisted for
HA purposes. The most up to date secret key is used to create tokens when
needed, but a few older keys are kept to allow for verification to already
issues tokens. A token consists of a TokenIdentifier, a Service, a Kind, and a
Password.
The TokenIdentifier is the a blob of bytes that you as a service want to
protect and identifies the user/job/etc, and the id of the secret used.
The Service is who issued this token, and can be used to know who to talk to
with the token.
The Kind is sort of a sub-service, like hdfs vs web-hdfs.
The Password is the signature of the TokenIdentifier. The Service and Kind are
not included in this, because they are used to help find the proper token to
use as a client, not to actually verify anything. In this case the signature
is a HmacSHA1 hash of the secret manager's secret along with the bytes of the
TokenIdentifier
SASL verification happens using the DIGEST-MD5 mechanism. The user name is the
token identifier, and the password is the password/signature from the token.
To add an extra layer of security and allow for individual tokens to expire
before the private keys rotates out of existence most tokens are persisted to a
store and when verification happens it can not only verify that the
TokenIdentifier is signed correctly, but also that it is still active and valid.
*Differences:*
There are some differences between storm and Hadoop. First we don't create
nearly as many delegation tokens as Hadoop. Second our topologies are much
more likely to be very long lived. We will try to take these into account in
our design.
*Storm Proposal:*
I propose that for each topology that is submitted we generate 2 new secrets.
One of them would be for generating a DRPC token, and the second one would be
for generating a Nimbus/Supervisor token.
These secrets would be stored in zookeeper in a location that only nimbus and
the corresponding services could have access to along with the user name and
topology id associated with the secret.
Nimbus would then use the secrets to create tokens for the DRPC service and for
the Nimbus/Supervisor service. These tokens would be inserted into the
Credentials for the topology and distributed to the topology.
When a topology is killed the corresponding secrets would be removed from
zookeeper as part of the topology cleanup.
Some AutoStormToken class would be added as an AutoCredentials handler for the
worker that would take the tokens and put them into the current Subject where a
client could find them.
The KerberoseSaslTransportPlugin would be updated so it could support tokens as
well.
On the client side when deciding which mechanism to use it would first look for
a valid token in the current Subject. If one was found it would create a
DIGEST-MD5 mechanism and start to connect using the same mechanism as Hadoop
does for the user name and password.
On the Server side we would register both the GSSAPI(kerberos) and
DIGEST-MD5(token) SaslServer mechanisms so the client could select whichever
they wanted. If the token mechanism is used it would get the user
name/topology id, and then attempt to look up the corresponding secret. If it
could not find the secret the connection would be rejected. For performance
reasons we will likely cache the secrets locally on the servers and update them
when ZK indicates that something has changed or after a given timeout.
*Drawbacks/Concerns + Additional security:*
I would like us to eventually have the ability to rotate the private keys
periodically. If we are using credentials we should be able to send updated
keys to the workers regularly. As such we could support keeping the last two
private keys around for each topology along with an id for which key it is that
was used to sign it. Then we can rotate the private keys and issue new tokens
that would be propagated to the workers. That way if a token is compromised it
is only valid for a limited period of time. We should design all of the thrift
object associated with this in such a way that they can support this, even if
we are not going to do it at the beginning.
One of the drawbacks with the current design is that if you compromise a
Supervisor you have access to the actual private key, and could in theory
generate more tokens.
To mitigate this we could use a combination of ZK ACLs + encrypting the
private keys so only nimbus instances have access to them. Then only
publishing the tokens themselves in locations that the different services have
access.
However, if you compromise a supervisor you can get access to all of a
worker's credentials anyways so it is game over, and if you compromise a DRPC
server you can already do anything a DRPC server can. So I am not sure it will
slow down an attacker.
We may want to do it anyways because in the future we may move away form ZK
so it could limit the exposure.
We might also want to look at distinguishing in our authorization between a
token for a topology and a given user, so we can restrict the operations that a
user can do from a topology. I don't really want a topology trying to modify
itself while it is running.
DIGEST-MD5 is susceptible to some active attacks. As this is only coming from
the topology and nothing too serious is being exchanged in this case it is not
that scary and Hadoop uses this for a lot more scary things.
To mitigate this we should look into using privileged ports where ever
possible.
We might want to look into enabling encryption for the RPC traffic as well to
help counter any passive attacks. I'm not sure exactly how we would do it in a
backwards compatible way though.
*Alternatives Explored:*
We looked at using some kind of public/private key signing, but even though
that would work well for DRPC it would be a lot of extra work for
Nimbus/Supervisors with no added benefit.
> Strom should support auth through delegation tokens for workers
> ---------------------------------------------------------------
>
> Key: STORM-2898
> URL: https://issues.apache.org/jira/browse/STORM-2898
> Project: Apache Storm
> Issue Type: New Feature
> Components: storm-client, storm-server
> Affects Versions: 2.0.0
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
>
> There are a lot of cases where it would be great for a worker to be able to
> communicate directly to nimbus, supervisors, or drpc servers in a secure way
> out of the box.
> This is currently a pain to make work. The user has to ship a TGT with their
> topology, and continually keep it up to date with credentials-push. They
> also need a kind of hacked up jaas.conf to grab the TGT from AutoTGT and put
> it in the place that he client wants it.
> We should just generate a signed data structure (aka delegation token from
> hadoop) that we can had off to the topologies to use when talking to nimbus,
> a supervisor, or drpc servers.
> We may want to split up the different services from each other to make an
> attack against one not hit all of them, but that is something we can think
> about with the design of this.
> I will try to come up with a design shortly.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)