[ 
https://issues.apache.org/jira/browse/STORM-2898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323160#comment-16323160
 ] 

Robert Joseph Evans commented on STORM-2898:
--------------------------------------------

*Storm Delegation Token Design.*

The goal is to provide a clean and secure way for workers to be able to 
authenticate to Nimbus, Supervisors, and DRPC servers without the need for 
kerberos credentials.

First off rolling your own security is almost always a bad idea.  Even in this 
case I would argue that this is a far from ideal situation, but we don't have 
an out of the box option that does what we want to do so we are going to base 
our solution off of Hadoop that has a working, although not ideal, 
implementation of something similar.  It would be great if we could just rely 
on the hadoop-common library, but their implementation relies on the Hadoop 
UserGroupInformation and Configuration classes that we don't currently use and 
could cause a lot of configuration issues for our users if we started to try 
and use them.  We also want to make some modifications/improvements that are 
either storm specific or Hadoop could not make because of backwards 
incompatibilities.

*Hadoop:*
Hadoop uses symmetric key signing for their tokens.  To manage these secrets 
they employ a SecretManager

https://github.com/apache/hadoop/blob/2e0a451a8dd64ccd7199b2b435e1b6c704a9d3ae/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/SecretManager.java

implementation, usually some subclass of an AbstractDelegationTokenSecretManager

https://github.com/apache/hadoop/blob/2e0a451a8dd64ccd7199b2b435e1b6c704a9d3ae/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/AbstractDelegationTokenSecretManager.java

The secret manager will create private keys periodically that are persisted for 
HA purposes.  The most up to date secret key is used to create tokens when 
needed, but a few older keys are kept to allow for verification to already 
issues tokens.  A token consists of a TokenIdentifier, a Service, a Kind, and a 
Password.

The TokenIdentifier is the a blob of bytes that you as a service want to 
protect and identifies the user/job/etc, and the id of the secret used.

The Service is who issued this token, and can be used to know who to talk to 
with the token.

The Kind is sort of a sub-service, like hdfs vs web-hdfs.

The Password is the signature of the TokenIdentifier.  The Service and Kind are 
not included in this, because they are used to help find the proper token to 
use as a client, not to actually verify anything.  In this case the signature 
is a HmacSHA1 hash of the secret manager's secret along with the bytes of the 
TokenIdentifier


SASL verification happens using the DIGEST-MD5 mechanism.  The user name is the 
token identifier, and the password is the password/signature from the token.

To add an extra layer of security and allow for individual tokens to expire 
before the private keys rotates out of existence most tokens are persisted to a 
store and when verification happens it can not only verify that the 
TokenIdentifier is signed correctly, but also that it is still active and valid.


*Differences:*

There are some differences between storm and Hadoop.  First we don't create 
nearly as many delegation tokens as Hadoop.  Second our topologies are much 
more likely to be very long lived.  We will try to take these into account in 
our design.  

*Storm Proposal:*

I propose that for each topology that is submitted we generate 2 new secrets.  
One of them would be for generating a DRPC token, and the second one would be 
for generating a Nimbus/Supervisor token.

These secrets would be stored in zookeeper in a location that only nimbus and 
the corresponding services could have access to along with the user name and 
topology id associated with the secret.

Nimbus would then use the secrets to create tokens for the DRPC service and for 
the Nimbus/Supervisor service.  These tokens would be inserted into the 
Credentials for the topology and distributed to the topology.

When a topology is killed the corresponding secrets would be removed from 
zookeeper as part of the topology cleanup.

Some AutoStormToken class would be added as an AutoCredentials handler for the 
worker that would take the tokens and put them into the current Subject where a 
client could find them.

The KerberoseSaslTransportPlugin would be updated so it could support tokens as 
well.


On the client side when deciding which mechanism to use it would first look for 
a valid token in the current Subject.  If one was found it would create a 
DIGEST-MD5 mechanism and start to connect using the same mechanism as Hadoop 
does for the user name and password.

On the Server side we would register both the GSSAPI(kerberos) and 
DIGEST-MD5(token) SaslServer mechanisms so the client could select whichever 
they wanted.  If the token mechanism is used it would get the user 
name/topology id, and then attempt to look up the corresponding secret.  If it 
could not find the secret the connection would be rejected.  For performance 
reasons we will likely cache the secrets locally on the servers and update them 
when ZK indicates that something has changed or after a given timeout.


*Drawbacks/Concerns + Additional security:*

I would like us to eventually have the ability to rotate the private keys 
periodically.  If we are using credentials we should be able to send updated 
keys to the workers regularly.  As such we could support keeping the last two 
private keys around for each topology along with an id for which key it is that 
was used to sign it.  Then we can rotate the private keys and issue new tokens 
that would be propagated to the workers.  That way if a token is compromised it 
is only valid for a limited period of time.  We should design all of the thrift 
object associated with this in such a way that they can support this, even if 
we are not going to do it at the beginning.

One of the drawbacks with the current design is that if you compromise a 
Supervisor you have access to the actual private key, and could in theory 
generate more tokens.
  To mitigate this we could use a combination of ZK ACLs + encrypting the 
private keys so only nimbus instances have access to them.  Then only 
publishing the tokens themselves in locations that the different services have 
access.
  However, if you compromise a supervisor you can get access to all of a 
worker's credentials anyways so it is game over, and if you compromise a DRPC 
server you can already do anything a DRPC server can.  So I am not sure it will 
slow down an attacker.
  We may want to do it anyways because in the future we may move away form ZK 
so it could limit the exposure.
  We might also want to look at distinguishing in our authorization between a 
token for a topology and a given user, so we can restrict the operations that a 
user can do from a topology.  I don't really want a topology trying to modify 
itself while it is running.

DIGEST-MD5 is susceptible to some active attacks. As this is only coming from 
the topology and nothing too serious is being exchanged in this case it is not 
that scary and Hadoop uses this for a lot more scary things.  
  To mitigate this we should look into using privileged ports where ever 
possible.

We might want to look into enabling encryption for the RPC traffic as well to 
help counter any passive attacks.  I'm not sure exactly how we would do it in a 
backwards compatible way though.


*Alternatives Explored:*

We looked at using some kind of public/private key signing, but even though 
that would work well for DRPC it would be a lot of extra work for 
Nimbus/Supervisors with no added benefit.

> Strom should support auth through delegation tokens for workers
> ---------------------------------------------------------------
>
>                 Key: STORM-2898
>                 URL: https://issues.apache.org/jira/browse/STORM-2898
>             Project: Apache Storm
>          Issue Type: New Feature
>          Components: storm-client, storm-server
>    Affects Versions: 2.0.0
>            Reporter: Robert Joseph Evans
>            Assignee: Robert Joseph Evans
>
> There are a lot of cases where it would be great for a worker to be able to 
> communicate directly to nimbus, supervisors, or drpc servers in a secure way 
> out of the box.
> This is currently a pain to make work.  The user has to ship a TGT with their 
> topology, and continually keep it up to date with credentials-push.  They 
> also need a kind of hacked up jaas.conf to grab the TGT from AutoTGT and put 
> it in the place that he client wants it.
> We should just generate a signed data structure (aka delegation token from 
> hadoop) that we can had off to the topologies to use when talking to nimbus, 
> a supervisor, or drpc servers.
> We may want to split up the different services from each other to make an 
> attack against one not hit all of them, but that is something we can think 
> about with the design of this.
> I will try to come up with a design shortly.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Reply via email to