Arun Mahadevan created STORM-2482:
-------------------------------------
Summary: Refactor the Storm auto credential plugins to be more
usable
Key: STORM-2482
URL: https://issues.apache.org/jira/browse/STORM-2482
Project: Apache Storm
Issue Type: Bug
Reporter: Arun Mahadevan
Assignee: Arun Mahadevan
Currently, the auto credential plugins are part of the respective external
modules like storm-hdfs, storm-hbase etc. If users want to use it, they need to
place the jars (storm-hdfs, storm-hbase) and its dependencies into ext lib.
Currently these plugins does not accept any hadoop configuration
programatically. These are set by placing config files like hdfs-site.xml in
the class path and this does not scale well nor does it allow users to connect
and fetch tokens from different clusters (say two different name nodes) with a
single topology.
To make the auto cred plugins more usable,
1. Refactor the AutoHdfs, AutoHbase etc into a separate storm external module
(say storm-autocreds). This jars along with its dependencies can be packaged
and extracted to a folder like lib-autocreds which can be loaded into the class
path when storm runs in secure mode (e.g. by setting STORM_EXT_CLASSPATH). The
required plugins would be loaded by nimubs/workers based on the user
configuration in storm.yaml.
2. Modify the plugins to accept "configKeys" via topology config. "configKeys"
would be a list of string "keys" that the user would pass in the topology
config.
{noformat}
// for hdfs
topoConf.set("hdfsCredentialsConfigKeys", Arrays.asList(new String[]
{"cluster1Key", "cluster2Key"}));
// put respective config map for the config keys,
topoConf.set("cluster1Key", configMap1);
topoConf.set("cluster2Key", configMap2);
{noformat}
This way we can support credentials from multiple clusters.
3. During topology submission, nimbus invokes "populateCredentials". If
"configKeys" are specified, the plugins will login to hadoop for each config
key and fetch the credentials (delegation tokens) and store it with respective
keys in the storm cluster state. Cluster state already stores the credentials
as a Map<String, String> so no changes are needed there.
The workers will download the credentials and invoke "populateSubject". The
plugin would populate all the credentials for all the configured "configKeys"
into the subject. Similar steps would be performed during "updateSubject"
4. Nimbus periodically invokes "renew" credentials. At this time the plugin
will fetch the credentials for the configured "configKeys" (i.e. for the users
from different clusters) and renew the respective credentials.
5. The user could specify different principal and keytab within the config key
map so that the plugin will use appropriate user for logging into the
respective cluster.
We also need to enhance the auto cred by adding more plugins. E.g for hbase and
kafka delegation tokens which are missing currently (this could be a separate
JIRAs).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
