[
https://issues.apache.org/jira/browse/STORM-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim resolved STORM-2482.
---------------------------------
Resolution: Fixed
Fix Version/s: 1.1.1
2.0.0
Thanks [~arunmahadevan], merged into master and 1.x-branch.
> Refactor the Storm auto credential plugins to be more usable
> ------------------------------------------------------------
>
> Key: STORM-2482
> URL: https://issues.apache.org/jira/browse/STORM-2482
> Project: Apache Storm
> Issue Type: Bug
> Reporter: Arun Mahadevan
> Assignee: Arun Mahadevan
> Fix For: 2.0.0, 1.1.1
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> Currently, the auto credential plugins are part of the respective external
> modules like storm-hdfs, storm-hbase etc. If users want to use it, they need
> to place the jars (storm-hdfs, storm-hbase) and its dependencies into ext
> lib. Currently these plugins does not accept any hadoop configuration
> programatically. These are set by placing config files like hdfs-site.xml in
> the class path and this does not scale well nor does it allow users to
> connect and fetch tokens from different clusters (say two different name
> nodes) with a single topology.
> To make the auto cred plugins more usable,
> 1. Refactor the AutoHdfs, AutoHbase etc into a separate storm external module
> (say storm-autocreds). This jars along with its dependencies can be packaged
> and extracted to a folder like lib-autocreds which can be loaded into the
> class path when storm runs in secure mode (e.g. by setting
> STORM_EXT_CLASSPATH). The required plugins would be loaded by nimubs/workers
> based on the user configuration in storm.yaml.
> 2. Modify the plugins to accept "configKeys" via topology config.
> "configKeys" would be a list of string "keys" that the user would pass in the
> topology config.
> {noformat}
> // for hdfs
> topoConf.set("hdfsCredentialsConfigKeys", Arrays.asList(new String[]
> {"cluster1Key", "cluster2Key"}));
> // put respective config map for the config keys,
> topoConf.set("cluster1Key", configMap1);
> topoConf.set("cluster2Key", configMap2);
>
> {noformat}
> This way we can support credentials from multiple clusters.
> 3. During topology submission, nimbus invokes "populateCredentials". If
> "configKeys" are specified, the plugins will login to hadoop for each config
> key and fetch the credentials (delegation tokens) and store it with
> respective keys in the storm cluster state. Cluster state already stores the
> credentials as a Map<String, String> so no changes are needed there.
> The workers will download the credentials and invoke "populateSubject". The
> plugin would populate all the credentials for all the configured "configKeys"
> into the subject. Similar steps would be performed during "updateSubject"
> 4. Nimbus periodically invokes "renew" credentials. At this time the plugin
> will fetch the credentials for the configured "configKeys" (i.e. for the
> users from different clusters) and renew the respective credentials.
> 5. The user could specify different principal and keytab within the config
> key map so that the plugin will use appropriate user for logging into the
> respective cluster.
> We also need to enhance the auto cred by adding more plugins. E.g for hbase
> and kafka delegation tokens which are missing currently (this could be a
> separate JIRAs).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
