[
https://issues.apache.org/jira/browse/GOBBLIN-1398?focusedWorklogId=607546&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-607546
]
ASF GitHub Bot logged work on GOBBLIN-1398:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 06/Jun/21 23:57
Start Date: 06/Jun/21 23:57
Worklog Time Spent: 10m
Work Description: jhsenjaliya commented on a change in pull request #3236:
URL: https://github.com/apache/gobblin/pull/3236#discussion_r646205907
##########
File path:
gobblin-utility/src/main/java/org/apache/gobblin/util/ConfigUtils.java
##########
@@ -569,4 +608,124 @@ public static Config resolveEncrypted(Config config,
Optional<String> encConfigP
}
return ConfigFactory.parseMap(tmpMap).withFallback(config);
}
+
+ /**
+ *
+ * @param config
+ * @return ConfigObject ( config in tree ) of the {@link
ConfigUtils#GOBBLIN_SYNC_SYSTEMS_KEY}
+ */
+ public static ConfigObject getAllSystemConfig(Config config) {
+ return (config.hasPath(GOBBLIN_SYNC_SYSTEMS_KEY) ?
config.getObject(GOBBLIN_SYNC_SYSTEMS_KEY) : getInstance().root() );
+ }
+
+ /**
+ *
+ * @param config at path: {@link ConfigUtils#GOBBLIN_SYNC_SYSTEMS_KEY}
+ * @return map of system name to that system's config tree for all the
systems that has security enabled
+ */
+ public static Map<String, ConfigObject> getAllSecureSystems(Config config) {
+ ConfigObject allSysConfig = getAllSystemConfig(config);
+ Map<String, ConfigObject> secureSystems = new HashMap<>();
+
+ if (allSysConfig == null) return secureSystems;
+
+ for (String systemName : allSysConfig.keySet()) {
+ ConfigObject sysConfig = allSysConfig.atPath(systemName).root();
+ sysConfig.get(IS_SECURE_KEY);
+ if (ConfigUtils.getBoolean(sysConfig.toConfig(), IS_SECURE_KEY, false)) {
+ secureSystems.put(systemName, sysConfig);
+ }
+ }
+ return secureSystems;
+ }
+
+ /**
+ *
+ * @param config at path: {@link ConfigUtils#GOBBLIN_SYNC_SYSTEMS_KEY}
+ * @return multiple map of system name and its all config files regardless
of the component.
+ */
+ public static Multimap<String, String> getAllSystemConfigFiles(Config
config) {
+
+ Multimap<String, String> allSysConfFiles = LinkedHashMultimap.create();
+
+ getAllSystemConfig(config).entrySet();
+ ConfigObject allSysConfig = getAllSystemConfig(config);
+
+ if (allSysConfig == null) return allSysConfFiles;
+
+ for (Map.Entry<String, ConfigValue> sysConfig : allSysConfig.entrySet()) {
+
+ if (sysConfig.getValue().valueType().equals(ConfigValueType.OBJECT)) {
+ ConfigObject allComponentConfigs = (ConfigObject) sysConfig.getValue();
+ for (Map.Entry<String, ConfigValue> componentConfig :
allComponentConfigs.entrySet()) {
+ if
(componentConfig.getValue().valueType().equals(ConfigValueType.OBJECT)) {
+ ConfigObject componentConfigValue = (ConfigObject)
componentConfig.getValue();
+ Config configValue = componentConfigValue.toConfig();
+ if (configValue.hasPath(GOBBLIN_SYNC_SYSTEM_CONFIG_FILES_KEY)) {
+
configValue.getList(GOBBLIN_SYNC_SYSTEM_CONFIG_FILES_KEY).forEach(item ->
allSysConfFiles.put(sysConfig.getKey(), item.unwrapped().toString()));
+ }
+ }
+ }
+ }
+ }
+ return allSysConfFiles;
+ }
+
+ /**
+ *
+ * @param config at path: {@link ConfigUtils#GOBBLIN_SYNC_SYSTEMS_KEY}
+ * @return map of all system -> config for which the token management {@link
ConfigUtils#IS_TOKEN_MGMT_ENABLED_KEY} is enabled.
+ */
+ public static Map<String, Config> getAllSecureHiveSystems(Config config){
+ Map<String, Config> secureSystems = new HashMap<>();
+
+ for (String systemName :
config.getObject(GOBBLIN_SYNC_SYSTEMS_KEY).keySet()){
+ Config sysConfig =
config.getConfig(GOBBLIN_SYNC_SYSTEMS_KEY).getConfig(systemName);
+ if(ConfigUtils.getBoolean(sysConfig, IS_SECURE_KEY, false) &&
ConfigUtils.getBoolean(sysConfig, IS_TOKEN_MGMT_ENABLED_KEY, false) &&
sysConfig.hasPath(HIVE_SYSTEM_KEY)){
+ secureSystems.put(systemName, sysConfig);
+ }
+ }
+ return secureSystems;
+ }
+
+ /**
+ *
+ * @param metastoreURI for which we need to find system and its config
+ * @param config at path: {@link ConfigUtils#GOBBLIN_SYNC_SYSTEMS_KEY}
+ * @return map of system name -> config paths for the given metastoreURI
+ */
+ public static Multimap<String, String> getSystemConfigForMetastore(String
metastoreURI, Config config) {
+ if (config == null) {
+ config = getInstance();
+ }
+
+ Multimap<String, String> sysConfFiles = LinkedHashMultimap.create();
+
+ ConfigObject allSysConfig = getAllSystemConfig(config);
+
+ if (allSysConfig == null) return sysConfFiles;
+
+ for (Map.Entry<String, ConfigValue> sysConfig : allSysConfig.entrySet()) {
+
+ if (sysConfig.getValue().valueType().equals(ConfigValueType.OBJECT)) {
+ ConfigObject allComponentConfigs = (ConfigObject) sysConfig.getValue();
Review comment:
this makes the traversing possible, only `ConfigObject` provides the
entrySet method which `ConfigValue` does not. I think there can be other ways
to obtain `ConfigObject` out of `ConfigValue` as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 607546)
Time Spent: 1.5h (was: 1h 20m)
> Gobblin kerberos token management for multiple remote hive metastores
> ---------------------------------------------------------------------
>
> Key: GOBBLIN-1398
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1398
> Project: Apache Gobblin
> Issue Type: Improvement
> Affects Versions: 0.15.0
> Reporter: Jay Sen
> Priority: Major
> Fix For: 0.16.0
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> GOBBLIN-1308 takes care of ability to connect to remote and secure hive
> metastore, but it still requires the management of hive-site to be provided
> to be container local file.
> When it comes to multiple hive clusters this manual management approach would
> not work, and requires special feature of providing system specific
> hive-site.xml without namespace collision.
>
> This ticket aims to do following things
> 1) define a way to provide remote system configuration ( keep giving flat
> config is more cumbersome )
> 2) based on system config and feature flag, copy config files to container
> local path automatically.
> 3) when creating metastoreClient, pick up the right config for the requested
> system ( identified by the metastore URI )
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
