This is an automated email from the ASF dual-hosted git repository.
oleewere pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/ambari.git
The following commit(s) were added to refs/heads/trunk by this push:
new b71d73f AMBARI-23945. Infra Solr Migration - helper command to
enable/disable/fix solr kerberos config.
b71d73f is described below
commit b71d73fe8dd8ab07b0dc6d2520722b6423fdf45e
Author: Oliver Szabo <[email protected]>
AuthorDate: Mon Jun 25 02:49:37 2018 +0200
AMBARI-23945. Infra Solr Migration - helper command to enable/disable/fix
solr kerberos config.
---
ambari-infra/ambari-infra-solr-client/README.md | 26 ++---
.../src/main/python/migrationHelper.py | 127 ++++++++++++++++++---
2 files changed, 125 insertions(+), 28 deletions(-)
diff --git a/ambari-infra/ambari-infra-solr-client/README.md
b/ambari-infra/ambari-infra-solr-client/README.md
index 2df113b..53c6a3e 100644
--- a/ambari-infra/ambari-infra-solr-client/README.md
+++ b/ambari-infra/ambari-infra-solr-client/README.md
@@ -764,9 +764,16 @@ See [transport old data to new
collections](#viii.-transport-old-data-to-new-col
#### <a id="if-solr-restarted">What to do if Solr instances restarted right
after Ambari upgrade but before upgrade Solr instance packages?</a>
-As Solr instances won't start with the new upgraded configs (only if kerberos
is enabled), you can do a small fix to make it work to just add this line to
`infra-solr-env/content`:
+If you restarted Solr before backup or upgrade Solr server packages, you can
fix the Solr config with the following command:
```bash
-SOLR_KERB_NAME_RULES="{{infra_solr_kerberos_name_rules}}"
+/usr/bin/python /usr/lib/ambari-infra-solr-client/migrationHelper.py
--ini-file $CONFIG_INI_LOCATION --action fix-solr5-kerberos-config
+```
+
+That is basically add `SOLR_KERB_NAME_RULES` back to `infra-solr-env/content`
and disable authorization for Solr. (upload a /security.json to /infra-solr
znode without the authorization config, then turn manually managed
/security.json on in order to not override /security.json again on Solr
restart) After the command finished successfully, you will need to restart Solr
instances.
+
+But if you added `SOLR_KERB_NAME_RULES` config to the
`infra-solr-env/content`, you will require to delete that after you upgraded
Solr package (and before restarting them). You can do that with the
`fix-solr7-kerberos-config` action:
+```bash
+/usr/bin/python /usr/lib/ambari-infra-solr-client/migrationHelper.py
--ini-file $CONFIG_INI_LOCATION --action fix-solr7-kerberos-config
```
#### <a id="get-core-/-shard-names-with-hosts">Get core / shard names with
hosts</a>
@@ -775,22 +782,15 @@ To get which hosts are related for your collections, you
can check the Solr UI (
#### <a id="turn-off-infra-solr-authorization">Turn off Infra Solr
Authorization</a>
-You can turn off Solr authorization plugin with setting
`infra-solr-security-json/content` Ambari configuration to `{"authentication":
{"class": "org.apache.solr.security.KerberosPlugin"}}` (with that
authentication will be still enabled). Then you will need to restart Solr, as
that config is uploaded to the `/infra-solr/security.json` znode during
startup. Other option is to use zkcli.sh of an Infra Solr to upload the
security.json to the right place:
+You can turn off Solr authorization plugin with the
`disable-solr-authorization` action (can be executed after config generation
[step](#0-gather-params)):
```bash
-# Setup env for zkcli.sh
-source /etc/ambari-infra-solr/conf/infra-solr-env.sh
-# Run that command only if kerberos is enabled.
-export SOLR_ZK_CREDS_AND_ACLS="${SOLR_AUTHENTICATION_OPTS}"
-ZK_CONN_STRING=... # connection string -> zookeeper server addresses with the
znode, e.g.: c7401.ambari.apache.org:2181/infra-solr
-
-/usr/lib/ambari-infra-solr/server/scripts/cloud-scripts/zkcli.sh -zkhost
$ZK_CONN_STRING -cmd put /security.json
- '{"authentication": {"class": "org.apache.solr.security.KerberosPlugin"}}'
+/usr/bin/python /usr/lib/ambari-infra-solr-client/migrationHelper.py
--ini-file $CONFIG_INI_LOCATION --action disable-solr-authorization
```
-Or you can also use the `migationHelper.py` script to disable the Solr
authorization (for that to keep this settings, you can disable the management
of the security.json in `infra-solr-security-json` config type)
+You can re-enable it with the following command: (or set
`infra-solr-security-json/infra_solr_security_manually_managed` configuration
to `false`, then restart Solr)
```bash
-/usr/bin/python /usr/lib/ambari-infra-solr-client/migrationHelper.py
--ini-file $CONFIG_INI_LOCATION --action disable-solr-authorization
+/usr/bin/python /usr/lib/ambari-infra-solr-client/migrationHelper.py
--ini-file $CONFIG_INI_LOCATION --action enable-solr-authorization
```
#### <a id="">Solr Migration Helper Script</a>
diff --git
a/ambari-infra/ambari-infra-solr-client/src/main/python/migrationHelper.py
b/ambari-infra/ambari-infra-solr-client/src/main/python/migrationHelper.py
index 4d63bdd..1e62b98 100755
--- a/ambari-infra/ambari-infra-solr-client/src/main/python/migrationHelper.py
+++ b/ambari-infra/ambari-infra-solr-client/src/main/python/migrationHelper.py
@@ -61,6 +61,8 @@ GET_HOSTS_COMPONENTS_URL =
'/services/{0}/components/{1}?fields=host_components'
REQUESTS_API_URL = '/requests'
BATCH_REQUEST_API_URL = "/api/v1/clusters/{0}/request_schedules"
+GET_ACTUAL_CONFIG_URL =
'/configurations/service_config_versions?service_name={0}&is_current=true'
+CREATE_CONFIGURATIONS_URL = '/configurations'
LIST_SOLR_COLLECTION_URL = '{0}/admin/collections?action=LIST&wt=json'
CREATE_SOLR_COLLECTION_URL =
'{0}/admin/collections?action=CREATE&name={1}&collection.configName={2}&numShards={3}&replicationFactor={4}&maxShardsPerNode={5}&wt=json'
@@ -295,7 +297,6 @@ def create_batch_command(command, hosts, cluster,
service_name, component_name,
return request_schedules
-
def create_command_request(command, parameters, hosts, cluster, context,
service=SOLR_SERVICE_NAME, component=SOLR_COMPONENT_NAME):
request = {}
request_info = {}
@@ -635,7 +636,84 @@ def filter_collections(options, collections):
else:
return collections
-def get_solr_urls(options, config, collection, collections_json):
+def get_infra_solr_props(config, accessor):
+ cluster = config.get('ambari_server', 'cluster')
+ service_configs = get_json(accessor, CLUSTERS_URL.format(cluster) +
GET_ACTUAL_CONFIG_URL.format(SOLR_SERVICE_NAME))
+ infra_solr_props = {}
+ infra_solr_env_properties = {}
+ infra_solr_security_json_properties = {}
+ if 'items' in service_configs and len(service_configs['items']) > 0:
+ if 'configurations' in service_configs['items'][0]:
+ for config in service_configs['items'][0]['configurations']:
+ if 'type' in config and config['type'] == 'infra-solr-env':
+ infra_solr_env_properties = config['properties']
+ if 'type' in config and config['type'] == 'infra-solr-security-json':
+ infra_solr_security_json_properties = config['properties']
+ infra_solr_props['infra-solr-env'] = infra_solr_env_properties
+ infra_solr_props['infra-solr-security-json'] =
infra_solr_security_json_properties
+ return infra_solr_props
+
+def insert_string_before(full_str, sub_str, insert_str):
+ idx = full_str.index(sub_str)
+ return full_str[:idx] + insert_str + full_str[idx:]
+
+def set_solr_security_management(infra_solr_props, accessor, enable = True):
+ security_props = infra_solr_props['infra-solr-security-json']
+ check_value = "false" if enable else "true"
+ set_value = "true" if enable else "false"
+ turn_status = "on" if enable else "off"
+ if 'infra_solr_security_manually_managed' in security_props and
security_props['infra_solr_security_manually_managed'] == check_value:
+ security_props['infra_solr_security_manually_managed'] = set_value
+ post_configuration = create_configs('infra-solr-security-json',
security_props, 'Turn {0} security.json manaul management by
migrationHelper.py'.format(turn_status))
+ apply_configs(config, accessor, post_configuration)
+ else:
+ print "Configuration
'infra-solr-security-json/infra_solr_security_manually_managed' has already set
to '{0}'".format(set_value)
+
+def set_solr_name_rules(infra_solr_props, accessor, add = False):
+ """
+ Set name rules in infra-solr-env/content if not set in add mode, in non-add
mode, remove it if exists
+ :param add: solr kerb name rules needs to be added (if false, it needs to be
removed)
+ """
+ infra_solr_env_props = infra_solr_props['infra-solr-env']
+ name_rules_param =
"SOLR_KERB_NAME_RULES=\"{{infra_solr_kerberos_name_rules}}\"\n"
+
+ if 'content' in infra_solr_env_props and (name_rules_param not in
infra_solr_env_props['content']) is add:
+ if add:
+ print "Adding 'SOLR_KERB_NAME_RULES' to 'infra-solr-env/content'"
+ new_content = insert_string_before(infra_solr_env_props['content'],
"SOLR_KERB_KEYTAB", name_rules_param)
+ infra_solr_env_props['content'] = new_content
+ post_configuration = create_configs('infra-solr-env',
infra_solr_env_props, 'Add "SOLR_KERB_NAME_RULES" by migrationHelper.py')
+ apply_configs(config, accessor, post_configuration)
+ else:
+ print "Removing 'SOLR_KERB_NAME_RULES' from 'infra-solr-env/content'"
+ new_content = infra_solr_env_props['content'].replace(name_rules_param,
'')
+ infra_solr_env_props['content'] = new_content
+ post_configuration = create_configs('infra-solr-env',
infra_solr_env_props, 'Remove "SOLR_KERB_NAME_RULES" by migrationHelper.py')
+ apply_configs(config, accessor, post_configuration)
+ else:
+ if add:
+ print "'SOLR_KERB_NAME_RULES' has already set in configuration
'infra-solr-env/content'"
+ else:
+ print "Configuration 'infra-solr-env/content' does not contain
'SOLR_KERB_NAME_RULES'"
+
+def apply_configs(config, accessor, post_configuration):
+ cluster = config.get('ambari_server', 'cluster')
+ desired_configs_post_body = {}
+ desired_configs_post_body["Clusters"] = {}
+ desired_configs_post_body["Clusters"]["desired_configs"] = post_configuration
+ accessor(CLUSTERS_URL.format(cluster), 'PUT',
json.dumps(desired_configs_post_body))
+
+def create_configs(config_type, properties, context):
+ configs_for_posts = {}
+ configuration = {}
+ configuration['type'] = config_type
+ configuration['tag'] = "version" + str(int(round(time.time() * 1000)))
+ configuration['properties'] = properties
+ configuration['service_config_version_note'] = context
+ configs_for_posts[config_type] = configuration
+ return configs_for_posts
+
+def get_solr_urls(options, config, collection, collections_json): # TODO: use
proper url from collections.json
solr_urls = []
solr_hosts = None
solr_port = "8886"
@@ -1521,7 +1599,7 @@ def update_state_jsons(options, accessor, parser, config,
service_filter):
else:
print "Collection '{0}' does not exist or filtered out. Skipping update
collection state operation.".format(backup_fulltext_index_name)
-def disable_solr_authorization(options, accessor, parser, config):
+def set_solr_authorization(options, accessor, parser, config,
enable_authorization, fix_kerberos_config = False):
solr_znode='/infra-solr'
if config.has_section('infra_solr') and config.has_option('infra_solr',
'znode'):
solr_znode=config.get('infra_solr', 'znode')
@@ -1529,11 +1607,24 @@ def disable_solr_authorization(options, accessor,
parser, config):
if config.has_section('cluster') and config.has_option('cluster',
'kerberos_enabled'):
kerberos_enabled=config.get('cluster', 'kerberos_enabled')
if kerberos_enabled == 'true':
- print "Disable Solr authorization by uploading a new security.json ..."
- copy_znode(options, config,
COLLECTIONS_DATA_JSON_LOCATION.format("security-without-authr.json"),
+ infra_solr_props = get_infra_solr_props(config, accessor)
+ if enable_authorization:
+ print "Enable Solr security.json management by Ambari ... "
+ set_solr_security_management(infra_solr_props, accessor, enable = False)
+ if fix_kerberos_config:
+ set_solr_name_rules(infra_solr_props, accessor, False)
+ else:
+ print "Disable Solr authorization by uploading a new security.json and
turn on security.json management by Ambari..."
+ set_solr_security_management(infra_solr_props, accessor, enable = True)
+ copy_znode(options, config,
COLLECTIONS_DATA_JSON_LOCATION.format("security-without-authr.json"),
"{0}/security.json".format(solr_znode), copy_from_local=True)
+ if fix_kerberos_config:
+ set_solr_name_rules(infra_solr_props, accessor, True)
else:
- print "Security is not enabled. Skipping disable Solr authorization
operation."
+ if fix_kerberos_config:
+ print "Security is not enabled. Skipping enable/disable Solr
authorization + fix infra-solr-env kerberos config operation."
+ else:
+ print "Security is not enabled. Skipping enable/disable Solr
authorization operation."
def summarize_shard_check_result(check_results, skip_warnings = False,
skip_index_size = False):
warnings = 0
@@ -1611,8 +1702,9 @@ if __name__=="__main__":
parser = optparse.OptionParser("usage: %prog [options]")
parser.add_option("-a", "--action", dest="action", type="string",
help="delete-collections | backup | cleanup-znodes | backup-and-cleanup |
migrate | restore |' \
- ' rolling-restart-solr | rolling-restart-atlas |
rolling-restart-ranger | check-shards | check-backup-shards |
disable-solr-authorization |'\
- ' upgrade-solr-clients | upgrade-solr-instances |
upgrade-logsearch-portal | upgrade-logfeeders | stop-logsearch | restart-solr
|restart-logsearch | restart-ranger | restart-atlas")
+ ' rolling-restart-solr | rolling-restart-atlas |
rolling-restart-ranger | check-shards | check-backup-shards |
enable-solr-authorization | disable-solr-authorization |'\
+ ' fix-solr5-kerberos-config | fix-solr7-kerberos-config |
upgrade-solr-clients | upgrade-solr-instances | upgrade-logsearch-portal |
upgrade-logfeeders | stop-logsearch |'\
+ ' restart-solr |restart-logsearch | restart-ranger |
restart-atlas")
parser.add_option("-i", "--ini-file", dest="ini_file", type="string",
help="Config ini file to parse (required)")
parser.add_option("-f", "--force", dest="force", default=False,
action="store_true", help="force index upgrade even if it's the right version")
parser.add_option("-v", "--verbose", dest="verbose", action="store_true",
help="use for verbose logging")
@@ -1745,8 +1837,14 @@ if __name__=="__main__":
print "ATLAS service has not found in the config or filtered out."
elif options.action.lower() == 'rolling-restart-solr':
rolling_restart(options, accessor, parser, config, SOLR_SERVICE_NAME,
SOLR_COMPONENT_NAME, "Rolling Restart Infra Solr Instances")
+ elif options.action.lower() == 'enable-solr-authorization':
+ set_solr_authorization(options, accessor, parser, config, True)
elif options.action.lower() == 'disable-solr-authorization':
- disable_solr_authorization(options, accessor, parser, config)
+ set_solr_authorization(options, accessor, parser, config, False)
+ elif options.action.lower() == 'fix-solr5-kerberos-config':
+ set_solr_authorization(options, accessor, parser, config, False, True)
+ elif options.action.lower() == 'fix-solr7-kerberos-config':
+ set_solr_authorization(options, accessor, parser, config, True, True)
elif options.action.lower() == 'check-shards':
check_shards(options, accessor, parser, config)
elif options.action.lower() == 'check-backup-shards':
@@ -1756,11 +1854,10 @@ if __name__=="__main__":
else:
parser.print_help()
print 'action option is invalid (available actions: delete-collections
| backup | cleanup-znodes | backup-and-cleanup | migrate | restore |' \
- ' rolling-restart-solr | rolling-restart-ranger |
rolling-restart-atlas | check-shards | check-backup-shards | check-docs |
disable-solr-authorization |' \
- ' upgrade-solr-clients | upgrade-solr-instances |
upgrade-logsearch-portal | upgrade-logfeeders | stop-logsearch | restart-solr
|' \
+ ' rolling-restart-solr | rolling-restart-ranger |
rolling-restart-atlas | check-shards | check-backup-shards | check-docs |
enable-solr-authorization |'\
+ ' disable-solr-authorization | fix-solr5-kerberos-config |
fix-solr7-kerberos-config | upgrade-solr-clients | upgrade-solr-instances |
upgrade-logsearch-portal |' \
+ ' upgrade-logfeeders | stop-logsearch | restart-solr |' \
' restart-logsearch | restart-ranger | restart-atlas)'
sys.exit(1)
- if options.verbose:
- print "Migration helper command ({0})
{1}FINISHED{2}".format(str(args), colors.OKGREEN, colors.ENDC)
- else:
- print "Migration helper command {0}FINISHED{1}".format(colors.OKGREEN,
colors.ENDC)
\ No newline at end of file
+
+ print "Migration helper command {0}FINISHED{1}".format(colors.OKGREEN,
colors.ENDC)
\ No newline at end of file
