[
https://issues.apache.org/jira/browse/METRON-1410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16330880#comment-16330880
]
Laurens Vets commented on METRON-1410:
--------------------------------------
[~cestella] suggested to rename 'metron_update', let Metron recreate the hbase
table and then put the original table back. (See
[https://stackoverflow.com/questions/27966072/how-do-you-rename-a-table-in-hbase).]
This works.
Procedure:
Rename 'old' table:
{code:java}
disable 'metron_update'
snapshot 'metron_update', 'metron_update_snap'
clone_snapshot 'metron_update_snap', 'metron_update_orig'
delete_snapshot 'metron_update_snap'
drop 'metron_update'{code}
Start Metron Indexing and let it recreate the 'metron_update' table.
Afterwards, stop Metron Indexing.
Replace the 'new' metron_update table with the old one:
{code:java}
disable 'metron_update'
snapshot 'metron_update', 'metron_update_snap'
clone_snapshot 'metron_update_snap', 'metron_update_2'
delete_snapshot 'metron_update_snap'
drop 'metron_update'
disable 'metron_update_orig'
snapshot 'metron_update_orig', 'metron_update_snap'
clone_snapshot 'metron_update_snap', 'metron_update'
delete_snapshot 'metron_update_snap'
drop 'metron_update_orig'{code}
Start Metron Indexing.
> Some more upgrade fallout... Can't restart Metron Indexing.
> -----------------------------------------------------------
>
> Key: METRON-1410
> URL: https://issues.apache.org/jira/browse/METRON-1410
> Project: Metron
> Issue Type: Bug
> Reporter: Laurens Vets
> Priority: Major
>
> After upgrading from 0.4.1 to 0.4.2, I can't seem to start or restart Metron
> Indexing. I get the following errors:
>
> stderr: /var/lib/ambari-agent/data/errors-2468.txt
> Traceback (most recent call last):
> File
> "/var/lib/ambari-agent/cache/common-services/METRON/0.4.2/package/scripts/indexing_master.py",
> line 160, in <module>
> Indexing().execute()
> File
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
> line 280, in execute
> method(env)
> File
> "/var/lib/ambari-agent/cache/common-services/METRON/0.4.2/package/scripts/indexing_master.py",
> line 82, in start
> self.configure(env)
> File
> "/var/lib/ambari-agent/cache/common-services/METRON/0.4.2/package/scripts/indexing_master.py",
> line 72, in configure
> commands.create_hbase_tables()
> File
> "/var/lib/ambari-agent/cache/common-services/METRON/0.4.2/package/scripts/indexing_commands.py",
> line 126, in create_hbase_tables
> user=self.__params.hbase_user
> File "/usr/lib/python2.6/site-packages/resource_management/core/base.py",
> line 155, in __init__
> self.env.run()
> File
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py",
> line 160, in run
> self.run_action(resource, action)
> File
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py",
> line 124, in run_action
> provider_action()
> File
> "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py",
> line 273, in action_run
> tries=self.resource.tries, try_sleep=self.resource.try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 70, in inner
> result = function(command, **kwargs)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 92, in checked_call
> tries=tries, try_sleep=try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 140, in _call_wrapper
> result = _call(command, **kwargs_copy)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 293, in _call
> raise ExecutionFailed(err_msg, code, out, err)
> resource_management.core.exceptions.ExecutionFailed: Execution of 'echo
> "create 'metron_update','t'" | hbase shell -n' returned 1. ERROR
> RuntimeError: Table already exists: metron_update!
>
> stdout: /var/lib/ambari-agent/data/output-2468.txt
>
> 2018-01-18 16:54:30,101 - Using hadoop conf dir:
> /usr/hdp/current/hadoop-client/conf
> 2018-01-18 16:54:30,301 - Using hadoop conf dir:
> /usr/hdp/current/hadoop-client/conf
> 2018-01-18 16:54:30,302 - Group['metron'] {}
> 2018-01-18 16:54:30,303 - Group['livy'] {}
> 2018-01-18 16:54:30,303 - Group['elasticsearch'] {}
> 2018-01-18 16:54:30,303 - Group['spark'] {}
> 2018-01-18 16:54:30,303 - Group['zeppelin'] {}
> 2018-01-18 16:54:30,304 - Group['hadoop'] {}
> 2018-01-18 16:54:30,304 - Group['kibana'] {}
> 2018-01-18 16:54:30,304 - Group['users'] {}
> 2018-01-18 16:54:30,304 - User['hive'] \{'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2018-01-18 16:54:30,305 - User['storm'] \{'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2018-01-18 16:54:30,306 - User['zookeeper'] \{'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2018-01-18 16:54:30,306 - User['infra-solr'] \{'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2018-01-18 16:54:30,307 - User['ams'] \{'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2018-01-18 16:54:30,307 - User['tez'] \{'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['users']}
> 2018-01-18 16:54:30,308 - User['zeppelin'] \{'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2018-01-18 16:54:30,309 - User['metron'] \{'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2018-01-18 16:54:30,309 - User['livy'] \{'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2018-01-18 16:54:30,310 - User['elasticsearch'] \{'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2018-01-18 16:54:30,310 - User['spark'] \{'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2018-01-18 16:54:30,311 - User['ambari-qa'] \{'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['users']}
> 2018-01-18 16:54:30,311 - User['flume'] \{'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2018-01-18 16:54:30,312 - User['kafka'] \{'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2018-01-18 16:54:30,312 - User['hdfs'] \{'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2018-01-18 16:54:30,313 - User['yarn'] \{'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2018-01-18 16:54:30,314 - User['kibana'] \{'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2018-01-18 16:54:30,314 - User['mapred'] \{'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2018-01-18 16:54:30,315 - User['hbase'] \{'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2018-01-18 16:54:30,315 - User['hcat'] \{'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2018-01-18 16:54:30,316 - File['/var/lib/ambari-agent/tmp/changeUid.sh']
> \{'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
> 2018-01-18 16:54:30,317 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh
> ambari-qa
> /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa']
> \{'not_if': '(test $(id -u ambari-qa) -gt 1000) || (false)'}
> 2018-01-18 16:54:30,323 - Skipping
> Execute['/var/lib/ambari-agent/tmp/changeUid.sh ambari-qa
> /tmp/hadoop-ambari-qa,/tmp/hsperfdata_ambari-qa,/home/ambari-qa,/tmp/ambari-qa,/tmp/sqoop-ambari-qa']
> due to not_if
> 2018-01-18 16:54:30,324 - Directory['/tmp/hbase-hbase'] \{'owner': 'hbase',
> 'create_parents': True, 'mode': 0775, 'cd_access': 'a'}
> 2018-01-18 16:54:30,325 - File['/var/lib/ambari-agent/tmp/changeUid.sh']
> \{'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
> 2018-01-18 16:54:30,326 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh
> hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase']
> \{'not_if': '(test $(id -u hbase) -gt 1000) || (false)'}
> 2018-01-18 16:54:30,331 - Skipping
> Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase
> /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase'] due
> to not_if
> 2018-01-18 16:54:30,332 - Group['hdfs'] {}
> 2018-01-18 16:54:30,332 - User['hdfs'] \{'fetch_nonlocal_groups': True,
> 'groups': ['hadoop', 'hdfs']}
> 2018-01-18 16:54:30,333 - FS Type:
> 2018-01-18 16:54:30,333 - Directory['/etc/hadoop'] \{'mode': 0755}
> 2018-01-18 16:54:30,346 -
> File['/usr/hdp/current/hadoop-client/conf/hadoop-env.sh'] \{'content':
> InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
> 2018-01-18 16:54:30,347 -
> Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] \{'owner':
> 'hdfs', 'group': 'hadoop', 'mode': 01777}
> 2018-01-18 16:54:30,365 - Execute[('setenforce', '0')] \{'not_if': '(! which
> getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo':
> True, 'only_if': 'test -f /selinux/enforce'}
> 2018-01-18 16:54:30,384 - Directory['/var/log/hadoop'] \{'owner': 'root',
> 'create_parents': True, 'group': 'hadoop', 'mode': 0775, 'cd_access': 'a'}
> 2018-01-18 16:54:30,386 - Directory['/var/run/hadoop'] \{'owner': 'root',
> 'create_parents': True, 'group': 'root', 'cd_access': 'a'}
> 2018-01-18 16:54:30,386 - Directory['/tmp/hadoop-hdfs'] \{'owner': 'hdfs',
> 'create_parents': True, 'cd_access': 'a'}
> 2018-01-18 16:54:30,390 -
> File['/usr/hdp/current/hadoop-client/conf/commons-logging.properties']
> \{'content': Template('commons-logging.properties.j2'), 'owner': 'hdfs'}
> 2018-01-18 16:54:30,392 -
> File['/usr/hdp/current/hadoop-client/conf/health_check'] \{'content':
> Template('health_check.j2'), 'owner': 'hdfs'}
> 2018-01-18 16:54:30,393 -
> File['/usr/hdp/current/hadoop-client/conf/log4j.properties'] \{'content':
> ..., 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
> 2018-01-18 16:54:30,403 -
> File['/usr/hdp/current/hadoop-client/conf/hadoop-metrics2.properties']
> \{'content': Template('hadoop-metrics2.properties.j2'), 'owner': 'hdfs',
> 'group': 'hadoop'}
> 2018-01-18 16:54:30,404 -
> File['/usr/hdp/current/hadoop-client/conf/task-log4j.properties']
> \{'content': StaticFile('task-log4j.properties'), 'mode': 0755}
> 2018-01-18 16:54:30,405 -
> File['/usr/hdp/current/hadoop-client/conf/configuration.xsl'] \{'owner':
> 'hdfs', 'group': 'hadoop'}
> 2018-01-18 16:54:30,409 - File['/etc/hadoop/conf/topology_mappings.data']
> \{'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'),
> 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop'}
> 2018-01-18 16:54:30,414 - File['/etc/hadoop/conf/topology_script.py']
> \{'content': StaticFile('topology_script.py'), 'only_if': 'test -d
> /etc/hadoop/conf', 'mode': 0755}
> 2018-01-18 16:54:30,667 - Using hadoop conf dir:
> /usr/hdp/current/hadoop-client/conf
> 2018-01-18 16:54:30,669 - Running indexing configure
> 2018-01-18 16:54:30,676 -
> File['/usr/metron/0.4.2/config/elasticsearch.properties'] \{'owner':
> 'metron', 'content': Template('elasticsearch.properties.j2'), 'group':
> 'metron'}
> 2018-01-18 16:54:30,678 - Patch global config in Zookeeper
> 2018-01-18 16:54:30,678 - Setup temporary global config JSON patch
> (formatting per RFC6902): /tmp/metron-global-config-patch.json
> 2018-01-18 16:54:30,681 - File['/tmp/metron-global-config-patch.json']
> \{'owner': 'metron', 'content': InlineTemplate(...), 'group': 'metron'}
> 2018-01-18 16:54:30,681 - Patching global config in ZooKeeper
> 2018-01-18 16:54:30,681 - Execute['/usr/metron/0.4.2/bin/zk_load_configs.sh
> --zk_quorum metron1:2181,metron2:2181 --mode PATCH --config_type GLOBAL
> --patch_file /tmp/metron-global-config-patch.json'] \{'path':
> ['/usr/jdk64/jdk1.8.0_77/bin']}
> 2018-01-18 16:55:19,874 - Done patching global config
> 2018-01-18 16:55:19,874 - Pull zookeeper config locally
> 2018-01-18 16:55:19,874 - Pulling all Metron configs down from ZooKeeper to
> local file system
> 2018-01-18 16:55:19,874 - NOTE - THIS IS OVERWRITING THE LOCAL METRON CONFIG
> DIR WITH ZOOKEEPER CONTENTS: /usr/metron/0.4.2/config/zookeeper
> 2018-01-18 16:55:19,875 - Execute['/usr/metron/0.4.2/bin/zk_load_configs.sh
> --zk_quorum metron1:2181,metron2:2181 --mode PULL --output_dir
> /usr/metron/0.4.2/config/zookeeper --force'] \{'path':
> ['/usr/jdk64/jdk1.8.0_77/bin']}
> 2018-01-18 16:56:08,286 - Creating HBase Tables for indexing
> 2018-01-18 16:56:08,287 - Execute['echo "create 'metron_update','t'" | hbase
> shell -n'] \{'logoutput': False, 'path':
> ['/usr/sbin:/sbin:/usr/local/bin:/bin:/usr/bin'], 'tries': 3, 'user':
> 'hbase', 'try_sleep': 5}
> 2018-01-18 16:56:59,943 - Retrying after 5 seconds. Reason: Execution of
> 'echo "create 'metron_update','t'" | hbase shell -n' returned 1. ERROR
> RuntimeError: Table already exists: metron_update!
> 2018-01-18 16:57:57,884 - Retrying after 5 seconds. Reason: Execution of
> 'echo "create 'metron_update','t'" | hbase shell -n' returned 1. ERROR
> RuntimeError: Table already exists: metron_update!
>
> Command failed after 1 tries
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
