Repository: atlas Updated Branches: refs/heads/branch-0.8 f6502c9e5 -> 0e80d9b37
ATLAS-2280: Recovering lost/bad index data Signed-off-by: Madhan Neethiraj <[email protected]> Project: http://git-wip-us.apache.org/repos/asf/atlas/repo Commit: http://git-wip-us.apache.org/repos/asf/atlas/commit/0e80d9b3 Tree: http://git-wip-us.apache.org/repos/asf/atlas/tree/0e80d9b3 Diff: http://git-wip-us.apache.org/repos/asf/atlas/diff/0e80d9b3 Branch: refs/heads/branch-0.8 Commit: 0e80d9b37df5654b341c0aeb129c86597147ed4f Parents: f6502c9 Author: Ashutosh Mestry <[email protected]> Authored: Sun Nov 26 20:28:22 2017 -0800 Committer: Madhan Neethiraj <[email protected]> Committed: Sun Nov 26 20:29:43 2017 -0800 ---------------------------------------------------------------------- tools/atlas-index-repair-kit/README.txt | 75 +++++++++++++++++++ .../atlas-index-repair-kit.tar.gz | Bin 0 -> 1252999 bytes 2 files changed, 75 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/atlas/blob/0e80d9b3/tools/atlas-index-repair-kit/README.txt ---------------------------------------------------------------------- diff --git a/tools/atlas-index-repair-kit/README.txt b/tools/atlas-index-repair-kit/README.txt new file mode 100644 index 0000000..4e10bdf --- /dev/null +++ b/tools/atlas-index-repair-kit/README.txt @@ -0,0 +1,75 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +Follow the instructions below to rebuid Atlas index data from the data +stored in data-store (like HBase). + +Please note that these steps have been verified in an Atlas environment +with Titan 0.5.4, HBase and Solr. For other variations, please refer to +the relevant documentation of the graph DB. + +1. Setup titan-0.5.4 in the host where Atlas is installed + - Download titan-0.5.4 Hadoop-2 distribution from http://s3.thinkaurelius.com/downloads/titan/titan-0.5.4-hadoop2.zip + - cd /tmp/ + - unzip titan-0.5.4-hadoop2.zip + +2. Add Atlas index repair kit to titan-0.5.4 installation, by running the following commands: + - cd /tmp/titan-0.5.4-hadoop2 + - tar xvf atlas-index-repair-kit.tar + +3. Update atlas-conf/atlas-titan.properties with details to connect to data-store (like HBase). For example: + storage.backend=hbase + storage.hostname=fqdn + storage.hbase.table=atlas_titan + +4. Update bin/atlas-gremlin.sh to set the following variables at the top. For example: + ATLAS_WEBAPP_DIR=/home/atlas/atlas-server/server/webapp + STORE_CONF_DIR=/etc/hbase/conf + +5. If Atlas is run in a kerberized environment, setup the following: + 5.1. Update atlas-conf/atlas-titan.properties with necessary kerberos details for the data-store and index. For example: + hbase.security.authentication=kerberos + hbase.security.authorization=true + hbase.rpc.engine=org.apache.hadoop.hbase.ipc.SecureRpcEngine + index.search.backend=solr5 + index.search.solr.mode=cloud + index.search.solr.zookeeper-url=fqdn:2181/infra-solr + + 5.2. Copy necessary configuration files from the deployment to atlas-conf folder. For example: + hadoop-client/conf/core-site.xml + hadoop-client/conf/hdfs-site.xml + hadoop-client/conf/yarn-site.xml + ambari-infra-solr/conf/infra_solr_jaas.conf + ambari-infra-solr/conf/security.json + + 5.3. Update bin/atlas-gremlin.sh to set the following variable at the top. For example: + JAAS_CONF_FILE=infra_solr_jaas.conf + + 5.4. Kinit as atlas user with the command like: + kinit -kt /etc/security/keytabs/atlas.service.keytab atlas/[email protected] + +6. Ensure home directory for 'atlas' user exists in HDFS and this directory is owned by 'atlas' user + su hdfs + hdfs dfs -mkdir /user/atlas + hdfs dfs -chown atlas:hdfs -R /user/atlas + +7. Start Gremlin shell by executing the following command: + bin/atlas-gremlin.sh bin/atlas-index-repair.groovy + +8. Start index repair by entering the following in the Gremlin shell: + repairAtlasIndex("atlas-conf/atlas-titan.properties") http://git-wip-us.apache.org/repos/asf/atlas/blob/0e80d9b3/tools/atlas-index-repair-kit/atlas-index-repair-kit.tar.gz ---------------------------------------------------------------------- diff --git a/tools/atlas-index-repair-kit/atlas-index-repair-kit.tar.gz b/tools/atlas-index-repair-kit/atlas-index-repair-kit.tar.gz new file mode 100644 index 0000000..f4d4525 Binary files /dev/null and b/tools/atlas-index-repair-kit/atlas-index-repair-kit.tar.gz differ
