Kylin with S3, cubes tables get in transition when new cluster booted

[Moisés Català]

Hi all,

I’ve followed carefully the instructions provided in 
http://kylin.apache.org/docs23/install/kylin_aws_emr.html 
<http://kylin.apache.org/docs23/install/kylin_aws_emr.html> 

My idea is to use s3 as the storage for Hbase, I have configured the cluster 
following the instructions but I get that tables that contain cube definition 
keep "on transition" when deploying new cluster and Kylie metadata seems 
outdated...

These are the steps I follow to create the cluster

Cluster creation command:

aws emr create-cluster \
--applications Name=Hadoop Name=Hue Name=Spark Name=Zeppelin Name=Ganglia 
Name=Hive Name=Hbase Name=HCatalog Name=Tez \
--tags 'hive=' 'spark=' 'zeppelin=' \
--ec2-attributes 'file://../config/ec2-attributes.json 
<file://../config/ec2-attributes.json>' \
--release-label emr-5.16.0 \
--log-uri 's3n://sns-da-logs/ <s3n://sns-da-logs/>' \
--instance-groups 'file://../config/instance-hive-datawarehouse.json 
<file://../config/instance-hive-datawarehouse.json>' \
--configurations  'file://../config/hive-hbase-s3.json 
<file://../config/hive-hbase-s3.json>' \
--auto-scaling-role EMR_AutoScaling_DefaultRole \
--ebs-root-volume-size 10 \
--service-role EMR_DefaultRole \
--enable-debugging \
--name 'hbase-hive-datawarehouse' \
--scale-down-behavior TERMINATE_AT_TASK_COMPLETION \
--region us-east-1


My configuration hive-hbase-s3.json:

[
  {
    "Classification": "hive-site",
    "Configurations": [],
    "Properties": {
      "hive.metastore.warehouse.dir": "s3://xxxxxxxx-datawarehouse/hive.db 
<s3://xxxxxxxx-datawarehouse/hive.db>",
      "javax.jdo.option.ConnectionDriverName": "org.mariadb.jdbc.Driver",
      "javax.jdo.option.ConnectionPassword": “xxxxx",
      "javax.jdo.option.ConnectionURL": 
"jdbc:mysql://xxxxxx:3306/hive_metastore?createDatabaseIfNotExist=true 
<mysql://xxxxxx:3306/hive_metastore?createDatabaseIfNotExist=true>",
      "javax.jdo.option.ConnectionUserName": “xxxx"
    }
  },
  {
    "Classification": "hbase",
    "Configurations": [],
    "Properties": {
      "hbase.emr.storageMode": "s3"
    }
  },
  {
    "Classification": "hbase-site",
    "Configurations": [],
    "Properties": {
      "hbase.rpc.timeout": "3600000",
      "hbase.rootdir": "s3://xxxxxx-hbase/ <s3://xxxxxx-hbase/>"
    }
  },
  {
      "Classification": "core-site",
      "Properties": {
        "io.file.buffer.size": "65536"
      }
  },
  {
      "Classification": "mapred-site",
      "Properties": {
        "mapred.map.tasks.speculative.execution": "false",
        "mapred.reduce.tasks.speculative.execution": "false",
        "mapreduce.map.speculative": "false",
        "mapreduce.reduce.speculative": "false"

      }
  } 
]

When I shut down the cluster I perform these commands:

../kylin_home/bin/kylin.sh stop
 

#Before you shutdown/restart the cluster, you must backup the “/kylin” data on 
HDFS to S3 with S3DistCp,
  
aws s3 rm s3://xxxxxx-config/metadata/kylin/* 
<s3://xxxxxx-config/metadata/kylin/*>
s3-dist-cp --src=hdfs:///kylin <hdfs:///kylin> 
--dest=s3://xxxxxx-config/metadata/kylin <s3://xxxxxx-config/metadata/kylin>

bash /usr/lib/hbase/bin/disable_all_tables.sh


Please, could you be so kind to indicate me what am I missing


Thanks in advance