hi Yaqian, thanks for the response. I am using this method from the documentation: "II. Migrate across two Hadoop clusters" using Kylin version 3.1.0 on EMR 5.29 on both source and destination clusters.
I tried in 2 ways: (1) Migrate from one kylin instance on emr 5.29 to another kylin instance on emr 5.29 with the Hbase S3 location same on both. (2) Migrate from one kylin instance on emr 5.29 to another kylin instance on emr 5.29 with Hbase s3 location different. In both ways the cube metadata/definition was migrated but not the cube data. Here is the log for (1): [hadoop@ip-172-31-90-136 apache-kylin-3.1.0-bin-hbase1x]$ ./bin/kylin.sh org.apache.kylin.tool.migration.CubeMigrationCrossClusterCLI -kylinUriSrc ADMIN:KYLIN@ip-172-31-90-136.ec2.internal:7070 -kylinUriDst ADMIN:KYLIN@ip-172-31-81-180.ec2.internal:7070 -cube kylin_sales_cube -execute true -overwrite true Using cached dependency... OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/local/kylin/apache-kylin-3.1.0-bin-hbase1x/tool/kylin-tool-3.1.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/lib/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/lib/spark/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] Running org.apache.kylin.tool.migration.CubeMigrationCrossClusterCLI -kylinUriSrc ADMIN:KYLIN@ip-172-31-90-136.ec2.internal:7070 -kylinUriDst ADMIN:KYLIN@ip-172-31-81-180.ec2.internal:7070 -cube kylin_sales_cube -execute true -overwrite true 2021-02-25 09:36:50,551 INFO [main] common.KylinConfig:118 : Loading kylin-defaults.properties from file:/usr/local/kylin/apache-kylin-3.1.0-bin-hbase1x/tool/kylin-tool-3.1.0.jar!/kylin-defaults.properties 2021-02-25 09:36:50,576 DEBUG [main] common.KylinConfig:346 : KYLIN_CONF property was not set, will seek KYLIN_HOME env variable 2021-02-25 09:36:50,579 INFO [main] common.KylinConfig:352 : Use KYLIN_HOME=/usr/local/kylin/apache-kylin-3.1.0-bin-hbase1x 2021-02-25 09:36:50,582 INFO [main] common.KylinConfig:153 : Initialized a new KylinConfig from getInstanceFromEnv : 245765246 2021-02-25 09:36:55,215 INFO [main] zookeeper.RecoverableZooKeeper:120 : Process identifier=hconnection-0x5d3b58ca connecting to ZooKeeper ensemble=ip-172-31-90-136.ec2.internal:2181 2021-02-25 09:36:55,224 INFO [main] zookeeper.ZooKeeper:100 : Client environment:zookeeper.version=3.4.14-5f788d5e8f90539ee331702c753fa250727128f4, built on 12/14/2019 00:35 GMT 2021-02-25 09:36:55,224 INFO [main] zookeeper.ZooKeeper:100 : Client environment:host.name=ip-172-31-90-136.ec2.internal 2021-02-25 09:36:55,225 INFO [main] zookeeper.ZooKeeper:100 : Client environment:java.version=1.8.0_272 2021-02-25 09:36:55,225 INFO [main] zookeeper.ZooKeeper:100 : Client environment:java.vendor=Red Hat, Inc. 2021-02-25 09:36:55,225 INFO [main] zookeeper.ZooKeeper:100 : Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.272.b10-1.56.amzn1.x86_64/jre 2021-02-25 09:36:55,225 INFO [main] zookeeper.ZooKeeper:100 : Client environment:java.class.path=/*removed*/ 2021-02-25 09:36:55,226 INFO [main] zookeeper.ZooKeeper:100 : Client environment:java.library.path=:/usr/lib/hadoop-lzo/lib/native:/usr/lib/hadoop/lib/native 2021-02-25 09:36:55,227 INFO [main] zookeeper.ZooKeeper:100 : Client environment:java.io.tmpdir=/tmp 2021-02-25 09:36:55,227 INFO [main] zookeeper.ZooKeeper:100 : Client environment:java.compiler=<NA> 2021-02-25 09:36:55,227 INFO [main] zookeeper.ZooKeeper:100 : Client environment:os.name=Linux 2021-02-25 09:36:55,228 INFO [main] zookeeper.ZooKeeper:100 : Client environment:os.arch=amd64 2021-02-25 09:36:55,229 INFO [main] zookeeper.ZooKeeper:100 : Client environment:os.version=4.14.154-99.181.amzn1.x86_64 2021-02-25 09:36:55,229 INFO [main] zookeeper.ZooKeeper:100 : Client environment:user.name=hadoop 2021-02-25 09:36:55,229 INFO [main] zookeeper.ZooKeeper:100 : Client environment:user.home=/home/hadoop 2021-02-25 09:36:55,229 INFO [main] zookeeper.ZooKeeper:100 : Client environment:user.dir=/usr/local/kylin/apache-kylin-3.1.0-bin-hbase1x 2021-02-25 09:36:55,230 INFO [main] zookeeper.ZooKeeper:442 : Initiating client connection, connectString=ip-172-31-90-136.ec2.internal:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@7e91ed74 2021-02-25 09:36:55,289 INFO [main-SendThread(ip-172-31-90-136.ec2.internal:2181)] zookeeper.ClientCnxn:1025 : Opening socket connection to server ip-172-31-90-136.ec2.internal/172.31.90.136:2181. Will not attempt to authenticate using SASL (unknown error) 2021-02-25 09:36:55,303 INFO [main-SendThread(ip-172-31-90-136.ec2.internal:2181)] zookeeper.ClientCnxn:879 : Socket connection established to ip-172-31-90-136.ec2.internal/172.31.90.136:2181, initiating session 2021-02-25 09:36:55,309 INFO [main-SendThread(ip-172-31-90-136.ec2.internal:2181)] zookeeper.ClientCnxn:1299 : Session establishment complete on server ip-172-31-90-136.ec2.internal/172.31.90.136:2181, sessionid = 0x10000029296003a, negotiated timeout = 40000 2021-02-25 09:36:55,787 DEBUG [main] hbase.HBaseConnection:316 : HTable 'kylin_metadata' already exists 2021-02-25 09:36:56,135 INFO [main] common.KylinConfig:476 : Creating new manager instance of class org.apache.kylin.metadata.TableMetadataManager 2021-02-25 09:36:56,142 INFO [main] persistence.ResourceStore:90 : Using metadata url kylin_metadata@hbase for resource store 2021-02-25 09:36:56,207 INFO [main] hbase.HBaseConnection:267 : connection is null or closed, creating a new one 2021-02-25 09:36:56,207 INFO [main] zookeeper.RecoverableZooKeeper:120 : Process identifier=hconnection-0x3c25cfe1 connecting to ZooKeeper ensemble=ip-172-31-90-136.ec2.internal:2181 2021-02-25 09:36:56,208 INFO [main] zookeeper.ZooKeeper:442 : Initiating client connection, connectString=ip-172-31-90-136.ec2.internal:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@1d3c112a 2021-02-25 09:36:56,210 INFO [main-SendThread(ip-172-31-90-136.ec2.internal:2181)] zookeeper.ClientCnxn:1025 : Opening socket connection to server ip-172-31-90-136.ec2.internal/172.31.90.136:2181. Will not attempt to authenticate using SASL (unknown error) 2021-02-25 09:36:56,211 INFO [main-SendThread(ip-172-31-90-136.ec2.internal:2181)] zookeeper.ClientCnxn:879 : Socket connection established to ip-172-31-90-136.ec2.internal/172.31.90.136:2181, initiating session 2021-02-25 09:36:56,213 INFO [main-SendThread(ip-172-31-90-136.ec2.internal:2181)] zookeeper.ClientCnxn:1299 : Session establishment complete on server ip-172-31-90-136.ec2.internal/172.31.90.136:2181, sessionid = 0x10000029296003b, negotiated timeout = 40000 2021-02-25 09:36:56,219 DEBUG [main] hbase.HBaseConnection:316 : HTable 'kylin_metadata' already exists 2021-02-25 09:36:56,478 DEBUG [main] cachesync.CachedCrudAssist:122 : Reloading TableDesc from kylin_metadata(key='/table')@kylin_metadata@hbase 2021-02-25 09:36:56,546 INFO [main] measure.MeasureTypeFactory:121 : Checking custom measure types from kylin config 2021-02-25 09:36:56,547 INFO [main] measure.MeasureTypeFactory:150 : registering COUNT_DISTINCT(hllc), class org.apache.kylin.measure.hllc.HLLCMeasureType$Factory 2021-02-25 09:36:56,551 INFO [main] measure.MeasureTypeFactory:150 : registering COUNT_DISTINCT(bitmap), class org.apache.kylin.measure.bitmap.BitmapMeasureType$Factory 2021-02-25 09:36:56,552 INFO [main] measure.MeasureTypeFactory:150 : registering TOP_N(topn), class org.apache.kylin.measure.topn.TopNMeasureType$Factory 2021-02-25 09:36:56,554 INFO [main] measure.MeasureTypeFactory:150 : registering RAW(raw), class org.apache.kylin.measure.raw.RawMeasureType$Factory 2021-02-25 09:36:56,554 INFO [main] measure.MeasureTypeFactory:150 : registering EXTENDED_COLUMN(extendedcolumn), class org.apache.kylin.measure.extendedcolumn.ExtendedColumnMeasureType$Factory 2021-02-25 09:36:56,555 INFO [main] measure.MeasureTypeFactory:150 : registering PERCENTILE_APPROX(percentile), class org.apache.kylin.measure.percentile.PercentileMeasureType$Factory 2021-02-25 09:36:56,556 INFO [main] measure.MeasureTypeFactory:150 : registering COUNT_DISTINCT(dim_dc), class org.apache.kylin.measure.dim.DimCountDistinctMeasureType$Factory 2021-02-25 09:36:56,557 INFO [main] measure.MeasureTypeFactory:150 : registering STDDEV_SUM(stddev_sum), class org.apache.kylin.measure.stddev.StdDevSumMeasureType$Factory 2021-02-25 09:36:56,560 INFO [main] measure.MeasureTypeFactory:150 : registering COUNT_DISTINCT(bitmap_map), class org.apache.kylin.measure.map.bitmap.BitmapMapMeasureType$Factory 2021-02-25 09:36:56,563 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded 6 TableDesc(s) out of 6 resource with 0 errors 2021-02-25 09:36:56,564 INFO [main] common.KylinConfig:476 : Creating new manager instance of class org.apache.kylin.metadata.cachesync.Broadcaster 2021-02-25 09:36:56,565 DEBUG [main] cachesync.Broadcaster:102 : 1 nodes in the cluster: [localhost:7070] 2021-02-25 09:36:56,566 DEBUG [main] cachesync.CachedCrudAssist:122 : Reloading TableExtDesc from kylin_metadata(key='/table_exd')@kylin_metadata@hbase 2021-02-25 09:36:56,568 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded 0 TableExtDesc(s) out of 0 resource with 0 errors 2021-02-25 09:36:56,568 DEBUG [main] cachesync.CachedCrudAssist:122 : Reloading ExternalFilterDesc from kylin_metadata(key='/ext_filter')@kylin_metadata@hbase 2021-02-25 09:36:56,570 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded 0 ExternalFilterDesc(s) out of 0 resource with 0 errors 2021-02-25 09:36:56,572 INFO [main] common.KylinConfig:476 : Creating new manager instance of class org.apache.kylin.metadata.model.DataModelManager 2021-02-25 09:36:56,573 DEBUG [main] cachesync.CachedCrudAssist:122 : Reloading DataModelDesc from kylin_metadata(key='/model_desc')@kylin_metadata@hbase 2021-02-25 09:36:56,593 INFO [main] common.KylinConfig:476 : Creating new manager instance of class org.apache.kylin.metadata.project.ProjectManager 2021-02-25 09:36:56,593 INFO [main] project.ProjectManager:85 : Initializing ProjectManager with metadata url kylin_metadata@hbase 2021-02-25 09:36:56,594 DEBUG [main] cachesync.CachedCrudAssist:122 : Reloading ProjectInstance from kylin_metadata(key='/project')@kylin_metadata@hbase 2021-02-25 09:36:56,602 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded 1 ProjectInstance(s) out of 1 resource with 0 errors 2021-02-25 09:36:56,609 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded 2 DataModelDesc(s) out of 2 resource with 0 errors 2021-02-25 09:36:56,611 INFO [main] common.KylinConfig:476 : Creating new manager instance of class org.apache.kylin.storage.hybrid.HybridManager 2021-02-25 09:36:56,611 INFO [main] hybrid.HybridManager:71 : Initializing HybridManager with config kylin_metadata@hbase 2021-02-25 09:36:56,611 DEBUG [main] cachesync.CachedCrudAssist:122 : Reloading HybridInstance from kylin_metadata(key='/hybrid')@kylin_metadata@hbase 2021-02-25 09:36:56,613 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded 0 HybridInstance(s) out of 0 resource with 0 errors 2021-02-25 09:36:56,616 INFO [main] common.KylinConfig:476 : Creating new manager instance of class org.apache.kylin.cube.CubeManager 2021-02-25 09:36:56,623 INFO [main] cube.CubeManager:136 : Initializing CubeManager with config kylin_metadata@hbase 2021-02-25 09:36:56,623 DEBUG [main] cachesync.CachedCrudAssist:122 : Reloading CubeInstance from kylin_metadata(key='/cube')@kylin_metadata@hbase 2021-02-25 09:36:56,703 INFO [main] common.KylinConfig:476 : Creating new manager instance of class org.apache.kylin.cube.CubeDescManager 2021-02-25 09:36:56,704 INFO [main] cube.CubeDescManager:91 : Initializing CubeDescManager with config kylin_metadata@hbase 2021-02-25 09:36:56,704 DEBUG [main] cachesync.CachedCrudAssist:122 : Reloading CubeDesc from kylin_metadata(key='/cube_desc')@kylin_metadata@hbase 2021-02-25 09:36:56,819 INFO [main] model.RowKeyColDesc:99 : Implicitly convert encoding to time 2021-02-25 09:36:56,819 INFO [main] model.RowKeyColDesc:99 : Implicitly convert encoding to time 2021-02-25 09:36:56,819 INFO [main] model.RowKeyColDesc:95 : Implicitly convert encoding to date 2021-02-25 09:36:56,820 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded 2 CubeDesc(s) out of 2 resource with 0 errors 2021-02-25 09:36:56,820 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded 2 CubeInstance(s) out of 2 resource with 0 errors 2021-02-25 09:36:56,821 INFO [main] common.KylinConfig:476 : Creating new manager instance of class org.apache.kylin.metadata.realization.RealizationRegistry 2021-02-25 09:36:56,821 INFO [main] realization.RealizationRegistry:54 : Initializing RealizationRegistry with metadata url kylin_metadata@hbase 2021-02-25 09:36:56,821 INFO [main] realization.RealizationRegistry:81 : RealizationRegistry is {CUBE=org.apache.kylin.cube.CubeManager@1a1f79ce, HYBRID=org.apache.kylin.storage.hybrid.HybridManager@6aa7e176} 2021-02-25 09:36:56,822 INFO [main] common.KylinConfig:476 : Creating new manager instance of class org.apache.kylin.dict.DictionaryManager 2021-02-25 09:36:56,823 INFO [main] common.KylinConfig:476 : Creating new manager instance of class org.apache.kylin.dict.lookup.SnapshotManager 2021-02-25 09:36:57,143 INFO [main] zookeeper.RecoverableZooKeeper:120 : Process identifier=hconnection-0x2589d787 connecting to ZooKeeper ensemble=ip-172-31-81-180.ec2.internal:2181 2021-02-25 09:36:57,143 INFO [main] zookeeper.ZooKeeper:442 : Initiating client connection, connectString=ip-172-31-81-180.ec2.internal:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@1788cb61 2021-02-25 09:36:57,153 INFO [main-SendThread(ip-172-31-81-180.ec2.internal:2181)] zookeeper.ClientCnxn:1025 : Opening socket connection to server ip-172-31-81-180.ec2.internal/172.31.81.180:2181. Will not attempt to authenticate using SASL (unknown error) 2021-02-25 09:36:57,154 INFO [main-SendThread(ip-172-31-81-180.ec2.internal:2181)] zookeeper.ClientCnxn:879 : Socket connection established to ip-172-31-81-180.ec2.internal/172.31.81.180:2181, initiating session 2021-02-25 09:36:57,156 INFO [main-SendThread(ip-172-31-81-180.ec2.internal:2181)] zookeeper.ClientCnxn:1299 : Session establishment complete on server ip-172-31-81-180.ec2.internal/172.31.81.180:2181, sessionid = 0x100000279680020, negotiated timeout = 40000 2021-02-25 09:36:57,170 DEBUG [main] hbase.HBaseConnection:316 : HTable 'kylin_metadata' already exists 2021-02-25 09:36:57,178 INFO [main] migration.CubeMigrationCrossClusterCLI:327 : start to migrate cube CUBE[name=kylin_sales_cube] 2021-02-25 09:36:59,671 INFO [main] migration.DstClusterUtil:249 : saved resource /table/DEFAULT.KYLIN_CAL_DT.json 2021-02-25 09:36:59,672 INFO [main] migration.DstClusterUtil:162 : saved table TableDesc{name='KYLIN_CAL_DT', columns= /*removed*/ 2021-02-25 09:36:59,676 INFO [main] migration.DstClusterUtil:249 : saved resource /table/DEFAULT.KYLIN_CATEGORY_GROUPINGS.json 2021-02-25 09:36:59,676 INFO [main] migration.DstClusterUtil:162 : saved table TableDesc{name='KYLIN_CATEGORY_GROUPINGS', columns= /*removed*/ 2021-02-25 09:36:59,678 INFO [main] migration.DstClusterUtil:249 : saved resource /table/DEFAULT.KYLIN_ACCOUNT.json 2021-02-25 09:36:59,679 INFO [main] migration.DstClusterUtil:162 : saved table TableDesc{name='KYLIN_ACCOUNT', columns=/*removed*/ 2021-02-25 09:36:59,681 INFO [main] migration.DstClusterUtil:249 : saved resource /table/DEFAULT.KYLIN_COUNTRY.json 2021-02-25 09:36:59,681 INFO [main] migration.DstClusterUtil:162 : saved table TableDesc{name='KYLIN_COUNTRY', columns=/*removed*/ 2021-02-25 09:36:59,683 INFO [main] migration.DstClusterUtil:249 : saved resource /table/DEFAULT.KYLIN_SALES.json 2021-02-25 09:36:59,683 INFO [main] migration.DstClusterUtil:162 : saved table TableDesc{name='KYLIN_SALES', columns=/*removed*/ 2021-02-25 09:36:59,706 INFO [main] migration.DstClusterUtil:249 : saved resource /model_desc/kylin_sales_model.json 2021-02-25 09:36:59,706 INFO [main] migration.DstClusterUtil:170 : saved model DataModelDesc [name=kylin_sales_model] 2021-02-25 09:36:59,726 INFO [main] migration.DstClusterUtil:249 : saved resource /cube_desc/kylin_sales_cube.json 2021-02-25 09:36:59,726 INFO [main] migration.DstClusterUtil:178 : saved cube desc CubeDesc [name=kylin_sales_cube] 2021-02-25 09:36:59,731 INFO [main] migration.DstClusterUtil:249 : saved resource /cube/kylin_sales_cube.json 2021-02-25 09:36:59,731 INFO [main] migration.DstClusterUtil:185 : saved cube instance CUBE[name=kylin_sales_cube] 2021-02-25 09:36:59,739 INFO [main] migration.CubeMigrationCrossClusterCLI:330 : finish migrating cube CUBE[name=kylin_sales_cube] 2021-02-25 09:36:59,745 INFO [main] migration.DstClusterUtil:249 : saved resource /project/learn_kylin.json 2021-02-25 09:36:59,745 INFO [main] migration.DstClusterUtil:146 : saved project ProjectDesc [name=learn_kylin] 2021-02-25 09:36:59,745 INFO [main] migration.DstClusterUtil:363 : update meta cache for org.apache.kylin.common.restclient.RestClient@58d4238e 2021-02-25 09:36:59,749 INFO [main] migration.CubeMigrationCrossClusterCLI:359 : Migration for cubes [CUBE[name=kylin_sales_cube]], hyrbids [] all succeed 2021-02-25 09:36:59,751 INFO [close-hbase-conn] hbase.HBaseConnection:137 : Closing HBase connections... 2021-02-25 09:36:59,751 INFO [close-hbase-conn] client.ConnectionManager$HConnectionImplementation:2259 : Closing master protocol: MasterService 2021-02-25 09:36:59,753 INFO [close-hbase-conn] client.ConnectionManager$HConnectionImplementation:1776 : Closing zookeeper sessionid=0x10000029296003b 2021-02-25 09:36:59,755 INFO [close-hbase-conn] zookeeper.ZooKeeper:693 : Session: 0x10000029296003b closed 2021-02-25 09:36:59,755 INFO [main-EventThread] zookeeper.ClientCnxn:522 : EventThread shut down for session: 0x10000029296003b Here is the log for (2): [hadoop@ip-172-31-90-136 ~]$ cd $KYLIN_HOME [hadoop@ip-172-31-90-136 apache-kylin-3.1.0-bin-hbase1x]$ ./bin/kylin.sh org.apache.kylin.tool.migration.CubeMigrationCrossClusterCLI -kylinUriSrc ADMIN:KYLIN@ip-172-31-90-136.ec2.internal:7070 -kylinUriDst ADMIN:KYLIN@ip-172-31-85-14.ec2.internal:7070 -cube kylin_sales_cube -execute true -overwrite true Using cached dependency... OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0 SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/local/kylin/apache-kylin-3.1.0-bin-hbase1x/tool/kylin-tool-3.1.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/lib/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/lib/spark/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] Running org.apache.kylin.tool.migration.CubeMigrationCrossClusterCLI -kylinUriSrc ADMIN:KYLIN@ip-172-31-90-136.ec2.internal:7070 -kylinUriDst ADMIN:KYLIN@ip-172-31-85-14.ec2.internal:7070 -cube kylin_sales_cube -execute true -overwrite true 2021-02-25 09:44:09,900 INFO [main] common.KylinConfig:118 : Loading kylin-defaults.properties from file:/usr/local/kylin/apache-kylin-3.1.0-bin-hbase1x/tool/kylin-tool-3.1.0.jar!/kylin-defaults.properties 2021-02-25 09:44:09,935 DEBUG [main] common.KylinConfig:346 : KYLIN_CONF property was not set, will seek KYLIN_HOME env variable 2021-02-25 09:44:09,937 INFO [main] common.KylinConfig:352 : Use KYLIN_HOME=/usr/local/kylin/apache-kylin-3.1.0-bin-hbase1x 2021-02-25 09:44:09,940 INFO [main] common.KylinConfig:153 : Initialized a new KylinConfig from getInstanceFromEnv : 245765246 2021-02-25 09:44:14,486 INFO [main] zookeeper.RecoverableZooKeeper:120 : Process identifier=hconnection-0x5d3b58ca connecting to ZooKeeper ensemble=ip-172-31-90-136.ec2.internal:2181 2021-02-25 09:44:14,492 INFO [main] zookeeper.ZooKeeper:100 : Client environment:zookeeper.version=3.4.14-5f788d5e8f90539ee331702c753fa250727128f4, built on 12/14/2019 00:35 GMT 2021-02-25 09:44:14,492 INFO [main] zookeeper.ZooKeeper:100 : Client environment:host.name=ip-172-31-90-136.ec2.internal 2021-02-25 09:44:14,493 INFO [main] zookeeper.ZooKeeper:100 : Client environment:java.version=1.8.0_272 2021-02-25 09:44:14,493 INFO [main] zookeeper.ZooKeeper:100 : Client environment:java.vendor=Red Hat, Inc. 2021-02-25 09:44:14,493 INFO [main] zookeeper.ZooKeeper:100 : Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.272.b10-1.56.amzn1.x86_64/jre 2021-02-25 09:44:14,493 INFO [main] zookeeper.ZooKeeper:100 : Client environment:java.class.path=/*REMOVED*/ 2021-02-25 09:44:14,494 INFO [main] zookeeper.ZooKeeper:100 : Client environment:java.library.path=:/usr/lib/hadoop-lzo/lib/native:/usr/lib/hadoop/lib/native 2021-02-25 09:44:14,494 INFO [main] zookeeper.ZooKeeper:100 : Client environment:java.io.tmpdir=/tmp 2021-02-25 09:44:14,494 INFO [main] zookeeper.ZooKeeper:100 : Client environment:java.compiler=<NA> 2021-02-25 09:44:14,495 INFO [main] zookeeper.ZooKeeper:100 : Client environment:os.name=Linux 2021-02-25 09:44:14,495 INFO [main] zookeeper.ZooKeeper:100 : Client environment:os.arch=amd64 2021-02-25 09:44:14,496 INFO [main] zookeeper.ZooKeeper:100 : Client environment:os.version=4.14.154-99.181.amzn1.x86_64 2021-02-25 09:44:14,496 INFO [main] zookeeper.ZooKeeper:100 : Client environment:user.name=hadoop 2021-02-25 09:44:14,496 INFO [main] zookeeper.ZooKeeper:100 : Client environment:user.home=/home/hadoop 2021-02-25 09:44:14,496 INFO [main] zookeeper.ZooKeeper:100 : Client environment:user.dir=/usr/local/kylin/apache-kylin-3.1.0-bin-hbase1x 2021-02-25 09:44:14,497 INFO [main] zookeeper.ZooKeeper:442 : Initiating client connection, connectString=ip-172-31-90-136.ec2.internal:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@7e91ed74 2021-02-25 09:44:14,564 INFO [main-SendThread(ip-172-31-90-136.ec2.internal:2181)] zookeeper.ClientCnxn:1025 : Opening socket connection to server ip-172-31-90-136.ec2.internal/172.31.90.136:2181. Will not attempt to authenticate using SASL (unknown error) 2021-02-25 09:44:14,569 INFO [main-SendThread(ip-172-31-90-136.ec2.internal:2181)] zookeeper.ClientCnxn:879 : Socket connection established to ip-172-31-90-136.ec2.internal/172.31.90.136:2181, initiating session 2021-02-25 09:44:14,574 INFO [main-SendThread(ip-172-31-90-136.ec2.internal:2181)] zookeeper.ClientCnxn:1299 : Session establishment complete on server ip-172-31-90-136.ec2.internal/172.31.90.136:2181, sessionid = 0x10000029296003c, negotiated timeout = 40000 2021-02-25 09:44:15,012 DEBUG [main] hbase.HBaseConnection:316 : HTable 'kylin_metadata' already exists 2021-02-25 09:44:15,356 INFO [main] common.KylinConfig:476 : Creating new manager instance of class org.apache.kylin.metadata.TableMetadataManager 2021-02-25 09:44:15,364 INFO [main] persistence.ResourceStore:90 : Using metadata url kylin_metadata@hbase for resource store 2021-02-25 09:44:15,426 INFO [main] hbase.HBaseConnection:267 : connection is null or closed, creating a new one 2021-02-25 09:44:15,426 INFO [main] zookeeper.RecoverableZooKeeper:120 : Process identifier=hconnection-0x3c25cfe1 connecting to ZooKeeper ensemble=ip-172-31-90-136.ec2.internal:2181 2021-02-25 09:44:15,427 INFO [main] zookeeper.ZooKeeper:442 : Initiating client connection, connectString=ip-172-31-90-136.ec2.internal:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@1d3c112a 2021-02-25 09:44:15,428 INFO [main-SendThread(ip-172-31-90-136.ec2.internal:2181)] zookeeper.ClientCnxn:1025 : Opening socket connection to server ip-172-31-90-136.ec2.internal/172.31.90.136:2181. Will not attempt to authenticate using SASL (unknown error) 2021-02-25 09:44:15,428 INFO [main-SendThread(ip-172-31-90-136.ec2.internal:2181)] zookeeper.ClientCnxn:879 : Socket connection established to ip-172-31-90-136.ec2.internal/172.31.90.136:2181, initiating session 2021-02-25 09:44:15,430 INFO [main-SendThread(ip-172-31-90-136.ec2.internal:2181)] zookeeper.ClientCnxn:1299 : Session establishment complete on server ip-172-31-90-136.ec2.internal/172.31.90.136:2181, sessionid = 0x10000029296003d, negotiated timeout = 40000 2021-02-25 09:44:15,436 DEBUG [main] hbase.HBaseConnection:316 : HTable 'kylin_metadata' already exists 2021-02-25 09:44:15,552 DEBUG [main] cachesync.CachedCrudAssist:122 : Reloading TableDesc from kylin_metadata(key='/table')@kylin_metadata@hbase 2021-02-25 09:44:15,626 INFO [main] measure.MeasureTypeFactory:121 : Checking custom measure types from kylin config 2021-02-25 09:44:15,627 INFO [main] measure.MeasureTypeFactory:150 : registering COUNT_DISTINCT(hllc), class org.apache.kylin.measure.hllc.HLLCMeasureType$Factory 2021-02-25 09:44:15,630 INFO [main] measure.MeasureTypeFactory:150 : registering COUNT_DISTINCT(bitmap), class org.apache.kylin.measure.bitmap.BitmapMeasureType$Factory 2021-02-25 09:44:15,632 INFO [main] measure.MeasureTypeFactory:150 : registering TOP_N(topn), class org.apache.kylin.measure.topn.TopNMeasureType$Factory 2021-02-25 09:44:15,633 INFO [main] measure.MeasureTypeFactory:150 : registering RAW(raw), class org.apache.kylin.measure.raw.RawMeasureType$Factory 2021-02-25 09:44:15,634 INFO [main] measure.MeasureTypeFactory:150 : registering EXTENDED_COLUMN(extendedcolumn), class org.apache.kylin.measure.extendedcolumn.ExtendedColumnMeasureType$Factory 2021-02-25 09:44:15,635 INFO [main] measure.MeasureTypeFactory:150 : registering PERCENTILE_APPROX(percentile), class org.apache.kylin.measure.percentile.PercentileMeasureType$Factory 2021-02-25 09:44:15,637 INFO [main] measure.MeasureTypeFactory:150 : registering COUNT_DISTINCT(dim_dc), class org.apache.kylin.measure.dim.DimCountDistinctMeasureType$Factory 2021-02-25 09:44:15,637 INFO [main] measure.MeasureTypeFactory:150 : registering STDDEV_SUM(stddev_sum), class org.apache.kylin.measure.stddev.StdDevSumMeasureType$Factory 2021-02-25 09:44:15,641 INFO [main] measure.MeasureTypeFactory:150 : registering COUNT_DISTINCT(bitmap_map), class org.apache.kylin.measure.map.bitmap.BitmapMapMeasureType$Factory 2021-02-25 09:44:15,644 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded 6 TableDesc(s) out of 6 resource with 0 errors 2021-02-25 09:44:15,645 INFO [main] common.KylinConfig:476 : Creating new manager instance of class org.apache.kylin.metadata.cachesync.Broadcaster 2021-02-25 09:44:15,646 DEBUG [main] cachesync.Broadcaster:102 : 1 nodes in the cluster: [localhost:7070] 2021-02-25 09:44:15,647 DEBUG [main] cachesync.CachedCrudAssist:122 : Reloading TableExtDesc from kylin_metadata(key='/table_exd')@kylin_metadata@hbase 2021-02-25 09:44:15,649 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded 0 TableExtDesc(s) out of 0 resource with 0 errors 2021-02-25 09:44:15,650 DEBUG [main] cachesync.CachedCrudAssist:122 : Reloading ExternalFilterDesc from kylin_metadata(key='/ext_filter')@kylin_metadata@hbase 2021-02-25 09:44:15,651 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded 0 ExternalFilterDesc(s) out of 0 resource with 0 errors 2021-02-25 09:44:15,653 INFO [main] common.KylinConfig:476 : Creating new manager instance of class org.apache.kylin.metadata.model.DataModelManager 2021-02-25 09:44:15,654 DEBUG [main] cachesync.CachedCrudAssist:122 : Reloading DataModelDesc from kylin_metadata(key='/model_desc')@kylin_metadata@hbase 2021-02-25 09:44:15,676 INFO [main] common.KylinConfig:476 : Creating new manager instance of class org.apache.kylin.metadata.project.ProjectManager 2021-02-25 09:44:15,677 INFO [main] project.ProjectManager:85 : Initializing ProjectManager with metadata url kylin_metadata@hbase 2021-02-25 09:44:15,678 DEBUG [main] cachesync.CachedCrudAssist:122 : Reloading ProjectInstance from kylin_metadata(key='/project')@kylin_metadata@hbase 2021-02-25 09:44:15,687 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded 1 ProjectInstance(s) out of 1 resource with 0 errors 2021-02-25 09:44:15,693 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded 2 DataModelDesc(s) out of 2 resource with 0 errors 2021-02-25 09:44:15,694 INFO [main] common.KylinConfig:476 : Creating new manager instance of class org.apache.kylin.storage.hybrid.HybridManager 2021-02-25 09:44:15,695 INFO [main] hybrid.HybridManager:71 : Initializing HybridManager with config kylin_metadata@hbase 2021-02-25 09:44:15,695 DEBUG [main] cachesync.CachedCrudAssist:122 : Reloading HybridInstance from kylin_metadata(key='/hybrid')@kylin_metadata@hbase 2021-02-25 09:44:15,697 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded 0 HybridInstance(s) out of 0 resource with 0 errors 2021-02-25 09:44:15,700 INFO [main] common.KylinConfig:476 : Creating new manager instance of class org.apache.kylin.cube.CubeManager 2021-02-25 09:44:15,707 INFO [main] cube.CubeManager:136 : Initializing CubeManager with config kylin_metadata@hbase 2021-02-25 09:44:15,708 DEBUG [main] cachesync.CachedCrudAssist:122 : Reloading CubeInstance from kylin_metadata(key='/cube')@kylin_metadata@hbase 2021-02-25 09:44:15,783 INFO [main] common.KylinConfig:476 : Creating new manager instance of class org.apache.kylin.cube.CubeDescManager 2021-02-25 09:44:15,783 INFO [main] cube.CubeDescManager:91 : Initializing CubeDescManager with config kylin_metadata@hbase 2021-02-25 09:44:15,783 DEBUG [main] cachesync.CachedCrudAssist:122 : Reloading CubeDesc from kylin_metadata(key='/cube_desc')@kylin_metadata@hbase 2021-02-25 09:44:15,873 INFO [main] model.RowKeyColDesc:99 : Implicitly convert encoding to time 2021-02-25 09:44:15,873 INFO [main] model.RowKeyColDesc:99 : Implicitly convert encoding to time 2021-02-25 09:44:15,873 INFO [main] model.RowKeyColDesc:95 : Implicitly convert encoding to date 2021-02-25 09:44:15,873 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded 2 CubeDesc(s) out of 2 resource with 0 errors 2021-02-25 09:44:15,874 DEBUG [main] cachesync.CachedCrudAssist:155 : Loaded 2 CubeInstance(s) out of 2 resource with 0 errors 2021-02-25 09:44:15,874 INFO [main] common.KylinConfig:476 : Creating new manager instance of class org.apache.kylin.metadata.realization.RealizationRegistry 2021-02-25 09:44:15,875 INFO [main] realization.RealizationRegistry:54 : Initializing RealizationRegistry with metadata url kylin_metadata@hbase 2021-02-25 09:44:15,875 INFO [main] realization.RealizationRegistry:81 : RealizationRegistry is {CUBE=org.apache.kylin.cube.CubeManager@1a1f79ce, HYBRID=org.apache.kylin.storage.hybrid.HybridManager@6aa7e176} 2021-02-25 09:44:15,876 INFO [main] common.KylinConfig:476 : Creating new manager instance of class org.apache.kylin.dict.DictionaryManager 2021-02-25 09:44:15,876 INFO [main] common.KylinConfig:476 : Creating new manager instance of class org.apache.kylin.dict.lookup.SnapshotManager 2021-02-25 09:44:16,138 INFO [main] zookeeper.RecoverableZooKeeper:120 : Process identifier=hconnection-0x2589d787 connecting to ZooKeeper ensemble=ip-172-31-85-14.ec2.internal:2181 2021-02-25 09:44:16,138 INFO [main] zookeeper.ZooKeeper:442 : Initiating client connection, connectString=ip-172-31-85-14.ec2.internal:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@1788cb61 2021-02-25 09:44:16,140 INFO [main-SendThread(ip-172-31-85-14.ec2.internal:2181)] zookeeper.ClientCnxn:1025 : Opening socket connection to server ip-172-31-85-14.ec2.internal/172.31.85.14:2181. Will not attempt to authenticate using SASL (unknown error) 2021-02-25 09:44:16,141 INFO [main-SendThread(ip-172-31-85-14.ec2.internal:2181)] zookeeper.ClientCnxn:879 : Socket connection established to ip-172-31-85-14.ec2.internal/172.31.85.14:2181, initiating session 2021-02-25 09:44:16,143 INFO [main-SendThread(ip-172-31-85-14.ec2.internal:2181)] zookeeper.ClientCnxn:1299 : Session establishment complete on server ip-172-31-85-14.ec2.internal/172.31.85.14:2181, sessionid = 0x10000028a4b001f, negotiated timeout = 40000 2021-02-25 09:44:16,150 DEBUG [main] hbase.HBaseConnection:316 : HTable 'kylin_metadata' already exists 2021-02-25 09:44:16,156 INFO [main] migration.CubeMigrationCrossClusterCLI:327 : start to migrate cube CUBE[name=kylin_sales_cube] 2021-02-25 09:44:18,950 INFO [main] migration.DstClusterUtil:249 : saved resource /table/DEFAULT.KYLIN_CAL_DT.json 2021-02-25 09:44:18,951 INFO [main] migration.DstClusterUtil:162 : saved table TableDesc{name='KYLIN_CAL_DT', columns=/*REMOVED*/ 2021-02-25 09:44:18,955 INFO [main] migration.DstClusterUtil:249 : saved resource /table/DEFAULT.KYLIN_CATEGORY_GROUPINGS.json 2021-02-25 09:44:18,955 INFO [main] migration.DstClusterUtil:162 : saved table TableDesc{name='KYLIN_CATEGORY_GROUPINGS', columns=/*REMOVED*/ 2021-02-25 09:44:18,959 INFO [main] migration.DstClusterUtil:249 : saved resource /table/DEFAULT.KYLIN_ACCOUNT.json 2021-02-25 09:44:18,959 INFO [main] migration.DstClusterUtil:162 : saved table TableDesc{name='KYLIN_ACCOUNT', columns=/*REMOVED*/ 2021-02-25 09:44:18,979 INFO [main] migration.DstClusterUtil:249 : saved resource /table/DEFAULT.KYLIN_COUNTRY.json 2021-02-25 09:44:18,979 INFO [main] migration.DstClusterUtil:162 : saved table TableDesc{name='KYLIN_COUNTRY', columns=/*REMOVED*/ 2021-02-25 09:44:18,982 INFO [main] migration.DstClusterUtil:249 : saved resource /table/DEFAULT.KYLIN_SALES.json 2021-02-25 09:44:18,982 INFO [main] migration.DstClusterUtil:162 : saved table TableDesc{name='KYLIN_SALES', columns=/*REMOVED*/ 2021-02-25 09:44:18,985 INFO [main] migration.DstClusterUtil:249 : saved resource /model_desc/kylin_sales_model.json 2021-02-25 09:44:18,985 INFO [main] migration.DstClusterUtil:170 : saved model DataModelDesc [name=kylin_sales_model] 2021-02-25 09:44:19,006 INFO [main] migration.DstClusterUtil:249 : saved resource /cube_desc/kylin_sales_cube.json 2021-02-25 09:44:19,007 INFO [main] migration.DstClusterUtil:178 : saved cube desc CubeDesc [name=kylin_sales_cube] 2021-02-25 09:44:19,012 INFO [main] migration.DstClusterUtil:249 : saved resource /cube/kylin_sales_cube.json 2021-02-25 09:44:19,012 INFO [main] migration.DstClusterUtil:185 : saved cube instance CUBE[name=kylin_sales_cube] 2021-02-25 09:44:19,020 INFO [main] migration.CubeMigrationCrossClusterCLI:330 : finish migrating cube CUBE[name=kylin_sales_cube] 2021-02-25 09:44:19,026 INFO [main] migration.DstClusterUtil:249 : saved resource /project/learn_kylin.json 2021-02-25 09:44:19,026 INFO [main] migration.DstClusterUtil:146 : saved project ProjectDesc [name=learn_kylin] 2021-02-25 09:44:19,026 INFO [main] migration.DstClusterUtil:363 : update meta cache for org.apache.kylin.common.restclient.RestClient@58d4238e 2021-02-25 09:44:19,030 INFO [main] migration.CubeMigrationCrossClusterCLI:359 : Migration for cubes [CUBE[name=kylin_sales_cube]], hyrbids [] all succeed 2021-02-25 09:44:19,032 INFO [close-hbase-conn] hbase.HBaseConnection:137 : Closing HBase connections... 2021-02-25 09:44:19,034 INFO [close-hbase-conn] client.ConnectionManager$HConnectionImplementation:2259 : Closing master protocol: MasterService 2021-02-25 09:44:19,037 INFO [close-hbase-conn] client.ConnectionManager$HConnectionImplementation:1776 : Closing zookeeper sessionid=0x10000029296003d 2021-02-25 09:44:19,039 INFO [main-EventThread] zookeeper.ClientCnxn:522 : EventThread shut down for session: 0x10000029296003d 2021-02-25 09:44:19,039 INFO [close-hbase-conn] zookeeper.ZooKeeper:693 : Session: 0x10000029296003d closed [hadoop@ip-172-31-90-136 apache-kylin-3.1.0-bin-hbase1x]$ -- Sent from: http://apache-kylin.74782.x6.nabble.com/
