[
https://issues.apache.org/jira/browse/KYLIN-3035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16249433#comment-16249433
]
Shawn Wang commented on KYLIN-3035:
-----------------------------------
Hi, I tried the steps.
My kylin.properties:
{noformat}
kylin.env.hdfs-working-dir=s3://kylin-emrfs/kylin-working-dir
kylin.server.mode=query
kylin.storage.hbase.cluster-fs=s3://kylin-emrfs
{noformat}
And in my hbase-site.xml:
{noformat}
<property>
<name>hbase.rootdir</name>
<value>s3://kylin-emrfs/hbase-production</value>
</property>
{noformat}
Redeploy coprocessor gives error:
{noformat}
[hadoop@ip-172-31-1-10 apache-kylin-2.2.0-bin]$ $KYLIN_HOME/bin/kylin.sh
org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI default all
Retrieving hadoop conf dir...
KYLIN_HOME is set to /usr/local/kylin/apache-kylin-2.2.0-bin
Retrieving hive dependency...
Retrieving hbase dependency...
Retrieving hadoop conf dir...
Retrieving kafka dependency...
Retrieving Spark dependency...
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/usr/local/kylin/apache-kylin-2.2.0-bin/tool/kylin-tool-2.2.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/lib/hive/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/local/kylin/apache-kylin-2.2.0-bin/spark/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2017-11-13 11:30:13,085 INFO [main] common.KylinConfig:315 : Loading
kylin-defaults.properties from
file:/usr/local/kylin/apache-kylin-2.2.0-bin/tool/kylin-tool-2.2.0.jar!/kylin-defaults.properties
2017-11-13 11:30:13,123 DEBUG [main] common.KylinConfig:274 : KYLIN_CONF
property was not set, will seek KYLIN_HOME env variable
2017-11-13 11:30:13,129 INFO [main] common.KylinConfig:95 : Initialized a new
KylinConfig from getInstanceFromEnv : 1285524499
2017-11-13 11:30:17,375 INFO [main] hbase.HBaseConnection:230 : connection is
null or closed, creating a new one
2017-11-13 11:30:17,504 INFO [main] zookeeper.RecoverableZooKeeper:120 :
Process identifier=hconnection-0x74170687 connecting to ZooKeeper
ensemble=ip-172-31-1-10.cn-north-1.compute.internal:2181
2017-11-13 11:30:17,512 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:zookeeper.version=3.4.10-0338e6621a7a828d8224b10e83a4d72395c41c29,
built on 04/14/2017 23:18 GMT
2017-11-13 11:30:17,512 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:host.name=ip-172-31-1-10.cn-north-1.compute.internal
2017-11-13 11:30:17,512 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:java.version=1.8.0_141
2017-11-13 11:30:17,512 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:java.vendor=Oracle Corporation
2017-11-13 11:30:17,513 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.141-1.b16.32.amzn1.x86_64/jre
2017-11-13 11:30:17,513 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:java.class.path=/ ......
2017-11-13 11:30:17,516 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:java.library.path=:/usr/lib/hadoop-lzo/lib/native:/usr/lib/hadoop/lib/native
2017-11-13 11:30:17,516 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:java.io.tmpdir=/tmp
2017-11-13 11:30:17,517 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:java.compiler=<NA>
2017-11-13 11:30:17,517 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:os.name=Linux
2017-11-13 11:30:17,517 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:os.arch=amd64
2017-11-13 11:30:17,518 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:os.version=4.4.35-33.55.amzn1.x86_64
2017-11-13 11:30:17,519 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:user.name=hadoop
2017-11-13 11:30:17,519 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:user.home=/home/hadoop
2017-11-13 11:30:17,519 INFO [main] zookeeper.ZooKeeper:100 : Client
environment:user.dir=/usr/local/kylin/apache-kylin-2.2.0-bin
2017-11-13 11:30:17,520 INFO [main] zookeeper.ZooKeeper:438 : Initiating
client connection,
connectString=ip-172-31-1-10.cn-north-1.compute.internal:2181
sessionTimeout=90000
watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@3e3315d9
2017-11-13 11:30:17,544 INFO
[main-SendThread(ip-172-31-1-10.cn-north-1.compute.internal:2181)]
zookeeper.ClientCnxn:1032 : Opening socket connection to server
ip-172-31-1-10.cn-north-1.compute.internal/172.31.1.10:2181. Will not attempt
to authenticate using SASL (unknown error)
2017-11-13 11:30:17,553 INFO
[main-SendThread(ip-172-31-1-10.cn-north-1.compute.internal:2181)]
zookeeper.ClientCnxn:876 : Socket connection established to
ip-172-31-1-10.cn-north-1.compute.internal/172.31.1.10:2181, initiating session
2017-11-13 11:30:17,561 INFO
[main-SendThread(ip-172-31-1-10.cn-north-1.compute.internal:2181)]
zookeeper.ClientCnxn:1299 : Session establishment complete on server
ip-172-31-1-10.cn-north-1.compute.internal/172.31.1.10:2181, sessionid =
0x15fb4173c1006a0, negotiated timeout = 40000
2017-11-13 11:30:17,713 INFO [main] util.DeployCoprocessorCLI:100 : Identify
coprocessor jar
/usr/local/kylin/apache-kylin-2.2.0-bin/lib/kylin-coprocessor-2.2.0.jar
2017-11-13 11:30:17,718 INFO [main] cube.CubeManager:140 : Initializing
CubeManager with config kylin_metadata@hbase
2017-11-13 11:30:17,720 INFO [main] persistence.ResourceStore:79 : Using
metadata url kylin_metadata@hbase for resource store
2017-11-13 11:30:17,800 INFO [main] hbase.HBaseConnection:230 : connection is
null or closed, creating a new one
2017-11-13 11:30:17,801 INFO [main] zookeeper.RecoverableZooKeeper:120 :
Process identifier=hconnection-0x7b32b129 connecting to ZooKeeper
ensemble=ip-172-31-1-10.cn-north-1.compute.internal:2181
2017-11-13 11:30:17,801 INFO [main] zookeeper.ZooKeeper:438 : Initiating
client connection,
connectString=ip-172-31-1-10.cn-north-1.compute.internal:2181
sessionTimeout=90000
watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@439e3cb4
2017-11-13 11:30:17,803 INFO
[main-SendThread(ip-172-31-1-10.cn-north-1.compute.internal:2181)]
zookeeper.ClientCnxn:1032 : Opening socket connection to server
ip-172-31-1-10.cn-north-1.compute.internal/172.31.1.10:2181. Will not attempt
to authenticate using SASL (unknown error)
2017-11-13 11:30:17,803 INFO
[main-SendThread(ip-172-31-1-10.cn-north-1.compute.internal:2181)]
zookeeper.ClientCnxn:876 : Socket connection established to
ip-172-31-1-10.cn-north-1.compute.internal/172.31.1.10:2181, initiating session
2017-11-13 11:30:17,805 INFO
[main-SendThread(ip-172-31-1-10.cn-north-1.compute.internal:2181)]
zookeeper.ClientCnxn:1299 : Session establishment complete on server
ip-172-31-1-10.cn-north-1.compute.internal/172.31.1.10:2181, sessionid =
0x15fb4173c1006a1, negotiated timeout = 40000
2017-11-13 11:30:18,085 DEBUG [main] hbase.HBaseConnection:279 : HTable
'kylin_metadata' already exists
2017-11-13 11:30:18,788 INFO [main] cube.CubeManager:744 : Loading Cube from
folder kylin_metadata(key='/cube')@kylin_metadata@hbase
2017-11-13 11:30:18,799 ERROR [main] cube.CubeManager:800 : Error during load
cube instance, skipping : /cube/test_goods_cube.json
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
attempts=1, exceptions:
Mon Nov 13 11:30:18 UTC 2017, RpcRetryingCaller{globalStartTime=1510572618788,
pause=100, retries=1},
org.apache.hadoop.hbase.regionserver.RegionServerAbortedException:
org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server
ip-172-31-4-89.cn-north-1.compute.internal,16020,1510572608521 aborting
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1168)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2069)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:34946)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2328)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:159)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:864)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:830)
at
org.apache.kylin.storage.hbase.HBaseResourceStore.internalGetFromHTable(HBaseResourceStore.java:385)
at
org.apache.kylin.storage.hbase.HBaseResourceStore.getFromHTable(HBaseResourceStore.java:363)
at
org.apache.kylin.storage.hbase.HBaseResourceStore.getResourceImpl(HBaseResourceStore.java:273)
at
org.apache.kylin.common.persistence.ResourceStore.getResource(ResourceStore.java:154)
at
org.apache.kylin.cube.CubeManager.reloadCubeLocalAt(CubeManager.java:765)
at
org.apache.kylin.cube.CubeManager.loadAllCubeInstance(CubeManager.java:749)
at org.apache.kylin.cube.CubeManager.<init>(CubeManager.java:145)
at org.apache.kylin.cube.CubeManager.getInstance(CubeManager.java:108)
at
org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI.getHTableNames(DeployCoprocessorCLI.java:465)
at
org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI.main(DeployCoprocessorCLI.java:102)
Caused by: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException:
org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server
ip-172-31-4-89.cn-north-1.compute.internal,16020,1510572608521 aborting
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1168)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2069)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:34946)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2328)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
at
org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332)
at org.apache.hadoop.hbase.client.HTable$3.call(HTable.java:860)
at org.apache.hadoop.hbase.client.HTable$3.call(HTable.java:847)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:137)
... 12 more
Caused by:
org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionServerAbortedException):
org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server
ip-172-31-4-89.cn-north-1.compute.internal,16020,1510572608521 aborting
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1168)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2069)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:34946)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2328)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
at
org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.get(ClientProtos.java:35372)
at org.apache.hadoop.hbase.client.HTable$3.call(HTable.java:856)
... 14 more
2017-11-13 11:30:18,806 ERROR [main] cube.CubeManager:800 : Error during load
cube instance, skipping : /cube/tt_test.json
org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after
attempts=1, exceptions:
Mon Nov 13 11:30:18 UTC 2017, RpcRetryingCaller{globalStartTime=1510572618801,
pause=100, retries=1},
org.apache.hadoop.hbase.regionserver.RegionServerAbortedException:
org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server
ip-172-31-4-89.cn-north-1.compute.internal,16020,1510572608521 aborting
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1168)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2069)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:34946)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2328)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:159)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:864)
at org.apache.hadoop.hbase.client.HTable.get(HTable.java:830)
at
org.apache.kylin.storage.hbase.HBaseResourceStore.internalGetFromHTable(HBaseResourceStore.java:385)
at
org.apache.kylin.storage.hbase.HBaseResourceStore.getFromHTable(HBaseResourceStore.java:363)
at
org.apache.kylin.storage.hbase.HBaseResourceStore.getResourceImpl(HBaseResourceStore.java:273)
at
org.apache.kylin.common.persistence.ResourceStore.getResource(ResourceStore.java:154)
at
org.apache.kylin.cube.CubeManager.reloadCubeLocalAt(CubeManager.java:765)
at
org.apache.kylin.cube.CubeManager.loadAllCubeInstance(CubeManager.java:749)
at org.apache.kylin.cube.CubeManager.<init>(CubeManager.java:145)
at org.apache.kylin.cube.CubeManager.getInstance(CubeManager.java:108)
at
org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI.getHTableNames(DeployCoprocessorCLI.java:465)
at
org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI.main(DeployCoprocessorCLI.java:102)
Caused by: org.apache.hadoop.hbase.regionserver.RegionServerAbortedException:
org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server
ip-172-31-4-89.cn-north-1.compute.internal,16020,1510572608521 aborting
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1168)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2069)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:34946)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2328)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
at
org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:332)
at org.apache.hadoop.hbase.client.HTable$3.call(HTable.java:860)
at org.apache.hadoop.hbase.client.HTable$3.call(HTable.java:847)
at
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:137)
... 12 more
Caused by:
org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.regionserver.RegionServerAbortedException):
org.apache.hadoop.hbase.regionserver.RegionServerAbortedException: Server
ip-172-31-4-89.cn-north-1.compute.internal,16020,1510572608521 aborting
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.checkOpen(RSRpcServices.java:1168)
at
org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2069)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:34946)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2328)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:188)
at
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:168)
at
org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1270)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
at
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.get(ClientProtos.java:35372)
at org.apache.hadoop.hbase.client.HTable$3.call(HTable.java:856)
... 14 more
2017-11-13 11:30:18,807 INFO [main] cube.CubeManager:757 : Loaded 0 cubes,
fail on 2 cubes
2017-11-13 11:30:18,809 DEBUG [main] cachesync.Broadcaster:122 : 1 nodes in the
cluster: [localhost:7070]
2017-11-13 11:30:18,810 INFO [main] util.DeployCoprocessorCLI:103 : Identify
tables []
2017-11-13 11:30:18,810 INFO [main] util.DeployCoprocessorCLI:116 : Will
execute tables []
2017-11-13 11:30:18,811 INFO [main] util.DeployCoprocessorCLI:120 : Old
coprocessor jar: []
2017-11-13 11:30:19,496 INFO [main] util.DeployCoprocessorCLI:123 : New
coprocessor jar:
s3://kylin-emrfs/kylin-working-dir/kylin_metadata/coprocessor/kylin-coprocessor-2.2.0-0.jar
2017-11-13 11:30:19,496 INFO [main] util.DeployCoprocessorCLI:132 : Processed
time: 685
2017-11-13 11:30:19,496 INFO [main] util.DeployCoprocessorCLI:133 : Processed
tables count: 0
2017-11-13 11:30:19,496 INFO [main] util.DeployCoprocessorCLI:134 : Processed
tables: []
2017-11-13 11:30:19,496 INFO [main] util.DeployCoprocessorCLI:135 : Active
coprocessor jar:
s3://kylin-emrfs/kylin-working-dir/kylin_metadata/coprocessor/kylin-coprocessor-2.2.0-0.jar
2017-11-13 11:30:19,498 INFO [Thread-0]
client.ConnectionManager$HConnectionImplementation:1718 : Closing zookeeper
sessionid=0x15fb4173c1006a0
2017-11-13 11:30:19,499 INFO [main-EventThread] zookeeper.ClientCnxn:519 :
EventThread shut down for session: 0x15fb4173c1006a0
2017-11-13 11:30:19,499 INFO [Thread-0] zookeeper.ZooKeeper:684 : Session:
0x15fb4173c1006a0 closed
2017-11-13 11:30:19,500 INFO [Thread-0]
client.ConnectionManager$HConnectionImplementation:2180 : Closing master
protocol: MasterService
2017-11-13 11:30:19,501 INFO [Thread-0]
client.ConnectionManager$HConnectionImplementation:1718 : Closing zookeeper
sessionid=0x15fb4173c1006a1
2017-11-13 11:30:19,502 INFO [Thread-0] zookeeper.ZooKeeper:684 : Session:
0x15fb4173c1006a1 closed
2017-11-13 11:30:19,502 INFO [main-EventThread] zookeeper.ClientCnxn:519 :
EventThread shut down for session: 0x15fb4173c1006a1
{noformat}
hbase hbck says:
{noformat}
2017-11-13 11:31:37,786 INFO [main] util.HBaseFsck: Checking and fixing region
consistency
ERROR: Region { meta =>
kylin_metadata,,1510301827569.2dcbceee014fe3d3b55d987f54ce9632., hdfs =>
s3://kylin-emrfs/hbase-production/data/default/kylin_metadata/2dcbceee014fe3d3b55d987f54ce9632,
deployed => , replicaId => 0 } not deployed on any region server.
ERROR: Region { meta =>
KYLIN_WSFX70K3YE,,1510308663611.c6506cd949d382880483dff35b236bad., hdfs =>
s3://kylin-emrfs/hbase-production/data/default/KYLIN_WSFX70K3YE/c6506cd949d382880483dff35b236bad,
deployed => , replicaId => 0 } not deployed on any region server.
ERROR: Region { meta =>
hbase:namespace,,1510301118352.f4593b0b8af80a90788fea3a96c25750., hdfs =>
s3://kylin-emrfs/hbase-production/data/hbase/namespace/f4593b0b8af80a90788fea3a96c25750,
deployed => , replicaId => 0 } not deployed on any region server.
2017-11-13 11:31:38,449 INFO [main] util.HBaseFsck: Handling overlap merges in
parallel. set hbasefsck.overlap.merge.parallel to false to run serially.
ERROR: There is a hole in the region chain between and . You need to create a
new .regioninfo and region dir in hdfs to plug the hole.
ERROR: Found inconsistency in table KYLIN_WSFX70K3YE
ERROR: There is a hole in the region chain between and . You need to create a
new .regioninfo and region dir in hdfs to plug the hole.
ERROR: Found inconsistency in table hbase:namespace
ERROR: There is a hole in the region chain between and . You need to create a
new .regioninfo and region dir in hdfs to plug the hole.
ERROR: Found inconsistency in table kylin_metadata
2017-11-13 11:31:38,455 INFO [main] util.HBaseFsck: Computing mapping of all
store files
......
Summary:
Table hbase:meta is okay.
Number of regions: 1
Deployed on: ip-172-31-1-10.cn-north-1.compute.internal,16000,1510554962161
Table KYLIN_WSFX70K3YE is okay.
Number of regions: 0
Deployed on:
Table hbase:namespace is okay.
Number of regions: 0
Deployed on:
Table kylin_metadata is okay.
Number of regions: 0
Deployed on:
6 inconsistencies detected.
Status: INCONSISTENT
{noformat}
If you need the complete hbase hbck output, I will post it below.
> How to use Kylin on EMR with S3 as hbase storage
> ------------------------------------------------
>
> Key: KYLIN-3035
> URL: https://issues.apache.org/jira/browse/KYLIN-3035
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
> Affects Versions: v2.2.0
> Environment: EMR 5.5.0, Kylin 2.2.0
> Reporter: Shawn Wang
> Assignee: Shaofeng SHI
>
> Can somebody give an example of how to use kylin on EMR with S3 as hbase
> storage, which support reuse the previously built cube on new EMR after the
> original EMR has been terminated.
> My purpose is simple:
> 1. use transient EMR cluster to build cubes
> 2. use a persistent cluster to handle query requests
> Of course, the culsters should share same hbase storage, so I setup the
> cluster to use S3 as hbase storage, after 2.2.0 fix the "HFile not written to
> S3" issue, I have been built a sample cube successfully, using configurations:
> EMR:
> {noformat}
> [
> {
> "Classification": "hbase-site",
> "Properties": {
> "hbase.rootdir": "s3://kylin-emrfs/hbase-production"
> }
> },
> {
> "Classification": "hbase",
> "Properties": {
> "hbase.emr.storageMode": "s3"
> }
> },
> {
> "Classification": "emrfs-site",
> "Properties": {
> "fs.s3.consistent": "true",
> "fs.s3.consistent.metadata.tableName":
> "KylinEmrFSMetadata"
> }
> }
> ]
> {noformat}
> kylin.propertities:
> {noformat}
> kylin.env.hdfs-working-dir=s3://kylin-emrfs/kylin-working-dir
> kylin.server.mode=all
> {noformat}
> Then I create a new cluster with same EMR configuration and query mode for
> kylin, kylin just can't startup with errors:
> {noformat}
> 2017-11-13 07:33:44,415 INFO
> [main-SendThread(ip-172-31-1-10.cn-north-1.compute.internal:2181)]
> zookeeper.ClientCnxn:876 : Socket connection established to
> ip-172-31-1-10.cn-north-1.compute.internal/172.31.1.10:2181, initiating
> session
> 2017-11-13 07:33:44,422 INFO
> [main-SendThread(ip-172-31-1-10.cn-north-1.compute.internal:2181)]
> zookeeper.ClientCnxn:1299 : Session establishment complete on server
> ip-172-31-1-10.cn-north-1.compute.internal/172.31.1.10:2181, sessionid =
> 0x15fb4173c100156, negotiated timeout = 40000
> 2017-11-13 07:33:48,380 DEBUG [main] hbase.HBaseConnection:279 : HTable
> 'kylin_metadata' already exists
> Exception in thread "main" java.lang.IllegalArgumentException: Failed to find
> metadata store by url: kylin_metadata@hbase
> at
> org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:89)
> at
> org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:101)
> at
> org.apache.kylin.rest.service.AclTableMigrationTool.checkIfNeedMigrate(AclTableMigrationTool.java:94)
> at
> org.apache.kylin.tool.AclTableMigrationCLI.main(AclTableMigrationCLI.java:41)
> Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed
> after attempts=1, exceptions:
> Mon Nov 13 07:33:48 UTC 2017,
> RpcRetryingCaller{globalStartTime=1510558428667, pause=100, retries=1},
> java.net.ConnectException: 拒绝连接
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:159)
> at org.apache.hadoop.hbase.client.HTable.get(HTable.java:864)
> at org.apache.hadoop.hbase.client.HTable.get(HTable.java:830)
> at
> org.apache.kylin.storage.hbase.HBaseResourceStore.internalGetFromHTable(HBaseResourceStore.java:385)
> at
> org.apache.kylin.storage.hbase.HBaseResourceStore.getFromHTable(HBaseResourceStore.java:363)
> at
> org.apache.kylin.storage.hbase.HBaseResourceStore.existsImpl(HBaseResourceStore.java:116)
> at
> org.apache.kylin.common.persistence.ResourceStore.exists(ResourceStore.java:144)
> at
> org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:84)
> ... 3 more
> Caused by: java.net.ConnectException: 拒绝连接
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
> at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupConnection(RpcClientImpl.java:416)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:722)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:909)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:873)
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1244)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)
> at
> org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.get(ClientProtos.java:35372)
> at org.apache.hadoop.hbase.client.HTable$3.call(HTable.java:856)
> at org.apache.hadoop.hbase.client.HTable$3.call(HTable.java:847)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:137)
> ... 10 more
> 2017-11-13 07:33:48,709 INFO [Thread-1]
> client.ConnectionManager$HConnectionImplementation:2180 : Closing master
> protocol: MasterService
> 2017-11-13 07:33:48,710 INFO [Thread-1]
> client.ConnectionManager$HConnectionImplementation:1718 : Closing zookeeper
> sessionid=0x15fb4173c100156
> 2017-11-13 07:33:48,712 INFO [Thread-1] zookeeper.ZooKeeper:684 : Session:
> 0x15fb4173c100156 closed
> 2017-11-13 07:33:48,712 INFO [main-EventThread] zookeeper.ClientCnxn:519 :
> EventThread shut down for session: 0x15fb4173c100156
> ERROR: Unknown error. Please check full log.
> {noformat}
> And if I change the Kylin server mode to all, Kylin can startup, but the page
> on port 7070 can not be opened, with similar errors as above.
> I am wondering If there are some other configuration I have missed, or I am
> just in the wrong way.
> It'll be many thanks if someone can give an complete example to show how to
> get this work done!
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
