Hello everyone:
I've been using kylin for two months ,there was no problem each time I
created projects,models and built cube,it always worked!
But today I created a project, model and build cube,when I build cube , an
error occurred when build cube at step 6:create HTable.
The specific errors are described as follows:
java.lang.IllegalArgumentException: table KYLIN_CIT4CEENL7 created, but is not
available due to some reasons
at
com.google.common.base.Preconditions.checkArgument(Preconditions.java:92)
at
org.apache.kylin.storage.hbase.steps.CubeHTableUtil.createHTable(CubeHTableUtil.java:106)
at
org.apache.kylin.storage.hbase.steps.CreateHTableJob.run(CreateHTableJob.java:103)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at
org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:64)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:142)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
result code:2
环境：
apache-kylin-2.0.0
Hadoop 2.6.0-cdh5.8.5
HBase 1.2.0-cdh5.8.5
Hive 1.1.0-cdh5.8.5
I found the same problems on the Internet in the
http://apache-kylin.74782.x6.nabble.com/an-error-occurred-when-build-a-sample-cube-at-step-5-create-HTable-td4102.html
.
it offered a solution as follows:
$KYLIN_HOME/conf/kylin_job_conf.xml by removing all configuration entries
related to compression(Just grep the keyword “compress”). To disable
compressing hbase tables you need to open $KYLIN_HOME/conf/kylin.properties and
remove the line starting with kylin.hbase.default.compression.codec.
I did ,but My Cube can't build succeed,the same error occurred.
他遇到的问题是：kylin默认使用snappy压缩，我用的CDH是支持snappy压缩的。
[root@master hadoop-hdfs]# hadoop checknative -a
17/08/18 15:45:01 INFO bzip2.Bzip2Factory: Successfully loaded & initialized
native-bzip2 library system-native
17/08/18 15:45:01 INFO zlib.ZlibFactory: Successfully loaded & initialized
native-zlib library
Native library checking:
hadoop: true
/opt/cloudera/parcels/CDH-5.8.5-1.cdh5.8.5.p0.5/lib/hadoop/lib/native/libhadoop.so.1.0.0
zlib: true /lib64/libz.so.1
snappy: true
/opt/cloudera/parcels/CDH-5.8.5-1.cdh5.8.5.p0.5/lib/hadoop/lib/native/libsnappy.so.1
lz4: true revision:10301
bzip2: true /lib64/libbz2.so.1
openssl: true /usr/lib64/libcrypto.so
Now ,I will list more detailed description of the problem :
--- --------------------
kylin.log
2017-08-18 13:49:00,802 INFO [Job c0b3d6de-99e5-4b91-a941-0fb03d89e40a-726]
util.DeployCoprocessorCLI:229 : Add coprocessor on KYLIN_CIT4CEENL7
2017-08-18 13:49:00,812 INFO [Job c0b3d6de-99e5-4b91-a941-0fb03d89e40a-726]
util.DeployCoprocessorCLI:209 : hbase table KYLIN_CIT4CEENL7 deployed with
coprocessor.
2017-08-18 13:49:29,059 INFO [Job c0b3d6de-99e5-4b91-a941-0fb03d89e40a-726]
client.HBaseAdmin:784 : Created KYLIN_CIT4CEENL7
2017-08-18 13:49:29,078 ERROR [Job c0b3d6de-99e5-4b91-a941-0fb03d89e40a-726]
common.HadoopShellExecutable:65 : error execute
HadoopShellExecutable{id=c0b3d6de-99e5-4b91-a941-0fb03d89e40a-05, name=Create
HTable, state=RUNNING}
java.lang.IllegalArgumentException: table KYLIN_CIT4CEENL7 created, but is not
available due to some reasons
at com.google.common.base.Preconditions.checkArgument(Preconditions.java:92)
at
org.apache.kylin.storage.hbase.steps.CubeHTableUtil.createHTable(CubeHTableUtil.java:106)
at
org.apache.kylin.storage.hbase.steps.CreateHTableJob.run(CreateHTableJob.java:103)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at
org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:64)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:142)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2017-08-18 13:49:29,080 DEBUG [Job c0b3d6de-99e5-4b91-a941-0fb03d89e40a-726]
dao.ExecutableDao:217 : updating job output, id:
c0b3d6de-99e5-4b91-a941-0fb03d89e40a-05
2017-08-18 13:49:29,087 DEBUG [Job c0b3d6de-99e5-4b91-a941-0fb03d89e40a-726]
dao.ExecutableDao:217 : updating job output, id:
c0b3d6de-99e5-4b91-a941-0fb03d89e40a-05
2017-08-18 13:49:29,091 INFO [Job c0b3d6de-99e5-4b91-a941-0fb03d89e40a-726]
execution.ExecutableManager:389 : job
id:c0b3d6de-99e5-4b91-a941-0fb03d89e40a-05 from RUNNING to ERROR
2017-08-18 13:49:29,100 DEBUG [Job c0b3d6de-99e5-4b91-a941-0fb03d89e40a-726]
dao.ExecutableDao:217 : updating job output, id:
c0b3d6de-99e5-4b91-a941-0fb03d89e40a
2017-08-18 13:49:29,106 DEBUG [Job c0b3d6de-99e5-4b91-a941-0fb03d89e40a-726]
dao.ExecutableDao:217 : updating job output, id:
c0b3d6de-99e5-4b91-a941-0fb03d89e40a
-----------------------------------------------------------------
hbase的web 控制台
------------------------------------------
hbase的日志信息如下： 里面STARTKEY => '', ENDKEY => '' 表示数据没有写入成功吗？？
2017-08-18 17:04:54,324 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Assigning KYLIN_CIT4CEENL7,,1503046982941.3b629cdbed8252f24ced8f6f71fe2d1f. to
slave3.cdh.com,60020,1503035643997
2017-08-18 17:04:54,324 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {3b629cdbed8252f24ced8f6f71fe2d1f state=OFFLINE, ts=1503047094320,
server=slave2.cdh.com,60020,1503035646921} to {3b629cdbed8252f24ced8f6f71fe2d1f
state=PENDING_OPEN, ts=1503047094324, server=slave3.cdh.com,60020,1503035643997}
2017-08-18 17:04:54,345 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {3b629cdbed8252f24ced8f6f71fe2d1f state=PENDING_OPEN,
ts=1503047094324, server=slave3.cdh.com,60020,1503035643997} to
{3b629cdbed8252f24ced8f6f71fe2d1f state=OPENING, ts=1503047094345,
server=slave3.cdh.com,60020,1503035643997}
2017-08-18 17:05:49,463 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {3b629cdbed8252f24ced8f6f71fe2d1f state=OPENING, ts=1503047094345,
server=slave3.cdh.com,60020,1503035643997} to {3b629cdbed8252f24ced8f6f71fe2d1f
state=CLOSED, ts=1503047149463, server=slave3.cdh.com,60020,1503035643997}
2017-08-18 17:05:49,465 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Setting node as OFFLINED in ZooKeeper for region {ENCODED =>
3b629cdbed8252f24ced8f6f71fe2d1f, NAME =>
'KYLIN_CIT4CEENL7,,1503046982941.3b629cdbed8252f24ced8f6f71fe2d1f.', STARTKEY
=> '', ENDKEY => ''}
2017-08-18 17:05:49,465 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {3b629cdbed8252f24ced8f6f71fe2d1f state=CLOSED, ts=1503047149465,
server=slave3.cdh.com,60020,1503035643997} to {3b629cdbed8252f24ced8f6f71fe2d1f
state=OFFLINE, ts=1503047149465, server=slave3.cdh.com,60020,1503035643997}
2017-08-18 17:05:49,470 INFO org.apache.hadoop.hbase.master.AssignmentManager:
Assigning KYLIN_CIT4CEENL7,,1503046982941.3b629cdbed8252f24ced8f6f71fe2d1f. to
slave2.cdh.com,60020,1503035646921
2017-08-18 17:05:49,470 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {3b629cdbed8252f24ced8f6f71fe2d1f state=OFFLINE, ts=1503047149465,
server=slave3.cdh.com,60020,1503035643997} to {3b629cdbed8252f24ced8f6f71fe2d1f
state=PENDING_OPEN, ts=1503047149470, server=slave2.cdh.com,60020,1503035646921}
2017-08-18 17:05:49,496 INFO org.apache.hadoop.hbase.master.RegionStates:
Transition {3b629cdbed8252f24ced8f6f71fe2d1f state=PENDING_OPEN,
ts=1503047149470, server=slave2.cdh.com,60020,1503035646921} to
{3b629cdbed8252f24ced8f6f71fe2d1f state=OPENING, ts=1503047149496,
server=slave2.cdh.com,60020,1503035646921}
---------------------------------------------------------------------------
最后查看 hdfs的日志信息：
里面有警告：KYLIN_CIT4CEENL7 这张表没有写入权限
2017-08-18 17:05:09,389 WARN org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:hbase (auth:SIMPLE)
cause:org.apache.hadoop.security.AccessControlException: Permission denied:
user=hbase, access=WRITE, inode="/hbase/data/default/2017-08-18 17:05:09,389
WARN org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:hbase (auth:SIMPLE)
cause:org.apache.hadoop.security.AccessControlException: Permission denied:
user=hbase, access=WRITE,
inode="/hbase/data/default/KYLIN_CIT4CEENL7/3b629cdbed8252f24ced8f6f71fe2d1f":hdfs:hbase:drwxr-xr-x
2017-08-18 17:05:09,389 INFO org.apache.hadoop.ipc.Server: IPC Server handler
12 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.mkdirs from
192.168.1.114:39396 Call#1504 Retry#0:
org.apache.hadoop.security.AccessControlException: Permission denied:
user=hbase, access=WRITE,
inode="/hbase/data/default/KYLIN_CIT4CEENL7/3b629cdbed8252f24ced8f6f71fe2d1f":hdfs:hbase:drwxr-xr-x
2017-08-18 17:05:10,849 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor:
Rescanning after 30000 milliseconds
2017-08-18 17:05:10,850 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned
0 directive(s) and 0 block(s) in 1 millisecond(s).
2017-08-18 17:05:15,395 WARN org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:hbase (auth:SIMPLE)
cause:org.apache.hadoop.security.AccessControlException: Permission denied:
user=hbase, access=WRITE,
inode="/hbase/data/default/KYLIN_CIT4CEENL7/3b629cdbed8252f24ced8f6f71fe2d1f":hdfs:hbase:drwxr-xr-x
2017-08-18 17:05:15,395 INFO org.apache.hadoop.ipc.Server: IPC Server handler
29 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.mkdirs from
192.168.1.114:39396 Call#1507 Retry#0:
org.apache.hadoop.security.AccessControlException: Permission denied:
user=hbase, access=WRITE,
inode="/hbase/data/default/KYLIN_CIT4CEENL7/3b629cdbed8252f24ced8f6f71fe2d1f":hdfs:hbase:drwxr-xr-x
2017-08-18 17:05:16,703 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog:
Number of transactions: 117 Total time for transactions(ms): 3 Number of
transactions batched in Syncs: 32 Number of syncs: 85 SyncTimes(ms): 107
2017-08-18 17:05:16,817 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
allocateBlock:
/tmp/.cloudera_health_monitoring_canary_files/.canary_file_2017_08_18-17_05_16.
BP-100649073-192.168.1.108-1499418425504
blk_1073896776_155980{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[[DISK]DS-291c404a-5f38-4850-9164-334da4e2f35b:NORMAL:192.168.1.113:50010|RBW],
ReplicaUnderConstruction[[DISK]DS-5d1d9d8e-bd8c-47c8-9554-f4643bcf26b7:NORMAL:192.168.1.112:50010|RBW],
ReplicaUnderConstruction[[DISK]DS-c6bb2c74-ddb4-48a4-aed4-971bd781af21:NORMAL:192.168.1.114:50010|RBW]]}
2017-08-18 17:05:09,389 WARN org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:hbase (auth:SIMPLE)
cause:org.apache.hadoop.security.AccessControlException: Permission denied:
user=hbase, access=WRITE,
inode="/hbase/data/default/KYLIN_CIT4CEENL7/3b629cdbed8252f24ced8f6f71fe2d1f":hdfs:hbase:drwxr-xr-x
2017-08-18 17:05:09,389 INFO org.apache.hadoop.ipc.Server: IPC Server handler
12 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.mkdirs from
192.168.1.114:39396 Call#1504 Retry#0:
org.apache.hadoop.security.AccessControlException: Permission denied:
user=hbase, access=WRITE,
inode="/hbase/data/default/KYLIN_CIT4CEENL7/3b629cdbed8252f24ced8f6f71fe2d1f":hdfs:hbase:drwxr-xr-x
2017-08-18 17:05:10,849 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor:
Rescanning after 30000 milliseconds
2017-08-18 17:05:10,850 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned
0 directive(s) and 0 block(s) in 1 millisecond(s).
2017-08-18 17:05:15,395 WARN org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:hbase (auth:SIMPLE)
cause:org.apache.hadoop.security.AccessControlException: Permission denied:
user=hbase, access=WRITE,
inode="/hbase/data/default/KYLIN_CIT4CEENL7/3b629cdbed8252f24ced8f6f71fe2d1f":hdfs:hbase:drwxr-xr-x
2017-08-18 17:05:15,395 INFO org.apache.hadoop.ipc.Server: IPC Server handler
29 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.mkdirs from
192.168.1.114:39396 Call#1507 Retry#0:
org.apache.hadoop.security.AccessControlException: Permission denied:
user=hbase, access=WRITE,
inode="/hbase/data/default/KYLIN_CIT4CEENL7/3b629cdbed8252f24ced8f6f71fe2d1f":hdfs:hbase:drwxr-xr-x
2017-08-18 17:05:16,703 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog:
Number of transactions: 117 Total time for transactions(ms): 3 Number of
transactions batched in Syncs: 32 Number of syncs: 85 SyncTimes(ms): 107
2017-08-18 17:05:16,817 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
allocateBlock:
/tmp/.cloudera_health_monitoring_canary_files/.canary_file_2017_08_18-17_05_16.
BP-100649073-192.168.1.108-1499418425504
blk_1073896776_155980{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[[DISK]DS-291c404a-5f38-4850-9164-334da4e2f35b:NORMAL:192.168.1.113:50010|RBW],
ReplicaUnderConstruction[[DISK]DS-5d1d9d8e-bd8c-47c8-9554-f4643bcf26b7:NORMAL:192.168.1.112:50010|RBW],
ReplicaUnderConstruction[[DISK]DS-c6bb2c74-ddb4-48a4-aed4-971bd781af21:NORMAL:192.168.1.114:50010|RBW]]}
/3b629cdbed8252f24ced8f6f71fe2d1f":hdfs:hbase:drwxr-xr-x
2017-08-18 17:05:09,389 INFO org.apache.hadoop.ipc.Server: IPC Server handler
12 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.mkdirs from
192.168.1.114:39396 Call#1504 Retry#0:
org.apache.hadoop.security.AccessControlException: Permission denied:
user=hbase, access=WRITE,
inode="/hbase/data/default/KYLIN_CIT4CEENL7/3b629cdbed8252f24ced8f6f71fe2d1f":hdfs:hbase:drwxr-xr-x
2017-08-18 17:05:10,849 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor:
Rescanning after 30000 milliseconds
2017-08-18 17:05:10,850 INFO
org.apache.hadoop.hdfs.server.blockmanagement.CacheReplicationMonitor: Scanned
0 directive(s) and 0 block(s) in 1 millisecond(s).
2017-08-18 17:05:15,395 WARN org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:hbase (auth:SIMPLE)
cause:org.apache.hadoop.security.AccessControlException: Permission denied:
user=hbase, access=WRITE,
inode="/hbase/data/default/KYLIN_CIT4CEENL7/3b629cdbed8252f24ced8f6f71fe2d1f":hdfs:hbase:drwxr-xr-x
2017-08-18 17:05:15,395 INFO org.apache.hadoop.ipc.Server: IPC Server handler
29 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.mkdirs from
192.168.1.114:39396 Call#1507 Retry#0:
org.apache.hadoop.security.AccessControlException: Permission denied:
user=hbase, access=WRITE,
inode="/hbase/data/default/KYLIN_CIT4CEENL7/3b629cdbed8252f24ced8f6f71fe2d1f":hdfs:hbase:drwxr-xr-x
2017-08-18 17:05:16,703 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog:
Number of transactions: 117 Total time for transactions(ms): 3 Number of
transactions batched in Syncs: 32 Number of syncs: 85 SyncTimes(ms): 107
2017-08-18 17:05:16,817 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
allocateBlock:
/tmp/.cloudera_health_monitoring_canary_files/.canary_file_2017_08_18-17_05_16.
BP-100649073-192.168.1.108-1499418425504
blk_1073896776_155980{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1,
replicas=[ReplicaUnderConstruction[[DISK]DS-291c404a-5f38-4850-9164-334da4e2f35b:NORMAL:192.168.1.113:50010|RBW],
ReplicaUnderConstruction[[DISK]DS-5d1d9d8e-bd8c-47c8-9554-f4643bcf26b7:NORMAL:192.168.1.112:50010|RBW],
ReplicaUnderConstruction[[DISK]DS-c6bb2c74-ddb4-48a4-aed4-971bd781af21:NORMAL:192.168.1.114:50010|RBW]]}
上面的警告，表KYLIN_CIT4CEENL7 没有写入权限 ，导致，创建表后无法向里写入数据，导致异常？是不是这里的问题？
（以前buid成功是有写权限的，今天build突然没权限了）
直接给表加写权限 ，重新build会报表已经存在的异常，所以，怎么能让kylin
创建表的时候，同时给KYLIN_CIT4CEENL7这样的表写权限？在哪里配置？求助中。。。。
在kylin 中配置 还是在hadoop的配置环境中配置：
网上说在hdfs-site.xml 中如下配置
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
试过，还是无法解决问题。。。
大家帮忙分析分析。。。。。
