hi Abhilash, Is this step blocking cube building? (is it crashing your hbase?) Checking LZO will try/catch exception, but it does no harm. If kylin finds out that LZO is not available, it will give up to use LZO and go on:
[pool-7-thread-10]:[2015-08-06 08:42:26,052][INFO]*[org.apache.kylin.job.hadoop.hbase.CreateHTableJob.run(CreateHTableJob.java:104)] - hbase will not use lzo to compress data* 2015-08-06 08:42:26,084 INFO [pool-7-thread-10] compress.CodecPool: Got brand-new decompressor [.snappy] 2015-08-06 08:42:26,085 INFO [pool-7-thread-10] compress.CodecPool: Got brand-new decompressor [.snappy] On Thu, Aug 6, 2015 at 3:13 PM, Abhilash L L <[email protected]> wrote: > We are also facing the same issue. > > Even though LZO is disabled in the conf files, its trying to check if it > can use compression etc. > > hbase org.apache.hadoop.hbase.util.CompressionTest > file:///PATH-TO-A-LOCAL-TMP-FILE lzo > > The above test works but > > create 'lzoTable', {NAME => 'colFam',COMPRESSION => 'LZO'} > > fails > > > > Regards, > Abhilash > > On Thu, Aug 6, 2015 at 10:32 AM, hongbin ma <[email protected]> wrote: > > > Did you make any changes to kylin.properties? > > If you meant to enable lzo, you should first make sure lzo is availabe in > > your cluster: > > http://kylin.incubator.apache.org/docs/install/advance_settings.html > > > > On Thu, Aug 6, 2015 at 9:16 AM, liangmeng <[email protected]> wrote: > > > > > cube building stop at step 10:create htable, and our cluster crashed > > > down,many region servers restarted! kylin.log is as follow: > > > > > > [2015-08-06 > > > > > > 08:42:26,050][ERROR][org.apache.kylin.job.tools.LZOSupportnessChecker.getSupportness(LZOSupportnessChecker.java:38)] > > > - Fail to compress file with lzo > > > java.lang.RuntimeException: java.lang.ClassNotFoundException: > > > com.hadoop.compression.lzo.LzoCodec > > > at > > > > > > org.apache.hadoop.hbase.io.compress.Compression$Algorithm$1.buildCodec(Compression.java:131) > > > at > > > > > > org.apache.hadoop.hbase.io.compress.Compression$Algorithm$1.getCodec(Compression.java:116) > > > at > > > > > > org.apache.hadoop.hbase.io.compress.Compression$Algorithm.getCompressor(Compression.java:310) > > > at > > > > > > org.apache.hadoop.hbase.io.encoding.HFileBlockDefaultEncodingContext.<init>(HFileBlockDefaultEncodingContext.java:92) > > > at > > > > > > org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.<init>(HFileBlock.java:690) > > > at > > > > > > org.apache.hadoop.hbase.io.hfile.HFileWriterV2.finishInit(HFileWriterV2.java:117) > > > at > > > > > > org.apache.hadoop.hbase.io.hfile.HFileWriterV2.<init>(HFileWriterV2.java:109) > > > at > > > > > > org.apache.hadoop.hbase.io.hfile.HFileWriterV2$WriterFactoryV2.createWriter(HFileWriterV2.java:97) > > > at > > > > > > org.apache.hadoop.hbase.io.hfile.HFile$WriterFactory.create(HFile.java:393) > > > at > > > > > > org.apache.hadoop.hbase.util.CompressionTest.doSmokeTest(CompressionTest.java:118) > > > at > > > > > > org.apache.hadoop.hbase.util.CompressionTest.main(CompressionTest.java:148) > > > at > > > > > > org.apache.kylin.job.tools.LZOSupportnessChecker.getSupportness(LZOSupportnessChecker.java:36) > > > at > > > > > > org.apache.kylin.job.hadoop.hbase.CreateHTableJob.run(CreateHTableJob.java:100) > > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > > > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > > > at > > > > > > org.apache.kylin.job.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63) > > > at > > > > > > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:106) > > > at > > > > > > org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50) > > > at > > > > > > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:106) > > > at > > > > > > org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:133) > > > at > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > > at > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > > at java.lang.Thread.run(Thread.java:745) > > > Caused by: java.lang.ClassNotFoundException: > > > com.hadoop.compression.lzo.LzoCodec > > > at java.net.URLClassLoader$1.run(URLClassLoader.java:366) > > > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > > > at java.security.AccessController.doPrivileged(Native Method) > > > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > > > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > > > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > > > at > > > > > > org.apache.hadoop.hbase.io.compress.Compression$Algorithm$1.buildCodec(Compression.java:125) > > > ... 22 more > > > [pool-7-thread-10]:[2015-08-06 > > > > > > 08:42:26,052][INFO][org.apache.kylin.job.hadoop.hbase.CreateHTableJob.run(CreateHTableJob.java:104)] > > > - hbase will not use lzo to compress data > > > 2015-08-06 08:42:26,084 INFO [pool-7-thread-10] compress.CodecPool: > Got > > > brand-new decompressor [.snappy] > > > 2015-08-06 08:42:26,085 INFO [pool-7-thread-10] compress.CodecPool: > Got > > > brand-new decompressor [.snappy] > > > 2015-08-06 08:42:26,085 INFO [pool-7-thread-10] compress.CodecPool: > Got > > > brand-new decompressor [.snappy] > > > 2015-08-06 08:42:26,085 INFO [pool-7-thread-10] compress.CodecPool: > Got > > > brand-new decompressor [.snappy] > > > [pool-7-thread-10]:[2015-08-06 > > > > > > 08:42:26,091][INFO][org.apache.kylin.job.hadoop.hbase.CreateHTableJob.getSplits(CreateHTableJob.java:161)] > > > - 1 regions > > > [pool-7-thread-10]:[2015-08-06 > > > > > > 08:42:26,091][INFO][org.apache.kylin.job.hadoop.hbase.CreateHTableJob.getSplits(CreateHTableJob.java:162)] > > > - 0 splits > > > 2015-08-06 08:42:26,096 INFO [pool-7-thread-10] zookeeper.ZooKeeper: > > > Initiating client connection, connectString= > > node41.cluster-a.gdyd.com:2181 > > > ,node22.cluster-a.gdyd.com:2181,node21.cluster-a.gdyd.com:2181, > > > node20.cluster-a.gdyd.com:2181,node42.cluster-a.gdyd.com:2181 > > > sessionTimeout=300000 watcher=catalogtracker-on-hconnection-0x66f0a71e, > > > quorum=node41.cluster-a.gdyd.com:2181,node22.cluster-a.gdyd.com:2181, > > > node21.cluster-a.gdyd.com:2181,node20.cluster-a.gdyd.com:2181, > > > node42.cluster-a.gdyd.com:2181, baseZNode=/hbase > > > 2015-08-06 08:42:26,097 INFO [pool-7-thread-10] > > > zookeeper.RecoverableZooKeeper: Process > > > identifier=catalogtracker-on-hconnection-0x66f0a71e connecting to > > ZooKeeper > > > ensemble=node41.cluster-a.gdyd.com:2181,node22.cluster-a.gdyd.com:2181 > , > > > node21.cluster-a.gdyd.com:2181,node20.cluster-a.gdyd.com:2181, > > > node42.cluster-a.gdyd.com:2181 > > > 2015-08-06 08:42:26,097 DEBUG [pool-7-thread-10] > catalog.CatalogTracker: > > > Starting catalog tracker > > > > > > > > > > > > 梁猛 > > > 中国移动广东公司 网管维护中心 网管支撑室 > > > 电话：13802880779 > > > 邮箱: [email protected] ，[email protected] > > > 地址：广东省广州市珠江新城珠江西路11号 广东全球通大厦北3楼 > > > 邮编：510623 > > > > > > > > > > > -- > > Regards, > > > > *Bin Mahone | 马洪宾* > > Apache Kylin: http://kylin.io > > Github: https://github.com/binmahone > > > -- Regards, *Bin Mahone | 马洪宾* Apache Kylin: http://kylin.io Github: https://github.com/binmahone
