Thanks a lot. I swear I googled it :-)
On 10/20/15, 1:56 PM, "Shi, Shaofeng" <[email protected]> wrote: >Hi Benjamin, please check if this JIRA matches your problem: >https://issues.apache.org/jira/browse/KYLIN-1083 > >On 10/20/15, 12:53 PM, "Benjamin Leonhardi" <[email protected]> >wrote: > >>Hello Everybody, >> >>Thanks for all the help/ >> >>I successfully build Kylin on HDP2.3 with the new updates to the branch >>and by changing some dependencies in the pom file ( got a HCAT mapred >>version conflict before ) When I changed the entries in the pom file to >>the exact version of HDP2.3 it worked. >> >> >> >> >><!-- Hadoop versions --> >> <hadoop2.version>2.7.1</hadoop2.version> >> <yarn.version>2.7.1</yarn.version> >> <zookeeper.version>3.4.6</zookeeper.version> >> <hive.version>1.2.1</hive.version> >> <hive-hcatalog.version>1.2.1</hive-hcatalog.version> >> <hbase-hadoop2.version>1.1.1</hbase-hadoop2.version> >> <curator.version>2.7.1</curator.version> >> >> >>However now I get errors in the test and also when I create a new cube >>from scratch during the Hfile creation. >> >>Any idea what this could be? During the test run I thought it might be a >>bad testcase but it also happens when I build a new cube from scratch. >> >>Parameters: >> >> -conf /kylin/kylin-1.1-incubating-SNAPSHOT/conf/kylin_job_conf.xml >>-cubename aggtest -input >>/kylin/kylin_metadata/kylin-0669bcc0-2a42-4189-a7c8-6abbb533da8c/aggtest/c >>uboid/* -output >>hdfs://sandbox.hortonworks.com:8020/kylin/kylin_metadata/kylin-0669bcc0-2a >>42-4189-a7c8-6abbb533da8c/aggtest/hfile -htablename KYLIN_0PV292NH5B >>-jobname Kylin_HFile_Generator_aggtest_Step >> >> >>Error message : >> >> >>java.lang.IllegalArgumentException: Can not create a Path from a null >>string >> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:122) >> at org.apache.hadoop.fs.Path.<init>(Path.java:134) >> at org.apache.hadoop.fs.Path.<init>(Path.java:88) >> at >>org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2.configurePartitioner( >>HFileOutputFormat2.java:591) >> at >>org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2.configureIncrementalL >>oad(HFileOutputFormat2.java:440) >> at >>org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2.configureIncrementalL >>oad(HFileOutputFormat2.java:405) >> at >>org.apache.hadoop.hbase.mapreduce.HFileOutputFormat.configureIncrementalLo >>ad(HFileOutputFormat.java:91) >> at >>org.apache.kylin.job.hadoop.cube.CubeHFileJob.run(CubeHFileJob.java:86) >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) >> at >>org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable >>.java:113) >> at >>org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutab >>le.java:107) >> at >>org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChai >>nedExecutable.java:51) >> at >>org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutab >>le.java:107) >> at >>org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(Defaul >>tScheduler.java:130) >> at >>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java: >>1145) >> at >>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java >>:615) >> at java.lang.Thread.run(Thread.java:745) >> >>result code:2 >> >> >> >>On 10/9/15, 1:11 PM, "Benjamin Leonhardi" <[email protected]> >>wrote: >> >>>Hello Everybody, >>> >>>We figured out the 1.2 coded comes from the first dependency. Will try >>>and update. >>> >>>Thanks for your help. >>> >>>Ben >>> >>> >>> >>>On 10/6/15, 12:47 AM, "周千昊" <[email protected]> wrote: >>> >>>>Hi, Benjamin, >>>> It is a bug, a jira ticket is created as follow: >>>>https://issues.apache.org/jira/browse/KYLIN-1059 >>>> and it should have been fixed. since I don't have the hdp 2.3 env >>>>currently, it would be appreciated if you can help to verify with the >>>>latest code(branch 1.x-HBase1.x) >>>> >>>>Luke Han <[email protected]>于2015年10月5日周一 下午8:35写道： >>>> >>>>> Hi Benjamin, >>>>> Did you generate binary package by yourself with below >>>>>instruction? or >>>>> just compile war/jar from maven? >>>>> >>>>> http://kylin.incubator.apache.org/development/howto_package.html >>>>> <http://kylin.incubator.apache.org/development/howto_package.html> >>>>> >>>>> Thanks. >>>>> >>>>> >>>>> Best Regards! >>>>> --------------------- >>>>> >>>>> Luke Han >>>>> >>>>> On Thu, Oct 1, 2015 at 1:30 PM, Benjamin Leonhardi < >>>>> [email protected]> wrote: >>>>> >>>>> > Hello All, >>>>> > >>>>> > I try to get Kylin to run with HDP 2.3 sandbox image. Using the >>>>>HBase1.x >>>>> > branch. >>>>> > >>>>> > https://github.com/apache/incubator-kylin/tree/1.x-HBase1.x >>>>> > >>>>> > It compiles fine and the tests run through until they reach the job >>>>> > creation. I followed the steps below. >>>>> > >>>>> > http://kylin.incubator.apache.org/development/dev_env.html >>>>> > >>>>> > When I run the tests he reaches the job part and then fails because >>>>>he >>>>> > misses a method in commons-codec.jar. The method is >>>>>encodeBase64(byte[], >>>>> > isChunk boolean, isUrlSafe boolean) and it is in commons-codec1.4+ >>>>>but >>>>> not >>>>> > in older versions. I checked but my environment only has codec jars >>>>>that >>>>> > are newer. >>>>> > >>>>> > However when I build kylin he for some reason compiled an older >>>>>codec >>>>> into >>>>> > the kylin-job-1.1-incubating-SNAPSHOT-job.jar. If I unzip this I >>>>>see a >>>>> > Base64 class that only has the method encodeBase64(byte[], isChunk >>>>> boolean >>>>> > ) but not the urlsafe method. ( so I suppose codec 1.3 or older ) I >>>>> tried >>>>> > to check the pom.xml because I thought I might find an older codec >>>>> > dependency inside but did not find anything. >>>>> > >>>>> > I also have to say I am not the best Maven person so I might be >>>>> > overlooking something easy. One question I would have is why he >>>>>compiles >>>>> > the code into the -job.jar at all. >>>>> > >>>>> > >>>>> > Any help welcome >>>>> > >>>>> > Best regards, >>>>> > >>>>> > Benjamin >>>>> > >>>>> > >>>>> > Starting: >>>>> > Kylin_Fact_Distinct_Columns_test_kylin_cube_without_slr_empty_Step >>>>> > L4J [2015-10-01 >>>>> > 03:51:24,166][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] - >>>>> append >>>>> > job jar: >>>>> > >>>>>/root/kylin_build/incubator-kylin/job/../job/target/kylin-job-1.1-incu >>>>> > bating-SNAPSHOT-job.jar >>>>> > L4J [2015-10-01 >>>>> > 03:51:24,169][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] - >>>>> append >>>>> > kylin.hive.dependency: null and kylin.hbase.dependency: null to >>>>> > mapreduce.appli >>>>> > cation.classpath >>>>> > L4J [2015-10-01 >>>>> > 03:51:24,167][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] - >>>>> append >>>>> > job jar: >>>>> > >>>>>/root/kylin_build/incubator-kylin/job/../job/target/kylin-job-1.1-incu >>>>> > bating-SNAPSHOT-job.jar >>>>> > L4J [2015-10-01 >>>>> > 03:51:24,170][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] - >>>>> append >>>>> > kylin.hive.dependency: null and kylin.hbase.dependency: null to >>>>> > mapreduce.appli >>>>> > cation.classpath >>>>> > L4J [2015-10-01 >>>>> > 03:51:24,170][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] - >>>>> Hadoop >>>>> > job classpath is: >>>>> > >>>>> >>>>>/tmp/kylin/*,$HADOOP_CONF_DIR,/usr/hdp/2.3.0.0-2530/hbase/lib/hbase-com >>>>>mon.jar,/usr/hdp/current/hive-client/conf/,/usr/hdp/2.3.0.0-2530/hive/l >>>>>ib/hive-metastore.jar,/usr/hdp/2.3.0.0-2530/hive/lib/hive-exec.jar,/usr >>>>>/hdp/2.3.0.0-2530/hive-hcatalog/share/hcatalog/*,$PWD/mr-framework/hado >>>>>op/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapre >>>>>duce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-frame >>>>>work/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/ha >>>>>doop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-fr >>>>>amework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoo >>>>>p/hdfs/lib/*:/usr/hdp/2.3.0.0-2530/hadoop/lib/hadoop-lzo-0.6.0.2.3.0.0- >>>>>2530.jar:/etc/hadoop/conf/secure >>>>> > L4J [2015-10-01 >>>>> > 03:51:24,169][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] - >>>>> Hadoop >>>>> > job classpath is: >>>>> > >>>>> >>>>>/tmp/kylin/*,$HADOOP_CONF_DIR,/usr/hdp/2.3.0.0-2530/hbase/lib/hbase-com >>>>>mon.jar,/usr/hdp/current/hive-client/conf/,/usr/hdp/2.3.0.0-2530/hive/l >>>>>ib/hive-metastore.jar,/usr/hdp/2.3.0.0-2530/hive/lib/hive-exec.jar,/usr >>>>>/hdp/2.3.0.0-2530/hive-hcatalog/share/hcatalog/*,$PWD/mr-framework/hado >>>>>op/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapre >>>>>duce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-frame >>>>>work/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/ha >>>>>doop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-fr >>>>>amework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoo >>>>>p/hdfs/lib/*:/usr/hdp/2.3.0.0-2530/hadoop/lib/hadoop-lzo-0.6.0.2.3.0.0- >>>>>2530.jar:/etc/hadoop/conf/secure >>>>> > L4J [2015-10-01 >>>>>03:51:24,229][WARN][org.apache.hadoop.hive.conf.HiveConf] >>>>> > - HiveConf of name hive.heapsize does not exist >>>>> > L4J [2015-10-01 >>>>>03:51:24,230][WARN][org.apache.hadoop.hive.conf.HiveConf] >>>>> > - HiveConf of name hive.server2.enable.impersonation does not exist >>>>> > L4J [2015-10-01 >>>>>03:51:24,236][WARN][org.apache.hadoop.hive.conf.HiveConf] >>>>> > - HiveConf of name hive.heapsize does not exist >>>>> > L4J [2015-10-01 >>>>>03:51:24,237][WARN][org.apache.hadoop.hive.conf.HiveConf] >>>>> > - HiveConf of name hive.server2.enable.impersonation does not exist >>>>> > L4J [2015-10-01 >>>>> > 03:51:24,265][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] - >>>>> > tempMetaFileString is : null >>>>> > L4J [2015-10-01 >>>>> > 03:51:24,266][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] - >>>>> > tempMetaFileString is : null >>>>> > L4J [2015-10-01 >>>>> > >>>>>03:51:24,267][ERROR][org.apache.kylin.job.execution.AbstractExecutable] >>>>> - >>>>> > error running Executable >>>>> > java.lang.NoSuchMethodError: >>>>> > org.apache.commons.codec.binary.Base64.encodeBase64([BZZ)[B >>>>> > at >>>>> > >>>>>org.apache.hive.hcatalog.common.HCatUtil.encodeBytes(HCatUtil.java:125) >>>>> > at >>>>> > >>>>>org.apache.hive.hcatalog.common.HCatUtil.serialize(HCatUtil.java:104) >>>>> > at >>>>> > >>>>>org.apache.hive.hcatalog.common.HCatUtil.getHiveConf(HCatUtil.java:585) >>>>> > at >>>>> > >>>>> >>>>>org.apache.hive.hcatalog.mapreduce.InitializeInput.getInputJobInfo(Init >>>>>ializeInput.java:100) >>>>> > at >>>>> > >>>>> >>>>>org.apache.hive.hcatalog.mapreduce.InitializeInput.setInput(InitializeI >>>>>nput.java:86) >>>>> > at >>>>> > >>>>> >>>>>org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFo >>>>>rmat.java:95) >>>>> > at >>>>> > >>>>> >>>>>org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFo >>>>>rmat.java:51) >>>>> > at >>>>> > >>>>> >>>>>org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.setupMapper(Fac >>>>>tDistinctColumnsJob.java:101) >>>>> > at >>>>> > >>>>> >>>>>org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.run(FactDistinc >>>>>tColumnsJob.java:77) >>>>> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) >>>>> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) >>>>> > at >>>>> > >>>>> >>>>>org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecuta >>>>>ble.java:113) >>>>> > at >>>>> > >>>>> >>>>>org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecu >>>>>table.java:107) >>>>> > at >>>>> > >>>>> >>>>>org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultC >>>>>hainedExecutable.java:51) >>>>> > at >>>>> > >>>>> >>>>>org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecu >>>>>table.java:107) >>>>> > at >>>>> > >>>>> >>>>>org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(Def >>>>>aultScheduler.java:130) >>>>> > at >>>>> > >>>>> >>>>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja >>>>>va:1145) >>>>> > at >>>>> > >>>>> >>>>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j >>>>>ava:615) >>>>> > at java.lang.Thread.run(Thread.java:745) >>>>> > L4J [2015-10-01 >>>>> > >>>>>03:51:24,267][ERROR][org.apache.kylin.job.execution.AbstractExecutable] >>>>> - >>>>> > error running Executable >>>>> > java.lang.NoSuchMethodError: >>>>> > org.apache.commons.codec.binary.Base64.encodeBase64([BZZ)[B >>>>> > at >>>>> > >>>>>org.apache.hive.hcatalog.common.HCatUtil.encodeBytes(HCatUtil.java:125) >>>>> > at >>>>> > >>>>>org.apache.hive.hcatalog.common.HCatUtil.serialize(HCatUtil.java:104) >>>>> > at >>>>> > >>>>>org.apache.hive.hcatalog.common.HCatUtil.getHiveConf(HCatUtil.java:585) >>>>> > at >>>>> > >>>>> >>>>>org.apache.hive.hcatalog.mapreduce.InitializeInput.getInputJobInfo(Init >>>>>ializeInput.java:100) >>>>> > at >>>>> > >>>>> >>>>>org.apache.hive.hcatalog.mapreduce.InitializeInput.setInput(InitializeI >>>>>nput.java:86) >>>>> > at >>>>> > >>>>> >>>>>org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFo >>>>>rmat.java:95) >>>>> > at >>>>> > >>>>> >>>>>org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFo >>>>>rmat.java:51) >>>>> > at >>>>> > >>>>> >>>>>org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.setupMapper(Fac >>>>>tDistinctColumnsJob.java:101) >>>>> > at >>>>> > >>>>> >>>>>org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.run(FactDistinc >>>>>tColumnsJob.java:77) >>>>> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) >>>>> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) >>>>> > at >>>>> > >>>>> >>>>>org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecuta >>>>>ble.java:113) >>>>> > at >>>>> > >>>>> >>>>>org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecu >>>>>table.java:107) >>>>> > at >>>>> > >>>>> >>>>>org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultC >>>>>hainedExecutable.java:51) >>>>> > at >>>>> > >>>>> >>>>>org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecu >>>>>table.java:107) >>>>> > at >>>>> > >>>>> >>>>>org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(Def >>>>>aultScheduler.java:130) >>>>> > at >>>>> > >>>>> >>>>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja >>>>>va:1145) >>>>> > at >>>>> > >>>>> >>>>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j >>>>>ava:615) >>>>> > >>>>> > at java.lang.Thread.run(Thread.java:745) >>>>> > >>>>> > >
