Hmm although adding hadoop.tmp.dir does not seem to work for me. I added it everywhere in hadoop (core-site,mapred-site and hbase-site 0 and in the kylin job config as well. Still same error. I will have a deeper look
On 10/20/15, 1:58 PM, "Benjamin Leonhardi" <[email protected]> wrote: >Thanks a lot. I swear I googled it :-) > > > > >On 10/20/15, 1:56 PM, "Shi, Shaofeng" <[email protected]> wrote: > >>Hi Benjamin, please check if this JIRA matches your problem: >>https://issues.apache.org/jira/browse/KYLIN-1083 >> >>On 10/20/15, 12:53 PM, "Benjamin Leonhardi" <[email protected]> >>wrote: >> >>>Hello Everybody, >>> >>>Thanks for all the help/ >>> >>>I successfully build Kylin on HDP2.3 with the new updates to the branch >>>and by changing some dependencies in the pom file ( got a HCAT mapred >>>version conflict before ) When I changed the entries in the pom file to >>>the exact version of HDP2.3 it worked. >>> >>> >>> >>> >>><!-- Hadoop versions --> >>> <hadoop2.version>2.7.1</hadoop2.version> >>> <yarn.version>2.7.1</yarn.version> >>> <zookeeper.version>3.4.6</zookeeper.version> >>> <hive.version>1.2.1</hive.version> >>> <hive-hcatalog.version>1.2.1</hive-hcatalog.version> >>> <hbase-hadoop2.version>1.1.1</hbase-hadoop2.version> >>> <curator.version>2.7.1</curator.version> >>> >>> >>>However now I get errors in the test and also when I create a new cube >>>from scratch during the Hfile creation. >>> >>>Any idea what this could be? During the test run I thought it might be a >>>bad testcase but it also happens when I build a new cube from scratch. >>> >>>Parameters: >>> >>> -conf /kylin/kylin-1.1-incubating-SNAPSHOT/conf/kylin_job_conf.xml >>>-cubename aggtest -input >>>/kylin/kylin_metadata/kylin-0669bcc0-2a42-4189-a7c8-6abbb533da8c/aggtest/c >>>uboid/* -output >>>hdfs://sandbox.hortonworks.com:8020/kylin/kylin_metadata/kylin-0669bcc0-2a >>>42-4189-a7c8-6abbb533da8c/aggtest/hfile -htablename KYLIN_0PV292NH5B >>>-jobname Kylin_HFile_Generator_aggtest_Step >>> >>> >>>Error message : >>> >>> >>>java.lang.IllegalArgumentException: Can not create a Path from a null >>>string >>> at org.apache.hadoop.fs.Path.checkPathArg(Path.java:122) >>> at org.apache.hadoop.fs.Path.<init>(Path.java:134) >>> at org.apache.hadoop.fs.Path.<init>(Path.java:88) >>> at >>>org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2.configurePartitioner( >>>HFileOutputFormat2.java:591) >>> at >>>org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2.configureIncrementalL >>>oad(HFileOutputFormat2.java:440) >>> at >>>org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2.configureIncrementalL >>>oad(HFileOutputFormat2.java:405) >>> at >>>org.apache.hadoop.hbase.mapreduce.HFileOutputFormat.configureIncrementalLo >>>ad(HFileOutputFormat.java:91) >>> at >>>org.apache.kylin.job.hadoop.cube.CubeHFileJob.run(CubeHFileJob.java:86) >>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) >>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) >>> at >>>org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable >>>.java:113) >>> at >>>org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutab >>>le.java:107) >>> at >>>org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChai >>>nedExecutable.java:51) >>> at >>>org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutab >>>le.java:107) >>> at >>>org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(Defaul >>>tScheduler.java:130) >>> at >>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java: >>>1145) >>> at >>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java >>>:615) >>> at java.lang.Thread.run(Thread.java:745) >>> >>>result code:2 >>> >>> >>> >>>On 10/9/15, 1:11 PM, "Benjamin Leonhardi" <[email protected]> >>>wrote: >>> >>>>Hello Everybody, >>>> >>>>We figured out the 1.2 coded comes from the first dependency. Will try >>>>and update. >>>> >>>>Thanks for your help. >>>> >>>>Ben >>>> >>>> >>>> >>>>On 10/6/15, 12:47 AM, "周千昊" <[email protected]> wrote: >>>> >>>>>Hi, Benjamin, >>>>> It is a bug, a jira ticket is created as follow: >>>>>https://issues.apache.org/jira/browse/KYLIN-1059 >>>>> and it should have been fixed. since I don't have the hdp 2.3 env >>>>>currently, it would be appreciated if you can help to verify with the >>>>>latest code(branch 1.x-HBase1.x) >>>>> >>>>>Luke Han <[email protected]>于2015年10月5日周一 下午8:35写道： >>>>> >>>>>> Hi Benjamin, >>>>>> Did you generate binary package by yourself with below >>>>>>instruction? or >>>>>> just compile war/jar from maven? >>>>>> >>>>>> http://kylin.incubator.apache.org/development/howto_package.html >>>>>> <http://kylin.incubator.apache.org/development/howto_package.html> >>>>>> >>>>>> Thanks. >>>>>> >>>>>> >>>>>> Best Regards! >>>>>> --------------------- >>>>>> >>>>>> Luke Han >>>>>> >>>>>> On Thu, Oct 1, 2015 at 1:30 PM, Benjamin Leonhardi < >>>>>> [email protected]> wrote: >>>>>> >>>>>> > Hello All, >>>>>> > >>>>>> > I try to get Kylin to run with HDP 2.3 sandbox image. Using the >>>>>>HBase1.x >>>>>> > branch. >>>>>> > >>>>>> > https://github.com/apache/incubator-kylin/tree/1.x-HBase1.x >>>>>> > >>>>>> > It compiles fine and the tests run through until they reach the job >>>>>> > creation. I followed the steps below. >>>>>> > >>>>>> > http://kylin.incubator.apache.org/development/dev_env.html >>>>>> > >>>>>> > When I run the tests he reaches the job part and then fails because >>>>>>he >>>>>> > misses a method in commons-codec.jar. The method is >>>>>>encodeBase64(byte[], >>>>>> > isChunk boolean, isUrlSafe boolean) and it is in commons-codec1.4+ >>>>>>but >>>>>> not >>>>>> > in older versions. I checked but my environment only has codec jars >>>>>>that >>>>>> > are newer. >>>>>> > >>>>>> > However when I build kylin he for some reason compiled an older >>>>>>codec >>>>>> into >>>>>> > the kylin-job-1.1-incubating-SNAPSHOT-job.jar. If I unzip this I >>>>>>see a >>>>>> > Base64 class that only has the method encodeBase64(byte[], isChunk >>>>>> boolean >>>>>> > ) but not the urlsafe method. ( so I suppose codec 1.3 or older ) I >>>>>> tried >>>>>> > to check the pom.xml because I thought I might find an older codec >>>>>> > dependency inside but did not find anything. >>>>>> > >>>>>> > I also have to say I am not the best Maven person so I might be >>>>>> > overlooking something easy. One question I would have is why he >>>>>>compiles >>>>>> > the code into the -job.jar at all. >>>>>> > >>>>>> > >>>>>> > Any help welcome >>>>>> > >>>>>> > Best regards, >>>>>> > >>>>>> > Benjamin >>>>>> > >>>>>> > >>>>>> > Starting: >>>>>> > Kylin_Fact_Distinct_Columns_test_kylin_cube_without_slr_empty_Step >>>>>> > L4J [2015-10-01 >>>>>> > 03:51:24,166][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] - >>>>>> append >>>>>> > job jar: >>>>>> > >>>>>>/root/kylin_build/incubator-kylin/job/../job/target/kylin-job-1.1-incu >>>>>> > bating-SNAPSHOT-job.jar >>>>>> > L4J [2015-10-01 >>>>>> > 03:51:24,169][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] - >>>>>> append >>>>>> > kylin.hive.dependency: null and kylin.hbase.dependency: null to >>>>>> > mapreduce.appli >>>>>> > cation.classpath >>>>>> > L4J [2015-10-01 >>>>>> > 03:51:24,167][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] - >>>>>> append >>>>>> > job jar: >>>>>> > >>>>>>/root/kylin_build/incubator-kylin/job/../job/target/kylin-job-1.1-incu >>>>>> > bating-SNAPSHOT-job.jar >>>>>> > L4J [2015-10-01 >>>>>> > 03:51:24,170][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] - >>>>>> append >>>>>> > kylin.hive.dependency: null and kylin.hbase.dependency: null to >>>>>> > mapreduce.appli >>>>>> > cation.classpath >>>>>> > L4J [2015-10-01 >>>>>> > 03:51:24,170][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] - >>>>>> Hadoop >>>>>> > job classpath is: >>>>>> > >>>>>> >>>>>>/tmp/kylin/*,$HADOOP_CONF_DIR,/usr/hdp/2.3.0.0-2530/hbase/lib/hbase-com >>>>>>mon.jar,/usr/hdp/current/hive-client/conf/,/usr/hdp/2.3.0.0-2530/hive/l >>>>>>ib/hive-metastore.jar,/usr/hdp/2.3.0.0-2530/hive/lib/hive-exec.jar,/usr >>>>>>/hdp/2.3.0.0-2530/hive-hcatalog/share/hcatalog/*,$PWD/mr-framework/hado >>>>>>op/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapre >>>>>>duce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-frame >>>>>>work/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/ha >>>>>>doop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-fr >>>>>>amework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoo >>>>>>p/hdfs/lib/*:/usr/hdp/2.3.0.0-2530/hadoop/lib/hadoop-lzo-0.6.0.2.3.0.0- >>>>>>2530.jar:/etc/hadoop/conf/secure >>>>>> > L4J [2015-10-01 >>>>>> > 03:51:24,169][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] - >>>>>> Hadoop >>>>>> > job classpath is: >>>>>> > >>>>>> >>>>>>/tmp/kylin/*,$HADOOP_CONF_DIR,/usr/hdp/2.3.0.0-2530/hbase/lib/hbase-com >>>>>>mon.jar,/usr/hdp/current/hive-client/conf/,/usr/hdp/2.3.0.0-2530/hive/l >>>>>>ib/hive-metastore.jar,/usr/hdp/2.3.0.0-2530/hive/lib/hive-exec.jar,/usr >>>>>>/hdp/2.3.0.0-2530/hive-hcatalog/share/hcatalog/*,$PWD/mr-framework/hado >>>>>>op/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapre >>>>>>duce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-frame >>>>>>work/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/ha >>>>>>doop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-fr >>>>>>amework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoo >>>>>>p/hdfs/lib/*:/usr/hdp/2.3.0.0-2530/hadoop/lib/hadoop-lzo-0.6.0.2.3.0.0- >>>>>>2530.jar:/etc/hadoop/conf/secure >>>>>> > L4J [2015-10-01 >>>>>>03:51:24,229][WARN][org.apache.hadoop.hive.conf.HiveConf] >>>>>> > - HiveConf of name hive.heapsize does not exist >>>>>> > L4J [2015-10-01 >>>>>>03:51:24,230][WARN][org.apache.hadoop.hive.conf.HiveConf] >>>>>> > - HiveConf of name hive.server2.enable.impersonation does not exist >>>>>> > L4J [2015-10-01 >>>>>>03:51:24,236][WARN][org.apache.hadoop.hive.conf.HiveConf] >>>>>> > - HiveConf of name hive.heapsize does not exist >>>>>> > L4J [2015-10-01 >>>>>>03:51:24,237][WARN][org.apache.hadoop.hive.conf.HiveConf] >>>>>> > - HiveConf of name hive.server2.enable.impersonation does not exist >>>>>> > L4J [2015-10-01 >>>>>> > 03:51:24,265][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] - >>>>>> > tempMetaFileString is : null >>>>>> > L4J [2015-10-01 >>>>>> > 03:51:24,266][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob] - >>>>>> > tempMetaFileString is : null >>>>>> > L4J [2015-10-01 >>>>>> > >>>>>>03:51:24,267][ERROR][org.apache.kylin.job.execution.AbstractExecutable] >>>>>> - >>>>>> > error running Executable >>>>>> > java.lang.NoSuchMethodError: >>>>>> > org.apache.commons.codec.binary.Base64.encodeBase64([BZZ)[B >>>>>> > at >>>>>> > >>>>>>org.apache.hive.hcatalog.common.HCatUtil.encodeBytes(HCatUtil.java:125) >>>>>> > at >>>>>> > >>>>>>org.apache.hive.hcatalog.common.HCatUtil.serialize(HCatUtil.java:104) >>>>>> > at >>>>>> > >>>>>>org.apache.hive.hcatalog.common.HCatUtil.getHiveConf(HCatUtil.java:585) >>>>>> > at >>>>>> > >>>>>> >>>>>>org.apache.hive.hcatalog.mapreduce.InitializeInput.getInputJobInfo(Init >>>>>>ializeInput.java:100) >>>>>> > at >>>>>> > >>>>>> >>>>>>org.apache.hive.hcatalog.mapreduce.InitializeInput.setInput(InitializeI >>>>>>nput.java:86) >>>>>> > at >>>>>> > >>>>>> >>>>>>org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFo >>>>>>rmat.java:95) >>>>>> > at >>>>>> > >>>>>> >>>>>>org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFo >>>>>>rmat.java:51) >>>>>> > at >>>>>> > >>>>>> >>>>>>org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.setupMapper(Fac >>>>>>tDistinctColumnsJob.java:101) >>>>>> > at >>>>>> > >>>>>> >>>>>>org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.run(FactDistinc >>>>>>tColumnsJob.java:77) >>>>>> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) >>>>>> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) >>>>>> > at >>>>>> > >>>>>> >>>>>>org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecuta >>>>>>ble.java:113) >>>>>> > at >>>>>> > >>>>>> >>>>>>org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecu >>>>>>table.java:107) >>>>>> > at >>>>>> > >>>>>> >>>>>>org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultC >>>>>>hainedExecutable.java:51) >>>>>> > at >>>>>> > >>>>>> >>>>>>org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecu >>>>>>table.java:107) >>>>>> > at >>>>>> > >>>>>> >>>>>>org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(Def >>>>>>aultScheduler.java:130) >>>>>> > at >>>>>> > >>>>>> >>>>>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja >>>>>>va:1145) >>>>>> > at >>>>>> > >>>>>> >>>>>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j >>>>>>ava:615) >>>>>> > at java.lang.Thread.run(Thread.java:745) >>>>>> > L4J [2015-10-01 >>>>>> > >>>>>>03:51:24,267][ERROR][org.apache.kylin.job.execution.AbstractExecutable] >>>>>> - >>>>>> > error running Executable >>>>>> > java.lang.NoSuchMethodError: >>>>>> > org.apache.commons.codec.binary.Base64.encodeBase64([BZZ)[B >>>>>> > at >>>>>> > >>>>>>org.apache.hive.hcatalog.common.HCatUtil.encodeBytes(HCatUtil.java:125) >>>>>> > at >>>>>> > >>>>>>org.apache.hive.hcatalog.common.HCatUtil.serialize(HCatUtil.java:104) >>>>>> > at >>>>>> > >>>>>>org.apache.hive.hcatalog.common.HCatUtil.getHiveConf(HCatUtil.java:585) >>>>>> > at >>>>>> > >>>>>> >>>>>>org.apache.hive.hcatalog.mapreduce.InitializeInput.getInputJobInfo(Init >>>>>>ializeInput.java:100) >>>>>> > at >>>>>> > >>>>>> >>>>>>org.apache.hive.hcatalog.mapreduce.InitializeInput.setInput(InitializeI >>>>>>nput.java:86) >>>>>> > at >>>>>> > >>>>>> >>>>>>org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFo >>>>>>rmat.java:95) >>>>>> > at >>>>>> > >>>>>> >>>>>>org.apache.hive.hcatalog.mapreduce.HCatInputFormat.setInput(HCatInputFo >>>>>>rmat.java:51) >>>>>> > at >>>>>> > >>>>>> >>>>>>org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.setupMapper(Fac >>>>>>tDistinctColumnsJob.java:101) >>>>>> > at >>>>>> > >>>>>> >>>>>>org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.run(FactDistinc >>>>>>tColumnsJob.java:77) >>>>>> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) >>>>>> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) >>>>>> > at >>>>>> > >>>>>> >>>>>>org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecuta >>>>>>ble.java:113) >>>>>> > at >>>>>> > >>>>>> >>>>>>org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecu >>>>>>table.java:107) >>>>>> > at >>>>>> > >>>>>> >>>>>>org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultC >>>>>>hainedExecutable.java:51) >>>>>> > at >>>>>> > >>>>>> >>>>>>org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecu >>>>>>table.java:107) >>>>>> > at >>>>>> > >>>>>> >>>>>>org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(Def >>>>>>aultScheduler.java:130) >>>>>> > at >>>>>> > >>>>>> >>>>>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja >>>>>>va:1145) >>>>>> > at >>>>>> > >>>>>> >>>>>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j >>>>>>ava:615) >>>>>> > >>>>>> > at java.lang.Thread.run(Thread.java:745) >>>>>> > >>>>>> >> >>
