Hbase table is always empty when build with spark
In kylin-2.5.2, the result hbase table is always table is always empty when I build cube with spark. I found that the step "Load HFile to HBase Table" has some warn log: /2019-01-27 00:49:30,067 WARN [Scheduler 448149092 Job 89a25959-e12d-7a5e-0ecb-80c978533eab-6419] mapreduce.LoadIncrementalHFiles:204 : Skipping non-directory hdfs://test/kylin/kylin_metadata/kylin-89a25959-e12d-7a5e-0ecb-80c978533eab/test_UUID_spark/hfile/_SUCCESS 2019-01-27 00:49:30,068 WARN [Scheduler 448149092 Job 89a25959-e12d-7a5e-0ecb-80c978533eab-6419] mapreduce.LoadIncrementalHFiles:204 : Skipping non-directory hdfs://test/kylin/kylin_metadata/kylin-89a25959-e12d-7a5e-0ecb-80c978533eab/test_UUID_spark/hfile/part-r-0 2019-01-27 00:49:30,068 WARN [Scheduler 448149092 Job 89a25959-e12d-7a5e-0ecb-80c978533eab-6419] mapreduce.LoadIncrementalHFiles:204 : Skipping non-directory hdfs://test/kylin/kylin_metadata/kylin-89a25959-e12d-7a5e-0ecb-80c978533eab/test_UUID_spark/hfile/part-r-1/ After read the source code, I found the step "Convert Cuboid Data to HFile" with spark has bug. The above step's outputdir should has subdirectory with column family. Indeed, SparkCubeHFile must set mapreduce.job.outputformat.class with HFileOutputFormat2.class. Please check if I am correct！ -- Sent from: http://apache-kylin.74782.x6.nabble.com/
[jira] [Created] (KYLIN-3819) kylin.sh run in the foreground
Temple Zhou created KYLIN-3819: -- Summary: kylin.sh run in the foreground Key: KYLIN-3819 URL: https://issues.apache.org/jira/browse/KYLIN-3819 Project: Kylin Issue Type: Improvement Components: Tools, Build and Test Affects Versions: v2.6.0, v2.5.0, v2.4.0 Reporter: Temple Zhou Assignee: Temple Zhou The "kylin.sh start" can only run kylin in the background, but it should be run in the foreground as well. Because only in this way, both streams can be captured if using a supervision system like daemontools. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Re: [Discuss] Moving toward Apache Kylin 3.0
Thanks for the feedbacks; Since there was no objection on this, the version on the current master branch has been updated to 3.0.0-SNAPSHOT. Best regards, Shaofeng Shi 史少锋 Apache Kylin PMC Work email: shaofeng@kyligence.io Kyligence Inc: https://kyligence.io/ Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html Join Kylin user mail group: user-subscr...@kylin.apache.org Join Kylin dev mail group: dev-subscr...@kylin.apache.org Yichen Zhou 于2019年1月25日周五 下午4:14写道： > +1 > > Regards, > Yichen > > On Thu, Jan 24, 2019 at 11:49 PM nichunen wrote: > > > +1 > > > > > > -- > > > > > > Best regards, > > > > > > > > Ni Chunen / George > > > > > > > > > > At 2019-01-25 15:48:21, "Billy Liu" wrote: > > >+1 > > > > > >That's cool. Let's move to the real-time scenario. > > > > > >With Warm regards > > > > > >Billy Liu > > > > > > > > >Temple Zhou 于2019年1月25日周五 下午3:32写道： > > > > > >> +1 > > >> Real-time streaming feature may be expected by many people. > > >> ——— > > >> Best wishes~ > > >> Temple Zhou > > >> > > >> Original Message > > >> *Sender:* zhan shaoxiong > > >> *Recipient:* dev@kylin.apache.org > > >> *Date:* Friday, Jan 25, 2019 15:27 > > >> *Subject:* Re: [Discuss] Moving toward Apache Kylin 3.0 > > >> > > >> +1 > > >> thanks > > >> > > >> 在 2019/1/23 下午3:57，“ShaoFeng Shi” 写入: > > >> > > >> Hi Kylin developers, > > >> > > >> In last week, Kylin released v2.6.0, with the enhanced & > > distributed query > > >> cache and JDBC data source SDK. After this release, the next batch > > >> candidate features include real-time streaming, parquet storage, > > and druid > > >> storage. These features were developed in the past 1-2 years by > > different > > >> Kylin players and were open sourced in the past 6 months. They > have > > already > > >> been staged in separate branches and are under evaluation by the > > community. > > >> We have received much feedback from the community. > > >> > > >> These candidate features are big supplements to as-is Kylin > > functions; For > > >> example, the real-time streaming feature will bring Kylin from > > batch & > > >> historical analytics into real-time analytics. The parquet storage > > will > > >> make the deployment more flexible and more cloud-friendly. Of > > course, > > >> stabilizing and improving these features need additional time and > > effort. > > >> > > >> So, when we merging and releasing them, we'd better give it a new > > version > > >> number so that user can clearly know the difference with current > 2.x > > >> versions. I discussed this with several developers offline, we > > think it is > > >> time to move toward Kylin 3.0. So, if one of the above features is > > merged, > > >> the version will be 3.0. The current 2.6 will be maintained until > > 3.x is > > >> ready for production use. > > >> > > >> Your comments, ideas, and suggestions are welcomed! > > >> > > >> Best regards, > > >> > > >> Shaofeng Shi 史少锋 > > >> Apache Kylin PMC > > >> Work email: shaofeng@kyligence.io > > >> Kyligence Inc: https://kyligence.io/ > > >> > > >> Apache Kylin FAQ: > > https://kylin.apache.org/docs/gettingstarted/faq.html > > >> Join Kylin user mail group: user-subscr...@kylin.apache.org > > >> Join Kylin dev mail group: dev-subscr...@kylin.apache.org > > >> > > >> > > >
Re: Kylin go to hdfs to find jar file
Did you configure to use HDFS as the defaultFS in core-site.xml? Best regards, Shaofeng Shi 史少锋 Apache Kylin PMC Work email: shaofeng@kyligence.io Kyligence Inc: https://kyligence.io/ Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html Join Kylin user mail group: user-subscr...@kylin.apache.org Join Kylin dev mail group: dev-subscr...@kylin.apache.org jiangxiaoma111 <369806...@qq.com> 于2019年2月19日周二 下午6:02写道： > Did you find the solution of this exception > > -- > Sent from: http://apache-kylin.74782.x6.nabble.com/ >
Re: Kylin go to hdfs to find jar file
Did you find the solution of this exception -- Sent from: http://apache-kylin.74782.x6.nabble.com/
Build cube exception: java.io.FileNotFoundException
hi, all. I am a beginner of kylin. My deployment env: os: max os 10.12.6 hadoop: 3.0.0 hive: 2.3.1 hbase: 1.2.9 kylin: apache-kylin-2.6.0-bin-hbase1x All project is deployed standlone. When I build a cube,there is a error occured in Step 3: 2019-02-19 17:22:33,757 ERROR [pool-10-thread-3] threadpool.DefaultScheduler:116 : ExecuteException job:f2fdcdd5-7c2b-df7c-89ea-8a61a85f8975 org.apache.kylin.job.exception.ExecuteException: org.apache.kylin.job.exception.ExecuteException: java.io.FileNotFoundException: File does not exist: hdfs://localhost:8020/usr/local/var/log/hadoop/hadoop-didi/mapred/staging/didi2113494170/.staging/job_local2113494170_0001/libjars/hive-hcatalog-core-2.3.1.jar at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:180) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.kylin.job.exception.ExecuteException: java.io.FileNotFoundException: File does not exist: hdfs://localhost:8020/usr/local/var/log/hadoop/hadoop-didi/mapred/staging/didi2113494170/.staging/job_local2113494170_0001/libjars/hive-hcatalog-core-2.3.1.jar at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:180) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:70) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165) ... 4 more Caused by: java.io.FileNotFoundException: File does not exist: hdfs://localhost:8020/usr/local/var/log/hadoop/hadoop-didi/mapred/staging/didi2113494170/.staging/job_local2113494170_0001/libjars/hive-hcatalog-core-2.3.1.jar at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1072) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1064) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) I searched a lot of posts, but can not solute the exception. Please help. -- Sent from: http://apache-kylin.74782.x6.nabble.com/
[jira] [Created] (KYLIN-3818) After Cube disabled, merge cube job still running
Na Zhai created KYLIN-3818: -- Summary: After Cube disabled, merge cube job still running Key: KYLIN-3818 URL: https://issues.apache.org/jira/browse/KYLIN-3818 Project: Kylin Issue Type: Bug Affects Versions: v2.6.0 Reporter: Na Zhai Assignee: Na Zhai There is a merge cube job with an error status, then I discard the job, then I disable the cube, but the merge cube job still running. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
