Re: Cube build error - file kylin_job_meta does not exist

[Li Yang]

Well, it's very old and stable.  :-)

We shall check the new Hadoop Job API.
http://hadoop.apache.org/docs/stable2/api/org/apache/hadoop/mapreduce/Job.html#addArchiveToClassPath(org.apache.hadoop.fs.Path)


On Wed, Jan 13, 2016 at 12:30 AM, hongbin ma  wrote:

> btw, @liyang is our usage of hadoop distributed cache typical? it's not
> easy to find documentations for it, is it a stable and sustainable function
> of hadoop?
>
> On Tue, Jan 12, 2016 at 6:50 PM, Li Yang  wrote:
>
> > Seems your hadoop has a problem when copying distributed cache.  However
> > not sure of the best way to troubleshoot the problem.  Maybe try a
> simpler
> > MR job that involves distributed cache?
> >
> > On Fri, Jan 8, 2016 at 7:05 AM, Jiunn Jye Ng  wrote:
> >
> > > Hi,
> > > I am trying out Kylin 1.2.
> > > While building the cube, the job has failed at Step 2 with error file
> > > "kylin_job_meta546890272332823064/meta" does not exist.
> > > Can someone provide some hint what could have gone wrong.
> > >
> > > Below are more detail of the error captured
> > >
> > > AM Container for appattempt_1452159080124_0008_01 exited with
> > exitCode:
> > > -1000
> > > For more detailed output, check application tracking page:
> > > http://:8088/cluster/app/application_1452159080124_0008Then, click
> > on
> > > links to logs of each attempt.
> > > Diagnostics: File
> > >
> > >
> >
> file:/root/Downloads/apache-kylin-1.2-bin/tomcat/temp/kylin_job_meta546890272332823064/meta
> > > does not exist
> > > java.io.FileNotFoundException: File
> > >
> > >
> >
> file:/root/Downloads/apache-kylin-1.2-bin/tomcat/temp/kylin_job_meta546890272332823064/meta
> > > does not exist
> > > at
> > >
> > >
> >
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:606)
> > > at
> > >
> > >
> >
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:819)
> > > at
> > >
> > >
> >
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:596)
> > > at
> > >
> > >
> >
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421)
> > > at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:253)
> > > at
> org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:63)
> > > at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:361)
> > > at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)
> > > at java.security.AccessController.doPrivileged(Native Method)
> > > at javax.security.auth.Subject.doAs(Subject.java:415)
> > > at
> > >
> > >
> >
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> > > at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:358)
> > > at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62)
> > > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> > > at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> > > at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> > > at
> > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> > > at
> > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> > > at java.lang.Thread.run(Thread.java:745)
> > > Failing this attempt
> > >
> > >
> > > Thank you very much.
> > >
> > > Rgds,
> > > Ng Jiunn Jye (黄俊杰）
> > >
> >
>
>
>
> --
> Regards,
>
> *Bin Mahone | 马洪宾*
> Apache Kylin: http://kylin.io
> Github: https://github.com/binmahone
>

Re: internal hive table and build the cube backward

[yu feng]

ShaoFeng Shi is right, kylin use hive command to generate  intermediate
table(take it as source data), and use hcatlog get data from hive in step2,
hive performance does have an impact on Kylin's performance, so a newer
version is recommended。

2016-01-20 8:05 GMT+08:00 ShaoFeng Shi :

> Only the first step actually, Kylin runs "hive -e" command to create an
> intermediate table; The following steps are running MR over the files under
> that table.
>
> 2016-01-20 4:18 GMT+08:00 zhong zhang :
>
> > Hi Yu and Everyone,
> >
> > Just a little bit supplement, Hive definitely involves in the step of
> > Create
> > Intermediate Flat Hive Table and Build Dimension Dictionary. The question
> > is that does Hive involve in the following steps of building cuboids?
> >
> > Best regards,
> > Zhong
> >
> > On Sun, Jan 17, 2016 at 10:35 PM, yu feng  wrote:
> >
> > > Firstly, kylin do not distinguish which kind table in hive,  if only
> you
> > > can query it in hive, so the table can be normal table, external table,
> > > view or table with some serdes.
> > > then I think it is hard to build cube backward along the time in kylin.
> > > maybe someone has some good ideas at this point.
> > >
> > > 2016-01-18 11:04 GMT+08:00 zhong zhang :
> > >
> > > > Hi All,
> > > >
> > > > I'm wondering can I build the Kylin cube backward along the time.
> More
> > > > specifically, can I build the cube from the current time to six
> months
> > > ago
> > > > and then from six months ago to 12 months ago and go on? In this
> way, I
> > > can
> > > > have the latest six months' cube result first.
> > > >
> > > > It's well known that the input of Kylin cube is hive table. Does it
> > make
> > > > any difference
> > > > between using internal hive table and external hive table when
> building
> > > the
> > > > cube?
> > > >
> > > > Best regards,
> > > > Zhong
> > > >
> > >
> >
>
>
>
> --
> Best regards,
>
> Shaofeng Shi
>

Re: internal hive table and build the cube backward

[ShaoFeng Shi]

Only the first step actually, Kylin runs "hive -e" command to create an
intermediate table; The following steps are running MR over the files under
that table.

2016-01-20 4:18 GMT+08:00 zhong zhang :

> Hi Yu and Everyone,
>
> Just a little bit supplement, Hive definitely involves in the step of
> Create
> Intermediate Flat Hive Table and Build Dimension Dictionary. The question
> is that does Hive involve in the following steps of building cuboids?
>
> Best regards,
> Zhong
>
> On Sun, Jan 17, 2016 at 10:35 PM, yu feng  wrote:
>
> > Firstly, kylin do not distinguish which kind table in hive,  if only you
> > can query it in hive, so the table can be normal table, external table,
> > view or table with some serdes.
> > then I think it is hard to build cube backward along the time in kylin.
> > maybe someone has some good ideas at this point.
> >
> > 2016-01-18 11:04 GMT+08:00 zhong zhang :
> >
> > > Hi All,
> > >
> > > I'm wondering can I build the Kylin cube backward along the time. More
> > > specifically, can I build the cube from the current time to six months
> > ago
> > > and then from six months ago to 12 months ago and go on? In this way, I
> > can
> > > have the latest six months' cube result first.
> > >
> > > It's well known that the input of Kylin cube is hive table. Does it
> make
> > > any difference
> > > between using internal hive table and external hive table when building
> > the
> > > cube?
> > >
> > > Best regards,
> > > Zhong
> > >
> >
>



-- 
Best regards,

Shaofeng Shi

关于 apache kylin 函数的问题

[王琳]

Dear

[jira] [Created] (KYLIN-1346) IllegalStateException happens in SparkCubing

[Dong Li (JIRA)]

Dong Li created KYLIN-1346:
--

 Summary: IllegalStateException happens in SparkCubing
 Key: KYLIN-1346
 URL: https://issues.apache.org/jira/browse/KYLIN-1346
 Project: Kylin
  Issue Type: Bug
  Components: Spark Engine
Affects Versions: v2.0
Reporter: Dong Li
Assignee: Dong Li
Priority: Minor


java.lang.IllegalStateException
at 
com.google.common.base.Preconditions.checkState(Preconditions.java:129)
at org.apache.kylin.cube.kv.RowKeyEncoder.encode(RowKeyEncoder.java:92)
at 
org.apache.kylin.engine.spark.cube.DefaultTupleConverter.convert(DefaultTupleConverter.java:87)
at 
org.apache.kylin.engine.spark.cube.BufferedCuboidWriter.write(BufferedCuboidWriter.java:46)
at 
org.apache.kylin.cube.inmemcubing.AbstractInMemCubeBuilder.outputCuboid(AbstractInMemCubeBuilder.java:89)
at 
org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$Merger.mergeAndOutput(DoggedCubeBuilder.java:308)
at 
org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.build(DoggedCubeBuilder.java:116)
at 
org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder.build(DoggedCubeBuilder.java:72)
at 
org.apache.kylin.cube.inmemcubing.AbstractInMemCubeBuilder$1.run(AbstractInMemCubeBuilder.java:74)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
16/01/19 03:34:15 ERROR inmemcubing.DoggedCubeBuilder: Error closing grid table 
org.apache.kylin.gridtable.GridTable@
java.lang.IllegalStateException
at 
org.apache.kylin.cube.inmemcubing.ConcurrentDiskStore.close(ConcurrentDiskStore.java:357)
at org.apache.kylin.gridtable.GridTable.close(GridTable.java:48)
at 
org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.closeGirdTables(DoggedCubeBuilder.java:140)
at 
org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder$BuildOnce.build(DoggedCubeBuilder.java:128)
at 
org.apache.kylin.cube.inmemcubing.DoggedCubeBuilder.build(DoggedCubeBuilder.java:72)
at 
org.apache.kylin.cube.inmemcubing.AbstractInMemCubeBuilder$1.run(AbstractInMemCubeBuilder.java:74)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KYLIN-1341) Display JSON of Data Model in the dialog

[Dong Li (JIRA)]

Dong Li created KYLIN-1341:
--

 Summary: Display JSON of Data Model in the dialog
 Key: KYLIN-1341
 URL: https://issues.apache.org/jira/browse/KYLIN-1341
 Project: Kylin
  Issue Type: Improvement
  Components: Web 
Affects Versions: v2.0
Reporter: Dong Li
Assignee: Zhong,Jason
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)