[jira] [Created] (KYLIN-1964) Add a companion tool of CubeMetaExtractor for cube importing

[hongbin ma (JIRA)]

hongbin ma created KYLIN-1964:
-

 Summary: Add a companion tool of CubeMetaExtractor for cube 
importing
 Key: KYLIN-1964
 URL: https://issues.apache.org/jira/browse/KYLIN-1964
 Project: Kylin
  Issue Type: Wish
Reporter: hongbin ma
Assignee: hongbin ma


Now that we have CubeMetaExtractor for cube exporting, additionally we need a 
importer to import the exported cube



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Cube query problem

[lxw]

Hi,experts:

   After create and build successfully, I use SQL to query the cube and got 
error:

Error while executing SQL "select advid_, sum(pv_) as pv  from lxw1234.t_table  
group by advid_   limit 10": Error in coprocessor 
   
   I use kylin-1.5.3??hbase-0.98.6-cdh5.2.0,  and some other cubes can be 
normal query .

   The error log is :

2016-08-19 13:24:55,862 WARN  [kylin-coproc--pool5-t19] 
ipc.CoprocessorRpcChannel:59 : Call failed on IOException
org.apache.hadoop.hbase.DoNotRetryIOException: 
org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.NoClassDefFoundError: 
Could not initialize class org.apache.kylin.metadata.datatype.DataType
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2067)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
at 
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.NoClassDefFoundError: Could not initialize class 
org.apache.kylin.metadata.datatype.DataType
at org.apache.kylin.gridtable.GTInfo$1.deserialize(GTInfo.java:328)
at org.apache.kylin.gridtable.GTInfo$1.deserialize(GTInfo.java:283)
at 
org.apache.kylin.gridtable.GTScanRequest$2.deserialize(GTScanRequest.java:322)
at 
org.apache.kylin.gridtable.GTScanRequest$2.deserialize(GTScanRequest.java:295)
at 
org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService.visitCube(CubeVisitService.java:189)
at 
org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitService.callMethod(CubeVisitProtos.java:4353)
at 
org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:5602)
at 
org.apache.hadoop.hbase.regionserver.HRegionServer.execServiceOnRegion(HRegionServer.java:3416)
at 
org.apache.hadoop.hbase.regionserver.HRegionServer.execService(HRegionServer.java:3398)
at 
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29591)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031)
... 4 more

at sun.reflect.GeneratedConstructorAccessor86.newInstance(Unknown 
Source)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at 
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at 
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
at 
org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:304)
at 
org.apache.hadoop.hbase.protobuf.ProtobufUtil.execService(ProtobufUtil.java:1627)
at 
org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:93)
at 
org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:90)
at 
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:114)
at 
org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:90)
at 
org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel.callExecService(RegionCoprocessorRpcChannel.java:96)
at 
org.apache.hadoop.hbase.ipc.CoprocessorRpcChannel.callMethod(CoprocessorRpcChannel.java:57)
at 
org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitService$Stub.visitCube(CubeVisitProtos.java:4400)
at 
org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$1$1.call(CubeHBaseEndpointRPC.java:371)
at 
org.apache.kylin.storage.hbase.cube.v2.CubeHBaseEndpointRPC$1$1.call(CubeHBaseEndpointRPC.java:367)
at org.apache.hadoop.hbase.client.HTable$16.call(HTable.java:1542)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: 
org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.DoNotRetryIOException):
 org.apache.hadoop.hbase.DoNotRetryIOException: java.lang.NoClassDefFoundError: 
Could not initialize class org.apache.kylin.metadata.datatype.DataType
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2067)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
at 
org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
at java.lang.Thread.run(Thread.java:744)

Re: Kylin Query Performance

[Mars J]

just 'select A,B from Fact f left join dima a on f.no=a.no...', when I
query this, it needs 10+s, then the same sql will return result in 0.0s,
but when I change the limit N or any column behind 'select' and 'group by'
and join table ,it costs more than 10+s again.
the source records in fact and dima is 160w , and count no in htable is
1.9billion.

2016-08-18 17:50 GMT+08:00 Li Yang :

> Performance troubleshoot is complicated and requires much information. If
> you can share a diagnosis pack, people maybe able to help.
>
> On Sat, Aug 13, 2016 at 10:32 PM, Yiming Liu 
> wrote:
>
> > What's kind of queries? How many records supposed to be returned? Could
> you
> > send out the query log together?
> >
> > 2016-08-12 10:28 GMT+08:00 Mars J :
> >
> > > Hi,
> > > I have run Kylin 1.5.2.1 and build a cube successfully,the cube
> size
> > is
> > > 3.2G for fact table and dimensional table have 1.5 million records
> > > seperately.HTable count is about 200 billion.
> > >My Cube Design includes 3 Derived Dims and 5 Normal Dims which
> formed
> > a
> > > hierachy dim in agg. Rowkeys are generated automatically,the first and
> > > second rowkey is the highest cardinality column in fact table and one
> dim
> > > table seperately.
> > >
> > > When I query it from kylin insight, it costs 11s,and the second
> same
> > > query is also 10+s, How can I optimize this ?
> > >
> >
> >
> >
> > --
> > With Warm regards
> >
> > Yiming Liu (刘一鸣)
> >
>

回复: SQL got unexpected result

[liuhua...@neusoft.com]

--I get the expect result when I use the where condition.
SQL1:
select a.*,b.aae019 from 
(SELECT  akc193,bkc231,akb020,akc190, aae072
  FROm kc21_24_60_ka06 a 
  where akb020='2189' and akc190 = 'MZ492539' and aae072 = 'MB492541'
group by  akc193,bkc231,akb020,akc190,aae072) a
join 
(SELECT akb020, akc190, aae072, SUm(aae019) aae019
  FROm kc22_test
 group by akb020, akc190, aae072) b
on a.akb020 = b.akb020
   AND b.akc190 = a.akc190
   AND b.aae072 = a.aae072;
result:
R05 01
咳嗽
2189
MZ492539
MB492541
289.6

--when I remove the where condition,the result is null,that's not my expection.
SQL2:
select a.*,b.aae019 from 
(SELECT  akc193,bkc231,akb020,akc190, aae072
  FROm kc21_24_60_ka06 a 
  --where akb020='2189' and akc190 = 'MZ492539' and aae072 = 'MB492541'
group by  akc193,bkc231,akb020,akc190,aae072) a
join 
(SELECT akb020, akc190, aae072, SUm(aae019) aae019
  FROm kc22_test
 group by akb020, akc190, aae072) b
on a.akb020 = b.akb020
   AND b.akc190 = a.akc190
   AND b.aae072 = a.aae072;
result:
 Results(0)
  


刘焕宇

技术发展部（TDD）
 
东软集团股份有限公司 
沈阳浑南新区新秀街2号 东软软件园  A2-105A
Mobile：18602471510
Email：liuhua...@neusoft.com
 
发件人： liuhua...@neusoft.com
发送时间： 2016-08-19 10:12
收件人： dev
主题： SQL got unexpected result
Hi,I found there's a bug when I execute SQL in Insight window:

--I get the expect result when I use the where condition.
SQL1:
select a.*,b.aae019 from 
(SELECT  akc193,bkc231,akb020,akc190, aae072
  FROm kc21_24_60_ka06 a 
  where akb020='2189' and akc190 = 'MZ492539' and aae072 = 'MB492541'
group by  akc193,bkc231,akb020,akc190,aae072) a
join 
(SELECT akb020, akc190, aae072, SUm(aae019) aae019
  FROm kc22_test
 group by akb020, akc190, aae072) b
on a.akb020 = b.akb020
   AND b.akc190 = a.akc190
   AND b.aae072 = a.aae072;
result:


--when I remove the where condition,the result is null,that's not my expection.
SQL2:
select a.*,b.aae019 from 
(SELECT  akc193,bkc231,akb020,akc190, aae072
  FROm kc21_24_60_ka06 a 
  --where akb020='2189' and akc190 = 'MZ492539' and aae072 = 'MB492541'
group by  akc193,bkc231,akb020,akc190,aae072) a
join 
(SELECT akb020, akc190, aae072, SUm(aae019) aae019
  FROm kc22_test
 group by akb020, akc190, aae072) b
on a.akb020 = b.akb020
   AND b.akc190 = a.akc190
   AND b.aae072 = a.aae072;
result:


who can tell me why this happened?

best regards.



刘焕宇

技术发展部（TDD）
 
东软集团股份有限公司 
沈阳浑南新区新秀街2号 东软软件园  A2-105A
Mobile：18602471510
Email：liuhua...@neusoft.com

---
Confidentiality Notice: The information contained in this e-mail and any 
accompanying attachment(s) 
is intended only for the use of the intended recipient and may be confidential 
and/or privileged of 
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of 
this communication is 
not the intended recipient, unauthorized use, forwarding, printing,  storing, 
disclosure or copying 
is strictly prohibited, and may be unlawful.If you have received this 
communication in error,please 
immediately notify the sender by return e-mail, and delete the original message 
and all copies from 
your system. Thank you. 
---


---
Confidentiality Notice: The information contained in this e-mail and any 
accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential 
and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of 
this communication is
not the intended recipient, unauthorized use, forwarding, printing,  storing, 
disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this 
communication in error,please
immediately notify the sender by return e-mail, and delete the original message 
and all copies from
your system. Thank you.
---

SQL got unexpected result

[liuhua...@neusoft.com]

Hi,I found there's a bug when I execute SQL in Insight window:

--I get the expect result when I use the where condition.
SQL1:
select a.*,b.aae019 from 
(SELECT  akc193,bkc231,akb020,akc190, aae072
  FROm kc21_24_60_ka06 a 
  where akb020='2189' and akc190 = 'MZ492539' and aae072 = 'MB492541'
group by  akc193,bkc231,akb020,akc190,aae072) a
join 
(SELECT akb020, akc190, aae072, SUm(aae019) aae019
  FROm kc22_test
 group by akb020, akc190, aae072) b
on a.akb020 = b.akb020
   AND b.akc190 = a.akc190
   AND b.aae072 = a.aae072;
result:


--when I remove the where condition,the result is null,that's not my expection.
SQL2:
select a.*,b.aae019 from 
(SELECT  akc193,bkc231,akb020,akc190, aae072
  FROm kc21_24_60_ka06 a 
  --where akb020='2189' and akc190 = 'MZ492539' and aae072 = 'MB492541'
group by  akc193,bkc231,akb020,akc190,aae072) a
join 
(SELECT akb020, akc190, aae072, SUm(aae019) aae019
  FROm kc22_test
 group by akb020, akc190, aae072) b
on a.akb020 = b.akb020
   AND b.akc190 = a.akc190
   AND b.aae072 = a.aae072;
result:


who can tell me why this happened?

best regards.



刘焕宇

技术发展部（TDD）
 
东软集团股份有限公司 
沈阳浑南新区新秀街2号 东软软件园  A2-105A
Mobile：18602471510
Email：liuhua...@neusoft.com


---
Confidentiality Notice: The information contained in this e-mail and any 
accompanying attachment(s)
is intended only for the use of the intended recipient and may be confidential 
and/or privileged of
Neusoft Corporation, its subsidiaries and/or its affiliates. If any reader of 
this communication is
not the intended recipient, unauthorized use, forwarding, printing,  storing, 
disclosure or copying
is strictly prohibited, and may be unlawful.If you have received this 
communication in error,please
immediately notify the sender by return e-mail, and delete the original message 
and all copies from
your system. Thank you.
---

Re: 答复: About lookup table fileds in filter

[Li Yang]

I read the description but don't understand the question

On Thu, Aug 18, 2016 at 9:04 AM, 胡志华(万里通科技及数据中心商务智能团队数据分析组) <
huzhihua...@pingan.com.cn> wrote:

> Hello,
>Anybody?
>
> 发件人: 胡志华
> 发送时间: 2016年8月17日 16:40
> 收件人: dev@kylin.apache.org
> 主题: About lookup table fileds in filter
>
> Hi all,
>
>I am encountering a problem, when I create model, I put lookup
> table fields as filter fields .But job monitor page told me “invalid table
> alias or column reference”，
>
> It means I can’t use lookup table fields as filter fields, I must use fact
> table fields as filter?
>
>If I want to filter lookup table, what can I do?
>
>
>
>
> 
> 
> The information in this email is confidential and may be legally
> privileged. If you have received this email in error or are not the
> intended recipient, please immediately notify the sender and delete this
> message from your computer. Any use, distribution, or copying of this email
> other than by the intended recipient is strictly prohibited. All messages
> sent to and from us may be monitored to ensure compliance with internal
> policies and to protect our business.
> Emails are not secure and cannot be guaranteed to be error free as they
> can be intercepted, amended, lost or destroyed, or contain viruses. Anyone
> who communicates with us by email is taken to accept these risks.
>
> 收发邮件者请注意：
> 本邮件含保密信息，若误收本邮件，请务必通知发送人并直接删去，不得使用、传播或复制本邮件。
> 进出邮件均受到本公司合规监控。邮件可能发生被截留、被修改、丢失、被破坏或包含计算机病毒等不安全情况。
> 
> 
>

Re: Kylin Bulid Cube OOM

[Li Yang]

Need more info, like in which process OOM occurs? In Kylin process or in a
mapper/reducer?

On Tue, Aug 16, 2016 at 2:52 PM, Roy  wrote:

> Hi Guys,
>
> I Create one cube, Data Size  507536771 rows , 10 dimension and 3
> measure.  when cube bulid in middled steps throw exception. check logs
> found OOM.
>
> anyone konw how to increase reduce task in kylin.it's modify server
> config parameter?
>
> Best Regards
>
> Roy
>

Re: Fail to query the very long SQL with Kylin1.5.3

[Li Yang]

Search more in the log, the posted stack trace is not the root cause.

On Tue, Aug 16, 2016 at 8:57 AM, echo  wrote:

> from the source code to find the problem that need more logs.
>
> may be the error is from QueryService.query
>
>
> 2016-08-16
>
> langren_qing
>
>
>
> 发件人：沙漠火狐 [via Apache Kylin] 
> 发送时间：2016-08-15 10:33
> 主题：Fail to query the very long SQL with Kylin1.5.3
> 收件人："echo"
> 抄送：
>
> Hi
> I got a error when I execute the very long sql, but I cut short the
> sql, it can query succsess.   The kylin version is 1.5.3 for CDH5.7.
>
>
> The SQL is :
>   select DIM_SITE.SITE_ID as c0, dim_page_entrance.PAGE_ID as
> c1, DIM_SHOP.SHOP_ID as c2, DIM_DATE.DATE_ID as c3,
> DIM_GOODS_TYPE.GOODS_TYPE_ID as c4, sum(sale.DETL_PAGE_PV) as m0,
> count(distinct sale.GU_ID) as m1 from FCT_ORDR_PATH_OLAP as sale join
> DIM_SITE as DIM_SITE on sale.SITE_ID = DIM_SITE.SITE_ID join DIM_PAGE as
> dim_page_entrance on sale.LAST_ENTRANCE_PAGE_ID = dim_page_entrance.PAGE_ID
> join DIM_SHOP as DIM_SHOP on sale.SHOP_ID = DIM_SHOP.SHOP_ID join DIM_DATE
> as DIM_DATE on sale.DATE_ID = DIM_DATE.DATE_ID join DIM_GOODS_TYPE as
> DIM_GOODS_TYPE on sale.GOODS_TYPE_ID = DIM_GOODS_TYPE.GOODS_TYPE_ID where
> DIM_SITE.SITE_ID in (1, 2) and dim_page_entrance.PAGE_ID in (-1, 1, 17, 33,
> 35, 36, 37, 38, 39, 40, 41, 42, 43, 45, 46, 47, 48, 49, 54, 63, 66, 133,
> 135, 177, 178, 179, 189, 219, 221, 222, 223, 224, 225, 226, 228, 229, 230,
> 233, 234, 235, 236, 254, 268, 279, 280, 316, 317, 318, 319, 320, 321, 322,
> 323, 324, 10002, 10006, 10012, 10013, 10014, 10015, 10016, 10017, 10018,
> 10020, 10040, 10041, 10058, 10059, 10060, 10062, 10063, 10065, 10066,
> 10067, 10068) and DIM_SHOP.SHOP_ID in (83, 104, 207, 360, 417, 450, 502,
> 618, 634, 655, 779, 829, 872, 896, 984, 1012, 1056, 1063, 1079, 1098, 1179,
> 1221, 1233, 1237, 1238, 1264, 1332, 1366, 1384, 1530, 1532, 1544, 1566,
> 1593, 1597, 1600, 1601, 1605, 1614, 1647, 1709, 1732, 1735, 1765, 1776,
> 1796, 1840, 1867, 1875, 1900, 1936, 1946, 1948, 1952, 1956, 1957, 1958,
> 1959, 1961, 1964, 1978, 1979, 1981, 1984, 1989, 1991, 2092, 2165, 2221,
> 2256, 2341, 2352, 2394, 2424, 2540, 2563, 2597, 2619, 2642, 2654, 2670,
> 2684, 2705, 2715, 2781, 2786, 2828, 2880, 2909, 2958, 2970, 3007, 3054,
> 3062, 3089, 3141, 3151, 3208, 3230, 3295, 3305, 3386, 3388, 3416, 3551,
> 3721, 3750, 3775, 3819, 3855, 3905, 3931, 3998, 4010, 4013, 4014, 4221,
> 4235, 4255, 4260, 4275, 4287, 4291, 4310, 4311, 4341, 4344, 4352, 4358,
> 4369, 4381, 4382, 4385, 4387, 4389, 4393, 4413, 4415, 4417, 4418, 4419,
> 4421, 4425, 4426, 4427, 4431, 4434, 4436, 4454, 4459, 4480, 4523, 4599,
> 4713, 4717, 4913, 4949, 4962, 4998, 5067, 5078, 5097, 5176, 5199, 5287,
> 5342, 5423, 5450, 5484, 5535, 5549, 5551, 5607, 5617, 5630, 5676, 5685,
> 5759, 5863, 5912, 5958, 6009, 6014, 6020, 6024, 6045, 6164, 6265, 6297,
> 6360, 6364, 6367, 6447, 6451, 6466, 6486, 6529, 6565, 6594, 6597, 6648,
> 6726, 6732, 6758, 6833, 6919, 7001, 7039, 7043, 7069, 7118, 7128, 7134,
> 7137, 7188, 7189, 7216, 7221, 7273, 7284, 7310, 7376, 7436, 7451, 7553,
> 7611, 7641, 7650, 7670, 7737, 7747, 7767, 7771, 7878, 7978, 7999, 8019,
> 8028, 8066, 8119, 8129, 8151, 8195, 8229, 8234, 8324, 8396, 8519, 8538,
> 8553, 8668, 8741, 8750, 8764, 8780, 8782, 8891, 8978, 9039, 9067, 9155,
> 9234, 9246, 9258, 9301, 9326, 9357, 9379, 9429, 9469, 9483, 9488, 9513,
> 9539, 9703, 9709, 9755, 9758, 9770, 9837, 9881, 9987, 10159, 10165, 10193,
> 10240, 10348, 10373, 10401, 10496, 10516, 10552, 10570, 10605, 10656,
> 10707, 10719, 10724, 10733, 10787, 10803, 10869, 10920, 11052, 11079,
> 11134, 11142, 11209, 11327, 11332, 11450, 11488, 11578, 11586, 11615,
> 11726, 11772, 11785, 11830, 11909, 11911, 11939, 11957, 11970, 11997,
> 12043, 12058, 12097, 12283, 12345, 12367, 12500, 12569, 12609) and
> DIM_DATE.DATE_ID = '2016-07-02' and DIM_GOODS_TYPE.GOODS_TYPE_ID = 2 group
> by DIM_SITE.SITE_ID, dim_page_entrance.PAGE_ID, DIM_SHOP.SHOP_ID,
> DIM_DATE.DATE_ID, DIM_GOODS_TYPE.GOODS_TYPE_ID
>
>
>  I cut short half  the DIM_SHOP.SHOP_ID condition, it query sucessful.
>
>
>  the Error I got is :
>==[QUERY]===
>
>
> 2016-08-15 09:47:37,382 ERROR [http-bio-7070-exec-2]
> controller.BasicController:44 :
> org.apache.kylin.rest.exception.InternalErrorException
> at org.apache.kylin.rest.controller.QueryController.doQueryWithCache(
> QueryController.java:224)
> at org.apache.kylin.rest.controller.QueryController.
> query(QueryController.java:94)
> at sun.reflect.GeneratedMethodAccessor171.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.springframework.web.method.support.InvocableHandlerMethod.invoke(
> InvocableHandlerMethod.java:213)
> at org.springframework.web.method.support.InvocableHandlerMethod.
> 

Re: 2015年7月16日 15:34:38 自动保存草稿

[Li Yang]

> incubator-kylin-0.7.2-release

The problem seems related with
https://issues.apache.org/jira/browse/KYLIN-1319

Consider upgrade to 1.5.3

On Mon, Aug 15, 2016 at 10:51 AM, 周千昊  wrote:

> -- Forwarded message -
> From: 
> Date: 2016年8月13日周六 上午5:27
> Subject: Re: 2015年7月16日 15:34:38 自动保存草稿
> To: 
>
>
> Hi,
>
> Could you please let me know how you solved the problem as i'm facing the
> same issue :(
>
> 
> that's interesting, kylin will not hard code some thing like that. So it
> should be some where in the classpath.
> in the bin/ directory, we have some script to locate the hive and hbase
> classpath, can you please dig into these path to check if the value 18032
> is configured anywhere in the classpath or some default settings for CDH
>
> streemwu 于2015年7月16日周四 下午6:50写道：
>
> > hi ，
> > from the log I found when i run the job of building the cube,kylin
> > get yarn.resourcemanager.address on localhost/127.0.0.1:18032.but in
> > my yarn-site.xml,the yarn.resourcemanager.address is
> > 
> >yarn.resourcemanager.address
> >ewayCEServer:8032
> >
> >
> > ---i am sure yarn had started successful. It seems like that  kylin
> > didn't get the yarn.resourcemanager.address from $hadoop_home/etc/hadoop/
> > yarn-site.xml.  but from the kylin.log I found Hadoop job classpath
> > contains
> >  hadoop conf ,hbase conf,hive conf
> >
> > [pool-7-thread-2]:[2015-07-16
> >
> 18:34:48,609][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob.
> setJobClasspath(AbstractHadoopJob.java:180)]
> > - Hadoop job classpath is: /home/eway/hadoop-2.6.0/etc/hadoop
> >
> :/home/eway/hadoop-2.6.0/share/hadoop/common/lib/*:/
> home/eway/hadoop-2.6.0/share/hadoop/common/*:/home/eway/
> hadoop-2.6.0/share/hadoop/hdfs:/home/eway/hadoop-2.6.0/
> share/hadoop/hdfs/lib/*:/home/eway/hadoop-2.6.0/share/
> hadoop/hdfs/*:/home/eway/hadoop-2.6.0/share/hadoop/
> yarn/lib/*:/home/eway/hadoop-2.6.0/share/hadoop/yarn/*:/
> home/eway/hadoop-2.6.0/share/hadoop/mapreduce/lib/*:(...)
> > ,/home/eway/hbase-0.98.13-hadoop2/lib/hbase-common-0.98.13-hadoop2.jar,
> > /home/eway/hive-0.13.1-cdh5.3.0/conf
> >
> ,/home/eway/hive-0.13.1-cdh5.3.0/lib/*,/home/eway/hive-0.
> 13.1-cdh5.3.0/hcatalog/share/hcatalog,/home/eway/hive-0.13.
> 1-cdh5.3.0/bin,/home/eway/hbase-0.98.13-hadoop2/conf,/
> home/eway/hive-0.13.1-cdh5.3.0/conf,/home/eway/hive-0.13.1-
> cdh5.3.0/lib/*,/home/eway/hive-0.13.1-cdh5.3.0/hcatalog/
> share/hcatalog,/home/eway/hive-0.13.1-cdh5.3.0/bin,
> > /home/eway/hbase-0.98.13-hadoop2/conf
> >
> > -BTW,i have searched the resource code from
> > incubator-kylin-0.7.2-release,but there is no  '18032'.
> > pls help me.
> > Regards.
> >
> >
> >
> >
> >
> > --
> Best Regard
> ZhouQianhao
>
> 
> Quoted from:
> http://apache-kylin.74782.x6.nabble.com/2015-7-16-15-34-38-tp542p543.html
>
>
> _
> Sent from http://apache-kylin.74782.x6.nabble.com
>
> --
> Best Regard
> ZhouQianhao
>

Re: Kylin Query Performance

[Li Yang]

Performance troubleshoot is complicated and requires much information. If
you can share a diagnosis pack, people maybe able to help.

On Sat, Aug 13, 2016 at 10:32 PM, Yiming Liu 
wrote:

> What's kind of queries? How many records supposed to be returned? Could you
> send out the query log together?
>
> 2016-08-12 10:28 GMT+08:00 Mars J :
>
> > Hi,
> > I have run Kylin 1.5.2.1 and build a cube successfully,the cube size
> is
> > 3.2G for fact table and dimensional table have 1.5 million records
> > seperately.HTable count is about 200 billion.
> >My Cube Design includes 3 Derived Dims and 5 Normal Dims which formed
> a
> > hierachy dim in agg. Rowkeys are generated automatically,the first and
> > second rowkey is the highest cardinality column in fact table and one dim
> > table seperately.
> >
> > When I query it from kylin insight, it costs 11s,and the second same
> > query is also 10+s, How can I optimize this ?
> >
>
>
>
> --
> With Warm regards
>
> Yiming Liu (刘一鸣)
>

Re: Derived measures in Kylin

[Alberto Ramón]

Hi ¡¡,  I'm not expert  in Kylin  :)

Perhaps, this can be useful for you:
https://issues.apache.org/jira/browse/KYLIN-976

The problem: this jira don't have example / doc or code changeSet  ... but:

   -
   
https://github.com/apache/kylin/blob/696fb89aa2eb684408d1009218f80fe323dffd3b/core-metadata/src/main/java/org/apache/kylin/measure/MeasureTypeFactory.java#L110


   -
   
https://github.com/apache/kylin/blob/6c298fac2acd7d311e82002e0854e896037883c8/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java#L455


and perhaps: the property in the "kylin.cube.measure.customMeasureType" you
can specify a list of strings of 'customFactory' of MeasureTypeFactory class



BR, Alberto



2016-08-18 10:13 GMT+02:00 Bhanu Prakash Sunkesula 
:

> Hello,
>
> we also had the same issue, where my business spokes has expected a Derived
> measure with normal measures, but as of now there is no direct solution for
> this in KYLIN (apart from SQL Query in Insight tab). Indirectly we had a
> workaround by creating a column TotalSalary  = SUM(HRA) + SUM(DA)
> in hive and need to populate it in KYLIN Cube.
>
> Regards,
> Bhanu Prakash.
>
> --
> View this message in context: http://apache-kylin.74782.x6.
> nabble.com/Derived-measures-in-Kylin-tp5513p5592.html
> Sent from the Apache Kylin mailing list archive at Nabble.com.
>

Re: Derived measures in Kylin

[Bhanu Prakash Sunkesula]

Hello,

we also had the same issue, where my business spokes has expected a Derived
measure with normal measures, but as of now there is no direct solution for
this in KYLIN (apart from SQL Query in Insight tab). Indirectly we had a
workaround by creating a column TotalSalary  = SUM(HRA) + SUM(DA) 
in hive and need to populate it in KYLIN Cube.

Regards,
Bhanu Prakash.

--
View this message in context: 
http://apache-kylin.74782.x6.nabble.com/Derived-measures-in-Kylin-tp5513p5592.html
Sent from the Apache Kylin mailing list archive at Nabble.com.