Re: Issue while query or building cube

[Kaige Liu]

Kylin does support running over VM. The problem you met might be caused by 
wrong distribution of Kylin.
Please ensure you chose the CDH version of Kylin. 
http://kylin.apache.org/download/

If you still met such issue when using correct distribution, then it should be 
a compatibility issue. Thanks.

Best regards,
 
Kaige Liu(刘凯歌)
Mail: kaige@kyligence.io
Shanghai Kyligence Information Technology Co., Ltd
上海市浦东新区亮秀路112号Y1座405室
 
"Do small things with great love." 
 


On 9/5/17, 11:28 AM, "gulatimahesh"  wrote:

So, Kylin installation is not supported over VM ? Please confirm.

--
Sent from: http://apache-kylin.74782.x6.nabble.com/

[jira] [Created] (KYLIN-2845) Merging cube get error IllegalStateException

[JiangQin (JIRA)]

JiangQin created KYLIN-2845:
---

 Summary: Merging cube get error IllegalStateException
 Key: KYLIN-2845
 URL: https://issues.apache.org/jira/browse/KYLIN-2845
 Project: Kylin
  Issue Type: Bug
  Components: Job Engine
Affects Versions: v2.0.0
Reporter: JiangQin
Assignee: Dong Li


When trying to merge cubes, I encountered following errors :
Error: java.lang.IllegalStateException at 
com.google.common.base.Preconditions.checkState(Preconditions.java:129) at 
org.apache.kylin.cube.kv.RowKeyEncoder.encode(RowKeyEncoder.java:123) at 
org.apache.kylin.engine.mr.steps.MergeCuboidMapper.doMap(MergeCuboidMapper.java:246)

I digged into the source code and found the following code: 
{code:java}
 public void encode(ByteArray bodyBytes, ByteArray outputBuf) {
Preconditions.checkState(bodyBytes.length() == bodyLength);
Preconditions.checkState(bodyBytes.length() + getHeaderLength() == 
outputBuf.length(), //
"bodybytes length: " + bodyBytes.length() + " outputBuf length: 
" + outputBuf.length() + " header length: " + getHeaderLength());
System.arraycopy(bodyBytes.array(), bodyBytes.offset(), 
outputBuf.array(), getHeaderLength(), bodyLength);

//fill shard and cuboid
fillHeader(outputBuf.array());
}
{code}

And also I found an old similar bug 
[https://issues.apache.org/jira/browse/KYLIN-1004].
The cube can build correctly, but the merge just does not work.

Here is the detail for this bug
2017-09-05 09:35:57,801 INFO [main] org.apache.kylin.cube.CubeManager: Reloaded 
cube cube_fact_dui_assemble_log being CUBE[name=cube_fact_dui_assemble_log] 
having 7 segments
2017-09-05 09:35:57,801 INFO [main] org.apache.kylin.cube.CubeManager: Loaded 1 
cubes, fail on 0 cubes
2017-09-05 09:35:57,805 INFO [main] org.apache.kylin.dict.DictionaryManager: 
DictionaryManager(1908999587) loading DictionaryInfo(loadDictObj:true) at 
/dict/DIM_DUI.DIM_CALENDAR/P_DAY/7dbd2f63-ffc6-4eb6-972a-b712954bcff0.dict
2017-09-05 09:35:57,821 INFO [main] org.apache.kylin.dict.DictionaryManager: 
DictionaryManager(1908999587) loading DictionaryInfo(loadDictObj:true) at 
/dict/DIM_DUI.DIM_DEVELOPER/DEVELOPER_ID/31e3578a-94fe-4d99-b00f-c523f59b8eb2.dict
2017-09-05 09:35:57,833 INFO [main] org.apache.kylin.dict.DictionaryManager: 
DictionaryManager(1908999587) loading DictionaryInfo(loadDictObj:true) at 
/dict/DIM_DUI.DIM_DEVELOPER/DEVELOPER_ID/9e365c29-52d8-4397-acb9-63cb9214a05e.dict
2017-09-05 09:35:57,834 INFO [main] org.apache.kylin.engine.mr.KylinMapper: 
Accepting Mapper Key with ordinal: 1
2017-09-05 09:35:57,837 ERROR [main] org.apache.kylin.engine.mr.KylinMapper: 
java.lang.IllegalStateException
at 
com.google.common.base.Preconditions.checkState(Preconditions.java:129)
at org.apache.kylin.cube.kv.RowKeyEncoder.encode(RowKeyEncoder.java:123)
at 
org.apache.kylin.engine.mr.steps.MergeCuboidMapper.doMap(MergeCuboidMapper.java:246)
at 
org.apache.kylin.engine.mr.steps.MergeCuboidMapper.doMap(MergeCuboidMapper.java:63)
at org.apache.kylin.engine.mr.KylinMapper.map(KylinMapper.java:48)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)
2017-09-05 09:35:57,866 INFO [main] org.apache.hadoop.mapred.MapTask: Starting 
flush of map output
2017-09-05 09:35:57,958 INFO [main] org.apache.hadoop.io.compress.CodecPool: 
Got brand-new compressor [.deflate]
2017-09-05 09:35:57,964 WARN [main] org.apache.hadoop.mapred.YarnChild: 
Exception running child : java.lang.IllegalStateException
at 
com.google.common.base.Preconditions.checkState(Preconditions.java:129)
at org.apache.kylin.cube.kv.RowKeyEncoder.encode(RowKeyEncoder.java:123)
at 
org.apache.kylin.engine.mr.steps.MergeCuboidMapper.doMap(MergeCuboidMapper.java:246)
at 
org.apache.kylin.engine.mr.steps.MergeCuboidMapper.doMap(MergeCuboidMapper.java:63)
at org.apache.kylin.engine.mr.KylinMapper.map(KylinMapper.java:48)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
 

Re: Issue while query or building cube

[gulatimahesh]

So, Kylin installation is not supported over VM ? Please confirm.

--
Sent from: http://apache-kylin.74782.x6.nabble.com/

#5 Step Name: Save Cuboid Statistics

[kylin_liu]

你好，我在使用 Kylin2.1 #5 Step Name: Save Cuboid Statistics 过程中
无法通过，但是也不是都失败，请问这个是什么问题呢，是我 hdfs 的问题吗？

日志主要如下：
Caused by: java.lang.IllegalArgumentException: Wrong FS: hdfs://  ,
expected: viewfs:/

我用 kylin 2.0 的时候就没这个问题了，这很奇怪了，感觉这个问题很奇怪。

求教！！Why?

--
Sent from: http://apache-kylin.74782.x6.nabble.com/

Re: Re: Can't edit cube

[apache_...@163.com]

 
   Thinks.

  version:   apache-kylin-2.1.0-bin-hbase1x.tar.gz 
  deployed:   single  .

   



apache_...@163.com
 
From: ShaoFeng Shi
Date: 2017-09-03 22:31
To: dev
Subject: Re: Can't edit cube
You'd better provide Kylin version, and how it be deployed (single or
clustered).
 
2017-09-03 19:26 GMT+08:00 Li Yang :
 
> As a workaround, you can reload metadata (under System tab) and edit again.
>
> On Thu, Aug 31, 2017 at 3:22 PM, Cheng Wang 
> wrote:
>
> > Sometimes it happens because one Kylin instance is used by multi-users.
> > Please make sure there are no others using the Kylin at the same time.
> >
> >
> >
> >
> > On 8/31/17, 3:19 PM, "apache_...@163.com"  wrote:
> >
> > >Hi,
> > >
> > >When edit the exist cube,can't save.Please help see what's the problem?
> > >
> > >Error message:
> > >
> > >
> > >2017-08-31 15:20:08,022 ERROR [http-bio-7070-exec-28]
> > controller.CubeController:516 : Failed to deal with the
> request:Overwriting
> > conflict /cube_desc/cube_0830.json, expect old TS 1504057946524, but it
> is
> > 1504163743354
> > >java.lang.IllegalStateException: Overwriting conflict
> > /cube_desc/cube_0830.json, expect old TS 1504057946524, but it is
> > 1504163743354
> > >at org.apache.kylin.storage.hbase.HBaseResourceStore.
> > checkAndPutResourceImpl(HBaseResourceStore.java:315)
> > >at org.apache.kylin.common.persistence.ResourceStore.
> > checkAndPutResourceCheckpoint(ResourceStore.java:294)
> > >at org.apache.kylin.common.persistence.ResourceStore.
> > putResource(ResourceStore.java:280)
> > >at org.apache.kylin.common.persistence.ResourceStore.
> > putResource(ResourceStore.java:260)
> > >at org.apache.kylin.cube.CubeDescManager.updateCubeDesc(
> > CubeDescManager.java:380)
> > >at org.apache.kylin.rest.service.CubeService.updateCubeAndDesc(
> > CubeService.java:255)
> > >at org.apache.kylin.rest.service.CubeService$$
> > FastClassBySpringCGLIB$$17a07c0e.invoke()
> > >at org.springframework.cglib.proxy.MethodProxy.invoke(
> > MethodProxy.java:204)
> > >at org.springframework.aop.framework.CglibAopProxy$
> > CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720)
> > >at org.springframework.aop.framework.
> ReflectiveMethodInvocation.
> > proceed(ReflectiveMethodInvocation.java:157)
> > >at org.springframework.security.access.intercept.aopalliance.
> > MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:68)
> > >at org.springframework.aop.framework.
> ReflectiveMethodInvocation.
> > proceed(ReflectiveMethodInvocation.java:179)
> > >at org.springframework.aop.framework.CglibAopProxy$
> > DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
> > >at org.apache.kylin.rest.service.CubeService$$
> > EnhancerBySpringCGLIB$$a843c409.updateCubeAndDesc()
> > >at org.apache.kylin.rest.controller.CubeController.
> > updateCubeDesc(CubeController.java:511)
> > >at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > >at sun.reflect.NativeMethodAccessorImpl.invoke(
> > NativeMethodAccessorImpl.java:62)
> > >at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> > DelegatingMethodAccessorImpl.java:43)
> > >at java.lang.reflect.Method.invoke(Method.java:498)
> > >
> > >
> > >
> > >apache_...@163.com
> >
>
 
 
 
-- 
Best regards,
 
Shaofeng Shi 史少锋

Re: Is it really deleted?

[Billy Liu]

The short answer is NO.

The StorageCleanupJob tool will clean up these "garbage"

2017-09-04 17:47 GMT+08:00 apache_...@163.com :

> Hi,
>
>Does it delete HBase data on hdfs  through api/kylin  API(DELETE submit
> )?
>
>Does it delete old  HBase data  on hdfs when refresh a cube?
>
>
>
>
> apache_...@163.com
>

[jira] [Created] (KYLIN-2842) 在升级到KYLIN 2.1之后，我使用JDBC，preparedstatement, 无法查询出任何数据。

[Jack (JIRA)]

Jack created KYLIN-2842:
---

 Summary: 在升级到KYLIN 2.1之后，我使用JDBC，preparedstatement, 无法查询出任何数据。
 Key: KYLIN-2842
 URL: https://issues.apache.org/jira/browse/KYLIN-2842
 Project: Kylin
  Issue Type: Bug
  Components: Query Engine
Affects Versions: v2.1.0
Reporter: Jack
Assignee: liyang


你好，我在升级到2.1之后，使用JDBC连接kylin进行查询的时候，使用preparedstatement设定参数查询，无法查询出任何数据，但是在使用普通的statement查询，是可以查询出数据的，请尽快帮忙解决。谢谢。



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: #5 Step Name: Save Cuboid Statistics

[Billy Liu]

Which file system were you using?

Try to define working-dir with full-qualified path.

在 2017年9月4日 下午5:24，kylin_liu 写道：

> 你好，我在使用 Kylin2.1 #5 Step Name: Save Cuboid Statistics 过程中
> 无法通过，但是也不是都失败，请问这个是什么问题呢，是我 hdfs 的问题吗？
>
> 日志主要如下：
> Caused by: java.lang.IllegalArgumentException: Wrong FS: hdfs://  ,
> expected: viewfs:/
>
> 我用 kylin 2.0 的时候就没这个问题了，这很奇怪了，感觉这个问题很奇怪。
>
> 求教！！Why?
>
> --
> Sent from: http://apache-kylin.74782.x6.nabble.com/

Re: when status of segment is 'NEW',the last_build_job_id is null

[周湘伦]

Thank you for your answer.
One more question:
how can i search for the failed job list? By command  or API?


2017-08-31 8:55 GMT+08:00 ShaoFeng Shi :

> Hi,
>
> If resume a job, you need get the job uuid; The uuid was returned when you
> submit the job request.
>
> Another way is to do a search for the failed job list, and get the job uuid
> from the result.
>
> The segment's "last_build_job_id" will be updated when there is a
> successful build. In your case as there is no success build, so it is null.
>
> If you're using Chrome browser, press F12 or right click "Inspect", in the
> "Network" tab, you will see how Kylin's frontend call Kylin REST server,
> that is the easist way to learn the REST API.
>
>
> 2017-08-30 18:36 GMT+08:00 周湘伦 :
>
> > Hi,all
> >
> > ​When the job execution failed,or pending,then i get cube information:
> > ​...
> > {"uuid":"523623e7-6233-4a91-af15-b8f846ae6063","name":"
> > 2017061000_2017061100","storage_location_identifier"
> > :"KYLIN_P1MD9NPHP3","date_range_start":149705280,"
> > date_range_end":149713920,"source_offset_start":0,"
> > source_offset_end":0,"status":"NEW","size_kb":0,"input_
> > records":0,"input_records_size":0,"last_build_time":0,"
> > last_build_job_id":null,"creat
> > e_time_utc":1504077397616,"cuboid_shard_nums":{},"total_
> > shards":1,"blackout_cuboids":[],"binary_signature":null,"
> > dictionaries":{"TBL
> > ...
> >
> > The 'last_build_job_id' is null.
> > I want resume the job  with restful API,so which field should I get?
> >
> > Best regards.
> >
> >
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>

Is it really deleted?

[apache_...@163.com]

Hi,

   Does it delete HBase data on hdfs  through api/kylin  API(DELETE submit )?

   Does it delete old  HBase data  on hdfs when refresh a cube?
   



apache_...@163.com

[jira] [Created] (KYLIN-2841) LIMIT is buggy with subquery

[Mu Kong (JIRA)]

Mu Kong created KYLIN-2841:
--

 Summary: LIMIT is buggy with subquery
 Key: KYLIN-2841
 URL: https://issues.apache.org/jira/browse/KYLIN-2841
 Project: Kylin
  Issue Type: Bug
Affects Versions: v2.1.0
Reporter: Mu Kong


Hi, all.

I found that limit in the web UI seems not behaving as expected.

When I run a query like the follows:

{code:sql}
SELECT
  SUM(col3) AS col4, 
  SUM(col5) AS total_col5,
  col1 
FROM
(
  SELECT
col1,
col2,
MAX(col3) AS col3,
COUNT(*) AS col5
  FROM db.table
  WHERE col6 = 'somestring'
  GROUP BY col1, col2
)
GROUP BY col1
{code}

When I specify the limit as 50, the result has 19 records, and when I specify 
the limit as 50, there are 90+ records in the result and each record has 
higher col4 and total_col5.
But for query that doesn't have subquery, the result remains the same no matter 
how I change the limit.

I guess for the query with subquery, limit somehow limits the number of the 
result from the inner query instead of the result of the outer query.





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 61022: KYLIN-2703 kylin supports managing access rights for project and cube through apache ranger.

[pengjianhua]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61022/
---

(Updated 九月 4, 2017, 7:52 a.m.)


Review request for kylin, Dayue Gao, Dong Li, hongbin ma, and Wang Xiaoyu.


Bugs: KYLIN-2703
https://issues.apache.org/jira/browse/KYLIN-2703


Repository: kylin


Description
---

Ranger is a framework to enable, monitor and manage comprehensive data security 
across the Hadoop platform. Apache Ranger has the following goals:
1. Centralized security administration to manage all security related tasks in 
a central UI or using REST APIs.
2. Fine grained authorization to do a specific action and/or operation with 
Hadoop component/tool and managed through a central administration tool
3. Standardize authorization method across all Hadoop components.
4. Enhanced support for different authorization methods - Role based access 
control, attribute based access control etc.
5. Centralize auditing of user access and administrative actions (security 
related) within all the components of Hadoop.
Ranger has supported enable, monitor and manage following components:
1. HDFS
2. HIVE
3. HBASE
4. KNOX
5. YARN
6. STORM
7. SOLR
8. KAFKA
9. ATLAS
In order to improve the flexibility of kylin privilege control and enhance 
value of kylin in the Apache Hadoop ecosystem, like hdfs, yarn, hive, hbase, 
Kylin should also support that using Ranger to control access rights for 
project and cube.
Specific implementation plan is as following:
On the ranger website, administrators can configure policies to control user 
access to projects and cube permissions.
Kylin provides an abstract class and authorization interfaces for use by the 
ranger plugin. kylin instantiates ranger plugin’s implementation class when 
starting(this class extends the abstract class provided by kylin).
Ranger plugin periodically polls ranger admin, updates the policy to the local, 
and updates project and cube access rights based on policy information.
In the Kylin side?
1. Kylin provides an abstract class that enables the ranger plugin's 
implementation class to extend.
2. Add configuration item. 1) ranger authorization switch, 2) ranger plugin 
implementation class's name.
3. Instantiate the ranger plugin implementation class when starting kylin.
4. kylin provides authorization interfaces for ranger plugin calls.
5. According to the ranger authorization configuration item, hide kylin's 
authorization management page.
6. Using ranger manager access rights of the kylin does not affect kylin's 
existing permissions functions and logic.
In the Ranger side?
1. Ranger plugin will periodically polls ranger admin, updates the policy to 
the local.
2. The ranger plugin invoking the authorization interfaces provided by kylin to 
updates the project and cube access rights based on the policy information.
reference link:https://issues.apache.org/jira/browse/RANGER-1672


Diffs (updated)
-

  build/deploy/context.xml 38c6ec833 
  core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java 
749b515c3 
  
core-common/src/main/java/org/apache/kylin/common/security/KylinAuthorizationProvider.java
 PRE-CREATION 
  
core-metadata/src/main/java/org/apache/kylin/metadata/project/ProjectManager.java
 df5450cf8 
  server-base/src/main/java/org/apache/kylin/rest/constant/Constant.java 
697a660b5 
  
server-base/src/main/java/org/apache/kylin/rest/controller/AccessController.java
 a88c34233 
  
server-base/src/main/java/org/apache/kylin/rest/security/KylinAclPermissionEvaluator.java
 PRE-CREATION 
  
server-base/src/main/java/org/apache/kylin/rest/security/KylinAuthorization.java
 PRE-CREATION 
  server-base/src/main/java/org/apache/kylin/rest/service/AccessService.java 
a46b86679 
  server-base/src/main/java/org/apache/kylin/rest/service/JobService.java 
d2180a76d 
  server/src/main/resources/kylinSecurity.xml ce068d7ac 
  webapp/app/js/services/kylinProperties.js 645ed2f3d 
  webapp/app/partials/cubes/cube_detail.html 0113c126d 
  webapp/app/partials/projects/project_detail.html 5cfc091ab 


Diff: https://reviews.apache.org/r/61022/diff/6/

Changes: https://reviews.apache.org/r/61022/diff/5-6/


Testing
---


Thanks,

pengjianhua

[jira] [Created] (KYLIN-2844) override max-visit-scanrange and max-fuzzykey-scan at cube level

[Dayue Gao (JIRA)]

Dayue Gao created KYLIN-2844:


 Summary: override max-visit-scanrange and max-fuzzykey-scan at 
cube level
 Key: KYLIN-2844
 URL: https://issues.apache.org/jira/browse/KYLIN-2844
 Project: Kylin
  Issue Type: Improvement
  Components: Query Engine
Affects Versions: v2.1.0
Reporter: Dayue Gao
Assignee: Dayue Gao


User should be able to override "kylin.storage.hbase.max-fuzzykey-scan" and 
"kylin.storage.hbase.max-visit-scanrange" at cube/project level.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)