[jira] [Created] (KYLIN-4112) Add hdfs keberos token delegation in Spark to support HBase and MR use different HDFS clusters

2019-07-24 Thread Liu Shaohui (JIRA) 
Liu Shaohui created KYLIN-4112:
--

 Summary: Add hdfs keberos token delegation in Spark to support 
HBase and MR use different HDFS clusters
 Key: KYLIN-4112
 URL: https://issues.apache.org/jira/browse/KYLIN-4112
 Project: Kylin
  Issue Type: Improvement
Reporter: Liu Shaohui
Assignee: Liu Shaohui


Currently the SparkExecutable only delegate the token for yarn hdfs cluster, 
not for the hdfs cluster used by the HBase cluster.

The spark job of Convert Cuboid Data to HFile will failed for kerberos issue.

 



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (KYLIN-4111) drop table failed with no valid privileges after KYLIN-3857

2019-07-24 Thread Liu Shaohui (JIRA) 
Liu Shaohui created KYLIN-4111:
--

 Summary: drop table failed with no valid privileges after 
KYLIN-3857
 Key: KYLIN-4111
 URL: https://issues.apache.org/jira/browse/KYLIN-4111
 Project: Kylin
  Issue Type: Bug
Reporter: Liu Shaohui
Assignee: Liu Shaohui


After KYLIN-3857, there will be quote ` around database and table.

The drop table sql will be:
{code:java}
DROP TABLE IF EXISTS 
`kylin_onebox.kylin_intermediate_kylin_sales_cube_7be84be1_a153_07c4_3ce6_270e8d99ff85`;{code}
Hive (1.2)with sentry will throw exception:
{code:java}
Error: Error while compiling statement: FAILED: HiveAccessControlException No 
valid privileges
 Required privileges for this query: 
Server=server1->Db=`kylin_onebox->Table=kylin_intermediate_kylin_sales_cube_7be84be1_a153_07c4_3ce6_270e8d99ff85`->action=drop;
Query log: 
http://zjy-hadoop-prc-ct14.bj:18201/log?qid=898c7878-a961-443d-b120-cca0e2667d15_f486bd16-4bbd-4014-a0a7-c2ebfdbe6668
 (state=42000,code=4)

{code}
The reason is that hive identify the databse be `kylin_onebox and table be: 
kylin_intermediate_kylin_sales_cube_7be84be1_a153_07c4_3ce6_270e8d99ff85`

May be we can fix it in hive and sentry. Just create a jira to show this 
problem.

 



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Using kylin in simplified way

2019-07-24 Thread Asim Ali 
Hi All,
I tried using Kylin on Hadoop environment, but overhead of hadoop is too
much
for our medium scale need.
Is there any way we can use kylin Olap engine with minimal requirements of
underlaying storage layer.
What are the best practices and architecture to support this, where we
possibly can use it without hadoop components.
Thanks

Asim Ali
*Software Developer*

Email: a...@easyemployer.com 

Phone: 1300 855 642 <1300855642>
Website: www.easyemployer.com

[image: easyemployer] 


On Fri, 21 Jun 2019 at 15:22, 敏丞  wrote:

> Hi,
> After check cube.json provided by you, I can reproduce this error in
> my development env. Looks like this kind exception occurs when you have a
> cube which *have both Raw measure and CountDistinct(Bitmap) on the same
> column*. I find the reason should be the Raw Measure choose the wrong
> dictionary (AppendTrieDictionary cannot used to decode).  Maybe you should
> try use two cubes in this situation.
> And if you don't mind, I have a question, have you ever use this type
> of query "select * from FACT_TABLE" in old version of kylin in such kind of
> cube(raw measure and count_distinct both on the same column) and get
> correct result successfully?
>
>
> If you have find anything wrong or other information, please let me
> know. Thank you.
>
>
>
>
> *-*
> *-*
> *Best wishes to you ! *
> *From ：**Xiaoxiang Yu*
>
>
> At 2019-06-20 16:00:10, "greatelvisw...@gmail.com"  
> wrote:
> >hi,all:
> >
> > I got an error like "AppendTrieDictionary can't retrieve value from id" 
> > while query the cube data, the following is the cube info and  exception 
> > info.
> >
> >I found the same error in this 
> >thread(https://lists.apache.org/thread.html/63981bc08ef7b97c41921ed692de79ef9a744f6329192e5199074ba3@%3Cdev.kylin.apache.org%3E),
> > but I just use the  bitmaps (count distinct) as  measure, and never use it 
> >as dimension.
> >
> >So please help me to resolve it.
> >
> >cube data:
> >{
> >  "uuid": "eb0b4a32-fbc0-b197-b3f0-4c9cd5fb3916",
> >  "last_modified": 1561014544528,
> >  "version": "2.6.2.0",
> >  "name": "dev_cube_user_currency",
> >  "is_draft": false,
> >  "model_name": "user_currency",
> >  "description": "",
> >  "null_string": null,
> >  "dimensions": [
> >{
> >  "name": "TYPE",
> >  "table": "DEV_DWD_USER_CURRENCY",
> >  "column": "TYPE",
> >  "derived": null
> >},
> >{
> >  "name": "SUB_TYPE",
> >  "table": "DEV_DWD_USER_CURRENCY",
> >  "column": "SUB_TYPE",
> >  "derived": null
> >},
> >{
> >  "name": "SOURCE_TYPE",
> >  "table": "DEV_DWD_USER_CURRENCY",
> >  "column": "SOURCE_TYPE",
> >  "derived": null
> >},
> >{
> >  "name": "SOURCE",
> >  "table": "DEV_DWD_USER_CURRENCY",
> >  "column": "SOURCE",
> >  "derived": null
> >},
> >{
> >  "name": "STATUS",
> >  "table": "DEV_DWD_USER_CURRENCY",
> >  "column": "STATUS",
> >  "derived": null
> >},
> >{
> >  "name": "CREATE_TIME",
> >  "table": "DEV_DWD_USER_CURRENCY",
> >  "column": "CREATE_TIME",
> >  "derived": null
> >},
> >{
> >  "name": "UPDATE_TIME",
> >  "table": "DEV_DWD_USER_CURRENCY",
> >  "column": "UPDATE_TIME",
> >  "derived": null
> >},
> >{
> >  "name": "END_TIME",
> >  "table": "DEV_DWD_USER_CURRENCY",
> >  "column": "END_TIME",
> >  "derived": null
> >},
> >{
> >  "name": "ACTIVATION_TIME",
> >  "table": "DEV_DWD_USER_CURRENCY",
> >  "column": "ACTIVATION_TIME",
> >  "derived": null
> >},
> >{
> >  "name": "CODE",
> >  "table": "DEV_DIM_CURRENCY_TYPE",
> >  "column": null,
> >  "derived": [
> >"CODE"
> >  ]
> >},
> >{
> >  "name": "SHORT_NAME",
> >  "table": "DEV_DIM_CURRENCY_TYPE",
> >  "column": null,
> >  "derived": [
> >"SHORT_NAME"
> >  ]
> >}
> >  ],
> >  "measures": [
> >{
> >  "name": "_COUNT_",
> >  "function": {
> >"expression": "COUNT",
> >"parameter": {
> >  "type": "constant",
> >  "value": "1"
> >},
> >"returntype": "bigint"
> >  }
> >},
> >{
> >  "name": "SUM_TOTAL",
> >  "function": {
> >"expression": "SUM",
> >"parameter": {
> >  "type": "column",
> >  "value": "DEV_DWD_USER_CURRENCY.TOTAL"
> >},
> >"returntype": "double"
> >  }
> >},
> >{
> >  "name": "SUM_BALANCE",
> >  "function": {
> >"expression": "SUM",
> >"parameter": {
> >  "type": "column",
> >  "value": "DEV_DWD_USER_CURRENCY.BALANCE"
> >},
> >"returntype": "double"
> >  }
> >},
> >{
> >  "name": "SUM_VALID_PERIOD",
> >  "function": {
> >"expression": "SUM",
> >"parameter": {
> >  "type": "column",
> >  "value":

[jira] [Created] (KYLIN-4110) Provider a config option to specify yarn queue

2019-07-24 Thread vinoyang (JIRA) 
vinoyang created KYLIN-4110:
---

 Summary: Provider a config option to specify yarn queue
 Key: KYLIN-4110
 URL: https://issues.apache.org/jira/browse/KYLIN-4110
 Project: Kylin
  Issue Type: Sub-task
  Components: Flink Engine
Reporter: vinoyang
Assignee: vinoyang


In Tencent, we have a requirement to specify a yarn queue for the flink cubing 
relevant applications.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Fwd: Aug 7 Meetup: Accelerate Your Big Data Analytics with Open Source Software on Wednesday, August 7

2019-07-24 Thread ShaoFeng Shi 
BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Meetup//Meetup Events v1.0//EN
CALSCALE:GREGORIAN
METHOD:PUBLISH
X-WR-CALNAME:Events - Apache Kylin Bay Area Group
X-MS-OLK-FORCEINSPECTOROPEN:TRUE
BEGIN:VTIMEZONE
TZID:America/Los_Angeles
TZURL:http://tzurl.org/zoneinfo-outlook/America/Los_Angeles
X-LIC-LOCATION:America/Los_Angeles
BEGIN:DAYLIGHT
TZOFFSETFROM:-0800
TZOFFSETTO:-0700
TZNAME:PDT
DTSTART:19700308T02
RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=2SU
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0700
TZOFFSETTO:-0800
TZNAME:PST
DTSTART:19701101T02
RRULE:FREQ=YEARLY;BYMONTH=11;BYDAY=1SU
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTAMP:20190724T224347Z
DTSTART;TZID=America/Los_Angeles:20190807T183000
DTEND;TZID=America/Los_Angeles:20190807T203000
STATUS:CONFIRMED
SUMMARY:Aug 7 Meetup: Accelerate Your Big Data Analytics with Open Source
  Software
DESCRIPTION:Apache Kylin Bay Area Group\nWednesday\, August 7 at 6:30 PM\
 n\nEnjoy a summer evening with pizza\, beer\, and some insightful tech t
 alks\, hosted and sponsored by Kyligence. AGENDA6:30 – 7:00 PM: Socializ
 e over food ...\n\nhttps://www.meetup.com/Apache-Kylin/events/263433976/
 
ORGANIZER;CN=Meetup Reminder:MAILTO:i...@meetup.com
CLASS:PUBLIC
CREATED:20190724T220409Z
GEO:37.33;-121.88
LOCATION:99 Almaden Blvd (99 Almaden Blvd\, San Jose\, CA 95110)
URL:https://www.meetup.com/Apache-Kylin/events/263433976/
SEQUENCE:4
LAST-MODIFIED:20190724T224345Z
UID:event_263433...@meetup.com
END:VEVENT
END:VCALENDAR


meetup.ics
Description: application/ics

[jira] [Created] (KYLIN-4109) CubeHFileMapperTest failed after commit: f4d2405f6aa978bbc3153c9ca9fa339b9d7e6c30

2019-07-24 Thread Liu Shaohui (JIRA) 
Liu Shaohui created KYLIN-4109:
--

 Summary: CubeHFileMapperTest failed after commit: 
f4d2405f6aa978bbc3153c9ca9fa339b9d7e6c30
 Key: KYLIN-4109
 URL: https://issues.apache.org/jira/browse/KYLIN-4109
 Project: Kylin
  Issue Type: Bug
Reporter: Liu Shaohui


CubeHFileMapperTest failed in 2.5.x-hadoop3.1 for following changes
{code:java}
-    assertEquals("cf1", new String(p1.getSecond().getFamily(), 
StandardCharsets.UTF_8));
-    assertEquals("usd_amt", new String(p1.getSecond().getQualifier(), 
StandardCharsets.UTF_8));
-    assertEquals("35.43", new String(p1.getSecond().getValue(), 
StandardCharsets.UTF_8));
+    assertEquals("cf1", new String(copy(p1.getSecond(;
+    assertEquals("usd_amt", new String(copy(p1.getSecond(;
+    assertEquals("35.43", new String(copy(p1.getSecond(;
 
 assertEquals(key, p2.getFirst());
-    assertEquals("cf1", new String(p2.getSecond().getFamily(), 
StandardCharsets.UTF_8));
-    assertEquals("item_count", new String(p2.getSecond().getQualifier(), 
StandardCharsets.UTF_8));
-    assertEquals("2", new String(p2.getSecond().getValue(), 
StandardCharsets.UTF_8));
+    assertEquals("cf1", new String(copy(p2.getSecond(;
+    assertEquals("item_count", new String(copy(p2.getSecond(;
+    assertEquals("2", new String(copy(p2.getSecond(;
{code}
 



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (KYLIN-4108) Show slow query hit cube in slow query page

2019-07-24 Thread Chao Long (JIRA) 
Chao Long created KYLIN-4108:


 Summary: Show slow query hit cube in slow query page
 Key: KYLIN-4108
 URL: https://issues.apache.org/jira/browse/KYLIN-4108
 Project: Kylin
  Issue Type: Improvement
  Components: Metrics
Reporter: Chao Long
Assignee: Chao Long
 Fix For: v3.0.0






--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

case when条件包含like异常

2019-07-24 Thread fanglie...@zhongan.com 
hi，各位大佬，社区版的kylin case when里面包含like条件好像有点问题。
比如这样(CASE WHEN kylin_sales.lstg_format_name like '%Auction%' THEN 'Auction'  
ELSE '其他' END)
select  distinct  (CASE WHEN kylin_sales.lstg_format_name like '%Auction%' THEN 
'Auction'  ELSE '其他' END) as zidingyiziduan,count(1)  from 
vehicle_sentry.kylin_sales as kylin_sales INNER JOIN 
vehicle_sentry.kylin_cal_dt as kylin_cal_dt on kylin_sales.part_dt = 
kylin_cal_dt.cal_dt group by zidingyiziduan  limit 1000;  这个查出来的结果是

换成(CASE WHEN kylin_sales.lstg_format_name = 'Auction' THEN 'Auction'  ELSE '其他' 
END)后 ，查出来的结果是



fanglie...@zhongan.com