[jira] [Commented] (KYLIN-5283) JobWorker execute方法无法捕获原始异常

[liyang (Jira)]

[ 
https://issues.apache.org/jira/browse/KYLIN-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17626226#comment-17626226
 ] 

liyang commented on KYLIN-5283:
---

值得修复，来个PR？

> JobWorker execute方法无法捕获原始异常
> ---
>
> Key: KYLIN-5283
> URL: https://issues.apache.org/jira/browse/KYLIN-5283
> Project: Kylin
>  Issue Type: Bug
>  Components: Job Engine, Spark Engine
>Affects Versions: v4.0.1
>Reporter: huangsheng
>Priority: Critical
> Attachments: 截屏2022-10-25 15.17.56.png, 截屏2022-10-25 15.18.35.png
>
>
> [SparkApplication.java|https://github.com/apache/kylin/blob/main/kylin-spark-project/kylin-spark-engine/src/main/java/org/apache/kylin/engine/spark/application/SparkApplication.java]
> 这里会捕获所有的原始异常，最终抛出一个RuntimeException
> !截屏2022-10-25 15.17.56.png!
> [JobWorker.scala|https://github.com/apache/kylin/blob/main/kylin-spark-project/kylin-spark-engine/src/main/scala/org/apache/spark/application/JobWorker.scala]
> 所以JobWorker无法捕获原始异常，所有异常都走ResourceLack方法处理，这里存在问题
> !截屏2022-10-25 15.18.35.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (KYLIN-5286) Improve QueryConnection.getConnection?

[liyang (Jira)]

[ 
https://issues.apache.org/jira/browse/KYLIN-5286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17626223#comment-17626223
 ] 

liyang commented on KYLIN-5286:
---

The QueryConnection is a lightweight object by design. That means there is no 
need to manage its lifecycle. Just create it when needed, and throw it away 
after it's used. That's the best management (unless we start to see performance 
issue in it).

> Improve QueryConnection.getConnection?
> --
>
> Key: KYLIN-5286
> URL: https://issues.apache.org/jira/browse/KYLIN-5286
> Project: Kylin
>  Issue Type: Wish
>  Components: Query Engine
>Affects Versions: v4.0.1
>Reporter: Liu Zhao
>Priority: Trivial
> Attachments: image-2022-10-29-00-51-39-725.png, 
> image-2022-10-29-00-52-05-786.png
>
>
> Is it possible to cache connections,  similar to BasicDataSouce, etc.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (KYLIN-5286) Improve QueryConnection.getConnection?

[liyang (Jira)]

[ 
https://issues.apache.org/jira/browse/KYLIN-5286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17626216#comment-17626216
 ] 

liyang commented on KYLIN-5286:
---

Technically, cache is always possible, at the cost of designing how to refresh 
the metadata linked by the connection object.

How many 'ms' it takes to get a new QueryConnection in your env? Let's check 
the potential benefit first. :)

> Improve QueryConnection.getConnection?
> --
>
> Key: KYLIN-5286
> URL: https://issues.apache.org/jira/browse/KYLIN-5286
> Project: Kylin
>  Issue Type: Wish
>  Components: Query Engine
>Affects Versions: v4.0.1
>Reporter: Liu Zhao
>Priority: Trivial
> Attachments: image-2022-10-29-00-51-39-725.png, 
> image-2022-10-29-00-52-05-786.png
>
>
> Is it possible to cache connections,  similar to BasicDataSouce, etc.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (KYLIN-5279) Default null collation for Kylin5.0 sqls should be null-last for ASC order and null-first for DESC order.

[liyang (Jira)]

[ 
https://issues.apache.org/jira/browse/KYLIN-5279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17622601#comment-17622601
 ] 

liyang commented on KYLIN-5279:
---

NULL is smaller or larger? It depends.

The answe depends on the SQL vendor. Below is a summary from
[https://learnsql.com/blog/how-to-order-rows-with-nulls/]
 * PostgreSQL / Oracle  — NULL is larger, by default
 * MySQL / SQL Server — NULL is smaller, by default

Of course, the default behavior is configurable in most of these products.

The PR looks good as it lets KYLIN behavior configurable as well. 

> Default null collation for Kylin5.0 sqls should be null-last for ASC order 
> and null-first for DESC order.
> -
>
> Key: KYLIN-5279
> URL: https://issues.apache.org/jira/browse/KYLIN-5279
> Project: Kylin
>  Issue Type: Bug
>  Components: Query Engine
>Reporter: Zhenning Zhang
>Assignee: Zhenning Zhang
>Priority: Major
> Fix For: 5.0-alpha
>
> Attachments: image-2022-10-20-15-12-12-967.png
>
>
> according to pgsql: 
> [https://www.postgresql.org/docs/current/queries-order.html]
> "By default, null values sort as if larger than any non-null value; that is, 
> {{NULLS FIRST}} is the default for {{DESC}} order, and {{NULLS LAST}} 
> otherwise."
> currently, the result is:
> !image-2022-10-20-15-12-12-967.png|width=1320,height=743!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (KYLIN-5278) Support Pivot feature of SQL

[liyang (Jira)]

[ 
https://issues.apache.org/jira/browse/KYLIN-5278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17622600#comment-17622600
 ] 

liyang commented on KYLIN-5278:
---

So PIVOT function gets +1

Let's see it will get more votes.

> Support Pivot feature of SQL
> 
>
> Key: KYLIN-5278
> URL: https://issues.apache.org/jira/browse/KYLIN-5278
> Project: Kylin
>  Issue Type: Wish
>  Components: Query Engine
>Reporter: yoonsung.lee
>Priority: Major
>
> HI. I'm using Kylin 3.1.3 for OLAP.
> It seems that there is no feature of Pivot. (e.g. 
> https://docs.data.world/documentation/sql/concepts/advanced/PIVOT.html ,
> https://calcite.apache.org/javadocAggregate/org/apache/calcite/sql/SqlPivot.html
> )
> I also tried to find source code related to Pivot, but there isn't. 
> Is there no feature for Pivot in Kylin(3 or 4)?
> If not, is there a plan?
> If not, is there a way to write a query like Pivot?
> Best regards :) 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (KYLIN-5267) force limit to protect the service when SQL has no limit

[liyang (Jira)]

[ 
https://issues.apache.org/jira/browse/KYLIN-5267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17612011#comment-17612011
 ] 

liyang commented on KYLIN-5267:
---

If what you need is something beyond the above two configs, feel free to 
propose and PR.

> force limit to protect the service when SQL has no limit
> 
>
> Key: KYLIN-5267
> URL: https://issues.apache.org/jira/browse/KYLIN-5267
> Project: Kylin
>  Issue Type: Improvement
>  Components: Query Engine
>Affects Versions: v4.0.1
>Reporter: Liu Zhao
>Priority: Minor
>
> Kylin4 queries use Spark, and the result data will be sent to the driver 
> (that is, query service). If the user's SQL returns a large data set, it may 
> cause the query oom to be unavailable. To solve this problem, can we provide 
> a means for users to enforce limit when SQL has no limit to protect services. 
> I am here https://issues.apache.org/jira/browse/KYLIN-2649 You can see the 
> idea above, but can  reduce the limit on the SQL structure？



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (KYLIN-5268) kylin4.0.1服务每天都会内存溢出java.lang.OutOfMemoryError: Java heap space

[liyang (Jira)]

[ 
https://issues.apache.org/jira/browse/KYLIN-5268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17612010#comment-17612010
 ] 

liyang commented on KYLIN-5268:
---

> 如果tableau发出的奇怪sql不多固定，似乎修改源码的 org.apache.kylin.rest.util.TableauInterceptor 
> 类更合理

+1

> kylin4.0.1服务每天都会内存溢出java.lang.OutOfMemoryError: Java heap space
> ---
>
> Key: KYLIN-5268
> URL: https://issues.apache.org/jira/browse/KYLIN-5268
> Project: Kylin
>  Issue Type: Bug
>Reporter: zhenkuan_zhang
>Priority: Major
> Attachments: kylin_query.log
>
>
> Hi，最近发现我们的kylin服务非常不稳定，三个节点，每天都会有节点出现内存溢出的状况，jvm从16G调到48G也同样会内存溢出。通过分析Dump 出 
> java_pid34184.hprof文件，发现内存会被两个查询完全占满导致jvm内存溢出，sql是通过tableau发送来的。sql如下（该事实表行数上亿）：
> 1.
> SELECT "自訂 SQL 查詢"."FAIL" AS "FAIL" FROM (   
>   SELECT DT  TEST_DATE,TEST_STATION_CODE,SUBSTRING(TEST_LINE,1,6) 
> TEST_FLOOR,TEST_LINE,CONFIG,REGION,LOCALLIZATION,
>   SUBSTRING(ATL FROM  1 for POSITION('*-' IN ATL from 1)-1 ) 
> VENDERATL,SUBSTRING(ATL FROM POSITION('-' IN ATL FROM 5)+1 FOR 3) 
> WEEKATL,SUBSTRING(ATL FROM POSITION('-' IN ATL FROM 6)+1 FOR 4) 
> DATEATL,SUBSTRING(ATL FROM POSITION('*-' IN ATL FROM 1)+7 FOR 4) ATL ,
>   INPUT,FAIL,RETEST FROM BI_DW.KPY_FACT_SUM_DATA
>   ) "自訂 SQL 查詢"
> 2.
> SELECT "自訂 SQL 查詢"."RETEST" AS "RETEST" FROM (   
>   SELECT DT  TEST_DATE,TEST_STATION_CODE,SUBSTRING(TEST_LINE,1,6) 
> TEST_FLOOR,TEST_LINE,CONFIG,REGION,LOCALLIZATION,
>   SUBSTRING(ATL FROM  1 for POSITION('*-' IN ATL from 1)-1 ) 
> VENDERATL,SUBSTRING(ATL FROM POSITION('-' IN ATL FROM 5)+1 FOR 3) 
> WEEKATL,SUBSTRING(ATL FROM POSITION('-' IN ATL FROM 6)+1 FOR 4) 
> DATEATL,SUBSTRING(ATL FROM POSITION('*-' IN ATL FROM 1)+7 FOR 4) ATL ,
>   INPUT,FAIL,RETEST FROM BI_DW.KPY_FACT_SUM_DATA
>   ) "自訂 SQL 查詢"
> 一方面我们在与tableau方联系，沟通为什么会发出这么奇怪的sql。
> 另一方面希望得到kylin的支持，看看有没有什么解决方法，
> 比如kylin.query.force-limit=1000 可以为 select * from table 加上 limit 1000，有没有办法限制 
> select a from table。
> 或者有没有其他的解决方法。
> 期待回复，十分感谢！！！



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (KYLIN-5267) force limit to protect the service when SQL has no limit

[liyang (Jira)]

[ 
https://issues.apache.org/jira/browse/KYLIN-5267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17612009#comment-17612009
 ] 

liyang commented on KYLIN-5267:
---

As a result of KYLIN-2649, a config parameter called "kylin.query.force-limit" 
is already added.

Does that help your use case?

 

Also "kylin.query.scan-threshold" is another config parameter that is often 
used to limit big queries going too wild.

> force limit to protect the service when SQL has no limit
> 
>
> Key: KYLIN-5267
> URL: https://issues.apache.org/jira/browse/KYLIN-5267
> Project: Kylin
>  Issue Type: Improvement
>  Components: Query Engine
>Affects Versions: v4.0.1
>Reporter: Liu Zhao
>Priority: Minor
>
> Kylin4 queries use Spark, and the result data will be sent to the driver 
> (that is, query service). If the user's SQL returns a large data set, it may 
> cause the query oom to be unavailable. To solve this problem, can we provide 
> a means for users to enforce limit when SQL has no limit to protect services. 
> I am here https://issues.apache.org/jira/browse/KYLIN-2649 You can see the 
> idea above, but can  reduce the limit on the SQL structure？



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (KYLIN-5272) The "case when" command may cause the field value to be different from the user's expectation.

[liyang (Jira)]

[ 
https://issues.apache.org/jira/browse/KYLIN-5272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17612006#comment-17612006
 ] 

liyang commented on KYLIN-5272:
---

Could you provide a complete SQL and related table structure, such that others 
can reproduce the issue and then be able to help?

Also likely, without knowing the exact problem, the issue maybe fixed in Kylin 
4.x as all calculations are moved into Spark. Calcite is only used for SQL 
parsing.

> The "case when" command may cause the field value to be different from the 
> user's expectation.
> --
>
> Key: KYLIN-5272
> URL: https://issues.apache.org/jira/browse/KYLIN-5272
> Project: Kylin
>  Issue Type: Bug
>  Components: Query Engine
>Affects Versions: v2.2.0, v2.6.3
>Reporter: GuKe
>Priority: Major
> Attachments: issue.jpg
>
>
> When using "Case then" to set a fixed character value for a field, Kylin uses 
> Spaces to complete the length of string to 32. This will cause the field to 
> display characters that are actually different from the user's expected 
> values, causing query errors, Example a failed Join operation.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (KYLIN-5263) replace log4j with logback

[liyang (Jira)]

[ 
https://issues.apache.org/jira/browse/KYLIN-5263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17605544#comment-17605544
 ] 

liyang commented on KYLIN-5263:
---

有没有办法评估一下换成 logback 能提升多少？

any estimate of the improvement after change to logback?

> replace log4j with logback
> --
>
> Key: KYLIN-5263
> URL: https://issues.apache.org/jira/browse/KYLIN-5263
> Project: Kylin
>  Issue Type: Improvement
>  Components: Query Engine
>Affects Versions: v4.0.1
>Reporter: zhaoliu
>Priority: Minor
> Attachments: screenshot-1.png
>
>
> 在查询并发高的时候，发现线程会阻塞在日志打印处，可以使用更高效的logback替换当前的log4j.
>  !screenshot-1.png! 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (KYLIN-5253) Is there any plan to add field processing method

[liyang (Jira)]

[ 
https://issues.apache.org/jira/browse/KYLIN-5253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17602996#comment-17602996
 ] 

liyang commented on KYLIN-5253:
---

any of these work?
 * SUBSTRING(string FROM integer1 FOR integer2)
 * concat(str1, str2)
 * left(str,n)

if yes, then the doc needs improvement

> Is there any plan to add field processing method
> 
>
> Key: KYLIN-5253
> URL: https://issues.apache.org/jira/browse/KYLIN-5253
> Project: Kylin
>  Issue Type: Wish
>  Components: Flink Engine
>Reporter: zys
>Priority: Major
>
> Hi! Is there any plan to add field processing method, such as concat, substr?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (KYLIN-5255) UTF-8 column name support

[liyang (Jira)]

[ 
https://issues.apache.org/jira/browse/KYLIN-5255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17602992#comment-17602992
 ] 

liyang commented on KYLIN-5255:
---

Nice improvement suggestion. 

> UTF-8 column name support
> -
>
> Key: KYLIN-5255
> URL: https://issues.apache.org/jira/browse/KYLIN-5255
> Project: Kylin
>  Issue Type: Wish
>Affects Versions: v4.0.1
>Reporter: Lilo
>Priority: Major
>
> Perhaps it's an appropriate time to support UTF-8 column names? (Kylin4 using 
> Spark to build Cube)
> Although the Hive documentation still says that UTF-8 characters are not 
> supported for table or column names, but after some simple sql tweaks to hive 
> metadata, we had no problems using UTF-8 column names in production (e.g. in 
> Hive / Spark2 / Spark3 / Impala / Presto / Flink).
>  
> (Tweaks like
> {code:sql}
> alter table COLUMNS_V2 modify column COLUMN_NAME varchar(255) character set 
> utf8;{code}
> )
>  
> also:
> https://issues.apache.org/jira/browse/KYLIN-2816  
> https://issues.apache.org/jira/browse/KYLIN-3668



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (KYLIN-5259) Support DAY*, WEEK*, MONTH*, YEAR* functions from SQL

[liyang (Jira)]

[ 
https://issues.apache.org/jira/browse/KYLIN-5259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17602983#comment-17602983
 ] 

liyang commented on KYLIN-5259:
---

try
 * DAYOFYEAR(date)
 * DAYOFMONTH(date)
 * DAYOFWEEK(date)
 * MONTH(date)
 * WEEK(date)
 * YEAR(date)

if these work, em, the doc needs improvements

> Support DAY*, WEEK*, MONTH*, YEAR* functions from SQL 
> --
>
> Key: KYLIN-5259
> URL: https://issues.apache.org/jira/browse/KYLIN-5259
> Project: Kylin
>  Issue Type: Wish
>Affects Versions: v3.1.3
>Reporter: yoonsung.lee
>Priority: Major
>
> Hi. I have been using Kylin 3.1.3. 
> Kylin's SQL seems to do not support DAY*, WEEK*, MONTH*, YEAR* functions. * 
> means series of functions such as day, day_of_week, day_of_month, 
> day_of_year. 
> Are there existing features to support these?
> If not, is there any plan to support these?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (KYLIN-5258) kylin5 HiveMetaStore

[liyang (Jira)]

[ 
https://issues.apache.org/jira/browse/KYLIN-5258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17602978#comment-17602978
 ] 

liyang commented on KYLIN-5258:
---

There must be deeper cause (likely caused by) not mentioned in the log. Please 
share the cause and solution together for other's reference.

> kylin5 HiveMetaStore
> 
>
> Key: KYLIN-5258
> URL: https://issues.apache.org/jira/browse/KYLIN-5258
> Project: Kylin
>  Issue Type: Bug
>  Components: Metadata
>Affects Versions: 5.0-alpha
>Reporter: zys
>Priority: Blocker
>
> Hi! i compile kylin5 and found that there are more settings in 
> kylin.properties, such as 
> spark.sql.hive.metastore. After start cmd, found errors in logs/check-env.out 
> like that:
>  
> 22/09/07 22:10:08 WARN metadata.Hive: Failed to register all functions.
> java.lang.RuntimeException: Unable to instantiate 
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
>         at 
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1662)
>         at 
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:67)
>         at 
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:82)
>         at 
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3411)
>         at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3430)
>         at 
> org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3655)
>         at 
> org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:231)
>         at 
> org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:215)
>  
> hive.metastore.version is 1.1.0. i have tried spark2.*, 3.*, and put 
> hive.site-xml in $SPARK_HOME/conf, the result is same.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (KYLIN-5261) Suggest to add group owner to identify the usergoup's owner

[liyang (Jira)]

[ 
https://issues.apache.org/jira/browse/KYLIN-5261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17602977#comment-17602977
 ] 

liyang commented on KYLIN-5261:
---

Good idea. Code commit is welcome.  :)

> Suggest to add group owner to identify the usergoup's owner
> ---
>
> Key: KYLIN-5261
> URL: https://issues.apache.org/jira/browse/KYLIN-5261
> Project: Kylin
>  Issue Type: Improvement
>  Components: Others
>Affects Versions: Future
>Reporter: Li Can
>Priority: Major
> Attachments: image-2022-09-09-16-45-37-415.png
>
>
> !image-2022-09-09-16-45-37-415.png|width=390,height=200!
> above this UserGroup authorization part, suggest to add group owner to 
> identify the group's owner, thus when someone need other group permission, 
> he/she can contact the owner quickly.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (KYLIN-5246) long running job's log staying in mem, may cause job server oom

[liyang (Jira)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-5246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liyang updated KYLIN-5246:
--
Summary: long running job's log staying in mem, may cause job server oom  
(was: long running jobs may cause memory problems in the job server)

> long running job's log staying in mem, may cause job server oom
> ---
>
> Key: KYLIN-5246
> URL: https://issues.apache.org/jira/browse/KYLIN-5246
> Project: Kylin
>  Issue Type: Improvement
>  Components: Job Engine
>Affects Versions: v4.0.1
>Reporter: zhaoliu
>Priority: Minor
>
> {code:java}
> CliCommandExecutor
> 
> BufferedReader reader = new BufferedReader(
>   new InputStreamReader(proc.getInputStream(), 
> StandardCharsets.UTF_8));
> String line;
> StringBuilder result = new StringBuilder();
> while ((line = reader.readLine()) != null && 
> !Thread.currentThread().isInterrupted()) {
>   result.append(line).append('\n');
>   if (logAppender != null) {
>   logAppender.log(line);
>   }
> }
> {code}
> job运行时间久，result 会非常大，可能会引起内存问题



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (KYLIN-5246) long running jobs may cause memory problems in the job server

[liyang (Jira)]

[ 
https://issues.apache.org/jira/browse/KYLIN-5246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17599820#comment-17599820
 ] 

liyang commented on KYLIN-5246:
---

 Good suggestion!

> long running jobs may cause memory problems in the job server
> -
>
> Key: KYLIN-5246
> URL: https://issues.apache.org/jira/browse/KYLIN-5246
> Project: Kylin
>  Issue Type: Improvement
>  Components: Job Engine
>Affects Versions: v4.0.1
>Reporter: zhaoliu
>Priority: Minor
>
> {code:java}
> CliCommandExecutor
> 
> BufferedReader reader = new BufferedReader(
>   new InputStreamReader(proc.getInputStream(), 
> StandardCharsets.UTF_8));
> String line;
> StringBuilder result = new StringBuilder();
> while ((line = reader.readLine()) != null && 
> !Thread.currentThread().isInterrupted()) {
>   result.append(line).append('\n');
>   if (logAppender != null) {
>   logAppender.log(line);
>   }
> }
> {code}
> job运行时间久，result 会非常大，可能会引起内存问题



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (KYLIN-5230) The bug from change password

[liyang (Jira)]

[ 
https://issues.apache.org/jira/browse/KYLIN-5230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17581445#comment-17581445
 ] 

liyang commented on KYLIN-5230:
---

Nice suggestion and thanks for the report!

> The bug from change password
> 
>
> Key: KYLIN-5230
> URL: https://issues.apache.org/jira/browse/KYLIN-5230
> Project: Kylin
>  Issue Type: Improvement
>  Components: Client - CLI
>Affects Versions: v4.0.1
>Reporter: Li Can
>Priority: Major
> Attachments: image.png
>
>
> 1.after login, you can change password by input any letter in current 
> password field that may not right current password with new password.
> 2.after changed password, old password can still work for a period of time if 
> the system hava cache function, and the new password dosen't work util cached 
> old password expired.
> 3.when the new password can work, if you don't restart the system, you can 
> only login but cann't access any service.(refer to the attachment).
>  
> The suggestion is to disable the change password function, because the 
> Authentication may integrated from external system, such as keystone.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (KYLIN-5114) Build cube error:java.lang.NoSuchMethodError

[liyang (Jira)]

[ 
https://issues.apache.org/jira/browse/KYLIN-5114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17581443#comment-17581443
 ] 

liyang commented on KYLIN-5114:
---

通常是数据量太大了，超过了集群现有的资源负载范围，建议用小数据量先试试。

 

对数据量和所需的资源有数了以后，也可以用人工指定 spark 资源（通过参数配置），来替代 Detect Resource。

> Build cube error:java.lang.NoSuchMethodError
> 
>
> Key: KYLIN-5114
> URL: https://issues.apache.org/jira/browse/KYLIN-5114
> Project: Kylin
>  Issue Type: Task
>  Components: Environment , Spark Engine
>Affects Versions: v4.0.0
> Environment: centos7
> Hive 3.1.2
> hadoop 3.1.3
> spark 2.4.5
>Reporter: qingquanzhang
>Assignee: mukvin
>Priority: Major
>  Labels: Question
> Fix For: Future
>
> Attachments: image-2021-11-04-16-36-25-537.png, 
> image-2021-11-04-17-34-11-875.png, kylin.log
>
>
> Build cube error:
> {code:java}
> Caused by: java.lang.NoSuchMethodError: 
> com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V{code}
> So I tryed move $HIVE_HOME/lib/guava-27.0-jre.jar into $KYLIN_HOME/tools/ and 
> $KYLIN_HOME/tomcat/lib/ and restart kylin,but still the same error.
>  
> !image-2021-11-04-16-36-25-537.png!
> 虽然报错了,但是该segment却有了记录,再次构建cube就会提示segment已存在.如果删除这个segment,还会报错,报错信息如下
> !image-2021-11-04-17-34-11-875.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (KYLIN-5231) RuntimeException in cube build with Spark engine

[liyang (Jira)]

[ 
https://issues.apache.org/jira/browse/KYLIN-5231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17581442#comment-17581442
 ] 

liyang commented on KYLIN-5231:
---

Most likely a env problem. Is this set of env used to work? Or it is a first 
attempt?

- Spark version : 3.2.1
- Hadoop version : 3.1.1.3.1.4.0
- Kylin version : 3.1.2

> RuntimeException in cube build with Spark engine
> 
>
> Key: KYLIN-5231
> URL: https://issues.apache.org/jira/browse/KYLIN-5231
> Project: Kylin
>  Issue Type: Bug
>Affects Versions: v3.1.2
>Reporter: Sandeep Putrevu
>Priority: Major
>
> Cube build is failing at *Step #7 :  Build Cube with Spark*
>  
> Our configuration:
> Spark version : 3.2.1
> Hadoop version : 3.1.1.3.1.4.0
> Kylin version : 3.1.2
> Error : 
> Exception in thread "main" java.lang.RuntimeException: error execute 
> org.apache.kylin.engine.spark.SparkCubingByLayer. Root cause: 
> scala.reflect.ClassTag$$anon$1
>     at 
> org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:42)
>     at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:44)
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>     at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>     at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>     at java.lang.reflect.Method.invoke(Method.java:498)
>     at 
> org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
>     at 
> org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955)
>     at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
>     at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
>     at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
>     at 
> org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
>     at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
>     at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Caused by: java.lang.ClassNotFoundException: scala.reflect.ClassTag$$anon$1
>     at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
>     at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
>     at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
>     at java.lang.Class.forName0(Native Method)
>     at java.lang.Class.forName(Class.java:264)
>     at 
> org.apache.kylin.engine.spark.SparkCubingByLayer.execute(SparkCubingByLayer.java:142)
>     at 
> org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
>     ... 13 more



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (KYLIN-5114) Build cube error:java.lang.NoSuchMethodError

[liyang (Jira)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-5114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liyang updated KYLIN-5114:
--
Labels: Question  (was: )

> Build cube error:java.lang.NoSuchMethodError
> 
>
> Key: KYLIN-5114
> URL: https://issues.apache.org/jira/browse/KYLIN-5114
> Project: Kylin
>  Issue Type: Task
>  Components: Environment , Spark Engine
>Affects Versions: v4.0.0
> Environment: centos7
> Hive 3.1.2
> hadoop 3.1.3
> spark 2.4.5
>Reporter: qingquanzhang
>Assignee: mukvin
>Priority: Major
>  Labels: Question
> Fix For: Future
>
> Attachments: image-2021-11-04-16-36-25-537.png, 
> image-2021-11-04-17-34-11-875.png, kylin.log
>
>
> Build cube error:
> {code:java}
> Caused by: java.lang.NoSuchMethodError: 
> com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V{code}
> So I tryed move $HIVE_HOME/lib/guava-27.0-jre.jar into $KYLIN_HOME/tools/ and 
> $KYLIN_HOME/tomcat/lib/ and restart kylin,but still the same error.
>  
> !image-2021-11-04-16-36-25-537.png!
> 虽然报错了,但是该segment却有了记录,再次构建cube就会提示segment已存在.如果删除这个segment,还会报错,报错信息如下
> !image-2021-11-04-17-34-11-875.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (KYLIN-5114) Build cube error:java.lang.NoSuchMethodError

[liyang (Jira)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-5114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liyang updated KYLIN-5114:
--
Issue Type: Task  (was: Bug)

> Build cube error:java.lang.NoSuchMethodError
> 
>
> Key: KYLIN-5114
> URL: https://issues.apache.org/jira/browse/KYLIN-5114
> Project: Kylin
>  Issue Type: Task
>  Components: Environment , Spark Engine
>Affects Versions: v4.0.0
> Environment: centos7
> Hive 3.1.2
> hadoop 3.1.3
> spark 2.4.5
>Reporter: qingquanzhang
>Assignee: mukvin
>Priority: Major
> Fix For: Future
>
> Attachments: image-2021-11-04-16-36-25-537.png, 
> image-2021-11-04-17-34-11-875.png, kylin.log
>
>
> Build cube error:
> {code:java}
> Caused by: java.lang.NoSuchMethodError: 
> com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V{code}
> So I tryed move $HIVE_HOME/lib/guava-27.0-jre.jar into $KYLIN_HOME/tools/ and 
> $KYLIN_HOME/tomcat/lib/ and restart kylin,but still the same error.
>  
> !image-2021-11-04-16-36-25-537.png!
> 虽然报错了,但是该segment却有了记录,再次构建cube就会提示segment已存在.如果删除这个segment,还会报错,报错信息如下
> !image-2021-11-04-17-34-11-875.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Comment Edited] (KYLIN-5114) Build cube error:java.lang.NoSuchMethodError

[liyang (Jira)]

[ 
https://issues.apache.org/jira/browse/KYLIN-5114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17580400#comment-17580400
 ] 

liyang edited comment on KYLIN-5114 at 8/16/22 4:41 PM:


这类是环境问题，假如不是一模一样的环境（centos7，Hive 3.1.2，hadoop 3.1.3，spark 2.4.5）即便是同样的报错解法也不一样。

只有解决思路是通用的：
 # 看日志先确认，是哪个进程报出的错。可能是 kylin 进程，也可能是远端的 spark 进程，解法会不同。
 # 根据具体 java 进程，再看它的 classpath 怎么组装来的。比如 [How can I view the classpath and jvm 
args of an executing java program in windows - Stack 
Overflow|https://stackoverflow.com/questions/17151058/how-can-i-view-the-classpath-and-jvm-args-of-an-executing-java-program-in-window]
 # 定位到有问题的 guava 包，再看怎么替换或者升级


was (Author: liyang.g...@gmail.com):
这类是环境问题，假如不是一模一样的环境（centos7，Hive 3.1.2，hadoop 3.1.3，spark 
2.4.5）即便是同样的报错解法也不一样。解决思路通常是这样：
 # 看日志先确认，是哪个进程报出的错。可能是 kylin 进程，也可能是远端的 spark 进程，解法会不同。
 # 根据具体 java 进程，再看它的 classpath 怎么组装来的。比如 [How can I view the classpath and jvm 
args of an executing java program in windows - Stack 
Overflow|https://stackoverflow.com/questions/17151058/how-can-i-view-the-classpath-and-jvm-args-of-an-executing-java-program-in-window]
 # 定位到有问题的 guava 包，再看怎么替换或者升级

> Build cube error:java.lang.NoSuchMethodError
> 
>
> Key: KYLIN-5114
> URL: https://issues.apache.org/jira/browse/KYLIN-5114
> Project: Kylin
>  Issue Type: Bug
>  Components: Environment , Spark Engine
>Affects Versions: v4.0.0
> Environment: centos7
> Hive 3.1.2
> hadoop 3.1.3
> spark 2.4.5
>Reporter: qingquanzhang
>Assignee: mukvin
>Priority: Major
> Fix For: Future
>
> Attachments: image-2021-11-04-16-36-25-537.png, 
> image-2021-11-04-17-34-11-875.png, kylin.log
>
>
> Build cube error:
> {code:java}
> Caused by: java.lang.NoSuchMethodError: 
> com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V{code}
> So I tryed move $HIVE_HOME/lib/guava-27.0-jre.jar into $KYLIN_HOME/tools/ and 
> $KYLIN_HOME/tomcat/lib/ and restart kylin,but still the same error.
>  
> !image-2021-11-04-16-36-25-537.png!
> 虽然报错了,但是该segment却有了记录,再次构建cube就会提示segment已存在.如果删除这个segment,还会报错,报错信息如下
> !image-2021-11-04-17-34-11-875.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (KYLIN-5114) Build cube error:java.lang.NoSuchMethodError

[liyang (Jira)]

[ 
https://issues.apache.org/jira/browse/KYLIN-5114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17580400#comment-17580400
 ] 

liyang commented on KYLIN-5114:
---

这类是环境问题，假如不是一模一样的环境（centos7，Hive 3.1.2，hadoop 3.1.3，spark 
2.4.5）即便是同样的报错解法也不一样。解决思路通常是这样：
 # 看日志先确认，是哪个进程报出的错。可能是 kylin 进程，也可能是远端的 spark 进程，解法会不同。
 # 根据具体 java 进程，再看它的 classpath 怎么组装来的。比如 [How can I view the classpath and jvm 
args of an executing java program in windows - Stack 
Overflow|https://stackoverflow.com/questions/17151058/how-can-i-view-the-classpath-and-jvm-args-of-an-executing-java-program-in-window]
 # 定位到有问题的 guava 包，再看怎么替换或者升级

> Build cube error:java.lang.NoSuchMethodError
> 
>
> Key: KYLIN-5114
> URL: https://issues.apache.org/jira/browse/KYLIN-5114
> Project: Kylin
>  Issue Type: Bug
>  Components: Environment , Spark Engine
>Affects Versions: v4.0.0
> Environment: centos7
> Hive 3.1.2
> hadoop 3.1.3
> spark 2.4.5
>Reporter: qingquanzhang
>Assignee: mukvin
>Priority: Major
> Fix For: Future
>
> Attachments: image-2021-11-04-16-36-25-537.png, 
> image-2021-11-04-17-34-11-875.png, kylin.log
>
>
> Build cube error:
> {code:java}
> Caused by: java.lang.NoSuchMethodError: 
> com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V{code}
> So I tryed move $HIVE_HOME/lib/guava-27.0-jre.jar into $KYLIN_HOME/tools/ and 
> $KYLIN_HOME/tomcat/lib/ and restart kylin,but still the same error.
>  
> !image-2021-11-04-16-36-25-537.png!
> 虽然报错了,但是该segment却有了记录,再次构建cube就会提示segment已存在.如果删除这个segment,还会报错,报错信息如下
> !image-2021-11-04-17-34-11-875.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Updated] (KYLIN-5224) kylin 4.0.1 does not support limit..offset (requires spark 3.4)

[liyang (Jira)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liyang updated KYLIN-5224:
--
Summary: kylin 4.0.1 does not support limit..offset (requires spark 3.4)  
(was: kylin 4.0.1 不支持分页查询)

> kylin 4.0.1 does not support limit..offset (requires spark 3.4)
> ---
>
> Key: KYLIN-5224
> URL: https://issues.apache.org/jira/browse/KYLIN-5224
> Project: Kylin
>  Issue Type: Bug
>Affects Versions: v4.0.1
>Reporter: liulei
>Priority: Major
>
> {
> "code": "999",
> "data": null,
> "msg": " mismatched input 'offset' expecting \{, ';'}(line 1, pos 
> 55)  == SQL == select * from MES_PROD.inspec_doc_report_data limit 10 offset 
> 5 ---^^^ ",
> "stacktrace": "org.apache.kylin.rest.exception.InternalErrorException:  
> mismatched input 'offset' expecting \{, ';'}(line 1, pos 55)  == SQL == 
> select * from MES_PROD.inspec_doc_report_data limit 10 offset 5 
> ---^^^ \n\tat 
> org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:497)\n\tat
>  
> org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:416)\n\tat
>  
> org.apache.kylin.rest.controller.QueryController.query(QueryController.java:93)\n\tat
>  sun.reflect.GeneratedMethodAccessor184.invoke(Unknown Source)\n\tat 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat
>  java.lang.reflect.Method.invoke(Method.java:498)\n\tat 
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)\n\tat
>  
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)\n\tat
>  
> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)\n\tat
>  
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:854)\n\tat
>  
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:765)\n\tat
>  
> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)\n\tat
>  
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)\n\tat
>  
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)\n\tat
>  
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)\n\tat
>  
> org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)\n\tat
>  javax.servlet.http.HttpServlet.service(HttpServlet.java:647)\n\tat 
> org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)\n\tat
>  javax.servlet.http.HttpServlet.service(HttpServlet.java:728)\n\tat 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)\n\tat
>  
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)\n\tat
>  org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)\n\tat 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)\n\tat
>  
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)\n\tat
>  
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)\n\tat
>  
> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)\n\tat
>  
> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)\n\tat
>  
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)\n\tat
>  
> org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)\n\tat
>  
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)\n\tat
>  
> org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)\n\tat
>  
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)\n\tat
>  
> org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)\n\tat
>  
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)\n\tat
>  
> 

[jira] [Commented] (KYLIN-5224) kylin 4.0.1 不支持分页查询

[liyang (Jira)]

[ 
https://issues.apache.org/jira/browse/KYLIN-5224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579441#comment-17579441
 ] 

liyang commented on KYLIN-5224:
---

then the request changes to support spark 3.4  :)

> kylin 4.0.1 不支持分页查询
> ---
>
> Key: KYLIN-5224
> URL: https://issues.apache.org/jira/browse/KYLIN-5224
> Project: Kylin
>  Issue Type: Bug
>Affects Versions: v4.0.1
>Reporter: liulei
>Priority: Major
>
> {
> "code": "999",
> "data": null,
> "msg": " mismatched input 'offset' expecting \{, ';'}(line 1, pos 
> 55)  == SQL == select * from MES_PROD.inspec_doc_report_data limit 10 offset 
> 5 ---^^^ ",
> "stacktrace": "org.apache.kylin.rest.exception.InternalErrorException:  
> mismatched input 'offset' expecting \{, ';'}(line 1, pos 55)  == SQL == 
> select * from MES_PROD.inspec_doc_report_data limit 10 offset 5 
> ---^^^ \n\tat 
> org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:497)\n\tat
>  
> org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:416)\n\tat
>  
> org.apache.kylin.rest.controller.QueryController.query(QueryController.java:93)\n\tat
>  sun.reflect.GeneratedMethodAccessor184.invoke(Unknown Source)\n\tat 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat
>  java.lang.reflect.Method.invoke(Method.java:498)\n\tat 
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)\n\tat
>  
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)\n\tat
>  
> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)\n\tat
>  
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:854)\n\tat
>  
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:765)\n\tat
>  
> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)\n\tat
>  
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)\n\tat
>  
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)\n\tat
>  
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)\n\tat
>  
> org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)\n\tat
>  javax.servlet.http.HttpServlet.service(HttpServlet.java:647)\n\tat 
> org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)\n\tat
>  javax.servlet.http.HttpServlet.service(HttpServlet.java:728)\n\tat 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)\n\tat
>  
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)\n\tat
>  org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)\n\tat 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)\n\tat
>  
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)\n\tat
>  
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)\n\tat
>  
> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)\n\tat
>  
> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)\n\tat
>  
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)\n\tat
>  
> org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)\n\tat
>  
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)\n\tat
>  
> org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)\n\tat
>  
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)\n\tat
>  
> org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)\n\tat
>  
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)\n\tat
>  
> org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)\n\tat
>  
> 

[jira] [Closed] (KYLIN-5227) Kylin association query does not use ad hoc query, but push down spark query

[liyang (Jira)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-5227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liyang closed KYLIN-5227.
-
Resolution: Not A Problem

> Kylin association query does not use ad hoc query, but push down spark query
> 
>
> Key: KYLIN-5227
> URL: https://issues.apache.org/jira/browse/KYLIN-5227
> Project: Kylin
>  Issue Type: Bug
>Reporter: huang song
>Priority: Major
> Attachments: image-2022-08-12-09-24-42-891.png
>
>
> The query statement is as follows
> select：
>  b.boname,
>  h.hdrcode
> from
>  b_booklist b
> inner join b_holding h 
>  on h.borecno=b.borecno
> Build cube fields as follows，
> All other steps are default
>  !image-2022-08-12-09-24-42-891.png! 
> Execute the push-down query of the above sql walk；
> take a long time
> I don't know where to build what's the problem



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Closed] (KYLIN-5212) cannot get HiveTableMeta

[liyang (Jira)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-5212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liyang closed KYLIN-5212.
-
Resolution: Not A Problem

> cannot get HiveTableMeta
> 
>
> Key: KYLIN-5212
> URL: https://issues.apache.org/jira/browse/KYLIN-5212
> Project: Kylin
>  Issue Type: Bug
>Reporter: huang song
>Priority: Major
> Attachments: screenshot-1.png
>
>
> version: kylin4.0.1+hive2.3.9+hadoop3.2.0+spark3.1.1
> 2022-07-19 11:42:25,982 ERROR [http-bio-7070-exec-2] 
> controller.BasicController:65 :
> org.apache.kylin.rest.exception.InternalErrorException: cannot get 
> HiveTableMeta
> at 
> org.apache.kylin.rest.controller.TableController.loadHiveTables(TableController.java:132)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
> at 
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
> at 
> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
> at 
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:854)
> at 
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:765)
> at 
> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
> at 
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
> at 
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
> at 
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
> at 
> org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
> at 
> org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
> at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
> at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> at 
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
> at 
> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
> at 
> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
> at 
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at 
> org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)
> at 
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at 
> org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
> at 
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at 
> org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
> at 
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at 
> org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)
> at 
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at 
> org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
> at 
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at 
> org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:215)
> at 
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at 
> 

[jira] [Commented] (KYLIN-5227) Kylin association query does not use ad hoc query, but push down spark query

[liyang (Jira)]

[ 
https://issues.apache.org/jira/browse/KYLIN-5227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579388#comment-17579388
 ] 

liyang commented on KYLIN-5227:
---

Without group by and aggregate function, this is a raw query. It is expected 
that raw query will go to spark pushdown and may take longer than aggregated 
queries that are answered by pre-computed data.

Things look normal suggest close this JIRA.
{code:java}
select：
b.boname,
h.hdrcode
from
b_booklist b
inner join b_holding h
on h.borecno=b.borecno{code}

> Kylin association query does not use ad hoc query, but push down spark query
> 
>
> Key: KYLIN-5227
> URL: https://issues.apache.org/jira/browse/KYLIN-5227
> Project: Kylin
>  Issue Type: Bug
>Reporter: huang song
>Priority: Major
> Attachments: image-2022-08-12-09-24-42-891.png
>
>
> The query statement is as follows
> select：
>  b.boname,
>  h.hdrcode
> from
>  b_booklist b
> inner join b_holding h 
>  on h.borecno=b.borecno
> Build cube fields as follows，
> All other steps are default
>  !image-2022-08-12-09-24-42-891.png! 
> Execute the push-down query of the above sql walk；
> take a long time
> I don't know where to build what's the problem



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (KYLIN-5215) The metadata information returned by Kylin pushdown query and Cube query is inconsistent

[liyang (Jira)]

[ 
https://issues.apache.org/jira/browse/KYLIN-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17574176#comment-17574176
 ] 

liyang commented on KYLIN-5215:
---

Kylin 的对外元数据默认都是大写，也就是比如 hive 数据源中用的是小写的表名和列名，从 Kylin 暴漏出来后将都是大写。

估计这也就是为什么原来代码里用了 upperCase() 把下推返回 name & label 也变成默认大写。

例子中的情况比较神奇是，cube 查询返回的 name & label 为什么是小写的值得研究一下。

> The metadata information returned by Kylin pushdown query and Cube query is 
> inconsistent
> 
>
> Key: KYLIN-5215
> URL: https://issues.apache.org/jira/browse/KYLIN-5215
> Project: Kylin
>  Issue Type: Bug
>  Components: Spark Engine
>Affects Versions: all, v4.0.1
> Environment: kylin 4.0.1
>Reporter: gaogao
>Priority: Major
> Attachments: cube_query.png, pushdown_query.png
>
>
> 问题：
>  在下压查询中返回的结果  和  cube 查询的结果不一致，下压查询中  columnMetas 的  label 和 name 字段会默认转换成大写！
> 对应源码类
> PushDownRunnerSparkImpl  50行
> ：
>  
> {code:java}
> // fill in selected column meta
> for (int i = 0; i < columnCount; ++i) {
> int nullable = fieldList.get(i).isNullable() ? 1 : 0;
> columnMetas.add(new SelectedColumnMeta(false, false, false, false, nullable, 
> true, Integer.MAX_VALUE,
> fieldList.get(i).getName().toUpperCase(Locale.ROOT), 
> fieldList.get(i).getName().toUpperCase(Locale.ROOT), null, null,
> null, fieldList.get(i).getPrecision(), fieldList.get(i).getScale(), 
> fieldList.get(i).getDataType(),
> fieldList.get(i).getDataTypeName(), false, false, false));
> }
>  
> {code}
>  
> pushdown query:
> !c:\Users\Administrator\AppData\Local\YNote\data\weixinobU7VjlOcNgcm0ZbmNGYzYRPZ__4\401fe7a7ea05495799145f2d413b0da4\clipboard.png!
> cube query:
> !c:\Users\Administrator\AppData\Local\YNote\data\weixinobU7VjlOcNgcm0ZbmNGYzYRPZ__4\0ef826f0229f42b59c4ad946a08bf6f3\clipboard.png!
>  
> 我认为这个问题应该修复，因为在尝试对接superset时发现，有些此类框架会强校验元数据列的大小写，如果默认进行转换会导致无法适配，且cube 
> query和pushdown query查询的元数据列结果不一致这也是无法接受的。
>  
> 如果讨论有需要的话，我可以修复他，谢谢~



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (KYLIN-3739) Use table alias rather than table identity for snapshots in CubeSegment, CubeInstance, CubeDesc

[liyang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16731520#comment-16731520
 ] 

liyang commented on KYLIN-3739:
---

Sounds fine to me if we can keep it backward compatible. I guess the logic will 
be, first check if the name is an alias, and if not then check if it's a full 
table name.

Should be a small change by line of code I assume.

> Use table alias rather than table identity for snapshots in CubeSegment, 
> CubeInstance, CubeDesc
> ---
>
> Key: KYLIN-3739
> URL: https://issues.apache.org/jira/browse/KYLIN-3739
> Project: Kylin
>  Issue Type: Improvement
>  Components: Metadata
>Reporter: Zhong Yanghong
>Priority: Major
>
> In 2.0.0, Kylin introduced table alias in DataModelDesc. Most of the elements 
> in CubeDesc, CubeInstance & CubeSegment use the table alias rather than the 
> table identity. However, for the lookup table snapshots, it still uses table 
> identity. 
> It's better for us to use only table alias instead of table identity in 
> CubeDesc, CubeInstance & CubeSegment. If so, it can provide several 
> advantages:
> # For CubeDesc, CubeInstance & CubeSegment, what exposed to them is only the 
> snowflake model and they don't need to care which real table is used.
> # If users want to change the table name in the snowflake model, we can still 
> keep the table alias unchanged and what we need to change is only the 
> DataModelDesc. And we don't need to do any change for CubeDesc, CubeInstance 
> & CubeSegment.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3423) Performance improvement in FactDistinctColumnsMapper

[liyang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liyang updated KYLIN-3423:
--
Description: 
Currently FactDistinctColumnsMapper writes every cell to mapper output. In 
spite of mapper side Combiner, we could do better de-dup using available mapper 
memory.

The situation becomes worse after KYLIN-3370, because not only dictionary 
columns, now it is every dimension column get written as mapper output.

Suggest
 * For non-dictionary dimension column, only write min/max value to mapper 
output.

 

  was:
Currently FactDistinctColumnsMapper writes every cell to mapper output. In 
spite of mapper side Combiner, we could do better de-dup using available mapper 
memory.

The situation becomes worse after KYLIN-3370, because not only dictionary 
columns, now it is every dimension column get written as mapper output.

Suggest
 * Use available mapper memory to de-dup before write to mapper output.
 * For non-dictionary dimension column, only write min/max value to mapper 
output.

 


> Performance improvement in FactDistinctColumnsMapper
> 
>
> Key: KYLIN-3423
> URL: https://issues.apache.org/jira/browse/KYLIN-3423
> Project: Kylin
>  Issue Type: Improvement
>Reporter: liyang
>Assignee: Shaoxiong Zhan
>Priority: Major
>
> Currently FactDistinctColumnsMapper writes every cell to mapper output. In 
> spite of mapper side Combiner, we could do better de-dup using available 
> mapper memory.
> The situation becomes worse after KYLIN-3370, because not only dictionary 
> columns, now it is every dimension column get written as mapper output.
> Suggest
>  * For non-dictionary dimension column, only write min/max value to mapper 
> output.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KYLIN-3423) Performance improvement in FactDistinctColumnsMapper

[liyang (JIRA)]

liyang created KYLIN-3423:
-

 Summary: Performance improvement in FactDistinctColumnsMapper
 Key: KYLIN-3423
 URL: https://issues.apache.org/jira/browse/KYLIN-3423
 Project: Kylin
  Issue Type: Improvement
Reporter: liyang


Currently FactDistinctColumnsMapper writes every cell to mapper output. In 
spite of mapper side Combiner, we could do better de-dup using available mapper 
memory.

The situation becomes worse after KYLIN-3370, because not only dictionary 
columns, now it is every dimension column get written as mapper output.

Suggest
 * Use available mapper memory to de-dup before write to mapper output.
 * For non-dictionary dimension column, only write min/max value to mapper 
output.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3364) Make the behavior of BigDecimalSumAggregator consistent with hive

[liyang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507223#comment-16507223
 ] 

liyang commented on KYLIN-3364:
---

This is fine, given we are aware that all aggregators must have consistent 
behavior in the end.

> Make the behavior of BigDecimalSumAggregator consistent with hive
> -
>
> Key: KYLIN-3364
> URL: https://issues.apache.org/jira/browse/KYLIN-3364
> Project: Kylin
>  Issue Type: Sub-task
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
>Priority: Major
> Fix For: v2.4.0
>
> Attachments: KYLIN-3364.patch
>
>
> When aggregating on a sequence of BigDecimal values, if all values are null, 
> the result should be null. Otherwise, null should be regarded as 0.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3399) Leaked lookup table in DictionaryGeneratorCLI#processSegment

[liyang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507222#comment-16507222
 ] 

liyang commented on KYLIN-3399:
---

The {{cubeMgr.getLookupTable(cubeSeg, join);}} is to check if any exception 
will be thrown by test getting the lookup table. Could use some comment if it 
confuses reader.

> Leaked lookup table in DictionaryGeneratorCLI#processSegment
> 
>
> Key: KYLIN-3399
> URL: https://issues.apache.org/jira/browse/KYLIN-3399
> Project: Kylin
>  Issue Type: Bug
>Reporter: Ted Yu
>Assignee: jiatao.tao
>Priority: Major
>
> {code}
> for (TableRef lookup : toCheckLookup) {
> logger.info("Checking snapshot of " + lookup);
> JoinDesc join = 
> cubeSeg.getModel().getJoinsTree().getJoinByPKSide(lookup);
> cubeMgr.getLookupTable(cubeSeg, join);
> {code}
> The lookup table from the last line doesn't seem to be used.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (KYLIN-3400) wipeCache and createCubeDesc make deadlock happenning

[liyang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507221#comment-16507221
 ] 

liyang edited comment on KYLIN-3400 at 6/10/18 1:05 AM:


This seems introduced by commit:

[https://github.com/apache/kylin/commit/a88403ae0050f6b1b2f903534f2330f03996603c]

 

Reverting the related lines in the commit is a fix candidate.

{{ public ProjectInstance reloadProjectQuietly(String project) throws 
IOException {}}

{{ try (AutoLock lock = prjMapLock.lockForWrite()) {}}
 {{ ProjectInstance prj = crud.reloadQuietly(project);}}
 {{-    clearL2Cache();}}
 {{+    reloadProjectL2Cache(project);}}
 {{ return prj;}}
 \{{ } }}
 \{{ } }}

 


was (Author: liyang.g...@gmail.com):
This seems introduced by commit:

[https://github.com/apache/kylin/commit/a88403ae0050f6b1b2f903534f2330f03996603c]

 

Reverting the related lines in the commit is a fix candidate.

{{ public ProjectInstance reloadProjectQuietly(String project) throws 
IOException {}}

{{ try (AutoLock lock = prjMapLock.lockForWrite()) {}}
{{ ProjectInstance prj = crud.reloadQuietly(project);}}
{{-    clearL2Cache();}}
{{+    reloadProjectL2Cache(project);}}
{{ return prj;}}
{{ }}}
{{ }}}

 

>  wipeCache and createCubeDesc make deadlock happenning
> --
>
> Key: KYLIN-3400
> URL: https://issues.apache.org/jira/browse/KYLIN-3400
> Project: Kylin
>  Issue Type: Bug
>  Components: REST Service
>Affects Versions: v2.3.0
>Reporter: powerinf
>Priority: Major
> Attachments: kylin_deadlock.log
>
>
> When using  the rest API  create cube , deadlock occasionally appears，the 
> kylin web ui hang，and cound not login ，See the [^kylin_deadlock.log]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (KYLIN-3400) wipeCache and createCubeDesc make deadlock happenning

[liyang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507221#comment-16507221
 ] 

liyang edited comment on KYLIN-3400 at 6/10/18 1:05 AM:


This seems introduced by commit:

[https://github.com/apache/kylin/commit/a88403ae0050f6b1b2f903534f2330f03996603c]

 

Reverting the related lines in the commit is a fix candidate.

{{ public ProjectInstance reloadProjectQuietly(String project) throws 
IOException {}}

{{ try (AutoLock lock = prjMapLock.lockForWrite()) {}}
 {{ ProjectInstance prj = crud.reloadQuietly(project);}}
 {{-    clearL2Cache();}}
 {{+    reloadProjectL2Cache(project);}}
 {{ return prj;}}


was (Author: liyang.g...@gmail.com):
This seems introduced by commit:

[https://github.com/apache/kylin/commit/a88403ae0050f6b1b2f903534f2330f03996603c]

 

Reverting the related lines in the commit is a fix candidate.

{{ public ProjectInstance reloadProjectQuietly(String project) throws 
IOException {}}

{{ try (AutoLock lock = prjMapLock.lockForWrite()) {}}
 {{ ProjectInstance prj = crud.reloadQuietly(project);}}
 {{-    clearL2Cache();}}
 {{+    reloadProjectL2Cache(project);}}
 {{ return prj;}}
 \{{ } }}
 \{{ } }}

 

>  wipeCache and createCubeDesc make deadlock happenning
> --
>
> Key: KYLIN-3400
> URL: https://issues.apache.org/jira/browse/KYLIN-3400
> Project: Kylin
>  Issue Type: Bug
>  Components: REST Service
>Affects Versions: v2.3.0
>Reporter: powerinf
>Priority: Major
> Attachments: kylin_deadlock.log
>
>
> When using  the rest API  create cube , deadlock occasionally appears，the 
> kylin web ui hang，and cound not login ，See the [^kylin_deadlock.log]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3400) wipeCache and createCubeDesc make deadlock happenning

[liyang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507221#comment-16507221
 ] 

liyang commented on KYLIN-3400:
---

This seems introduced by commit:

[https://github.com/apache/kylin/commit/a88403ae0050f6b1b2f903534f2330f03996603c]

 

Reverting the related lines in the commit is a fix candidate.

{{ public ProjectInstance reloadProjectQuietly(String project) throws 
IOException {}}

{{ try (AutoLock lock = prjMapLock.lockForWrite()) {}}
{{ ProjectInstance prj = crud.reloadQuietly(project);}}
{{-    clearL2Cache();}}
{{+    reloadProjectL2Cache(project);}}
{{ return prj;}}
{{ }}}
{{ }}}

 

>  wipeCache and createCubeDesc make deadlock happenning
> --
>
> Key: KYLIN-3400
> URL: https://issues.apache.org/jira/browse/KYLIN-3400
> Project: Kylin
>  Issue Type: Bug
>  Components: REST Service
>Affects Versions: v2.3.0
>Reporter: powerinf
>Priority: Major
> Attachments: kylin_deadlock.log
>
>
> When using  the rest API  create cube , deadlock occasionally appears，the 
> kylin web ui hang，and cound not login ，See the [^kylin_deadlock.log]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3389) intersect_count error

[liyang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507204#comment-16507204
 ] 

liyang commented on KYLIN-3389:
---

We need the cube_desc json to help. Also before certain it is a bug, better 
post in mailing list to ask. That is where get most people's attention.

> intersect_count error
> -
>
> Key: KYLIN-3389
> URL: https://issues.apache.org/jira/browse/KYLIN-3389
> Project: Kylin
>  Issue Type: Bug
>  Components: Measure - Count Distinct
>Affects Versions: v2.2.0
> Environment: hive+kylin+hbase
>Reporter: QingLi
>Priority: Major
>  Labels: easyfix
> Fix For: Future
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> select 
>  city_name,
>  intersect_count(server_user, dt,array['2018-05-10'] ) aaa
>  from
>  table 
>  where dt>='2018-05-10'
>  group by city_name
>  return: Error while compiling statement: FAILED: SemanticException [Error 
> 10025]: Line 3:0 Expression not in GROUP BY key ''2018-05-10''
>  
>  
>  
> here is cube json:
> {
>  "uuid": "5fe19005-7b31-4225-b7b3-62fde2c72256",
>  "last_modified": 1527489049614,
>  "version": "2.2.0.0",
>  "name": "CUBE_RETENTION_RD_LQ",
>  "owner": "SY_DW_U_RD",
>  "descriptor": "CUBE_RETENTION_RD_LQ",
>  "cost": 50,
>  "status": "READY",
>  "segments": [
>  {
>  "uuid": "75336a8d-a7b6-4465-bb63-fc3ff59b379e",
>  "name": "2018030100_2018052800",
>  "storage_location_identifier": "KYLIN_JVERXYS4CG",
>  "date_range_start": 151986240,
>  "date_range_end": 152746560,
>  "source_offset_start": 0,
>  "source_offset_end": 0,
>  "status": "READY",
>  "size_kb": 99,
>  "input_records": 7534006,
>  "input_records_size": 136364517,
>  "last_build_time": 1527489049614,
>  "last_build_job_id": "729f584e-66aa-46e3-a5b5-837aef69253c",
>  "create_time_utc": 1527488801260,
>  "cuboid_shard_nums": {},
>  "total_shards": 1,
>  "blackout_cuboids": [],
>  "binary_signature": null,
>  "dictionaries": {
>  "M_ORDER_DETAIL.CITY_NAME": 
> "/dict/SY_MART_KYLIN.M_ORDER_DETAIL/CITY_NAME/e7373276-ee9a-4308-93cf-ba5f3fafcf23.dict",
>  "M_ORDER_DETAIL.DT": 
> "/dict/SY_MART_KYLIN.M_ORDER_DETAIL/DT/29ec1575-8497-4df2-bb51-dcb9e7dda1a1.dict"
>  },
>  "snapshots": null,
>  "rowkey_stats": [
>  [
>  "M_ORDER_DETAIL.CITY_NAME",
>  336,
>  2
>  ],
>  [
>  "M_ORDER_DETAIL.DT",
>  57,
>  1
>  ]
>  ]
>  }
>  ],
>  "create_time_utc": 1527488778311,
>  "cuboid_bytes": null,
>  "cuboid_bytes_recommend": null,
>  "cuboid_last_optimized": 0,
>  "size_kb": 99,
>  "input_records_count": 7534006,
>  "input_records_size": 136364517
> }
>  
>  
> what's the problem?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3388) Data may become not correct if mappers fail during the cube building step, "distribute by rand()"

[liyang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16507197#comment-16507197
 ] 

liyang commented on KYLIN-3388:
---

Really!! This sounds like a critical bug in hive.

> Data may become not correct if mappers fail during the cube building step, 
> "distribute by rand()"
> -
>
> Key: KYLIN-3388
> URL: https://issues.apache.org/jira/browse/KYLIN-3388
> Project: Kylin
>  Issue Type: Bug
>Reporter: Zhong Yanghong
>Priority: Critical
> Attachments: Hive Issue - distribute by rand().png
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3364) Make the behavior of BigDecimalSumAggregator consistent with hive

[liyang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493008#comment-16493008
 ] 

liyang commented on KYLIN-3364:
---

Besides BigDecimalSumAggregator, other aggregators have the same issue I think.

Also at value ingestion time of MeasureType, the NULL value is already replaced 
by 0... so we still need a more complete solution of handling NULLs.

> Make the behavior of BigDecimalSumAggregator consistent with hive
> -
>
> Key: KYLIN-3364
> URL: https://issues.apache.org/jira/browse/KYLIN-3364
> Project: Kylin
>  Issue Type: Sub-task
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
>Priority: Major
> Attachments: KYLIN-3364.patch
>
>
> When aggregating on a sequence of BigDecimal values, if all values are null, 
> the result should be null. Otherwise, null should be regarded as 0.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KYLIN-1729) Error at #3 Step Name: Build Dimension Dictionary Duration

[liyang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liyang resolved KYLIN-1729.
---
   Resolution: Fixed
 Assignee: liyang  (was: Dong Li)
Fix Version/s: v2.0.0

This has been fixed long ago, however I cannot recall the exact fix version..

> Error at  #3 Step Name: Build Dimension Dictionary Duration
> ---
>
> Key: KYLIN-1729
> URL: https://issues.apache.org/jira/browse/KYLIN-1729
> Project: Kylin
>  Issue Type: Bug
>  Components: Job Engine
>Affects Versions: 1.x-HBase1.1.3, v1.5.0
>Reporter: Chaozhong Yang
>Assignee: liyang
>Priority: Major
> Fix For: v2.0.0
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> stack trace as follows:
> java.lang.NegativeArraySizeException
> at 
> org.apache.kylin.dict.TrieDictionaryBuilder.buildTrieBytes(TrieDictionaryBuilder.java:472)
> at 
> org.apache.kylin.dict.TrieDictionaryBuilder.build(TrieDictionaryBuilder.java:439)
> at 
> org.apache.kylin.dict.DictionaryGenerator.buildStringDict(DictionaryGenerator.java:163)
> at 
> org.apache.kylin.dict.DictionaryGenerator.buildDictionaryFromValueEnumerator(DictionaryGenerator.java:75)
> at 
> org.apache.kylin.dict.DictionaryGenerator.buildDictionary(DictionaryGenerator.java:108)
> at 
> org.apache.kylin.dict.DictionaryManager.buildDictionary(DictionaryManager.java:300)
> at org.apache.kylin.cube.CubeManager.buildDictionary(CubeManager.java:165)
> at 
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:51)
> at 
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:42)
> at 
> org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:56)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at 
> org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:60)
> at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
> at 
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50)
> at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
> at 
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:124)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> result code:2



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-2662) NegativeArraySizeException in "Extract Fact Table Distinct Columns"

[liyang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-2662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liyang updated KYLIN-2662:
--
Fix Version/s: v2.4.0

> NegativeArraySizeException in "Extract Fact Table Distinct Columns"
> ---
>
> Key: KYLIN-2662
> URL: https://issues.apache.org/jira/browse/KYLIN-2662
> Project: Kylin
>  Issue Type: Bug
>Reporter: liyang
>Assignee: liyang
>Priority: Major
>  Labels: scope
> Fix For: v2.4.0
>
>
> The full exception. The root cause suspect to be a cell goes over 32 KB. Need 
> verification and fix.
> {code}
> java.lang.NegativeArraySizeException
> at 
> org.apache.kylin.dict.TrieDictionary.getValueBytesFromIdWithoutCache(SourceFile:239)
> at 
> org.apache.kylin.dict.TrieDictionaryForestBuilder.addTree(SourceFile:134)
> at 
> org.apache.kylin.dict.TrieDictionaryForestBuilder.build(SourceFile:110)
> at 
> org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder.build(SourceFile:218)
> at 
> org.apache.kylin.engine.mr.steps.FactDistinctColumnsReducer.doCleanup(SourceFile:231)
> at 
> org.apache.kylin.engine.mr.KylinReducer.cleanup(SourceFile:71)
> at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:179)
> at 
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
> at 
> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
> at 
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (KYLIN-1729) Error at #3 Step Name: Build Dimension Dictionary Duration

[liyang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liyang closed KYLIN-1729.
-

> Error at  #3 Step Name: Build Dimension Dictionary Duration
> ---
>
> Key: KYLIN-1729
> URL: https://issues.apache.org/jira/browse/KYLIN-1729
> Project: Kylin
>  Issue Type: Bug
>  Components: Job Engine
>Affects Versions: 1.x-HBase1.1.3, v1.5.0
>Reporter: Chaozhong Yang
>Assignee: liyang
>Priority: Major
> Fix For: v2.0.0
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> stack trace as follows:
> java.lang.NegativeArraySizeException
> at 
> org.apache.kylin.dict.TrieDictionaryBuilder.buildTrieBytes(TrieDictionaryBuilder.java:472)
> at 
> org.apache.kylin.dict.TrieDictionaryBuilder.build(TrieDictionaryBuilder.java:439)
> at 
> org.apache.kylin.dict.DictionaryGenerator.buildStringDict(DictionaryGenerator.java:163)
> at 
> org.apache.kylin.dict.DictionaryGenerator.buildDictionaryFromValueEnumerator(DictionaryGenerator.java:75)
> at 
> org.apache.kylin.dict.DictionaryGenerator.buildDictionary(DictionaryGenerator.java:108)
> at 
> org.apache.kylin.dict.DictionaryManager.buildDictionary(DictionaryManager.java:300)
> at org.apache.kylin.cube.CubeManager.buildDictionary(CubeManager.java:165)
> at 
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:51)
> at 
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:42)
> at 
> org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:56)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at 
> org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:60)
> at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
> at 
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50)
> at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
> at 
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:124)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> result code:2



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (KYLIN-2662) NegativeArraySizeException in "Extract Fact Table Distinct Columns"

[liyang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-2662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liyang reassigned KYLIN-2662:
-

Assignee: liyang

> NegativeArraySizeException in "Extract Fact Table Distinct Columns"
> ---
>
> Key: KYLIN-2662
> URL: https://issues.apache.org/jira/browse/KYLIN-2662
> Project: Kylin
>  Issue Type: Bug
>Reporter: liyang
>Assignee: liyang
>Priority: Major
>  Labels: scope
> Fix For: v2.4.0
>
>
> The full exception. The root cause suspect to be a cell goes over 32 KB. Need 
> verification and fix.
> {code}
> java.lang.NegativeArraySizeException
> at 
> org.apache.kylin.dict.TrieDictionary.getValueBytesFromIdWithoutCache(SourceFile:239)
> at 
> org.apache.kylin.dict.TrieDictionaryForestBuilder.addTree(SourceFile:134)
> at 
> org.apache.kylin.dict.TrieDictionaryForestBuilder.build(SourceFile:110)
> at 
> org.apache.kylin.dict.DictionaryGenerator$StringTrieDictForestBuilder.build(SourceFile:218)
> at 
> org.apache.kylin.engine.mr.steps.FactDistinctColumnsReducer.doCleanup(SourceFile:231)
> at 
> org.apache.kylin.engine.mr.KylinReducer.cleanup(SourceFile:71)
> at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:179)
> at 
> org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
> at 
> org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
> at 
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3071) Add config to reuse dict to reduce dict size

[liyang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490410#comment-16490410
 ] 

liyang commented on KYLIN-3071:
---

I see the difference between Growing Dict and Reuse Dict. It is useful.

However there is a penalty of Reuse Dict, which could impact segment pruning at 
query time. Currently dictionary is used to transform filters. For example, a 
filter like {{A='non-exist-value'}} would be transformed to {{false}}, and such 
segment can be pruned to improve query performance. Having extra values in 
dictionary will weaken the effectiveness of such pruning.

Given the above side effect, I'd suggest the Reuse Dict feature be off by 
default.

> Add config to reuse dict to reduce dict size 
> -
>
> Key: KYLIN-3071
> URL: https://issues.apache.org/jira/browse/KYLIN-3071
> Project: Kylin
>  Issue Type: Improvement
>  Components: Metadata
>Reporter: Yang Hao
>Assignee: Yang Hao
>Priority: Major
> Fix For: Future
>
> Attachments: KYLIN-3071.apache-master.001.patch
>
>
> When calling DictionaryManager.trySaveNewDict, and growing dict is not 
> enabled, it only use the history dict which is equal, it may generate many 
> dict. We should supply a config to use contains instead of equal to reuse old 
> dict.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3071) Add config to reuse dict to reduce dict size

[liyang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465584#comment-16465584
 ] 

liyang commented on KYLIN-3071:
---

How is this different from {{KylinConfigBase.isGrowingDictEnabled()}} ?

> Add config to reuse dict to reduce dict size 
> -
>
> Key: KYLIN-3071
> URL: https://issues.apache.org/jira/browse/KYLIN-3071
> Project: Kylin
>  Issue Type: Improvement
>  Components: Metadata
>Reporter: Yang Hao
>Assignee: Yang Hao
>Priority: Major
> Fix For: Future
>
> Attachments: KYLIN-3071.apache-master.001.patch
>
>
> When calling DictionaryManager.trySaveNewDict, and growing dict is not 
> enabled, it only use the history dict which is equal, it may generate many 
> dict. We should supply a config to use contains instead of equal to reuse old 
> dict.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KYLIN-3356) Constant in SecretKeySpec

[liyang (JIRA)]

liyang created KYLIN-3356:
-

 Summary: Constant in SecretKeySpec
 Key: KYLIN-3356
 URL: https://issues.apache.org/jira/browse/KYLIN-3356
 Project: Kylin
  Issue Type: Improvement
Reporter: liyang


Reported by Rumen Paletov : 


 As part of some research about the common crypto mistakes that developers
 make <[https://cs.ucsb.edu/~chris/research/doc/ccs13_cryptolint.pdf]>, I
 noticed that your application has one of them.
 
 In particular, there's a violation of Rule 3 in
 org.apache.kylin.common.util.EncryptUtil
 
<[https://github.com/apache/kylin/blob/5552164ba09eba989b9ddccdf3f1e4f83ed0b799/core-common/src/main/java/org/apache/kylin/common/util/EncryptUtil.java#L36]>.
 That is, SecretKeySpec is being initialized with a constant key
 
<[https://github.com/apache/kylin/blob/5552164ba09eba989b9ddccdf3f1e4f83ed0b799/core-common/src/main/java/org/apache/kylin/common/util/EncryptUtil.java#L30]>
 instead of a randomly generated one.
 
 One solution would be to generate a key using SecureRandom:
 
 > byte[] key = new byte[16];
 > new SecureRandom.nextBytes(key);
 
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3335) Add project & cube related info to the job id for better filtering

[liyang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458515#comment-16458515
 ] 

liyang commented on KYLIN-3335:
---

+1 for upgrade, prefer simpler approach  :)

> Add project & cube related info to the job id for better filtering
> --
>
> Key: KYLIN-3335
> URL: https://issues.apache.org/jira/browse/KYLIN-3335
> Project: Kylin
>  Issue Type: Improvement
>Reporter: Zhong Yanghong
>Priority: Major
>
> Currently it's painful to search cube or project related jobs, since those 
> infos are hidden in values. Especially, when users want to list job in a 
> period under one project, by current design, all of the job output info have 
> to be read into memory. If this kind of operation is done very often within a 
> short period, it's easy to get OOM.
> If the job id is prefixed with project and cube names, then we can push down 
> prefix filters, which is efficient and safe.
> This kind of change will cause backward compatibility issue. There're two 
> ways to deal with this:
> * Set a milestone with time tag, if a search relates to data earlier than 
> this time, then just need to do one scan with prefix filter. Otherwise, two 
> scans are needed. One with prefix filter and the other use current strategy. 
> As time goes on, old job infos will be deleted. Once there's no data older 
> than the time, only one scan is needed.
> * Do migration for the old data once.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3308) Improvement exception in REST

[liyang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458509#comment-16458509
 ] 

liyang commented on KYLIN-3308:
---

Nice proposal~

> Improvement exception in REST
> -
>
> Key: KYLIN-3308
> URL: https://issues.apache.org/jira/browse/KYLIN-3308
> Project: Kylin
>  Issue Type: Improvement
>  Components: REST Service
>Reporter: Pan, Julian
>Assignee: Pan, Julian
>Priority: Major
>
> There are kinds of exception throw by rest call, and we need better defined 
> the exception. 
> For example:
> 1> There are three types of exception for cube equals null in CubeController:
> NotFoundException (deleteCube)
> InternalErrorException(getCube)
> BadRequestException (cloneCube)
> 2> And sometimes InternalErrorException will warp the BadRequestException in 
> CubeController.saveCubeDesc
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3352) Segment pruning bug, e.g. date_col > "max_date+1"

[liyang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liyang updated KYLIN-3352:
--
Summary: Segment pruning bug, e.g. date_col > "max_date+1"  (was: Segment 
pruning bug when date_col > "max_date+1")

> Segment pruning bug, e.g. date_col > "max_date+1"
> -
>
> Key: KYLIN-3352
> URL: https://issues.apache.org/jira/browse/KYLIN-3352
> Project: Kylin
>  Issue Type: Bug
>Reporter: liyang
>Assignee: liyang
>Priority: Major
>
> Currently {{date_col > "max_date+1"}} is rounded down to {{date_col > 
> "max_date"}} during encoding and further evaluated as {{date_col >= 
> "max_date"}} during segment pruning. This causes a segment can be pruned is 
> not pruned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KYLIN-3352) Segment pruning bug when date_col > "max_date+1"

[liyang (JIRA)]

liyang created KYLIN-3352:
-

 Summary: Segment pruning bug when date_col > "max_date+1"
 Key: KYLIN-3352
 URL: https://issues.apache.org/jira/browse/KYLIN-3352
 Project: Kylin
  Issue Type: Bug
Reporter: liyang
Assignee: liyang


Currently {{date_col > "max_date+1"}} is rounded down to {{date_col > 
"max_date"}} during encoding and further evaluated as {{date_col >= 
"max_date"}} during segment pruning. This causes a segment can be pruned is not 
pruned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KYLIN-3348) "missing LastBuildJobID" error when building new cube segment

[liyang (JIRA)]

liyang created KYLIN-3348:
-

 Summary: "missing LastBuildJobID" error when building new cube 
segment
 Key: KYLIN-3348
 URL: https://issues.apache.org/jira/browse/KYLIN-3348
 Project: Kylin
  Issue Type: Bug
Reporter: liyang


An unstable exception. Likely to happen when there are multiple concurrent 
builds.

{{2018-04-18 20:11:16,856 ERROR [pool-33-thread-11] threadpool.DefaultScheduler 
: ExecuteException job:cc08da19-f53e-4344-a6c5-05e764834924}}
 {{ org.apache.kylin.job.exception.ExecuteException: 
org.apache.kylin.job.exception.ExecuteException: 
java.lang.IllegalStateException: For cube CUBE[name=cube2], segment 
cube2[2018041423000_2018041423001] missing LastBuildJobID}}
 \{{ at 
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:140)}}
 \{{ at 
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:307)}}
 \{{ at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)}}
 \{{ at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)}}
 \{{ at java.lang.Thread.run(Thread.java:748)}}
 {{ Caused by: org.apache.kylin.job.exception.ExecuteException: 
java.lang.IllegalStateException: For cube CUBE[name=cube2], segment 
cube2[2018041423000_2018041423001] missing LastBuildJobID}}
 \{{ at 
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:140)}}
 \{{ at 
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:67)}}
 \{{ at 
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:129)}}
 \{{ ... 4 more}}
 {{ Caused by: java.lang.IllegalStateException: For cube CUBE[name=cube2], 
segment cube2[2018041423000_2018041423001] missing LastBuildJobID}}
 \{{ at 
org.apache.kylin.cube.CubeManager$SegmentAssist.promoteNewlyBuiltSegments(CubeManager.java:810)}}
 \{{ at 
org.apache.kylin.cube.CubeManager.promoteNewlyBuiltSegments(CubeManager.java:535)}}
 \{{ at 
org.apache.kylin.engine.mr.steps.UpdateCubeInfoAfterBuildStep.doWork(UpdateCubeInfoAfterBuildStep.java:78)}}
 \{{ at 
io.kyligence.kap.engine.mr.steps.KapUpdateCubeInfoAfterBuildStep.doWork(SourceFile:47)}}
 \{{ at 
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:129)}}
 \{{ ... 6 more}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3342) Cubing level calculation inconsistent?

[liyang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442154#comment-16442154
 ] 

liyang commented on KYLIN-3342:
---

I see. Then this Jira is invalid. Closing it.

Thanks [~yaho]!

> Cubing level calculation inconsistent?
> --
>
> Key: KYLIN-3342
> URL: https://issues.apache.org/jira/browse/KYLIN-3342
> Project: Kylin
>  Issue Type: Bug
>Reporter: liyang
>Priority: Major
> Attachments: KYLIN-3342.patch
>
>
> Got below exception during cube build.
> {{ Caused by: java.lang.IndexOutOfBoundsException: Index: 7, Size: 7}}
> {{ at java.util.ArrayList.rangeCheck(ArrayList.java:635)}}
> {{ at java.util.ArrayList.get(ArrayList.java:411)}}
> {{ at 
> org.apache.kylin.engine.mr.common.CubeStatsReader.estimateLayerSize(SourceFile:280)}}
> {{ at 
> org.apache.kylin.engine.spark.SparkCubingByLayer.estimateRDDPartitionNum(SourceFile:219)}}
> {{ at 
> org.apache.kylin.engine.spark.SparkCubingByLayer.execute(SourceFile:199)}}
> {{ at 
> org.apache.kylin.common.util.AbstractApplication.execute(SourceFile:37)}}
> {{ ... 6 more}}
>  
> Found two way of calculating the level of cuboids
>  * via CuboidScheduler.getBuildLevel()
>  * via CuboidUtil.getLongestDepth(...)
> We should settle down on one approach.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KYLIN-3342) Cubing level calculation inconsistent?

[liyang (JIRA)]

liyang created KYLIN-3342:
-

 Summary: Cubing level calculation inconsistent?
 Key: KYLIN-3342
 URL: https://issues.apache.org/jira/browse/KYLIN-3342
 Project: Kylin
  Issue Type: Bug
Reporter: liyang


Got below exception during cube build.

{{ Caused by: java.lang.IndexOutOfBoundsException: Index: 7, Size: 7}}
{{ at java.util.ArrayList.rangeCheck(ArrayList.java:635)}}
{{ at java.util.ArrayList.get(ArrayList.java:411)}}
{{ at 
org.apache.kylin.engine.mr.common.CubeStatsReader.estimateLayerSize(SourceFile:280)}}
{{ at 
org.apache.kylin.engine.spark.SparkCubingByLayer.estimateRDDPartitionNum(SourceFile:219)}}
{{ at org.apache.kylin.engine.spark.SparkCubingByLayer.execute(SourceFile:199)}}
{{ at org.apache.kylin.common.util.AbstractApplication.execute(SourceFile:37)}}
{{ ... 6 more}}

 

Found two way of calculating the level of cuboids
 * via CuboidScheduler.getBuildLevel()
 * via CuboidUtil.getLongestDepth(...)

We should settle down on one approach.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KYLIN-3337) Refactor KylinConfig.setKylinConfigThreadLocal()

[liyang (JIRA)]

liyang created KYLIN-3337:
-

 Summary: Refactor KylinConfig.setKylinConfigThreadLocal()
 Key: KYLIN-3337
 URL: https://issues.apache.org/jira/browse/KYLIN-3337
 Project: Kylin
  Issue Type: Improvement
Reporter: liyang
Assignee: liyang


Use can auto-closeable style API to replace 
\{{KylinConfig.setKylinConfigThreadLocal()}}. Remind developer that a thread 
local config must always be removed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3337) Refactor KylinConfig.setKylinConfigThreadLocal()

[liyang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liyang updated KYLIN-3337:
--
Description: Use an auto-closeable style API to replace 
{{KylinConfig.setKylinConfigThreadLocal()}}. Remind developer that a thread 
local config must always be removed.  (was: Use can auto-closeable style API to 
replace \{{KylinConfig.setKylinConfigThreadLocal()}}. Remind developer that a 
thread local config must always be removed.)

> Refactor KylinConfig.setKylinConfigThreadLocal()
> 
>
> Key: KYLIN-3337
> URL: https://issues.apache.org/jira/browse/KYLIN-3337
> Project: Kylin
>  Issue Type: Improvement
>Reporter: liyang
>Assignee: liyang
>Priority: Major
>
> Use an auto-closeable style API to replace 
> {{KylinConfig.setKylinConfigThreadLocal()}}. Remind developer that a thread 
> local config must always be removed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-2899) Enable segment level query cache

[liyang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-2899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423397#comment-16423397
 ] 

liyang commented on KYLIN-2899:
---

Very good document! Thanks Ma Gang!

Very helpful for understanding.

> Enable segment level query cache
> 
>
> Key: KYLIN-2899
> URL: https://issues.apache.org/jira/browse/KYLIN-2899
> Project: Kylin
>  Issue Type: Sub-task
>  Components: Query Engine
>Affects Versions: v2.1.0
>Reporter: Zhong Yanghong
>Assignee: Ma Gang
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3221) Some improvements for lookup table

[liyang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423395#comment-16423395
 ] 

liyang commented on KYLIN-3221:
---

+1 Nice proposal. The global snapshot table can be one way to support slowly 
changing dimension tables.

> Some improvements for lookup table 
> ---
>
> Key: KYLIN-3221
> URL: https://issues.apache.org/jira/browse/KYLIN-3221
> Project: Kylin
>  Issue Type: Improvement
>  Components: Job Engine, Metadata, Query Engine
>Reporter: Ma Gang
>Assignee: Ma Gang
>Priority: Major
>
> There are two limitations for current look table design:
>  # lookup table size is limited, because table snapshot need to be cached in 
> Kylin server, too large snapshot table will break the server.
>  # lookup table snapshot references are stored in all segments of the cube, 
> cannot support global snapshot table, the global snapshot table means when 
> the lookup table is updated, it will take effective for all segments.
> To resolve the above limitations, we decide to do some improvements for the 
> existing lookup table design, below is the initial document, any comments and 
> suggestions are welcome.
> h2. Metadata
> Will add a new property in CubeDesc to describe how lookup tables will be 
> snapshot, it can be defined during the cube design
> |{{@JsonProperty}}{{(}}{{"snapshot_table_desc_list"}}{{)}}
>  {{private}} {{List snapshotTableDescList = 
> Collections.emptyList();}}|
>  SnapshotTableDesc defines how table is stored and whether it is global or 
> not, currently we can support two types of store:
>  # "metaStore",  table snapshot is stored in the metadata store, it is the 
> same as current design, and this is the default option.
>  # "hbaseStore', table snapshot is stored in an additional hbase table.
> |{{@JsonProperty}}{{(}}{{"table_name"}}{{)}}
>  {{private}} {{String tableName;}}
>   
>  {{@JsonProperty}}{{(}}{{"store_type"}}{{)}}
>  {{private}} {{String snapshotStorageType = }}{{"metaStore"}}{{;}}
>   
>  {{@JsonProperty}}{{(}}{{"global"}}{{)}}
>  {{private}} {{boolean}} {{global = }}{{false}}{{;}}|
>  
> Add 'snapshots' property in CubeInstance, to store snapshots resource path 
> for each table, when the table snapshot is set to global in cube design:
> |{{@JsonProperty}}{{(}}{{"snapshots"}}{{)}}
>  {{private}} {{Map snapshots; }}{{// tableName -> 
> tableResoucePath mapping}}|
>  
> Add new meta model ExtTableSnapshot to describe the extended table snapshot 
> information, the information is stored in a new metastore path: 
> /ext_table_snapshot/\{tableName}/\{uuid}.snapshot, the metadata including 
> following info:
> |{{@JsonProperty}}{{(}}{{"tableName"}}{{)}}
>  {{private}} {{String tableName;}}
>   
>  {{@JsonProperty}}{{(}}{{"signature"}}{{)}}
>  {{private}} {{TableSignature signature;}}
>   
>  {{@JsonProperty}}{{(}}{{"storage_location_identifier"}}{{)}}
>  {{private}} {{String storageLocationIdentifier;}}
>   
> @JsonProperty("key_columns")
>  private String[] keyColumns;  // the key columns of the table
>  
> @JsonProperty("storage_type")
>  private String storageType;
>  
>  {{@JsonProperty}}{{(}}{{"size"}}{{)}}
>  {{private}} {{long}} {{size;}}
>   
>  {{@JsonProperty}}{{(}}{{"row_cnt"}}{{)}}
>  {{private}} {{long}} {{rowCnt;}}|
>  
> Add new section in 'Advance Setting' tab when do cube design, user can set 
> table snapshot properties for each table, and by default, it is segment level 
> and store to metadata store
> h2. Build
> If user specify 'hbaseStore' storageType for any lookup table, will use 
> MapReduce job convert the hive source table to hfiles, and then bulk load 
> hfiles to HTable. So it will add two job steps to do the lookup table 
> materialization.
> h2. HBase Lookup Table Schema
> all data are stored in raw value
> suppose the lookup table has primary keys: key1,key2
> rowkey will be:
> ||2bytes||2 bytes||len1 bytes||2 bytes||len2 bytes||
> |shard|key1 value length(len1)|key1 value|key 2 value length(len2)|key2 value|
> the first 2 bytes is shard number, HBase table can be pre-split, the shard 
> size is configurable through Kylin's properties: 
> "kylin.snapshot.ext.shard-mb", default size is 500MB.
> 1 column family c, multiple columns which column name is the index of the 
> column in the table definition
> |c|
> |1|2|...|
>  
> h2. Query
> For key lookup query, directly call hbase get api to get entire row according 
> to key.
> For queries that need fetch keys according to the derived columns, iterate 
> all rows to get related keys.
> For queries that only hit the lookup table, iterate all rows and let calcite 
> to do aggregation and filter.
> h2. Management
> For each lookup table, admin can view how many snapshots it has in Kylin, and 
> can view each snapshot type/size information and which cube/segments 

[jira] [Commented] (KYLIN-3321) Set MALLOC_ARENA_MAX in script

[liyang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423373#comment-16423373
 ] 

liyang commented on KYLIN-3321:
---

Interesting information!! Thank you Ted!

> Set MALLOC_ARENA_MAX in script
> --
>
> Key: KYLIN-3321
> URL: https://issues.apache.org/jira/browse/KYLIN-3321
> Project: Kylin
>  Issue Type: Task
>Reporter: Ted Yu
>Priority: Major
>
> conf/setenv.sh would be good place to set MALLOC_ARENA_MAX which prevents 
> native memory OOM.
> See https://github.com/prestodb/presto/issues/8993



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3001) Fix the wrong Cache key issue

[liyang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423344#comment-16423344
 ] 

liyang commented on KYLIN-3001:
---

Sorry I don't understand the patch. Replace "[ ]" with " ", replace a space 
with a space, what is the use?

> Fix the wrong Cache key issue 
> --
>
> Key: KYLIN-3001
> URL: https://issues.apache.org/jira/browse/KYLIN-3001
> Project: Kylin
>  Issue Type: Bug
>  Components: Query Engine
>Affects Versions: v2.1.0
>Reporter: Pan, Julian
>Assignee: Pan, Julian
>Priority: Major
> Fix For: v2.3.0
>
> Attachments: KYLIN_3001_Cache_key_issue.patch
>
>
> Currently the cacheKey generate by SQLRequest getCacheKey:
> cacheKey = Lists.newArrayList(sql.replaceAll("\\s+", ""), project, offset, 
> limit, acceptPartial,
> backdoorToggles);
> Which will remove all the space and break line.
> If I have a sql:
> select a, sum(c) from table 
> --where a > 0 group by a;
> and another sql:
> select a, sum(c) from table 
> --where a > 0 
> group by a;
> The two sql will share same cache key.
> But the first sql should be break. If you query second sql, and then query 
> the first again. It will return result.
> And another issue is if remove all space, which will mix a as b and aasb.
> I think the better way is just replace multiple space to one space
> Should I create a patch? Or just provide the fix:
> cacheKey = Lists.newArrayList(sql.replaceAll("[ ]", " "), project, offset, 
> limit, acceptPartial,
> backdoorToggles);



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KYLIN-3324) NegativeArraySizeException in CreateDictionaryJob$2.getDictionary()

[liyang (JIRA)]

liyang created KYLIN-3324:
-

 Summary: NegativeArraySizeException in 
CreateDictionaryJob$2.getDictionary()
 Key: KYLIN-3324
 URL: https://issues.apache.org/jira/browse/KYLIN-3324
 Project: Kylin
  Issue Type: Bug
Reporter: liyang
Assignee: liyang


During cube build, got following exception:
{noformat}
2018-01-31 09:58:14,982 ERROR [Scheduler 311988476 Job 
b931b081-3b44-462b-9bce-e15bf245f1b9-862] common.HadoopShellExecutable : error 
execute HadoopShellExecutable{id=b931b081-3b44-462b-9bce-e15bf245f1b9-03, 
name=Build Dimension Dictionary, state=RUNNING}
java.lang.NegativeArraySizeException
at 
org.apache.hadoop.io.BytesWritable.setCapacity(BytesWritable.java:144)
at org.apache.hadoop.io.BytesWritable.setSize(BytesWritable.java:123)
at org.apache.hadoop.io.BytesWritable.readFields(BytesWritable.java:179)
at 
org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:2259)
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2387)
at 
org.apache.kylin.engine.mr.steps.CreateDictionaryJob$2.getDictionary(CreateDictionaryJob.java:87)
at 
org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:62)
at 
org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:49)
at 
org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:66)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at 
org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:62)
at 
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125)
at 
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:64)
at 
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125)
at 
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:156)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745){noformat}
Root cause seems to be 
https://stackoverflow.com/questions/24127304/negativearraysizeexception-when-creating-a-sequencefile-with-large-1gb-bytesw



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KYLIN-3322) TopN requires a SUM to work

[liyang (JIRA)]

liyang created KYLIN-3322:
-

 Summary: TopN requires a SUM to work
 Key: KYLIN-3322
 URL: https://issues.apache.org/jira/browse/KYLIN-3322
 Project: Kylin
  Issue Type: Bug
Reporter: liyang
Assignee: liyang


Currently if user creates a measure of TopN seller by sum of price, it is 
required that user also creates a measure of SUM(price). Otherwise, NPE will be 
thrown at query time.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3311) Segments overlap error (refactor write conflict exception)

[liyang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liyang updated KYLIN-3311:
--
Summary: Segments overlap error (refactor write conflict exception)  (was: 
Segments overlap error)

> Segments overlap error (refactor write conflict exception)
> --
>
> Key: KYLIN-3311
> URL: https://issues.apache.org/jira/browse/KYLIN-3311
> Project: Kylin
>  Issue Type: Bug
>  Components: Metadata
>Affects Versions: v2.3.0
>Reporter: xujing
>Priority: Major
>  Labels: build
> Attachments: Segments_Overlap_ErrorLog.txt
>
>
> when "updateCubeWithRetry" method be called at first time ,
> line newSegs.validate();was passed .
> then 
> cube = crud.save(cube);seem with error throw exception
> write conflict to update cube  at try 0 ,will retry...
> while retry "updateCubeWithRetry" start
> line newSegs.validate();was not passed .
> throw exception 
> Segments overlap: [2018031800_2018031900] and 
> sales_order_channel[2018031800_201803
> 1900]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (KYLIN-3311) Segments overlap error

[liyang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16412880#comment-16412880
 ] 

liyang edited comment on KYLIN-3311 at 3/25/18 5:26 AM:


Thanks for reporting. I'm not sure of the root cause yet. It seems the first 
update was actually successful, however an IllegalStateException was thrown 
after that, caused a retry, and caused the same segment being added twice.

We need to enhance the code to produce more logs first. Then if the problem 
re-occurs, we can hunt down the root cause.


was (Author: liyang.g...@gmail.com):
Thanks for reporting. I'm not sure of the root cause yet. It seems the first 
update was actually successfully, but however an IllegalStateException was 
thrown after that, caused a retry, and caused the same segment being added 
twice.

I guess we need to enhance the code to produce more logs first. Then if the 
problem re-occurs, we can hunt down the root cause.

> Segments overlap error
> --
>
> Key: KYLIN-3311
> URL: https://issues.apache.org/jira/browse/KYLIN-3311
> Project: Kylin
>  Issue Type: Bug
>  Components: Metadata
>Affects Versions: v2.3.0
>Reporter: xujing
>Priority: Major
>  Labels: build
> Attachments: Segments_Overlap_ErrorLog.txt
>
>
> when "updateCubeWithRetry" method be called at first time ,
> line newSegs.validate();was passed .
> then 
> cube = crud.save(cube);seem with error throw exception
> write conflict to update cube  at try 0 ,will retry...
> while retry "updateCubeWithRetry" start
> line newSegs.validate();was not passed .
> throw exception 
> Segments overlap: [2018031800_2018031900] and 
> sales_order_channel[2018031800_201803
> 1900]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3311) Segments overlap error

[liyang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16412880#comment-16412880
 ] 

liyang commented on KYLIN-3311:
---

Thanks for reporting. I'm not sure of the root cause yet. It seems the first 
update was actually successfully, but however an IllegalStateException was 
thrown after that, caused a retry, and caused the same segment being added 
twice.

I guess we need to enhance the code to produce more logs first. Then if the 
problem re-occurs, we can hunt down the root cause.

> Segments overlap error
> --
>
> Key: KYLIN-3311
> URL: https://issues.apache.org/jira/browse/KYLIN-3311
> Project: Kylin
>  Issue Type: Bug
>  Components: Metadata
>Affects Versions: v2.3.0
>Reporter: xujing
>Priority: Major
>  Labels: build
> Attachments: Segments_Overlap_ErrorLog.txt
>
>
> when "updateCubeWithRetry" method be called at first time ,
> line newSegs.validate();was passed .
> then 
> cube = crud.save(cube);seem with error throw exception
> write conflict to update cube  at try 0 ,will retry...
> while retry "updateCubeWithRetry" start
> line newSegs.validate();was not passed .
> throw exception 
> Segments overlap: [2018031800_2018031900] and 
> sales_order_channel[2018031800_201803
> 1900]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3287) When a shard by column is in dict encoding, dict building error.

[liyang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395155#comment-16395155
 ] 

liyang commented on KYLIN-3287:
---

+1 patch verified

> When a shard by column is in dict encoding, dict building error.
> 
>
> Key: KYLIN-3287
> URL: https://issues.apache.org/jira/browse/KYLIN-3287
> Project: Kylin
>  Issue Type: Bug
>Reporter: jiatao.tao
>Assignee: jiatao.tao
>Priority: Major
> Attachments: image-2018-03-12-14-14-50-112.png, 
> image-2018-03-12-14-14-59-193.png, image-2018-03-12-14-15-42-046.png, 
> image-2018-03-12-14-16-38-657.png, image-2018-03-12-14-17-46-780.png
>
>
> In dict encoding.
> !image-2018-03-12-14-14-50-112.png|width=544,height=84!
> !image-2018-03-12-14-14-59-193.png|width=535,height=631!
> In int encoding
> !image-2018-03-12-14-15-42-046.png|width=544,height=87!
> !image-2018-03-12-14-16-38-657.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-2899) Enable segment level query cache

[liyang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-2899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379763#comment-16379763
 ] 

liyang commented on KYLIN-2899:
---

Segment level cache is a new idea. Can we have an elaboration of design and 
some performance comparison here?

> Enable segment level query cache
> 
>
> Key: KYLIN-2899
> URL: https://issues.apache.org/jira/browse/KYLIN-2899
> Project: Kylin
>  Issue Type: Sub-task
>  Components: Query Engine
>Affects Versions: v2.1.0
>Reporter: Zhong Yanghong
>Assignee: Ma Gang
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3002) Use Spark as default engine in web

[liyang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16377967#comment-16377967
 ] 

liyang commented on KYLIN-3002:
---

Recently verified that the measures are already supported by Spark engine. So 
the remaining gap is KYLIN-2484 mostly.

> Use Spark as default engine in web
> --
>
> Key: KYLIN-3002
> URL: https://issues.apache.org/jira/browse/KYLIN-3002
> Project: Kylin
>  Issue Type: Improvement
>  Components: Spark Engine, Web 
>Reporter: kangkaisen
>Assignee: kangkaisen
>Priority: Major
>
> After KYLIN-2997, like KYLIN-2963, we could use Spark as default engine for 
> none-global-dict cube.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KYLIN-3266) Improve CI coverage

[liyang (JIRA)]

liyang created KYLIN-3266:
-

 Summary: Improve CI coverage
 Key: KYLIN-3266
 URL: https://issues.apache.org/jira/browse/KYLIN-3266
 Project: Kylin
  Issue Type: Improvement
Reporter: liyang
Assignee: liyang


The current CI, aka BuildCubeWithEngine, only test merge on the MR engine. The 
merge operation is not tested on Spark engine. Need to improve the test 
coverage.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-2672) Only clean necessary cache for CubeMigrationCLI

[liyang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16352228#comment-16352228
 ] 

liyang commented on KYLIN-2672:
---

I rebased and merged the patch into branch KYLIN-2672. The code looks good. The 
only concern is the lack of test cases.

[~kangkaisen], could you add test cases to cover the newly added feature?

 

 

> Only clean necessary cache for CubeMigrationCLI
> ---
>
> Key: KYLIN-2672
> URL: https://issues.apache.org/jira/browse/KYLIN-2672
> Project: Kylin
>  Issue Type: Improvement
>  Components: Tools, Build and Test
>Affects Versions: v2.0.0
>Reporter: kangkaisen
>Assignee: kangkaisen
>Priority: Major
> Fix For: v2.3.0
>
> Attachments: KYLIN-2672.patch
>
>
> Currently, we simply clear ALL cache in  CubeMigrationCLI. which will make a 
> few of queries slower in prod env when we have many tables, models, cubes and 
> migrate cube often.
> So, we could only clean necessary cache for CubeMigrationCLI.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3045) Too many open files raised from new Configuration()

[liyang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351989#comment-16351989
 ] 

liyang commented on KYLIN-3045:
---

The "fixed version=2.3.0" removed.

> Too many open files raised from new Configuration()
> ---
>
> Key: KYLIN-3045
> URL: https://issues.apache.org/jira/browse/KYLIN-3045
> Project: Kylin
>  Issue Type: Bug
>Reporter: liyang
>Assignee: liyang
>Priority: Major
>  Labels: scope
>
> Try to refactor MapReduceExecutable to avoid re-creating Configuration too 
> many times.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3045) Too many open files raised from new Configuration()

[liyang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liyang updated KYLIN-3045:
--
Fix Version/s: (was: v2.3.0)

> Too many open files raised from new Configuration()
> ---
>
> Key: KYLIN-3045
> URL: https://issues.apache.org/jira/browse/KYLIN-3045
> Project: Kylin
>  Issue Type: Bug
>Reporter: liyang
>Assignee: liyang
>Priority: Major
>  Labels: scope
>
> Try to refactor MapReduceExecutable to avoid re-creating Configuration too 
> many times.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3045) Too many open files raised from new Configuration()

[liyang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351988#comment-16351988
 ] 

liyang commented on KYLIN-3045:
---

My attempt didn't work out. Seems it did not hit the root cause.

> Too many open files raised from new Configuration()
> ---
>
> Key: KYLIN-3045
> URL: https://issues.apache.org/jira/browse/KYLIN-3045
> Project: Kylin
>  Issue Type: Bug
>Reporter: liyang
>Assignee: liyang
>Priority: Major
>  Labels: scope
> Fix For: v2.3.0
>
>
> Try to refactor MapReduceExecutable to avoid re-creating Configuration too 
> many times.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KYLIN-3194) Tolerate broken job metadata caused by executable ClassNotFoundException

[liyang (JIRA)]

liyang created KYLIN-3194:
-

 Summary: Tolerate broken job metadata caused by executable 
ClassNotFoundException
 Key: KYLIN-3194
 URL: https://issues.apache.org/jira/browse/KYLIN-3194
 Project: Kylin
  Issue Type: Improvement
Reporter: liyang
Assignee: liyang


As Kylin evolves, many executable class was renamed or deprecated. These lead 
to broken job metadata and ClassNotFoundException can be thrown from many 
places in the code. Better to tolerate these error metadata once for all.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3164) HBase connection must be closed when clearing connection pool

[liyang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liyang updated KYLIN-3164:
--
Description: Currently {{HBaseConnection.clearConnCache()}} does not invoke 
{{close()}} on hbase connections. But it should.

> HBase connection must be closed when clearing connection pool
> -
>
> Key: KYLIN-3164
> URL: https://issues.apache.org/jira/browse/KYLIN-3164
> Project: Kylin
>  Issue Type: Bug
>Reporter: liyang
>Priority: Major
>
> Currently {{HBaseConnection.clearConnCache()}} does not invoke {{close()}} on 
> hbase connections. But it should.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KYLIN-3164) HBase connection must be closed when clearing connection pool

[liyang (JIRA)]

liyang created KYLIN-3164:
-

 Summary: HBase connection must be closed when clearing connection 
pool
 Key: KYLIN-3164
 URL: https://issues.apache.org/jira/browse/KYLIN-3164
 Project: Kylin
  Issue Type: Bug
Reporter: liyang






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KYLIN-3158) Metadata broadcast should only retry failed node

[liyang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liyang updated KYLIN-3158:
--
Description: In commit 
https://github.com/apache/kylin/commit/ecc01458c4ad361aaf863505884d51474a8fec9d,
 Kylin starts to retry failed metadata sync event, however it re-posts the 
failed event to all nodes. The retry SHOULD only apply to the failed node only.

> Metadata broadcast should only retry failed node
> 
>
> Key: KYLIN-3158
> URL: https://issues.apache.org/jira/browse/KYLIN-3158
> Project: Kylin
>  Issue Type: Improvement
>Reporter: liyang
>Assignee: liyang
>
> In commit 
> https://github.com/apache/kylin/commit/ecc01458c4ad361aaf863505884d51474a8fec9d,
>  Kylin starts to retry failed metadata sync event, however it re-posts the 
> failed event to all nodes. The retry SHOULD only apply to the failed node 
> only.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KYLIN-3158) Metadata broadcast should only retry failed node

[liyang (JIRA)]

liyang created KYLIN-3158:
-

 Summary: Metadata broadcast should only retry failed node
 Key: KYLIN-3158
 URL: https://issues.apache.org/jira/browse/KYLIN-3158
 Project: Kylin
  Issue Type: Improvement
Reporter: liyang
Assignee: liyang






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KYLIN-2990) Add warning window of exist model names for other project selected

[liyang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-2990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16307337#comment-16307337
 ] 

liyang commented on KYLIN-2990:
---

Note I reverted the check-dup logic in ModelService. Many direct REST API calls 
still relies on it. Please check if this impact the wanted behavior under this 
JIRA.

> Add warning window of exist model names for other project selected
> --
>
> Key: KYLIN-2990
> URL: https://issues.apache.org/jira/browse/KYLIN-2990
> Project: Kylin
>  Issue Type: Improvement
>  Components: Web 
>Reporter: peng.jianhua
>Assignee: peng.jianhua
> Fix For: v2.3.0
>
> Attachments: 
> 0001-KYLIN-2990-Add-warning-window-of-exist-model-names-f.patch, 01.PNG, 
> 02.PNG, 03.PNG
>
>
> I noticed that when we intend to build a new model and named it as  
> 'kylin_sales_model' in project 'learn_kylin'.
> we will get the prompt like 01.png.
> Cause we have already own the same name model.
> But if we build the same name model 'kylin_sales_model' in another project 
> such as project_1,
> we can't get the warning prompt until the last step. please refer to 02.png.
> I found that we just check the model in the selected project not all project.
> It's improper that we recognize the minor mistake until the final step,and we 
> should change the name in the first step and click 'next' buttun four times.
> We should offer the same prompt for other projects.
> So I modify the code in 'modelSchema.js',we can get the prompt in the first 
> step,please refer to 03.png.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KYLIN-2726) Introduce a dashboard for showing kylin service related metrics, like query count, query latency, job count, etc

[liyang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-2726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16307338#comment-16307338
 ] 

liyang commented on KYLIN-2726:
---

FYI, I reverted the {{QueryService.queryWithoutSecure()}} during a big code 
merge.
https://github.com/apache/kylin/commit/4d50b26972bb7bbaff852172990e0f189f987673

The question is can we achieve the same in a secured way?

> Introduce a dashboard for showing kylin service related metrics, like query 
> count, query latency, job count, etc
> 
>
> Key: KYLIN-2726
> URL: https://issues.apache.org/jira/browse/KYLIN-2726
> Project: Kylin
>  Issue Type: New Feature
>Reporter: Zhong Yanghong
>Assignee: Pan, Julian
> Fix For: v2.3.0
>
>
> Dashboard shows query count, query latency, job count with many dimensions, 
> like server level, project level, cube level, different time granularity



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KYLIN-2764) Build the dict for UHC column with MR

[liyang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16307336#comment-16307336
 ] 

liyang commented on KYLIN-2764:
---

Note I have renamed the config to 
"kylin.engine.mr.build-uhc-dict-in-additional-step" for better readability. FYI.

> Build the dict for UHC column with MR
> -
>
> Key: KYLIN-2764
> URL: https://issues.apache.org/jira/browse/KYLIN-2764
> Project: Kylin
>  Issue Type: Improvement
>  Components: Job Engine
>Affects Versions: v2.0.0
>Reporter: kangkaisen
>Assignee: kangkaisen
> Fix For: v2.3.0
>
> Attachments: job-memory-after.png, job-memory-before.png
>
>
> KYLIN-2217 has built dict for  normal column with MR,  but the UHC column 
> still build dict in JobServer. Like KYLIN-2217, we also could use MR build 
> dict for UHC column. which could thoroughly release the memory pressure and  
> improve job concurrent for JobServer  as well as speed up multi UHC columns 
> procedure.
> The MR input is the output of  "Extract Fact Table Distinct Columns", the MR 
> output is the UHC column dict. Because it is very hard build global dict with 
> multi reducers, I use one reducer handle one UHC column and allocate enough 
> memory to the reducer. According to my test, 8G memory is enough.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Comment Edited] (KYLIN-2764) Build the dict for UHC column with MR

[liyang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16307336#comment-16307336
 ] 

liyang edited comment on KYLIN-2764 at 1/1/18 12:17 AM:


Note I have renamed the config to 
"kylin.engine.mr.build-uhc-dict-in-additional-step" for better readability. The 
last chance before 2.3.0 releases. FYI.


was (Author: liyang.g...@gmail.com):
Note I have renamed the config to 
"kylin.engine.mr.build-uhc-dict-in-additional-step" for better readability. FYI.

> Build the dict for UHC column with MR
> -
>
> Key: KYLIN-2764
> URL: https://issues.apache.org/jira/browse/KYLIN-2764
> Project: Kylin
>  Issue Type: Improvement
>  Components: Job Engine
>Affects Versions: v2.0.0
>Reporter: kangkaisen
>Assignee: kangkaisen
> Fix For: v2.3.0
>
> Attachments: job-memory-after.png, job-memory-before.png
>
>
> KYLIN-2217 has built dict for  normal column with MR,  but the UHC column 
> still build dict in JobServer. Like KYLIN-2217, we also could use MR build 
> dict for UHC column. which could thoroughly release the memory pressure and  
> improve job concurrent for JobServer  as well as speed up multi UHC columns 
> procedure.
> The MR input is the output of  "Extract Fact Table Distinct Columns", the MR 
> output is the UHC column dict. Because it is very hard build global dict with 
> multi reducers, I use one reducer handle one UHC column and allocate enough 
> memory to the reducer. According to my test, 8G memory is enough.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KYLIN-2723) Introduce metrics collector for query & job metrics

[liyang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-2723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16307334#comment-16307334
 ] 

liyang commented on KYLIN-2723:
---

FYI. I have renamed config "kylin.core.metrics.xxx" to "kylin.metrics.xxx" for 
consistency.

> Introduce metrics collector for query & job metrics
> ---
>
> Key: KYLIN-2723
> URL: https://issues.apache.org/jira/browse/KYLIN-2723
> Project: Kylin
>  Issue Type: Sub-task
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
> Fix For: v2.3.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KYLIN-3022) Add clone() for ISegment, which is needed when updating cube segment

[liyang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16307328#comment-16307328
 ] 

liyang commented on KYLIN-3022:
---

Many issues (if not all) in KYLIN-3022 has been resolved by KYLIN-3085, which 
requires a clone at cube level before update. Sorry, but KYLIN-3022 may need a 
rework.

> Add clone() for ISegment, which is needed when updating cube segment
> 
>
> Key: KYLIN-3022
> URL: https://issues.apache.org/jira/browse/KYLIN-3022
> Project: Kylin
>  Issue Type: Improvement
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
> Attachments: APACHE-KYLIN-3022.patch
>
>
> We should not update the memory state until the storage state is updated. 
> Otherwise when it fails to update the storage state, inconsistency issue will 
> occur.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KYLIN-2986) Segments overlap issue may happen when concurrent merge or build

[liyang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-2986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16307327#comment-16307327
 ] 

liyang commented on KYLIN-2986:
---

Yes [~yaho], I just merged https://issues.apache.org/jira/browse/KYLIN-3085

Many issues (if not all) in KYLIN-3022 has been resolved by KYLIN-3085, which 
requires a clone at cube level before update. Sorry, but KYLIN-3022 may need a 
rework.

> Segments overlap issue may happen when concurrent merge or build
> 
>
> Key: KYLIN-2986
> URL: https://issues.apache.org/jira/browse/KYLIN-2986
> Project: Kylin
>  Issue Type: Bug
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
>Priority: Trivial
>  Labels: scope
> Attachments: APACHE-KYLIN-2986.patch, screenshot-1.png
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KYLIN-3059) Query got IllegalArgumentException

[liyang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16307324#comment-16307324
 ] 

liyang commented on KYLIN-3059:
---

With https://issues.apache.org/jira/browse/KYLIN-3092
I think this issue should be eased, though cannot verify since it cannot 
reproduce stably.

> Query got IllegalArgumentException
> --
>
> Key: KYLIN-3059
> URL: https://issues.apache.org/jira/browse/KYLIN-3059
> Project: Kylin
>  Issue Type: Bug
>  Components: Query Engine
>Affects Versions: v1.6.0
> Environment: apache-kylin-1.6.0-hbase1.x
> apache hadoop 2.7.2
> hbase-1.2.4
>Reporter: Xingxing Di
>Assignee: liyang
> Attachments: query.log
>
>
> A query got an exception , the error trace was :
> {code:java}
> Caused by: java.lang.IllegalArgumentException
> at 
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:76)
> at 
> org.apache.kylin.metadata.model.TblColRef.(TblColRef.java:105)
> at 
> org.apache.kylin.metadata.model.TblColRef.columnForUnknownModel(TblColRef.java:74)
> at 
> org.apache.kylin.query.relnode.OLAPTableScan.buildColumnRowType(OLAPTableScan.java:231)
> {code}
> the code is : 
> {code:java}
> TblColRef(TableRef table, ColumnDesc column) {
> checkArgument(table.getTableDesc() == column.getTable()); // this 
> line cause exception
> this.table = table;
> this.column = column;
> } 
> {code}
> and after we did "Reload metadata", query executed successful.
> I dont know what will cause this exception,  i think it may related to 
> ProjectL2Cache or ProjectManager.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KYLIN-3125) Support SparkSql in Cube building step "Create Intermediate Flat Hive Table"

[liyang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liyang resolved KYLIN-3125.
---
   Resolution: Fixed
Fix Version/s: v2.3.0

> Support SparkSql in Cube building step "Create Intermediate Flat Hive Table"
> 
>
> Key: KYLIN-3125
> URL: https://issues.apache.org/jira/browse/KYLIN-3125
> Project: Kylin
>  Issue Type: Improvement
>Reporter: liyang
>Assignee: liyang
> Fix For: v2.3.0
>
>
> It is said that SparkSQL is much faster then Hive when used to create flat 
> table.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KYLIN-3125) Support SparkSql in Cube building step "Create Intermediate Flat Hive Table"

[liyang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16307320#comment-16307320
 ] 

liyang commented on KYLIN-3125:
---

Code merge done.
https://github.com/apache/kylin/commit/4d50b26972bb7bbaff852172990e0f189f987673

> Support SparkSql in Cube building step "Create Intermediate Flat Hive Table"
> 
>
> Key: KYLIN-3125
> URL: https://issues.apache.org/jira/browse/KYLIN-3125
> Project: Kylin
>  Issue Type: Improvement
>Reporter: liyang
>Assignee: liyang
>
> It is said that SparkSQL is much faster then Hive when used to create flat 
> table.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KYLIN-3092) Synchronize read/write operations on Managers

[liyang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liyang updated KYLIN-3092:
--
Description: To prevent abnormal behavior when reading cache (those getXXX 
methods) and writing cache (broadcast reload) happens at the same time in 
managers.

> Synchronize read/write operations on Managers
> -
>
> Key: KYLIN-3092
> URL: https://issues.apache.org/jira/browse/KYLIN-3092
> Project: Kylin
>  Issue Type: Improvement
>Reporter: liyang
>Assignee: liyang
> Fix For: v2.3.0
>
>
> To prevent abnormal behavior when reading cache (those getXXX methods) and 
> writing cache (broadcast reload) happens at the same time in managers.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KYLIN-3092) Synchronize read/write operations on Managers

[liyang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16307319#comment-16307319
 ] 

liyang commented on KYLIN-3092:
---

Code merge done.
https://github.com/apache/kylin/commit/4d50b26972bb7bbaff852172990e0f189f987673

> Synchronize read/write operations on Managers
> -
>
> Key: KYLIN-3092
> URL: https://issues.apache.org/jira/browse/KYLIN-3092
> Project: Kylin
>  Issue Type: Improvement
>Reporter: liyang
>Assignee: liyang
> Fix For: v2.3.0
>
>
> To prevent abnormal behavior when reading cache (those getXXX methods) and 
> writing cache (broadcast reload) happens at the same time in managers.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KYLIN-3092) Synchronize read/write operations on Managers

[liyang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liyang resolved KYLIN-3092.
---
   Resolution: Fixed
 Assignee: liyang
Fix Version/s: v2.3.0

> Synchronize read/write operations on Managers
> -
>
> Key: KYLIN-3092
> URL: https://issues.apache.org/jira/browse/KYLIN-3092
> Project: Kylin
>  Issue Type: Improvement
>Reporter: liyang
>Assignee: liyang
> Fix For: v2.3.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KYLIN-3090) Refactor to consolidate all caches and managers under KylinConfig

[liyang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liyang resolved KYLIN-3090.
---
   Resolution: Fixed
 Assignee: liyang
Fix Version/s: v2.3.0

> Refactor to consolidate all caches and managers under KylinConfig
> -
>
> Key: KYLIN-3090
> URL: https://issues.apache.org/jira/browse/KYLIN-3090
> Project: Kylin
>  Issue Type: Improvement
>Reporter: liyang
>Assignee: liyang
> Fix For: v2.3.0
>
>
> Refactor to consolidate all caches and managers under KylinConfig.
> Such that there is a single place to clear (and GC) all cache under a 
> KylinConfig.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KYLIN-3090) Refactor to consolidate all caches and managers under KylinConfig

[liyang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16307316#comment-16307316
 ] 

liyang commented on KYLIN-3090:
---

Code merge done.
https://github.com/apache/kylin/commit/4d50b26972bb7bbaff852172990e0f189f987673

> Refactor to consolidate all caches and managers under KylinConfig
> -
>
> Key: KYLIN-3090
> URL: https://issues.apache.org/jira/browse/KYLIN-3090
> Project: Kylin
>  Issue Type: Improvement
>Reporter: liyang
>
> Refactor to consolidate all caches and managers under KylinConfig.
> Such that there is a single place to clear (and GC) all cache under a 
> KylinConfig.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KYLIN-3085) CubeManager.updateCube() must not update the cached CubeInstance

[liyang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liyang resolved KYLIN-3085.
---
   Resolution: Fixed
Fix Version/s: v2.3.0

> CubeManager.updateCube() must not update the cached CubeInstance
> 
>
> Key: KYLIN-3085
> URL: https://issues.apache.org/jira/browse/KYLIN-3085
> Project: Kylin
>  Issue Type: Bug
>Reporter: liyang
>Assignee: liyang
> Fix For: v2.3.0
>
>
> Currently CubeManager.updateCube() modifies the pass-in CubeInstance which is 
> the cache shared object. Causes metadata inconsistency during the period of 
> update, and may leave dirty metadata in memory if update fails.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KYLIN-3085) CubeManager.updateCube() must not update the cached CubeInstance

[liyang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16307315#comment-16307315
 ] 

liyang commented on KYLIN-3085:
---

Code merge done.
https://github.com/apache/kylin/commit/4d50b26972bb7bbaff852172990e0f189f987673

> CubeManager.updateCube() must not update the cached CubeInstance
> 
>
> Key: KYLIN-3085
> URL: https://issues.apache.org/jira/browse/KYLIN-3085
> Project: Kylin
>  Issue Type: Bug
>Reporter: liyang
>Assignee: liyang
>
> Currently CubeManager.updateCube() modifies the pass-in CubeInstance which is 
> the cache shared object. Causes metadata inconsistency during the period of 
> update, and may leave dirty metadata in memory if update fails.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KYLIN-3040) Refresh a non-partitioned cube changes the segment name to "19700101000000_2922789940817071255"

[liyang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KYLIN-3040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16307314#comment-16307314
 ] 

liyang commented on KYLIN-3040:
---

Code merge done.
https://github.com/apache/kylin/commit/4d50b26972bb7bbaff852172990e0f189f987673

> Refresh a non-partitioned cube changes the segment name to 
> "1970010100_2922789940817071255"
> ---
>
> Key: KYLIN-3040
> URL: https://issues.apache.org/jira/browse/KYLIN-3040
> Project: Kylin
>  Issue Type: Bug
>Reporter: liyang
>Assignee: liyang
> Fix For: v2.3.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KYLIN-3040) Refresh a non-partitioned cube changes the segment name to "19700101000000_2922789940817071255"

[liyang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liyang resolved KYLIN-3040.
---
Resolution: Fixed

> Refresh a non-partitioned cube changes the segment name to 
> "1970010100_2922789940817071255"
> ---
>
> Key: KYLIN-3040
> URL: https://issues.apache.org/jira/browse/KYLIN-3040
> Project: Kylin
>  Issue Type: Bug
>Reporter: liyang
>Assignee: liyang
> Fix For: v2.3.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KYLIN-3040) Refresh a non-partitioned cube changes the segment name to "19700101000000_2922789940817071255"

[liyang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-3040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liyang updated KYLIN-3040:
--
Fix Version/s: v2.3.0

> Refresh a non-partitioned cube changes the segment name to 
> "1970010100_2922789940817071255"
> ---
>
> Key: KYLIN-3040
> URL: https://issues.apache.org/jira/browse/KYLIN-3040
> Project: Kylin
>  Issue Type: Bug
>Reporter: liyang
>Assignee: liyang
> Fix For: v2.3.0
>
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Resolved] (KYLIN-2991) Query hit NumberFormatException if partitionDateFormat is not yyyy-MM-dd

[liyang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KYLIN-2991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

liyang resolved KYLIN-2991.
---
Resolution: Fixed

> Query hit NumberFormatException if partitionDateFormat is not -MM-dd
> 
>
> Key: KYLIN-2991
> URL: https://issues.apache.org/jira/browse/KYLIN-2991
> Project: Kylin
>  Issue Type: Bug
>  Components: Query Engine
>Affects Versions: v2.2.0
>Reporter: liyang
>Assignee: liyang
> Fix For: v2.3.0
>
>
> For example, partition date column is {{MONTH_ID}}, partition date format is 
> {{MMdd}}, query can hit following exception:
> {Code}
> Caused by: java.lang.NumberFormatException: For input string: "2016-01-01"
> at 
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> at java.lang.Long.parseLong(Long.java:589)
> at java.lang.Long.parseLong(Long.java:631)
> at org.apache.kylin.dimension.IntegerDimEnc.encode(IntegerDimEnc.java:99)
> at 
> org.apache.kylin.dimension.IntegerDimEnc$IntegerSerializer.serialize(IntegerDimEnc.java:153)
> at 
> org.apache.kylin.cube.gridtable.CubeCodeSystem.encodeColumnValue(CubeCodeSystem.java:135)
> at 
> org.apache.kylin.cube.gridtable.SegmentGTStartAndEnd.encodeTime(SegmentGTStartAndEnd.java:86)
> at 
> org.apache.kylin.cube.gridtable.SegmentGTStartAndEnd.getSegmentStartAndEnd(SegmentGTStartAndEnd.java:51)
> at 
> org.apache.kylin.storage.gtrecord.CubeScanRangePlanner.(CubeScanRangePlanner.java:114)
> at 
> org.apache.kylin.storage.gtrecord.CubeSegmentScanner.(CubeSegmentScanner.java:73)
> at org.apache.kylin.storage.gtrecord.GTC
> {Code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)