[jira] [Commented] (KYLIN-5267) force limit to protect the service when SQL has no limit
[ https://issues.apache.org/jira/browse/KYLIN-5267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17612011#comment-17612011 ] liyang commented on KYLIN-5267: --- If what you need is something beyond the above two configs, feel free to propose and PR. > force limit to protect the service when SQL has no limit > > > Key: KYLIN-5267 > URL: https://issues.apache.org/jira/browse/KYLIN-5267 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Affects Versions: v4.0.1 >Reporter: Liu Zhao >Priority: Minor > > Kylin4 queries use Spark, and the result data will be sent to the driver > (that is, query service). If the user's SQL returns a large data set, it may > cause the query oom to be unavailable. To solve this problem, can we provide > a means for users to enforce limit when SQL has no limit to protect services. > I am here https://issues.apache.org/jira/browse/KYLIN-2649 You can see the > idea above, but can reduce the limit on the SQL structure？ -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KYLIN-5268) kylin4.0.1服务每天都会内存溢出java.lang.OutOfMemoryError: Java heap space
[
https://issues.apache.org/jira/browse/KYLIN-5268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17612010#comment-17612010
]
liyang commented on KYLIN-5268:
---
> 如果tableau发出的奇怪sql不多固定，似乎修改源码的 org.apache.kylin.rest.util.TableauInterceptor
> 类更合理
+1
> kylin4.0.1服务每天都会内存溢出java.lang.OutOfMemoryError: Java heap space
> ---
>
> Key: KYLIN-5268
> URL: https://issues.apache.org/jira/browse/KYLIN-5268
> Project: Kylin
> Issue Type: Bug
>Reporter: zhenkuan_zhang
>Priority: Major
> Attachments: kylin_query.log
>
>
> Hi，最近发现我们的kylin服务非常不稳定，三个节点，每天都会有节点出现内存溢出的状况，jvm从16G调到48G也同样会内存溢出。通过分析Dump 出
> java_pid34184.hprof文件，发现内存会被两个查询完全占满导致jvm内存溢出，sql是通过tableau发送来的。sql如下（该事实表行数上亿）：
> 1.
> SELECT "自訂 SQL 查詢"."FAIL" AS "FAIL" FROM (
> SELECT DT TEST_DATE,TEST_STATION_CODE,SUBSTRING(TEST_LINE,1,6)
> TEST_FLOOR,TEST_LINE,CONFIG,REGION,LOCALLIZATION,
> SUBSTRING(ATL FROM 1 for POSITION('*-' IN ATL from 1)-1 )
> VENDERATL,SUBSTRING(ATL FROM POSITION('-' IN ATL FROM 5)+1 FOR 3)
> WEEKATL,SUBSTRING(ATL FROM POSITION('-' IN ATL FROM 6)+1 FOR 4)
> DATEATL,SUBSTRING(ATL FROM POSITION('*-' IN ATL FROM 1)+7 FOR 4) ATL ,
> INPUT,FAIL,RETEST FROM BI_DW.KPY_FACT_SUM_DATA
> ) "自訂 SQL 查詢"
> 2.
> SELECT "自訂 SQL 查詢"."RETEST" AS "RETEST" FROM (
> SELECT DT TEST_DATE,TEST_STATION_CODE,SUBSTRING(TEST_LINE,1,6)
> TEST_FLOOR,TEST_LINE,CONFIG,REGION,LOCALLIZATION,
> SUBSTRING(ATL FROM 1 for POSITION('*-' IN ATL from 1)-1 )
> VENDERATL,SUBSTRING(ATL FROM POSITION('-' IN ATL FROM 5)+1 FOR 3)
> WEEKATL,SUBSTRING(ATL FROM POSITION('-' IN ATL FROM 6)+1 FOR 4)
> DATEATL,SUBSTRING(ATL FROM POSITION('*-' IN ATL FROM 1)+7 FOR 4) ATL ,
> INPUT,FAIL,RETEST FROM BI_DW.KPY_FACT_SUM_DATA
> ) "自訂 SQL 查詢"
> 一方面我们在与tableau方联系，沟通为什么会发出这么奇怪的sql。
> 另一方面希望得到kylin的支持，看看有没有什么解决方法，
> 比如kylin.query.force-limit=1000 可以为 select * from table 加上 limit 1000，有没有办法限制
> select a from table。
> 或者有没有其他的解决方法。
> 期待回复，十分感谢！！！
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KYLIN-5267) force limit to protect the service when SQL has no limit
[ https://issues.apache.org/jira/browse/KYLIN-5267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17612009#comment-17612009 ] liyang commented on KYLIN-5267: --- As a result of KYLIN-2649, a config parameter called "kylin.query.force-limit" is already added. Does that help your use case? Also "kylin.query.scan-threshold" is another config parameter that is often used to limit big queries going too wild. > force limit to protect the service when SQL has no limit > > > Key: KYLIN-5267 > URL: https://issues.apache.org/jira/browse/KYLIN-5267 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Affects Versions: v4.0.1 >Reporter: Liu Zhao >Priority: Minor > > Kylin4 queries use Spark, and the result data will be sent to the driver > (that is, query service). If the user's SQL returns a large data set, it may > cause the query oom to be unavailable. To solve this problem, can we provide > a means for users to enforce limit when SQL has no limit to protect services. > I am here https://issues.apache.org/jira/browse/KYLIN-2649 You can see the > idea above, but can reduce the limit on the SQL structure？ -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KYLIN-5272) The "case when" command may cause the field value to be different from the user's expectation.
[ https://issues.apache.org/jira/browse/KYLIN-5272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17612006#comment-17612006 ] liyang commented on KYLIN-5272: --- Could you provide a complete SQL and related table structure, such that others can reproduce the issue and then be able to help? Also likely, without knowing the exact problem, the issue maybe fixed in Kylin 4.x as all calculations are moved into Spark. Calcite is only used for SQL parsing. > The "case when" command may cause the field value to be different from the > user's expectation. > -- > > Key: KYLIN-5272 > URL: https://issues.apache.org/jira/browse/KYLIN-5272 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: v2.2.0, v2.6.3 >Reporter: GuKe >Priority: Major > Attachments: issue.jpg > > > When using "Case then" to set a fixed character value for a field, Kylin uses > Spaces to complete the length of string to 32. This will cause the field to > display characters that are actually different from the user's expected > values, causing query errors, Example a failed Join operation. -- This message was sent by Atlassian Jira (v8.20.10#820010)
