[jira] [Created] (KYLIN-5790) Security of kafka-clients
pengfei.zhan created KYLIN-5790: --- Summary: Security of kafka-clients Key: KYLIN-5790 URL: https://issues.apache.org/jira/browse/KYLIN-5790 Project: Kylin Issue Type: Bug Components: Query Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0-beta -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5789) Clean sparder history and spark history automatically
pengfei.zhan created KYLIN-5789: --- Summary: Clean sparder history and spark history automatically Key: KYLIN-5789 URL: https://issues.apache.org/jira/browse/KYLIN-5789 Project: Kylin Issue Type: Bug Components: Job Engine, Query Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0-beta -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5788) Enhance global dict on flat table encoding stage logging & retry
pengfei.zhan created KYLIN-5788: --- Summary: Enhance global dict on flat table encoding stage logging & retry Key: KYLIN-5788 URL: https://issues.apache.org/jira/browse/KYLIN-5788 Project: Kylin Issue Type: Bug Components: Job Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0-beta -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5787) Use t-digest as spark percentile_approx function
pengfei.zhan created KYLIN-5787: --- Summary: Use t-digest as spark percentile_approx function Key: KYLIN-5787 URL: https://issues.apache.org/jira/browse/KYLIN-5787 Project: Kylin Issue Type: Improvement Components: Job Engine, Query Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0-beta -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5786) Add a write lock when merging v3 dict
pengfei.zhan created KYLIN-5786: --- Summary: Add a write lock when merging v3 dict Key: KYLIN-5786 URL: https://issues.apache.org/jira/browse/KYLIN-5786 Project: Kylin Issue Type: Bug Components: Job Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0-beta -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5785) Support implicit query of split function
pengfei.zhan created KYLIN-5785: --- Summary: Support implicit query of split function Key: KYLIN-5785 URL: https://issues.apache.org/jira/browse/KYLIN-5785 Project: Kylin Issue Type: Bug Components: Query Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0-beta -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5784) Optimization for query history filters
pengfei.zhan created KYLIN-5784: --- Summary: Optimization for query history filters Key: KYLIN-5784 URL: https://issues.apache.org/jira/browse/KYLIN-5784 Project: Kylin Issue Type: Improvement Components: Query Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0-beta KYLIN-5763 related -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5783) Round query result of decimal zero by config
pengfei.zhan created KYLIN-5783: --- Summary: Round query result of decimal zero by config Key: KYLIN-5783 URL: https://issues.apache.org/jira/browse/KYLIN-5783 Project: Kylin Issue Type: Improvement Components: Query Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0-beta -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5782) Duplicated join keys result empty query result
pengfei.zhan created KYLIN-5782: --- Summary: Duplicated join keys result empty query result Key: KYLIN-5782 URL: https://issues.apache.org/jira/browse/KYLIN-5782 Project: Kylin Issue Type: Bug Components: Query Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0-beta -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5781) Adjust the query result of union order
pengfei.zhan created KYLIN-5781: --- Summary: Adjust the query result of union order Key: KYLIN-5781 URL: https://issues.apache.org/jira/browse/KYLIN-5781 Project: Kylin Issue Type: Improvement Components: Query Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0-beta -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5780) Dimension capacity does not work
pengfei.zhan created KYLIN-5780: --- Summary: Dimension capacity does not work Key: KYLIN-5780 URL: https://issues.apache.org/jira/browse/KYLIN-5780 Project: Kylin Issue Type: Bug Components: Modeling Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0-beta -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5779) Without project permission，calling "kylin/api/access/acls" api occurs NPE
pengfei.zhan created KYLIN-5779: --- Summary: Without project permission，calling "kylin/api/access/acls" api occurs NPE Key: KYLIN-5779 URL: https://issues.apache.org/jira/browse/KYLIN-5779 Project: Kylin Issue Type: Bug Components: Query Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0-beta -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5778) Upgrade spark: parquet footer read cache
pengfei.zhan created KYLIN-5778: --- Summary: Upgrade spark: parquet footer read cache Key: KYLIN-5778 URL: https://issues.apache.org/jira/browse/KYLIN-5778 Project: Kylin Issue Type: Improvement Components: Job Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0-beta -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5776) Support parameter type is string for timestampadd
pengfei.zhan created KYLIN-5776: --- Summary: Support parameter type is string for timestampadd Key: KYLIN-5776 URL: https://issues.apache.org/jira/browse/KYLIN-5776 Project: Kylin Issue Type: Bug Components: Query Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0-beta -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5777) Change spark.yarn.submit.file.replication from 1 to 3
pengfei.zhan created KYLIN-5777: --- Summary: Change spark.yarn.submit.file.replication from 1 to 3 Key: KYLIN-5777 URL: https://issues.apache.org/jira/browse/KYLIN-5777 Project: Kylin Issue Type: Bug Components: Job Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0-beta -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5775) Upgrade testcontainers to version 1.16.2
pengfei.zhan created KYLIN-5775: --- Summary: Upgrade testcontainers to version 1.16.2 Key: KYLIN-5775 URL: https://issues.apache.org/jira/browse/KYLIN-5775 Project: Kylin Issue Type: Bug Components: Others Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0-beta -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5774) Optimize Calcite plan to convert spark logical plan
pengfei.zhan created KYLIN-5774: --- Summary: Optimize Calcite plan to convert spark logical plan Key: KYLIN-5774 URL: https://issues.apache.org/jira/browse/KYLIN-5774 Project: Kylin Issue Type: Improvement Components: Query Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0-beta -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KYLIN-5765) Optimization of registering udf operators
[ https://issues.apache.org/jira/browse/KYLIN-5765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pengfei.zhan updated KYLIN-5765: Summary: Optimization of registering udf operators (was: Optimization of register udf operators) > Optimization of registering udf operators > - > > Key: KYLIN-5765 > URL: https://issues.apache.org/jira/browse/KYLIN-5765 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Affects Versions: 5.0-beta >Reporter: pengfei.zhan >Assignee: pengfei.zhan >Priority: Major > Fix For: 5.0.0 > > Attachments: 42118-2.png, 42118.png > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KYLIN-5772) Fix vlunerability and excluding ekio
[ https://issues.apache.org/jira/browse/KYLIN-5772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832366#comment-17832366 ] pengfei.zhan commented on KYLIN-5772: - caused by [https://snyk.io/vuln/SNYK-JAVA-COMSQUAREUPOKIO-5773320] upgrade & excluding some package to solve this problem. > Fix vlunerability and excluding ekio > > > Key: KYLIN-5772 > URL: https://issues.apache.org/jira/browse/KYLIN-5772 > Project: Kylin > Issue Type: Bug > Components: Others >Affects Versions: 5.0-beta >Reporter: pengfei.zhan >Assignee: pengfei.zhan >Priority: Major > Fix For: 5.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KYLIN-5771) Query cannot be interrupted during Segment pruning phase
[ https://issues.apache.org/jira/browse/KYLIN-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832364#comment-17832364 ] pengfei.zhan commented on KYLIN-5771: - h1. Design Add timeout detection, directly terminate the query if timeout at the segment pruning stage. > Query cannot be interrupted during Segment pruning phase > > > Key: KYLIN-5771 > URL: https://issues.apache.org/jira/browse/KYLIN-5771 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: 5.0-beta >Reporter: pengfei.zhan >Assignee: pengfei.zhan >Priority: Major > Fix For: 5.0.0 > > > When SQL is in the planning stage and pruning segment optimization is > performed, it may cause the query to cross the expected 300s and trigger the > timeout cancellation logic due to the existence of a large number of > segments, and a large number of filter expressions for simplification issues. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KYLIN-5771) Query cannot be interrupted during Segment pruning phase
[
https://issues.apache.org/jira/browse/KYLIN-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832363#comment-17832363
]
pengfei.zhan commented on KYLIN-5771:
-
How to reproduce this problem? Add a large number of Segments to a model, for
example, 5000; execute the query SQL and SQL with the filter condition of time
partition column; at the same time, set the query timeout about 10 seconds, you
can observe that the log execution prunning setgments process will not be
interrupted, and will continuously output the following log:
{code:java}
logger.info("Find candidates by table {} and project={} : {}",
olapContext.firstTableScan.getTableName(),
olapContext.olapSchema.getProjectName(), candidate);
log.info("Scan segment.size: {} after segment pruning",
selectedSegments.size()); {code}
> Query cannot be interrupted during Segment pruning phase
>
>
> Key: KYLIN-5771
> URL: https://issues.apache.org/jira/browse/KYLIN-5771
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: 5.0-beta
>Reporter: pengfei.zhan
>Assignee: pengfei.zhan
>Priority: Major
> Fix For: 5.0.0
>
>
> When SQL is in the planning stage and pruning segment optimization is
> performed, it may cause the query to cross the expected 300s and trigger the
> timeout cancellation logic due to the existence of a large number of
> segments, and a large number of filter expressions for simplification issues.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KYLIN-5771) Query cannot be interrupted during Segment pruning phase
[ https://issues.apache.org/jira/browse/KYLIN-5771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pengfei.zhan updated KYLIN-5771: Description: When SQL is in the planning stage and pruning segment optimization is performed, it may cause the query to cross the expected 300s and trigger the timeout cancellation logic due to the existence of a large number of segments, and a large number of filter expressions for simplification issues. > Query cannot be interrupted during Segment pruning phase > > > Key: KYLIN-5771 > URL: https://issues.apache.org/jira/browse/KYLIN-5771 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: 5.0-beta >Reporter: pengfei.zhan >Assignee: pengfei.zhan >Priority: Major > Fix For: 5.0.0 > > > When SQL is in the planning stage and pruning segment optimization is > performed, it may cause the query to cross the expected 300s and trigger the > timeout cancellation logic due to the existence of a large number of > segments, and a large number of filter expressions for simplification issues. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KYLIN-5770) Update spring-security-config from 5.7.8 to 5.7.10
[
https://issues.apache.org/jira/browse/KYLIN-5770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832361#comment-17832361
]
pengfei.zhan commented on KYLIN-5770:
-
CVE-2023-34034 spring-security-config bugfix
Affected releases：
* 6.1.0 to 6.1.1
* 6.0.0 to 6.0.4
* 5.8.0 to 5.8.4
* 5.7.0 to 5.7.9
* 5.6.0 to 5.6.11
{quote}[WARNING] Using Maven 2 dependency tree to get verbose output, which may
be inconsistent with actual Maven 3 resolution
[INFO] org.apache.kylin:kylin-common-booter:jar:5.0.0-SNAPSHOT
[INFO] \- org.apache.kylin:kylin-common-server:jar:5.0.0-SNAPSHOT:compile
[INFO] \- org.apache.kylin:kylin-query-service:jar:5.0.0-SNAPSHOT:compile
[INFO] \- org.apache.kylin:kylin-common-service:jar:5.0.0-SNAPSHOT:compile
[INFO] \- org.springframework.security:spring-security-config:jar:5.7.8:compile
{quote}
> Update spring-security-config from 5.7.8 to 5.7.10
> --
>
> Key: KYLIN-5770
> URL: https://issues.apache.org/jira/browse/KYLIN-5770
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: 5.0-beta
>Reporter: pengfei.zhan
>Assignee: pengfei.zhan
>Priority: Major
> Fix For: 5.0.0
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (KYLIN-5769) Support split function
[ https://issues.apache.org/jira/browse/KYLIN-5769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832360#comment-17832360 ] pengfei.zhan edited comment on KYLIN-5769 at 3/30/24 2:48 AM: -- h1. Design The split function is supported, see [https://spark.apache.org/docs/3.2.4/api/sql/index.html] , for the two-parameter and three-parameter cases. It is mainly for type and data conversion processing. * The array obtained from the downward-pressing query supports type and data conversion. * min/max metadata query for array data conversion. * Conversion of array by hit aggregate index / detail index. * Handling the above cases naturally supports computable column definitions. was (Author: JIRAUSER294653): h1. Design The split function is supported, see [https://spark.apache.org/docs/3.2.4/api/sql/index.html] , for the two-parameter and three-parameter cases. It is mainly for type and data conversion processing. * The array obtained from the downward-pressing query supports type and data conversion. * min/max metadata query for array data conversion. * Conversion of array by hit aggregate index / detail index. * Handling the above cases naturally supports computable column definitions. > Support split function > -- > > Key: KYLIN-5769 > URL: https://issues.apache.org/jira/browse/KYLIN-5769 > Project: Kylin > Issue Type: Improvement > Components: Job Engine, Query Engine >Affects Versions: 5.0-beta >Reporter: pengfei.zhan >Assignee: pengfei.zhan >Priority: Major > Fix For: 5.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KYLIN-5769) Support split function
[ https://issues.apache.org/jira/browse/KYLIN-5769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832360#comment-17832360 ] pengfei.zhan commented on KYLIN-5769: - h1. Design The split function is supported, see Spark SQL, Built-in Functions, for the two-parameter and three-parameter cases. It is mainly for type and data conversion processing. * The array obtained from the downward-pressing query supports type and data conversion. * min/max metadata query for array data conversion. * Conversion of array by hit aggregate index / detail index. * Handling the above cases naturally supports computable column definitions. > Support split function > -- > > Key: KYLIN-5769 > URL: https://issues.apache.org/jira/browse/KYLIN-5769 > Project: Kylin > Issue Type: Improvement > Components: Job Engine, Query Engine >Affects Versions: 5.0-beta >Reporter: pengfei.zhan >Assignee: pengfei.zhan >Priority: Major > Fix For: 5.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (KYLIN-5769) Support split function
[ https://issues.apache.org/jira/browse/KYLIN-5769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832360#comment-17832360 ] pengfei.zhan edited comment on KYLIN-5769 at 3/30/24 2:47 AM: -- h1. Design The split function is supported, see [https://spark.apache.org/docs/3.2.4/api/sql/index.html] , for the two-parameter and three-parameter cases. It is mainly for type and data conversion processing. * The array obtained from the downward-pressing query supports type and data conversion. * min/max metadata query for array data conversion. * Conversion of array by hit aggregate index / detail index. * Handling the above cases naturally supports computable column definitions. was (Author: JIRAUSER294653): h1. Design The split function is supported, see Spark SQL, Built-in Functions, for the two-parameter and three-parameter cases. It is mainly for type and data conversion processing. * The array obtained from the downward-pressing query supports type and data conversion. * min/max metadata query for array data conversion. * Conversion of array by hit aggregate index / detail index. * Handling the above cases naturally supports computable column definitions. > Support split function > -- > > Key: KYLIN-5769 > URL: https://issues.apache.org/jira/browse/KYLIN-5769 > Project: Kylin > Issue Type: Improvement > Components: Job Engine, Query Engine >Affects Versions: 5.0-beta >Reporter: pengfei.zhan >Assignee: pengfei.zhan >Priority: Major > Fix For: 5.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KYLIN-5768) Refine some code for localcache
[ https://issues.apache.org/jira/browse/KYLIN-5768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832359#comment-17832359 ] pengfei.zhan commented on KYLIN-5768: - h1. Design Code adjustments to ensure compatibility with the commercial version. > Refine some code for localcache > --- > > Key: KYLIN-5768 > URL: https://issues.apache.org/jira/browse/KYLIN-5768 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Affects Versions: 5.0-beta >Reporter: pengfei.zhan >Assignee: pengfei.zhan >Priority: Major > Fix For: 5.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KYLIN-5768) Refine some code for localcache
[
https://issues.apache.org/jira/browse/KYLIN-5768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832358#comment-17832358
]
pengfei.zhan commented on KYLIN-5768:
-
KylinCacheFileSystem is the key class for enabling localcache, and is
internally a proxy for DistributedFileSystem in HDFS scenarios.
However, at runtime, due to the FileSystem CACHE, the initialized class is not
KylinCacheFileSystem, so localcache does not take effect as expected.
Configuring KylinCacheFileSystem only in kylin.properties.override does not
take effect, as followsKylinCacheFileSystem is the key class for enabling
localcache, and is internally a proxy for DistributedFileSystem in HDFS
scenarios.
However, at runtime, due to the FileSystem CACHE, the initialized class is not
KylinCacheFileSystem, so localcache does not take effect as expected.
Configuring KylinCacheFileSystem only in kylin.properties.override does not
take effect, as follows
{code:java}
kylin.storage.columnar.spark-conf.spark.hadoop.fs.hdfs.impl=io.kyligence.kap.cache.kylin.KylinCacheFileSystem
{code}
> Refine some code for localcache
> ---
>
> Key: KYLIN-5768
> URL: https://issues.apache.org/jira/browse/KYLIN-5768
> Project: Kylin
> Issue Type: Improvement
> Components: Query Engine
>Affects Versions: 5.0-beta
>Reporter: pengfei.zhan
>Assignee: pengfei.zhan
>Priority: Major
> Fix For: 5.0.0
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (KYLIN-5767) Calculating total rows abnormal when jdbc datasource is connnected
[
https://issues.apache.org/jira/browse/KYLIN-5767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832357#comment-17832357
]
pengfei.zhan edited comment on KYLIN-5767 at 3/30/24 2:37 AM:
--
h1. Design
add the method `getCountData` to
org.apache.kylin.source.jdbc.ISourceConnector, put the execution of sql `select
count \* from table` to the datasource.
was (Author: JIRAUSER294653):
h1. Design
add the method `getCountData` to
org.apache.kylin.source.jdbc.ISourceConnector, put the execution of sql `select
count(*) from table` to the datasource.
> Calculating total rows abnormal when jdbc datasource is connnected
> --
>
> Key: KYLIN-5767
> URL: https://issues.apache.org/jira/browse/KYLIN-5767
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: 5.0-beta
>Reporter: pengfei.zhan
>Assignee: pengfei.zhan
>Priority: Major
> Fix For: 5.0.0
>
>
> {{When the JDBC data source is connected, the snapshot management function is
> enabled and the dimension table is not sampled, optimize the build logic to
> ensure that the job can be executed normally when the dimension table data
> volume is large}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KYLIN-5767) Calculating total rows abnormal when jdbc datasource is connnected
[
https://issues.apache.org/jira/browse/KYLIN-5767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832357#comment-17832357
]
pengfei.zhan commented on KYLIN-5767:
-
h1. Design
add the method `getCountData` to
org.apache.kylin.source.jdbc.ISourceConnector, put the execution of sql `select
count(*) from table` to the datasource.
> Calculating total rows abnormal when jdbc datasource is connnected
> --
>
> Key: KYLIN-5767
> URL: https://issues.apache.org/jira/browse/KYLIN-5767
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: 5.0-beta
>Reporter: pengfei.zhan
>Assignee: pengfei.zhan
>Priority: Major
> Fix For: 5.0.0
>
>
> {{When the JDBC data source is connected, the snapshot management function is
> enabled and the dimension table is not sampled, optimize the build logic to
> ensure that the job can be executed normally when the dimension table data
> volume is large}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KYLIN-5767) Calculating total rows abnormal when jdbc datasource is connnected
[
https://issues.apache.org/jira/browse/KYLIN-5767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832356#comment-17832356
]
pengfei.zhan commented on KYLIN-5767:
-
h1. Problem
With Snapshot Management turned on, the Model Snapshot Build task did not skip
the Build Snapshot step. KYLIN will get the table sampling information
according to "tableManager.getTableExtIfExists(tableDesc)". If the tables's
sampling information is empty or the number of rows sampled is equal to 0, it
will calculate the total rows. This step will be queried at the JDBC data
source level by the way of "select *". However, if table from the customer
environment is too large, the build stage in this step will spent too much
time. Usually, the large dimension data and the partition column is null may
lead to this situation.
If the sampling information is empty or the number of rows sampled is equal to
0, then the total rows will be calculated. This step will be queried at the
JDBC data source level through the "select *" method.
> Calculating total rows abnormal when jdbc datasource is connnected
> --
>
> Key: KYLIN-5767
> URL: https://issues.apache.org/jira/browse/KYLIN-5767
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: 5.0-beta
>Reporter: pengfei.zhan
>Assignee: pengfei.zhan
>Priority: Major
> Fix For: 5.0.0
>
>
> {{When the JDBC data source is connected, the snapshot management function is
> enabled and the dimension table is not sampled, optimize the build logic to
> ensure that the job can be executed normally when the dimension table data
> volume is large}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KYLIN-5767) Calculating total rows abnormal when jdbc datasource is connnected
[
https://issues.apache.org/jira/browse/KYLIN-5767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
pengfei.zhan updated KYLIN-5767:
Description: {{When the JDBC data source is connected, the snapshot
management function is enabled and the dimension table is not sampled, optimize
the build logic to ensure that the job can be executed normally when the
dimension table data volume is large}} (was: h2. When the JDBC data source is
connected, the snapshot management function is enabled and the dimension table
is not sampled, optimize the build logic to ensure that the job can be executed
normally when the dimension table data volume is large)
> Calculating total rows abnormal when jdbc datasource is connnected
> --
>
> Key: KYLIN-5767
> URL: https://issues.apache.org/jira/browse/KYLIN-5767
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: 5.0-beta
>Reporter: pengfei.zhan
>Assignee: pengfei.zhan
>Priority: Major
> Fix For: 5.0.0
>
>
> {{When the JDBC data source is connected, the snapshot management function is
> enabled and the dimension table is not sampled, optimize the build logic to
> ensure that the job can be executed normally when the dimension table data
> volume is large}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KYLIN-5767) Calculating total rows abnormal when jdbc datasource is connnected
[ https://issues.apache.org/jira/browse/KYLIN-5767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pengfei.zhan updated KYLIN-5767: Description: h2. When the JDBC data source is connected, the snapshot management function is enabled and the dimension table is not sampled, optimize the build logic to ensure that the job can be executed normally when the dimension table data volume is large > Calculating total rows abnormal when jdbc datasource is connnected > -- > > Key: KYLIN-5767 > URL: https://issues.apache.org/jira/browse/KYLIN-5767 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: 5.0-beta >Reporter: pengfei.zhan >Assignee: pengfei.zhan >Priority: Major > Fix For: 5.0.0 > > > h2. When the JDBC data source is connected, the snapshot management function > is enabled and the dimension table is not sampled, optimize the build logic > to ensure that the job can be executed normally when the dimension table data > volume is large -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (KYLIN-5766) Normalize query cache key
[
https://issues.apache.org/jira/browse/KYLIN-5766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832355#comment-17832355
]
pengfei.zhan edited comment on KYLIN-5766 at 3/30/24 2:19 AM:
--
h1. Design
Parses the sql with javaCC, gets the "normalized" sql, and uses that sql as the
key. Among them, "normalization" specific form:
* Remove general comments (already implemented in the previous sql parsing
step)
* Replacing any number of spaces, line feeds, tabs, returns, and page breaks
with a single whitespace character;
* Replace "+", "-", "*", "/", "%", "=", ">=", "<=", "! =", "<>", "||" Single
operators are replaced with one space to the left and one space to the right;
* Replace ( ), the parentheses, with a single space to the left and right of
each;
* Converting , i.e. English comma to the left and replacing it with a single
space on the right, in the form of test ,test1 to test, test1.
* For strings with escaped identifiers, such as `2 + 3 `, no changes will be
made, leaving them as they are, so `2 + 3 ` and `2 + 3 ` are different sql, and
can't hit each other's caches.
For example, these two queries are the same after transformation.
{code:sql}
-- sql1
select user ,
count(*) from /*comments
comments
*/demo group by user
-- sql2
select user, count(*) -- comments from demo group by user
{code}
the normalized cache key is
{code:sql}
select user, count ( * ) from demo group by user
{code}
was (Author: JIRAUSER294653):
h1. Design
Parses the sql with javaCC, gets the "normalized" sql, and uses that sql as the
key. Among them, "normalization" specific form:
* Remove general comments (already implemented in the previous sql parsing
step)
* Replacing any number of spaces, line feeds, tabs, returns, and page breaks
with a single whitespace character;
* Replace "+", "-", "*", "/", "%", "=", ">=", "<=", "! =", "<>", "||" Single
operators are replaced with one space to the left and one space to the right;
* Replace ( ), the parentheses, with a single space to the left and right of
each;
* Converting , i.e. English comma to the left and replacing it with a single
space on the right, in the form of test ,test1 to test, test1.
* For strings with escaped identifiers, such as `2 + 3 `, no changes will be
made, leaving them as they are, so `2 + 3 ` and `2 + 3 ` are different sql, and
can't hit each other's caches.
> Normalize query cache key
> -
>
> Key: KYLIN-5766
> URL: https://issues.apache.org/jira/browse/KYLIN-5766
> Project: Kylin
> Issue Type: Improvement
> Components: Query Engine
>Affects Versions: 5.0-beta
>Reporter: pengfei.zhan
>Assignee: pengfei.zhan
>Priority: Major
> Fix For: 5.0.0
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KYLIN-5766) Normalize query cache key
[ https://issues.apache.org/jira/browse/KYLIN-5766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832355#comment-17832355 ] pengfei.zhan commented on KYLIN-5766: - h1. Design Parses the sql with javaCC, gets the "normalized" sql, and uses that sql as the key. Among them, "normalization" specific form: * Remove general comments (already implemented in the previous sql parsing step) * Replacing any number of spaces, line feeds, tabs, returns, and page breaks with a single whitespace character; * Replace "+", "-", "*", "/", "%", "=", ">=", "<=", "! =", "<>", "||" Single operators are replaced with one space to the left and one space to the right; * Replace ( ), the parentheses, with a single space to the left and right of each; * Converting , i.e. English comma to the left and replacing it with a single space on the right, in the form of test ,test1 to test, test1. * For strings with escaped identifiers, such as `2 + 3 `, no changes will be made, leaving them as they are, so `2 + 3 ` and `2 + 3 ` are different sql, and can't hit each other's caches. > Normalize query cache key > - > > Key: KYLIN-5766 > URL: https://issues.apache.org/jira/browse/KYLIN-5766 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Affects Versions: 5.0-beta >Reporter: pengfei.zhan >Assignee: pengfei.zhan >Priority: Major > Fix For: 5.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KYLIN-5765) Optimization of register udf operators
[ https://issues.apache.org/jira/browse/KYLIN-5765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832353#comment-17832353 ] pengfei.zhan commented on KYLIN-5765: - h1. Design The Issue description has clearly stated that the main reason for this performance issue is due to frequent calls to the reflection method RexToTblColRefTranslator.OLAPRexSqlStandardConvertletTable#registerUdfOperator. OLAPRexSqlStandardConvertletTable#registerUdfOperator is frequently called due to instantiation of OLAPRexSqlStandardConvertletTable. So we move the method registerUdfOperator out, to avoid OLAPRexSqlStandardConvertletTable instantiation leads to registerUdfOperator frequently call. > Optimization of register udf operators > -- > > Key: KYLIN-5765 > URL: https://issues.apache.org/jira/browse/KYLIN-5765 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Affects Versions: 5.0-beta >Reporter: pengfei.zhan >Assignee: pengfei.zhan >Priority: Major > Fix For: 5.0.0 > > Attachments: 42118-2.png, 42118.png > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (KYLIN-5765) Optimization of register udf operators
[ https://issues.apache.org/jira/browse/KYLIN-5765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832352#comment-17832352 ] pengfei.zhan edited comment on KYLIN-5765 at 3/30/24 2:08 AM: -- !42118.png! Most time consumption is in RexToTblColRefTranslator.OLAPRexSqlStandardConvertletTable#registerUdfOperator. Using Arthas to monitor the execution of this method, we found that it executed more than 10,000 times in a single query. !42118-2.png! was (Author: JIRAUSER294653): !42118.png! Most time consumption is in RexToTblColRefTranslator.OLAPRexSqlStandardConvertletTable#registerUdfOperator. Using Arthas to monitor the execution of this method, we found that it executed more than 10,000 times in a single query. !42118-2.png! > Optimization of register udf operators > -- > > Key: KYLIN-5765 > URL: https://issues.apache.org/jira/browse/KYLIN-5765 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Affects Versions: 5.0-beta >Reporter: pengfei.zhan >Assignee: pengfei.zhan >Priority: Major > Fix For: 5.0.0 > > Attachments: 42118-2.png, 42118.png > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KYLIN-5765) Optimization of register udf operators
[ https://issues.apache.org/jira/browse/KYLIN-5765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pengfei.zhan updated KYLIN-5765: Attachment: 42118-2.png > Optimization of register udf operators > -- > > Key: KYLIN-5765 > URL: https://issues.apache.org/jira/browse/KYLIN-5765 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Affects Versions: 5.0-beta >Reporter: pengfei.zhan >Assignee: pengfei.zhan >Priority: Major > Fix For: 5.0.0 > > Attachments: 42118-2.png, 42118.png > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KYLIN-5765) Optimization of register udf operators
[ https://issues.apache.org/jira/browse/KYLIN-5765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832352#comment-17832352 ] pengfei.zhan commented on KYLIN-5765: - !42118.png! Most time consumption is in RexToTblColRefTranslator.OLAPRexSqlStandardConvertletTable#registerUdfOperator. Using Arthas to monitor the execution of this method, we found that it executed more than 10,000 times in a single query. !42118-2.png! > Optimization of register udf operators > -- > > Key: KYLIN-5765 > URL: https://issues.apache.org/jira/browse/KYLIN-5765 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Affects Versions: 5.0-beta >Reporter: pengfei.zhan >Assignee: pengfei.zhan >Priority: Major > Fix For: 5.0.0 > > Attachments: 42118.png > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KYLIN-5765) Optimization of register udf operators
[ https://issues.apache.org/jira/browse/KYLIN-5765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pengfei.zhan updated KYLIN-5765: Attachment: 42118.png > Optimization of register udf operators > -- > > Key: KYLIN-5765 > URL: https://issues.apache.org/jira/browse/KYLIN-5765 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Affects Versions: 5.0-beta >Reporter: pengfei.zhan >Assignee: pengfei.zhan >Priority: Major > Fix For: 5.0.0 > > Attachments: 42118.png > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KYLIN-5764) Sum expression supports if condition
[ https://issues.apache.org/jira/browse/KYLIN-5764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832350#comment-17832350 ] pengfei.zhan commented on KYLIN-5764: - h1. Design The sum expression recognition that has been implemented is only for case when expressions. When detecting if expressions, treat them as case when. > Sum expression supports if condition > > > Key: KYLIN-5764 > URL: https://issues.apache.org/jira/browse/KYLIN-5764 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Affects Versions: 5.0-beta >Reporter: pengfei.zhan >Assignee: pengfei.zhan >Priority: Major > Fix For: 5.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KYLIN-5764) Sum expression supports if condition
[
https://issues.apache.org/jira/browse/KYLIN-5764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832349#comment-17832349
]
pengfei.zhan commented on KYLIN-5764:
-
The following query cannot route to sum-expression to rewrite the queryPlan.
{code:sql}
select LO_CUSTKEY, count(distinct if(LO_CUSTKEY=74, LINEORDER.LO_SUPPLYCOST,0))
from ssb.LINEORDER where LO_CUSTKEY = 74
group by LO_CUSTKEY {code}
> Sum expression supports if condition
>
>
> Key: KYLIN-5764
> URL: https://issues.apache.org/jira/browse/KYLIN-5764
> Project: Kylin
> Issue Type: Improvement
> Components: Query Engine
>Affects Versions: 5.0-beta
>Reporter: pengfei.zhan
>Assignee: pengfei.zhan
>Priority: Major
> Fix For: 5.0.0
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KYLIN-5759) Log governance of job building log
[ https://issues.apache.org/jira/browse/KYLIN-5759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pengfei.zhan updated KYLIN-5759: Summary: Log governance of job building log (was: Log governance) > Log governance of job building log > -- > > Key: KYLIN-5759 > URL: https://issues.apache.org/jira/browse/KYLIN-5759 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Affects Versions: 5.0-beta >Reporter: pengfei.zhan >Assignee: pengfei.zhan >Priority: Major > Fix For: 5.0-beta > > > Improve the log governance. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KYLIN-5759) Log governance
[
https://issues.apache.org/jira/browse/KYLIN-5759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832345#comment-17832345
]
pengfei.zhan commented on KYLIN-5759:
-
h1. Root Cause
Some of the job building logs are in the file of {{kylin.log}} because the
{{logCategory}} is set by try-resource at the beginning of the building task.
After setting {{logCategory}}, when the {{logCategory}} were setted again with
try-resource, {{logCategory}} will be reset to null when {{logCategory}} closes
automatically. However, the {{logCategory}} that was set for the first time
won't be saved, so the subsequent logs will be printed to kylin.log.
> Log governance
> --
>
> Key: KYLIN-5759
> URL: https://issues.apache.org/jira/browse/KYLIN-5759
> Project: Kylin
> Issue Type: Improvement
> Components: Job Engine
>Affects Versions: 5.0-beta
>Reporter: pengfei.zhan
>Assignee: pengfei.zhan
>Priority: Major
> Fix For: 5.0-beta
>
>
> Improve the log governance.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KYLIN-5763) Very slow to filter query history
[
https://issues.apache.org/jira/browse/KYLIN-5763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832341#comment-17832341
]
pengfei.zhan commented on KYLIN-5763:
-
h1. Design
The response body payload is transformed from a list of strings to an object,
where
* `search_count` is the total number of matches for front-end display,
* `total_model_count` is the total number of unfiltered models without limit
for front-end display,
* `engines` is the list of engines, and `models` is the list of models.
`engines` is the list of engines and `models` is the list of models.
Example:
* before modification: `\{"code": "000", "data":["HIVE", "model1", "model2"],
"msg":""}`
* after modification: `\{"code": "000", "data":{"search_count":10,
"total_model_count":20, "engines":[], "models":["MODEL1", "MODEL2"]}, "msg":""}`
> Very slow to filter query history
> -
>
> Key: KYLIN-5763
> URL: https://issues.apache.org/jira/browse/KYLIN-5763
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: 5.0-beta
>Reporter: pengfei.zhan
>Assignee: pengfei.zhan
>Priority: Major
> Fix For: 5.0-beta
>
>
> If there are many queryHistory records,
> 1) filtering them by query object is very slow;
> 2) searching them by key(the model name) cannot work.
>
> However, executing query in mysql, the result responses very fast.
> {code:sql}
> select distinct model from kylin_query_history_realization
> where project_name = "abc";
> select distinct engine_type from kylin_query_history where engine_type <>
> 'NATIVE' and project_name = "abc";
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KYLIN-5757) No way to deselect query objects after selecting all of them on this page
[
https://issues.apache.org/jira/browse/KYLIN-5757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832339#comment-17832339
]
pengfei.zhan commented on KYLIN-5757:
-
h3. Dev design
The backend uses not in model id's for inverse selection, the frontend passes
the models to be excluded to the backend.
Add parameter to QueryHistoryRequest to record models to be excluded
(deselected models)
Add parameter {{exclude_realization}} (excludeRealization) to tell the system
to ignore query histories linked to this parameter. The excluded model
list(realizations) can just be specific model, i.e., {{model1}}, {{model2}}.
All related APIs
* {{GET kylin/api/query/history_queries}}
* {{GET kylin/api/query/download_query_histories}}
* {{GET kylin/api/query/download_query_histories_sql}}
> No way to deselect query objects after selecting all of them on this page
> -
>
> Key: KYLIN-5757
> URL: https://issues.apache.org/jira/browse/KYLIN-5757
> Project: Kylin
> Issue Type: Improvement
> Components: Others
>Affects Versions: 5.0-beta
>Reporter: pengfei.zhan
>Assignee: pengfei.zhan
>Priority: Major
> Fix For: 5.0-beta
>
>
> The API of listing models very slow at sometime, improve it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (KYLIN-5757) No way to deselect query objects after selecting all of them on this page
[
https://issues.apache.org/jira/browse/KYLIN-5757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832338#comment-17832338
]
pengfei.zhan edited comment on KYLIN-5757 at 3/30/24 1:21 AM:
--
h1. Description
The filtering of the query object (`realization`) is an `or` relationship,
i.e., it is sufficient to satisfy one of them
| |All model|hive|constants|model1|Result to show|
|check all models|Y| | | |records of all models|
|check all models & hive|Y|Y| | |records of all models & hive|
|check all models & constants|Y| |Y| |records of all models & constants|
|check all models & model1|Y| | |Y|records of all models(model1 included)|
|just check model1| | | |Y|records of model1|
Why the front-end can't select and deselect all: The models in the query object
are queried by the interface
`kylin/api/query/query_history_models?project=test_size=100 `, and only
100 of them are shown here, i.e., the displayed models may not be complete.
was (Author: JIRAUSER294653):
h1. Description
The filtering of the query object (`realization`) is an `or` relationship,
i.e., it is sufficient to satisfy one of them
| |All model|hive|constants|model1|Result to show|
|check all models|Y| | | |records of all models|
|check all models & hive|Y|Y| | |records of all models & hive|
|check all models & constants|Y| |Y| |records of all models & constants|
|check all models & model1|Y| | |Y|records of all models(model1 included)|
|just check model1| | | |Y|records of model1|
*{*}The reason why the front-end can't select and deselect all:{*}* The models
in the query object are queried by the interface
`kylin/api/query/query_history_models?project=test_size=100 `, and only
100 of them are shown here, i.e., the *{*}displayed models may not be
complete{*}*.
> No way to deselect query objects after selecting all of them on this page
> -
>
> Key: KYLIN-5757
> URL: https://issues.apache.org/jira/browse/KYLIN-5757
> Project: Kylin
> Issue Type: Improvement
> Components: Others
>Affects Versions: 5.0-beta
>Reporter: pengfei.zhan
>Assignee: pengfei.zhan
>Priority: Major
> Fix For: 5.0-beta
>
>
> The API of listing models very slow at sometime, improve it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KYLIN-5757) No way to deselect query objects after selecting all of them on this page
[
https://issues.apache.org/jira/browse/KYLIN-5757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832338#comment-17832338
]
pengfei.zhan commented on KYLIN-5757:
-
h1. Description
The filtering of the query object (`realization`) is an `or` relationship,
i.e., it is sufficient to satisfy one of them
| |All model|hive|constants|model1|Result to show|
|check all models|Y| | | |records of all models|
|check all models & hive|Y|Y| | |records of all models & hive|
|check all models & constants|Y| |Y| |records of all models & constants|
|check all models & model1|Y| | |Y|records of all models(model1 included)|
|just check model1| | | |Y|records of model1|
*{*}The reason why the front-end can't select and deselect all:{*}* The models
in the query object are queried by the interface
`kylin/api/query/query_history_models?project=test_size=100 `, and only
100 of them are shown here, i.e., the *{*}displayed models may not be
complete{*}*.
> No way to deselect query objects after selecting all of them on this page
> -
>
> Key: KYLIN-5757
> URL: https://issues.apache.org/jira/browse/KYLIN-5757
> Project: Kylin
> Issue Type: Improvement
> Components: Others
>Affects Versions: 5.0-beta
>Reporter: pengfei.zhan
>Assignee: pengfei.zhan
>Priority: Major
> Fix For: 5.0-beta
>
>
> The API of listing models very slow at sometime, improve it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] The batch bug fix of 4.6.13 to KYLIN5 [kylin]
pfzhan closed pull request #2166: The batch bug fix of 4.6.13 to KYLIN5 URL: https://github.com/apache/kylin/pull/2166 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@kylin.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (KYLIN-5773) Upgrade Spark to 4.6.13.0
pengfei.zhan created KYLIN-5773: --- Summary: Upgrade Spark to 4.6.13.0 Key: KYLIN-5773 URL: https://issues.apache.org/jira/browse/KYLIN-5773 Project: Kylin Issue Type: Bug Components: Spark Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0-beta -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KYLIN-5754) SAML and LDAP Issue
[
https://issues.apache.org/jira/browse/KYLIN-5754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832176#comment-17832176
]
pengfei.zhan commented on KYLIN-5754:
-
What's the version of KYLIN?
> SAML and LDAP Issue
> ---
>
> Key: KYLIN-5754
> URL: https://issues.apache.org/jira/browse/KYLIN-5754
> Project: Kylin
> Issue Type: Bug
>Reporter: dennis lucero
>Priority: Blocker
>
> When i activate LDAP with the following Config am able to access the admin
> account.
>
> {color:#569cd6}kylin.security.ldap.user-search-base{color}{color:#cc}={color}{color:#569cd6}ou{color}{color:#cc}=People,{color}{color:#569cd6}dc{color}{color:#cc}=datasymphony,{color}{color:#569cd6}dc{color}{color:#cc}=ddns,{color}{color:#569cd6}dc{color}{color:#cc}=net{color}
> {color:#569cd6}kylin.security.ldap.user-search-pattern{color}{color:#cc}=(&({color}{color:#569cd6}cn{color}{color:#cc}=\{0})){color}
> {color:#569cd6}kylin.security.ldap.user-group-search-base{color}{color:#cc}={color}{color:#569cd6}ou{color}{color:#cc}=Groups,{color}{color:#569cd6}dc{color}{color:#cc}=datasymphony,{color}{color:#569cd6}dc{color}{color:#cc}=ddns,{color}{color:#569cd6}dc{color}{color:#cc}=net{color}
> {color:#569cd6}group-search-filter{color}{color:#cc}=(|({color}{color:#569cd6}member{color}{color:#cc}=\{0})({color}{color:#569cd6}memberUid{color}{color:#cc}=\{1})){color}
>
> But with the same config if i change profile to saml it doesnt loginto just
> say user or password invalid. thisis my saml config
> {color:#569cd6}kylin.security.saml.metadata-entity-base-url{color}{color:#cc}=https://kylin.datasymphony.com/kylin{color}
> {color:#569cd6}kylin.security.saml.context-scheme{color}{color:#cc}=https{color}
> {color:#569cd6}kylin.security.saml.context-server-name{color}{color:#cc}=kylin.datasymphony.com{color}
> {color:#569cd6}kylin.security.saml.context-server-port{color}{color:#cc}=443{color}
> {color:#569cd6}kylin.security.saml.context-path{color}{color:#cc}=/kylin{color}
>
> Am using wso2 as the dientity server, when i login it takes me to the
> identity server login but when i autorize and go back to kylin it just show
> kylin login page. been stuck on this for a while now i need help. thanks
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KYLIN-5746) On the page, select online model operation offline, click the model online again, and put the model online button into ash.
[ https://issues.apache.org/jira/browse/KYLIN-5746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pengfei.zhan resolved KYLIN-5746. - Fix Version/s: 5.0-beta (was: 5.0.0) Resolution: Fixed > On the page, select online model operation offline, click the model online > again, and put the model online button into ash. > --- > > Key: KYLIN-5746 > URL: https://issues.apache.org/jira/browse/KYLIN-5746 > Project: Kylin > Issue Type: Bug >Affects Versions: 5.0-beta >Reporter: zhong.zhu >Assignee: zhong.zhu >Priority: Blocker > Fix For: 5.0-beta > > Attachments: image-2023-12-11-17-33-31-889.png > > > Repeat step: > 1:Create model, build data, model for online. > 2:Operate the model offline > 3:Click on the model again and select the model to go online. > Actual result > The model on-line button is grayed out, indicating that the model is not > available for on-line operation. > !image-2023-12-11-17-33-31-889.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KYLIN-5744) Enable the parameter kylin.query.using-metadata-answer-minmax-of-dimension by default, to answer the max/min query through metadata
[ https://issues.apache.org/jira/browse/KYLIN-5744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pengfei.zhan resolved KYLIN-5744. - Fix Version/s: 5.0-beta (was: 5.0.0) Resolution: Fixed > Enable the parameter kylin.query.using-metadata-answer-minmax-of-dimension by > default, to answer the max/min query through metadata > --- > > Key: KYLIN-5744 > URL: https://issues.apache.org/jira/browse/KYLIN-5744 > Project: Kylin > Issue Type: Bug >Affects Versions: 5.0-beta >Reporter: zhong.zhu >Assignee: zhong.zhu >Priority: Minor > Fix For: 5.0-beta > > > Enable the parameter > *_kylin.query.using-metadata-answer-minmax-of-dimension_* by default to > answer max/min queries via metadata -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KYLIN-5756) Concat string and number with '+' gives unexpected result
[
https://issues.apache.org/jira/browse/KYLIN-5756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
pengfei.zhan resolved KYLIN-5756.
-
Resolution: Fixed
> Concat string and number with '+' gives unexpected result
> -
>
> Key: KYLIN-5756
> URL: https://issues.apache.org/jira/browse/KYLIN-5756
> Project: Kylin
> Issue Type: Improvement
> Components: Query Engine
>Affects Versions: 5.0-beta
>Reporter: pengfei.zhan
>Assignee: pengfei.zhan
>Priority: Major
> Fix For: 5.0-beta
>
>
> Concat string and number gives unexpected result.
> For example:
> 1. concat constant with plus operator, gives unexpected result
> {code:java}
> 1' + 3 + 3 // 7 (correct)
> '1' + 3 + '3' // 4.03 (wrong result)
> '1' + '3' + 'a' // error
> {code}
> 2. concat string and number, the first '+' produces null
> {code:java}
> 'q' + 1 + 1 // error
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KYLIN-5743) Set kylin.query.convert-sum-expression-enabled=true, fail to completely hit the aggregate index when the query contains sum (case when) expressions
[
https://issues.apache.org/jira/browse/KYLIN-5743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
pengfei.zhan resolved KYLIN-5743.
-
Fix Version/s: 5.0-beta
(was: 5.0.0)
Resolution: Fixed
> Set kylin.query.convert-sum-expression-enabled=true, fail to completely hit
> the aggregate index when the query contains sum (case when) expressions
> ---
>
> Key: KYLIN-5743
> URL: https://issues.apache.org/jira/browse/KYLIN-5743
> Project: Kylin
> Issue Type: Bug
>Affects Versions: 5.0-beta
>Reporter: zhong.zhu
>Assignee: zhong.zhu
>Priority: Major
> Fix For: 5.0-beta
>
>
> {code:sql}
> select
> sum(
> case
> when LO_COMMITDATE = '20230501' then LO_DISCOUNT
> end
> )
> from
> (
> select
> LO_COMMITDATE,
> LO_DISCOUNT,
> LINEORDER.LO_ORDERDATE
> from
> ssb.LINEORDER
> ) a
> where
> LO_COMMITDATE = (
> select
> max(LO_COMMITDATE)
> from
> ssb.LINEORDER
> )
> LIMIT
> 500
> {code}
> Fix the sum case when in this scenario so that it hits aggregated indexes
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KYLIN-5741) When using the API in project settings API to update the linking relationship between projects and job engines, an error is reported when the projects parameter is empty
[ https://issues.apache.org/jira/browse/KYLIN-5741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pengfei.zhan resolved KYLIN-5741. - Fix Version/s: 5.0-beta (was: 5.0.0) Resolution: Fixed > When using the API in project settings API to update the linking relationship > between projects and job engines, an error is reported when the projects > parameter is empty > - > > Key: KYLIN-5741 > URL: https://issues.apache.org/jira/browse/KYLIN-5741 > Project: Kylin > Issue Type: Bug >Affects Versions: 5.0-beta >Reporter: zhong.zhu >Assignee: zhong.zhu >Priority: Critical > Fix For: 5.0-beta > > Attachments: image-2023-12-11-14-44-04-923.png > > > !image-2023-12-11-14-44-04-923.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KYLIN-5742) When the "Group by" group has duplicate values, the result of Grouping Set query is inconsistent with that in SparkSQL
[
https://issues.apache.org/jira/browse/KYLIN-5742?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
pengfei.zhan resolved KYLIN-5742.
-
Fix Version/s: 5.0-beta
(was: 5.0.0)
Resolution: Fixed
> When the "Group by" group has duplicate values, the result of Grouping Set
> query is inconsistent with that in SparkSQL
> --
>
> Key: KYLIN-5742
> URL: https://issues.apache.org/jira/browse/KYLIN-5742
> Project: Kylin
> Issue Type: Bug
>Affects Versions: 5.0-beta
>Reporter: zhong.zhu
>Assignee: zhong.zhu
>Priority: Major
> Fix For: 5.0-beta
>
> Attachments: image-2023-12-11-14-54-38-652.png,
> image-2023-12-11-14-55-46-222.png, image-2023-12-11-14-57-32-037.png,
> image-2023-12-11-14-57-56-771.png
>
>
> {code:sql}
> -- sql1
> select C_NAME,C_CITY,C_NATION,C_REGION,count(*)
> FROM SSB.LINEORDER as LINEORDER
> INNER JOIN SSB.CUSTOMER as CUSTOMER
> ON LINEORDER.LO_CUSTKEY = CUSTOMER.C_CUSTKEY
> where C_NATION = 'CHINA' and C_CITY = 'CHINA 0'
> group by
> GROUPING SETS ((),(C_NAME,C_CITY),(C_NATION,C_REGION))
> order by C_NAME;
> -- sql2
> select C_NAME,C_CITY,C_NATION,C_REGION,count(*)
> FROM SSB.LINEORDER as LINEORDER
> INNER JOIN SSB.CUSTOMER as CUSTOMER
> ON LINEORDER.LO_CUSTKEY = CUSTOMER.C_CUSTKEY
> where C_NATION = 'CHINA' and C_CITY = 'CHINA 0'
> group by
> C_NAME,C_CITY,C_NATION,C_REGION,
> GROUPING SETS ((),(C_NAME,C_CITY),(C_NATION,C_REGION))
> order by C_NAME;
> -- sql3
> select C_NAME,C_CITY,C_NATION,C_REGION,count(*)
> FROM SSB.LINEORDER as LINEORDER
> INNER JOIN SSB.CUSTOMER as CUSTOMER
> ON LINEORDER.LO_CUSTKEY = CUSTOMER.C_CUSTKEY
> where C_NATION = 'CHINA' and C_CITY = 'CHINA 0'
> group by
> C_NAME,C_CITY,C_NATION,C_REGION
> GROUPING SETS ((),(C_NAME,C_CITY),(C_NATION,C_REGION))
> order by C_NAME
> {code}
> In spark-sql, sql1 and sql3 query results are consistent as follows:
> !image-2023-12-11-14-54-38-652.png!
> In spark-sql, sql 2 the query results are as follows.
> !image-2023-12-11-14-55-46-222.png!
> In KYLIN, the query result of sql1 is as follows, which is consistent with
> the result of spark-sql sql sql1 sql2:
> !image-2023-12-11-14-57-32-037.png!
> The query result of sql2 is as follows, which is inconsistent with the
> spark-sql sql2 result:
> !image-2023-12-11-14-57-56-771.png!
> The syntax of sql3 is not supported
> Hive does not support commas before grouping sets, that is, sql2 is not
> supported, and the query results of sql1 and sql3 are consistent with
> spark-sql
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KYLIN-5760) Support matching model exactly for list jobs
[ https://issues.apache.org/jira/browse/KYLIN-5760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pengfei.zhan resolved KYLIN-5760. - Resolution: Fixed > Support matching model exactly for list jobs > > > Key: KYLIN-5760 > URL: https://issues.apache.org/jira/browse/KYLIN-5760 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: 5.0-beta >Reporter: pengfei.zhan >Assignee: pengfei.zhan >Priority: Major > Fix For: 5.0-beta > > > h1. Description > At present the API of list jobs (/kylin/api/jobs) will return all job lists > that fuzzy match the given key. Sometimes, for better performance, the key is > applied to match jobs exactly, however, it's not supported yet. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KYLIN-5737) Fix high-level security vulnerabilities
[ https://issues.apache.org/jira/browse/KYLIN-5737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pengfei.zhan resolved KYLIN-5737. - Fix Version/s: 5.0-beta (was: 5.0.0) Resolution: Fixed > Fix high-level security vulnerabilities > --- > > Key: KYLIN-5737 > URL: https://issues.apache.org/jira/browse/KYLIN-5737 > Project: Kylin > Issue Type: Bug >Affects Versions: 5.0-beta >Reporter: zhong.zhu >Assignee: zhong.zhu >Priority: Critical > Fix For: 5.0-beta > > > * maven:org.apache.tomcat.embed:tomcat-embed-core@9.0.74/CVE-2023-34981 > * [https://security.snyk.io/vuln/SNYK-JAVA-ORGAPACHETOMCATEMBED-5726234] > > * maven:io.netty:netty-handler@4.1.92.Final /CVE-2023-34462 > * [https://security.snyk.io/vuln/SNYK-JAVA-IONETTY-5725787] > > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KYLIN-5740) Equivalent aggregations matches index failed when using the strategy of dimension as measure.
[
https://issues.apache.org/jira/browse/KYLIN-5740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
pengfei.zhan resolved KYLIN-5740.
-
Fix Version/s: 5.0-beta
(was: 5.0.0)
Resolution: Fixed
> Equivalent aggregations matches index failed when using the strategy of
> dimension as measure.
> -
>
> Key: KYLIN-5740
> URL: https://issues.apache.org/jira/browse/KYLIN-5740
> Project: Kylin
> Issue Type: Bug
>Affects Versions: 5.0-beta
>Reporter: zhong.zhu
>Assignee: zhong.zhu
>Priority: Critical
> Fix For: 5.0-beta
>
> Attachments: image-2023-12-11-14-25-14-409.png,
> image-2023-12-11-14-32-11-954.png, image-2023-12-11-14-36-45-263.png,
> image-2023-12-11-14-37-08-556.png
>
>
> Reproduction method:
> Table Building Statements
> {code:sql}
> CREATE EXTERNAL TABLE LT859 (
> `PRO_MT_SEASON` varchar(4096),
> `ARRIVAL_TRANSIT_IN_PRM_AMT` decimal(18,4),
> `BIG_REGION_NAME` varchar(4096),
> `REGION_NAME` varchar(4096)
> ) ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
> STORED AS TEXTFILE;LOAD DATA INPATH '/path/to/data/LT859.csv' OVERWRITE INTO
> TABLE LT859;
> {code}
> Data（LT859.csv）
> {code:bash}
> 22Q1,150.12,东区,华东区
> 22Q2,200.59,中区,华中区
> 22Q3,301.55,上海区,华东区
> {code}
> Select SQL:
> {code:sql}
> select
> max(BIG_REGION_NAME) "max1",
> max(
> case when 1=1 then "BIG_REGION_NAME"
> end
> ) "max2"
> from XXX.LT859
> {code}
> Create model, dimension 4 columns selected, add metrics MAX(BIG_REGION_NAME)
> Save the model and delete the base aggregation index, manually add an
> aggregation group as a replacement for the base aggregation, but it can't
> have the MAX(BIG_REGION_NAME) metric in it to mimic the dim as measure
> scenario, and build the full amount.
> Using the above select sql query, you can see that NPE is reported in the
> background:
> !image-2023-12-11-14-25-14-409.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KYLIN-5762) Initialize job scheduler encounters NPE
[ https://issues.apache.org/jira/browse/KYLIN-5762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pengfei.zhan resolved KYLIN-5762. - Resolution: Fixed > Initialize job scheduler encounters NPE > --- > > Key: KYLIN-5762 > URL: https://issues.apache.org/jira/browse/KYLIN-5762 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: 5.0-beta >Reporter: pengfei.zhan >Assignee: pengfei.zhan >Priority: Major > Fix For: 5.0-beta > > > Given `kylin.job.max-concurrent-jobs=0`, the initialization of job scheduler > encounters NPE. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KYLIN-5732) High-level security vulnerabilities:maven:org.xerial.snappy:snappy-java@1.1.8.4/CVE-2023-34455
[ https://issues.apache.org/jira/browse/KYLIN-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pengfei.zhan resolved KYLIN-5732. - Fix Version/s: 5.0-beta (was: 5.0.0) Resolution: Fixed > High-level security > vulnerabilities:maven:org.xerial.snappy:snappy-java@1.1.8.4/CVE-2023-34455 > -- > > Key: KYLIN-5732 > URL: https://issues.apache.org/jira/browse/KYLIN-5732 > Project: Kylin > Issue Type: Bug >Affects Versions: 5.0-beta >Reporter: zhong.zhu >Assignee: zhong.zhu >Priority: Critical > Fix For: 5.0-beta > > Attachments: image-2023-12-05-19-01-48-480.png > > > Vulnerability Details: > !image-2023-12-05-19-01-48-480.png! > snyk link: [https://snyk.io/vuln/SNYK-JAVA-ORGXERIALSNAPPY-5710960] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KYLIN-5733) Export model TDS file in English interface, including Chinese when opening the file in text mode
[ https://issues.apache.org/jira/browse/KYLIN-5733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pengfei.zhan resolved KYLIN-5733. - Fix Version/s: 5.0-beta (was: 5.0.0) Resolution: Fixed > Export model TDS file in English interface, including Chinese when opening > the file in text mode > > > Key: KYLIN-5733 > URL: https://issues.apache.org/jira/browse/KYLIN-5733 > Project: Kylin > Issue Type: Bug >Affects Versions: 5.0-beta >Reporter: zhong.zhu >Assignee: zhong.zhu >Priority: Critical > Fix For: 5.0-beta > > > *Steps to reproduce the issue:* > # Go to a model and click “Export TDS”. > # Open up the file in a text editor and look at the bottom. There are > Chinese characters. See attachment. > I confirmed this issue is present in 4.5.4 and .11, and likely exists in > other versions. The TDS file seems to work fine though. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KYLIN-5762) Initialize job scheduler encounters NPE
[ https://issues.apache.org/jira/browse/KYLIN-5762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832156#comment-17832156 ] ASF subversion and git services commented on KYLIN-5762: Commit 1d665a82829a82fc9a51181378aa87de1a98dea2 in kylin's branch refs/heads/kylin5 from huangsheng [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=1d665a8282 ] KYLIN-5762 Initialize job scheduler encounters NPE > Initialize job scheduler encounters NPE > --- > > Key: KYLIN-5762 > URL: https://issues.apache.org/jira/browse/KYLIN-5762 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: 5.0-beta >Reporter: pengfei.zhan >Assignee: pengfei.zhan >Priority: Major > Fix For: 5.0-beta > > > Given `kylin.job.max-concurrent-jobs=0`, the initialization of job scheduler > encounters NPE. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KYLIN-5740) Equivalent aggregations matches index failed when using the strategy of dimension as measure.
[
https://issues.apache.org/jira/browse/KYLIN-5740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832157#comment-17832157
]
ASF subversion and git services commented on KYLIN-5740:
Commit 0e3354414d89c4b4b81d0ec1db8d39705e760a40 in kylin's branch
refs/heads/kylin5 from huangsheng
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=0e3354414d ]
KYLIN-5740 Equivalent aggregations matches index failed
Equivalent aggregations matches index failed when using
the strategy of dimension as measure.
> Equivalent aggregations matches index failed when using the strategy of
> dimension as measure.
> -
>
> Key: KYLIN-5740
> URL: https://issues.apache.org/jira/browse/KYLIN-5740
> Project: Kylin
> Issue Type: Bug
>Affects Versions: 5.0-beta
>Reporter: zhong.zhu
>Assignee: zhong.zhu
>Priority: Critical
> Fix For: 5.0.0
>
> Attachments: image-2023-12-11-14-25-14-409.png,
> image-2023-12-11-14-32-11-954.png, image-2023-12-11-14-36-45-263.png,
> image-2023-12-11-14-37-08-556.png
>
>
> Reproduction method:
> Table Building Statements
> {code:sql}
> CREATE EXTERNAL TABLE LT859 (
> `PRO_MT_SEASON` varchar(4096),
> `ARRIVAL_TRANSIT_IN_PRM_AMT` decimal(18,4),
> `BIG_REGION_NAME` varchar(4096),
> `REGION_NAME` varchar(4096)
> ) ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
> STORED AS TEXTFILE;LOAD DATA INPATH '/path/to/data/LT859.csv' OVERWRITE INTO
> TABLE LT859;
> {code}
> Data（LT859.csv）
> {code:bash}
> 22Q1,150.12,东区,华东区
> 22Q2,200.59,中区,华中区
> 22Q3,301.55,上海区,华东区
> {code}
> Select SQL:
> {code:sql}
> select
> max(BIG_REGION_NAME) "max1",
> max(
> case when 1=1 then "BIG_REGION_NAME"
> end
> ) "max2"
> from XXX.LT859
> {code}
> Create model, dimension 4 columns selected, add metrics MAX(BIG_REGION_NAME)
> Save the model and delete the base aggregation index, manually add an
> aggregation group as a replacement for the base aggregation, but it can't
> have the MAX(BIG_REGION_NAME) metric in it to mimic the dim as measure
> scenario, and build the full amount.
> Using the above select sql query, you can see that NPE is reported in the
> background:
> !image-2023-12-11-14-25-14-409.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KYLIN-5744) Enable the parameter kylin.query.using-metadata-answer-minmax-of-dimension by default, to answer the max/min query through metadata
[ https://issues.apache.org/jira/browse/KYLIN-5744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832162#comment-17832162 ] ASF subversion and git services commented on KYLIN-5744: Commit d13199db7407552226a58060ceb9b41a7f859c36 in kylin's branch refs/heads/kylin5 from Pengfei Zhan [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=d13199db74 ] KYLIN-5744 Answering snapshot first, then answering by metadata Hitting snapshot has a higher priority than answering min/max with metadata. Change the default behavior of routing query to metadata to true 1. Fix timezone problem. Given the fact of min(cal_dt) = '2012-01-01', the result of `select min(cal_dt) from test_kylin_fact` may give '2011-12-31' rather than '2012-01-01'. 2. Fix min/max query hit a index without any dimensions. For example: `select min(cal_dt) as min_cal_dt from test_kylin_fact ` hits index just has a measure of min(cal_dt), the SparkPlan is wrong: ` project --- project --- tableScan `; but the desired SparkPlan is: ` aggregate --- project --- tableScan `. > Enable the parameter kylin.query.using-metadata-answer-minmax-of-dimension by > default, to answer the max/min query through metadata > --- > > Key: KYLIN-5744 > URL: https://issues.apache.org/jira/browse/KYLIN-5744 > Project: Kylin > Issue Type: Bug >Affects Versions: 5.0-beta >Reporter: zhong.zhu >Assignee: zhong.zhu >Priority: Minor > Fix For: 5.0.0 > > > Enable the parameter > *_kylin.query.using-metadata-answer-minmax-of-dimension_* by default to > answer max/min queries via metadata -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KYLIN-5747) Calcite constant folding, adding strings to numbers, results not as expected when multiple plus signs are used together
[ https://issues.apache.org/jira/browse/KYLIN-5747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832161#comment-17832161 ] ASF subversion and git services commented on KYLIN-5747: Commit 572d78a45b6e0cabb248b6772a9b434221d9f323 in kylin's branch refs/heads/kylin5 from Zhiting Guo [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=572d78a45b ] KYLIN-5756 Concat string and number with + gives unexpected result KYLIN-5747 duplicated Co-authored-by: Zhiting Guo > Calcite constant folding, adding strings to numbers, results not as expected > when multiple plus signs are used together > --- > > Key: KYLIN-5747 > URL: https://issues.apache.org/jira/browse/KYLIN-5747 > Project: Kylin > Issue Type: Bug >Affects Versions: 5.0-beta >Reporter: zhong.zhu >Assignee: zhong.zhu >Priority: Critical > Fix For: 5.0.0 > > > Phenomenon: > When more than one plus sign is used in a row and the parameters on both > sides of the plus sign are constants, the result is not as expected > '1' + 3 + 3 → 7 (correct) > '1' + 3 + '3' → 4.03 (wrong result) > '1' + '3' + 'a' → error > When multiple plus signs are used in a row, and the arguments on both sides > of the plus sign are constants, and the first plus sign results in null, the > use of plus signs in a row is not supported. > e.g. 'q' + 1 + 1 -> error -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KYLIN-5760) Support matching model exactly for list jobs
[ https://issues.apache.org/jira/browse/KYLIN-5760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832153#comment-17832153 ] ASF subversion and git services commented on KYLIN-5760: Commit b15367e3e2682b195966f5b59dbf09f8867a792e in kylin's branch refs/heads/kylin5 from Hang Jia [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=b15367e3e2 ] KYLIN-5760 Support matching model exactly for list jobs > Support matching model exactly for list jobs > > > Key: KYLIN-5760 > URL: https://issues.apache.org/jira/browse/KYLIN-5760 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: 5.0-beta >Reporter: pengfei.zhan >Assignee: pengfei.zhan >Priority: Major > Fix For: 5.0-beta > > > h1. Description > At present the API of list jobs (/kylin/api/jobs) will return all job lists > that fuzzy match the given key. Sometimes, for better performance, the key is > applied to match jobs exactly, however, it's not supported yet. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KYLIN-5762) Initialize job scheduler encounters NPE
[ https://issues.apache.org/jira/browse/KYLIN-5762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832152#comment-17832152 ] ASF subversion and git services commented on KYLIN-5762: Commit 6816947de43cc1f3a7101c06a1d32d623a36ef06 in kylin's branch refs/heads/kylin5 from sibingzhang [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=6816947de4 ] KYLIN-5762 Initialize job scheduler encounters NPE Co-authored-by: sibing.zhang > Initialize job scheduler encounters NPE > --- > > Key: KYLIN-5762 > URL: https://issues.apache.org/jira/browse/KYLIN-5762 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: 5.0-beta >Reporter: pengfei.zhan >Assignee: pengfei.zhan >Priority: Major > Fix For: 5.0-beta > > > Given `kylin.job.max-concurrent-jobs=0`, the initialization of job scheduler > encounters NPE. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KYLIN-5741) When using the API in project settings API to update the linking relationship between projects and job engines, an error is reported when the projects parameter is empt
[ https://issues.apache.org/jira/browse/KYLIN-5741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832158#comment-17832158 ] ASF subversion and git services commented on KYLIN-5741: Commit 7cd3fd0ecc8ce95ef4f21a84abd1ed251d4e54a4 in kylin's branch refs/heads/kylin5 from Yu Gan [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=7cd3fd0ecc ] KYLIN-5741 change all epoch when empty projects request > When using the API in project settings API to update the linking relationship > between projects and job engines, an error is reported when the projects > parameter is empty > - > > Key: KYLIN-5741 > URL: https://issues.apache.org/jira/browse/KYLIN-5741 > Project: Kylin > Issue Type: Bug >Affects Versions: 5.0-beta >Reporter: zhong.zhu >Assignee: zhong.zhu >Priority: Critical > Fix For: 5.0.0 > > Attachments: image-2023-12-11-14-44-04-923.png > > > !image-2023-12-11-14-44-04-923.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KYLIN-5742) When the "Group by" group has duplicate values, the result of Grouping Set query is inconsistent with that in SparkSQL
[
https://issues.apache.org/jira/browse/KYLIN-5742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832155#comment-17832155
]
ASF subversion and git services commented on KYLIN-5742:
Commit c396134127b17e741e6ead1197589afe7bb773d7 in kylin's branch
refs/heads/kylin5 from fengguangyuan
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=c396134127 ]
KYLIN-5742 Make the query result of duplicate group sets same as Spark
Co-authored-by: Guangyuan Feng
> When the "Group by" group has duplicate values, the result of Grouping Set
> query is inconsistent with that in SparkSQL
> --
>
> Key: KYLIN-5742
> URL: https://issues.apache.org/jira/browse/KYLIN-5742
> Project: Kylin
> Issue Type: Bug
>Affects Versions: 5.0-beta
>Reporter: zhong.zhu
>Assignee: zhong.zhu
>Priority: Major
> Fix For: 5.0.0
>
> Attachments: image-2023-12-11-14-54-38-652.png,
> image-2023-12-11-14-55-46-222.png, image-2023-12-11-14-57-32-037.png,
> image-2023-12-11-14-57-56-771.png
>
>
> {code:sql}
> -- sql1
> select C_NAME,C_CITY,C_NATION,C_REGION,count(*)
> FROM SSB.LINEORDER as LINEORDER
> INNER JOIN SSB.CUSTOMER as CUSTOMER
> ON LINEORDER.LO_CUSTKEY = CUSTOMER.C_CUSTKEY
> where C_NATION = 'CHINA' and C_CITY = 'CHINA 0'
> group by
> GROUPING SETS ((),(C_NAME,C_CITY),(C_NATION,C_REGION))
> order by C_NAME;
> -- sql2
> select C_NAME,C_CITY,C_NATION,C_REGION,count(*)
> FROM SSB.LINEORDER as LINEORDER
> INNER JOIN SSB.CUSTOMER as CUSTOMER
> ON LINEORDER.LO_CUSTKEY = CUSTOMER.C_CUSTKEY
> where C_NATION = 'CHINA' and C_CITY = 'CHINA 0'
> group by
> C_NAME,C_CITY,C_NATION,C_REGION,
> GROUPING SETS ((),(C_NAME,C_CITY),(C_NATION,C_REGION))
> order by C_NAME;
> -- sql3
> select C_NAME,C_CITY,C_NATION,C_REGION,count(*)
> FROM SSB.LINEORDER as LINEORDER
> INNER JOIN SSB.CUSTOMER as CUSTOMER
> ON LINEORDER.LO_CUSTKEY = CUSTOMER.C_CUSTKEY
> where C_NATION = 'CHINA' and C_CITY = 'CHINA 0'
> group by
> C_NAME,C_CITY,C_NATION,C_REGION
> GROUPING SETS ((),(C_NAME,C_CITY),(C_NATION,C_REGION))
> order by C_NAME
> {code}
> In spark-sql, sql1 and sql3 query results are consistent as follows:
> !image-2023-12-11-14-54-38-652.png!
> In spark-sql, sql 2 the query results are as follows.
> !image-2023-12-11-14-55-46-222.png!
> In KYLIN, the query result of sql1 is as follows, which is consistent with
> the result of spark-sql sql sql1 sql2:
> !image-2023-12-11-14-57-32-037.png!
> The query result of sql2 is as follows, which is inconsistent with the
> spark-sql sql2 result:
> !image-2023-12-11-14-57-56-771.png!
> The syntax of sql3 is not supported
> Hive does not support commas before grouping sets, that is, sql2 is not
> supported, and the query results of sql1 and sql3 are consistent with
> spark-sql
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KYLIN-5756) Concat string and number with '+' gives unexpected result
[
https://issues.apache.org/jira/browse/KYLIN-5756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832160#comment-17832160
]
ASF subversion and git services commented on KYLIN-5756:
Commit 572d78a45b6e0cabb248b6772a9b434221d9f323 in kylin's branch
refs/heads/kylin5 from Zhiting Guo
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=572d78a45b ]
KYLIN-5756 Concat string and number with + gives unexpected result
KYLIN-5747 duplicated
Co-authored-by: Zhiting Guo
> Concat string and number with '+' gives unexpected result
> -
>
> Key: KYLIN-5756
> URL: https://issues.apache.org/jira/browse/KYLIN-5756
> Project: Kylin
> Issue Type: Improvement
> Components: Query Engine
>Affects Versions: 5.0-beta
>Reporter: pengfei.zhan
>Assignee: pengfei.zhan
>Priority: Major
> Fix For: 5.0-beta
>
>
> Concat string and number gives unexpected result.
> For example:
> 1. concat constant with plus operator, gives unexpected result
> {code:java}
> 1' + 3 + 3 // 7 (correct)
> '1' + 3 + '3' // 4.03 (wrong result)
> '1' + '3' + 'a' // error
> {code}
> 2. concat string and number, the first '+' produces null
> {code:java}
> 'q' + 1 + 1 // error
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KYLIN-5746) On the page, select online model operation offline, click the model online again, and put the model online button into ash.
[ https://issues.apache.org/jira/browse/KYLIN-5746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832164#comment-17832164 ] ASF subversion and git services commented on KYLIN-5746: Commit 3cbcead2ed5cb3046647ef2a6617c9480254b1b9 in kylin's branch refs/heads/kylin5 from liang.huang [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=3cbcead2ed ] KYLIN-5746 model list get optimize fix > On the page, select online model operation offline, click the model online > again, and put the model online button into ash. > --- > > Key: KYLIN-5746 > URL: https://issues.apache.org/jira/browse/KYLIN-5746 > Project: Kylin > Issue Type: Bug >Affects Versions: 5.0-beta >Reporter: zhong.zhu >Assignee: zhong.zhu >Priority: Blocker > Fix For: 5.0.0 > > Attachments: image-2023-12-11-17-33-31-889.png > > > Repeat step: > 1:Create model, build data, model for online. > 2:Operate the model offline > 3:Click on the model again and select the model to go online. > Actual result > The model on-line button is grayed out, indicating that the model is not > available for on-line operation. > !image-2023-12-11-17-33-31-889.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KYLIN-5741) When using the API in project settings API to update the linking relationship between projects and job engines, an error is reported when the projects parameter is empt
[ https://issues.apache.org/jira/browse/KYLIN-5741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832163#comment-17832163 ] ASF subversion and git services commented on KYLIN-5741: Commit 227e7a068360978133f8188cff859723cea458bf in kylin's branch refs/heads/kylin5 from zhong.zhu [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=227e7a0683 ] KYLIN-5741 punctuation optimized > When using the API in project settings API to update the linking relationship > between projects and job engines, an error is reported when the projects > parameter is empty > - > > Key: KYLIN-5741 > URL: https://issues.apache.org/jira/browse/KYLIN-5741 > Project: Kylin > Issue Type: Bug >Affects Versions: 5.0-beta >Reporter: zhong.zhu >Assignee: zhong.zhu >Priority: Critical > Fix For: 5.0.0 > > Attachments: image-2023-12-11-14-44-04-923.png > > > !image-2023-12-11-14-44-04-923.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KYLIN-5743) Set kylin.query.convert-sum-expression-enabled=true, fail to completely hit the aggregate index when the query contains sum (case when) expressions
[
https://issues.apache.org/jira/browse/KYLIN-5743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832159#comment-17832159
]
ASF subversion and git services commented on KYLIN-5743:
Commit 553219830193ec43f7b6d68fe626b04456633120 in kylin's branch
refs/heads/kylin5 from Pengfei Zhan
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=5532198301 ]
KYLIN-5743 optimization of SumExpression prevents ProjectJoinTransposeRule to
hit AggIndex
> Set kylin.query.convert-sum-expression-enabled=true, fail to completely hit
> the aggregate index when the query contains sum (case when) expressions
> ---
>
> Key: KYLIN-5743
> URL: https://issues.apache.org/jira/browse/KYLIN-5743
> Project: Kylin
> Issue Type: Bug
>Affects Versions: 5.0-beta
>Reporter: zhong.zhu
>Assignee: zhong.zhu
>Priority: Major
> Fix For: 5.0.0
>
>
> {code:sql}
> select
> sum(
> case
> when LO_COMMITDATE = '20230501' then LO_DISCOUNT
> end
> )
> from
> (
> select
> LO_COMMITDATE,
> LO_DISCOUNT,
> LINEORDER.LO_ORDERDATE
> from
> ssb.LINEORDER
> ) a
> where
> LO_COMMITDATE = (
> select
> max(LO_COMMITDATE)
> from
> ssb.LINEORDER
> )
> LIMIT
> 500
> {code}
> Fix the sum case when in this scenario so that it hits aggregated indexes
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KYLIN-5733) Export model TDS file in English interface, including Chinese when opening the file in text mode
[ https://issues.apache.org/jira/browse/KYLIN-5733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832151#comment-17832151 ] ASF subversion and git services commented on KYLIN-5733: Commit 387e68d0a669b84d269341a8a4127c5b70d6bf55 in kylin's branch refs/heads/kylin5 from fanfanAlice [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=387e68d0a6 ] KYLIN-5733 TDS template modify to English > Export model TDS file in English interface, including Chinese when opening > the file in text mode > > > Key: KYLIN-5733 > URL: https://issues.apache.org/jira/browse/KYLIN-5733 > Project: Kylin > Issue Type: Bug >Affects Versions: 5.0-beta >Reporter: zhong.zhu >Assignee: zhong.zhu >Priority: Critical > Fix For: 5.0.0 > > > *Steps to reproduce the issue:* > # Go to a model and click “Export TDS”. > # Open up the file in a text editor and look at the bottom. There are > Chinese characters. See attachment. > I confirmed this issue is present in 4.5.4 and .11, and likely exists in > other versions. The TDS file seems to work fine though. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KYLIN-5732) High-level security vulnerabilities:maven:org.xerial.snappy:snappy-java@1.1.8.4/CVE-2023-34455
[ https://issues.apache.org/jira/browse/KYLIN-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832150#comment-17832150 ] ASF subversion and git services commented on KYLIN-5732: Commit 3ec3063135f6b99cb4f7934d056a6ca7fd4a1e5e in kylin's branch refs/heads/kylin5 from Yinghao Lin [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=3ec3063135 ] KYLIN-5732 Upgrade snappy-java to fix vulnerabilities > High-level security > vulnerabilities:maven:org.xerial.snappy:snappy-java@1.1.8.4/CVE-2023-34455 > -- > > Key: KYLIN-5732 > URL: https://issues.apache.org/jira/browse/KYLIN-5732 > Project: Kylin > Issue Type: Bug >Affects Versions: 5.0-beta >Reporter: zhong.zhu >Assignee: zhong.zhu >Priority: Critical > Fix For: 5.0.0 > > Attachments: image-2023-12-05-19-01-48-480.png > > > Vulnerability Details: > !image-2023-12-05-19-01-48-480.png! > snyk link: [https://snyk.io/vuln/SNYK-JAVA-ORGXERIALSNAPPY-5710960] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] Batch fix to Kylin5 [kylin]
liyang-kylin merged PR #2165: URL: https://github.com/apache/kylin/pull/2165 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@kylin.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (KYLIN-5772) Fix vlunerability and excluding ekio
pengfei.zhan created KYLIN-5772: --- Summary: Fix vlunerability and excluding ekio Key: KYLIN-5772 URL: https://issues.apache.org/jira/browse/KYLIN-5772 Project: Kylin Issue Type: Bug Components: Others Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5771) Query cannot be interrupted during Segment pruning phase
pengfei.zhan created KYLIN-5771: --- Summary: Query cannot be interrupted during Segment pruning phase Key: KYLIN-5771 URL: https://issues.apache.org/jira/browse/KYLIN-5771 Project: Kylin Issue Type: Bug Components: Query Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KYLIN-5770) Update spring-security-config from 5.7.8 to 5.7.10
[ https://issues.apache.org/jira/browse/KYLIN-5770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pengfei.zhan updated KYLIN-5770: Component/s: Query Engine > Update spring-security-config from 5.7.8 to 5.7.10 > -- > > Key: KYLIN-5770 > URL: https://issues.apache.org/jira/browse/KYLIN-5770 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: 5.0-beta >Reporter: pengfei.zhan >Assignee: pengfei.zhan >Priority: Major > Fix For: 5.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KYLIN-5770) Update spring-security-config from 5.7.8 to 5.7.10
[ https://issues.apache.org/jira/browse/KYLIN-5770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pengfei.zhan updated KYLIN-5770: Affects Version/s: 5.0-beta > Update spring-security-config from 5.7.8 to 5.7.10 > -- > > Key: KYLIN-5770 > URL: https://issues.apache.org/jira/browse/KYLIN-5770 > Project: Kylin > Issue Type: Bug >Affects Versions: 5.0-beta >Reporter: pengfei.zhan >Assignee: pengfei.zhan >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KYLIN-5770) Update spring-security-config from 5.7.8 to 5.7.10
[ https://issues.apache.org/jira/browse/KYLIN-5770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pengfei.zhan reassigned KYLIN-5770: --- Assignee: pengfei.zhan > Update spring-security-config from 5.7.8 to 5.7.10 > -- > > Key: KYLIN-5770 > URL: https://issues.apache.org/jira/browse/KYLIN-5770 > Project: Kylin > Issue Type: Bug >Reporter: pengfei.zhan >Assignee: pengfei.zhan >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5770) Update spring-security-config from 5.7.8 to 5.7.10
pengfei.zhan created KYLIN-5770: --- Summary: Update spring-security-config from 5.7.8 to 5.7.10 Key: KYLIN-5770 URL: https://issues.apache.org/jira/browse/KYLIN-5770 Project: Kylin Issue Type: Bug Reporter: pengfei.zhan -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KYLIN-5770) Update spring-security-config from 5.7.8 to 5.7.10
[ https://issues.apache.org/jira/browse/KYLIN-5770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pengfei.zhan updated KYLIN-5770: Fix Version/s: 5.0.0 > Update spring-security-config from 5.7.8 to 5.7.10 > -- > > Key: KYLIN-5770 > URL: https://issues.apache.org/jira/browse/KYLIN-5770 > Project: Kylin > Issue Type: Bug >Affects Versions: 5.0-beta >Reporter: pengfei.zhan >Assignee: pengfei.zhan >Priority: Major > Fix For: 5.0.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5769) Support split function
pengfei.zhan created KYLIN-5769: --- Summary: Support split function Key: KYLIN-5769 URL: https://issues.apache.org/jira/browse/KYLIN-5769 Project: Kylin Issue Type: Improvement Components: Job Engine, Query Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5768) Refine some code for localcache
pengfei.zhan created KYLIN-5768: --- Summary: Refine some code for localcache Key: KYLIN-5768 URL: https://issues.apache.org/jira/browse/KYLIN-5768 Project: Kylin Issue Type: Improvement Components: Query Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5767) Calculating total rows abnormal when jdbc datasource is connnected
pengfei.zhan created KYLIN-5767: --- Summary: Calculating total rows abnormal when jdbc datasource is connnected Key: KYLIN-5767 URL: https://issues.apache.org/jira/browse/KYLIN-5767 Project: Kylin Issue Type: Bug Components: Job Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5766) Normalize query cache key
pengfei.zhan created KYLIN-5766: --- Summary: Normalize query cache key Key: KYLIN-5766 URL: https://issues.apache.org/jira/browse/KYLIN-5766 Project: Kylin Issue Type: Improvement Components: Query Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5765) Optimization of register udf operators
pengfei.zhan created KYLIN-5765: --- Summary: Optimization of register udf operators Key: KYLIN-5765 URL: https://issues.apache.org/jira/browse/KYLIN-5765 Project: Kylin Issue Type: Improvement Components: Query Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5764) Sum expression supports if condition
pengfei.zhan created KYLIN-5764: --- Summary: Sum expression supports if condition Key: KYLIN-5764 URL: https://issues.apache.org/jira/browse/KYLIN-5764 Project: Kylin Issue Type: Improvement Components: Query Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KYLIN-5757) No way to deselect query objects after selecting all of them on this page
[ https://issues.apache.org/jira/browse/KYLIN-5757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pengfei.zhan updated KYLIN-5757: Summary: No way to deselect query objects after selecting all of them on this page (was: Optimization of the API of listing models ) > No way to deselect query objects after selecting all of them on this page > - > > Key: KYLIN-5757 > URL: https://issues.apache.org/jira/browse/KYLIN-5757 > Project: Kylin > Issue Type: Improvement > Components: Others >Affects Versions: 5.0-beta >Reporter: pengfei.zhan >Assignee: pengfei.zhan >Priority: Major > Fix For: 5.0-beta > > > The API of listing models very slow at sometime, improve it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KYLIN-5757) Optimization of the API of listing models
[ https://issues.apache.org/jira/browse/KYLIN-5757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pengfei.zhan updated KYLIN-5757: Attachment: (was: 1.png) > Optimization of the API of listing models > -- > > Key: KYLIN-5757 > URL: https://issues.apache.org/jira/browse/KYLIN-5757 > Project: Kylin > Issue Type: Improvement > Components: Others >Affects Versions: 5.0-beta >Reporter: pengfei.zhan >Assignee: pengfei.zhan >Priority: Major > Fix For: 5.0-beta > > > The API of listing models very slow at sometime, improve it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KYLIN-5763) Very slow to filter query history
[
https://issues.apache.org/jira/browse/KYLIN-5763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
pengfei.zhan updated KYLIN-5763:
Description:
If there are many queryHistory records,
1) filtering them by query object is very slow;
2) searching them by key(the model name) cannot work.
However, executing query in mysql, the result responses very fast.
{code:java}
select distinct model from kylin_query_history_realization
where project_name = "abc";
select distinct engine_type from kylin_query_history where engine_type <>
'NATIVE' and project_name = "abc";
{code}
was:
The following sql can not match index just has dimension lstg_format_name.
{code:sql}
select max(lstg_format_name) max1,
max( case when 1=1 then lstg_format_nameend ) max2
from test_kylin_fact{code}
> Very slow to filter query history
> -
>
> Key: KYLIN-5763
> URL: https://issues.apache.org/jira/browse/KYLIN-5763
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: 5.0-beta
>Reporter: pengfei.zhan
>Assignee: pengfei.zhan
>Priority: Major
> Fix For: 5.0-beta
>
>
> If there are many queryHistory records,
> 1) filtering them by query object is very slow;
> 2) searching them by key(the model name) cannot work.
>
> However, executing query in mysql, the result responses very fast.
> {code:java}
> select distinct model from kylin_query_history_realization
> where project_name = "abc";
> select distinct engine_type from kylin_query_history where engine_type <>
> 'NATIVE' and project_name = "abc";
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KYLIN-5763) Very slow to filter query history
[
https://issues.apache.org/jira/browse/KYLIN-5763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
pengfei.zhan updated KYLIN-5763:
Description:
If there are many queryHistory records,
1) filtering them by query object is very slow;
2) searching them by key(the model name) cannot work.
However, executing query in mysql, the result responses very fast.
{code:sql}
select distinct model from kylin_query_history_realization
where project_name = "abc";
select distinct engine_type from kylin_query_history where engine_type <>
'NATIVE' and project_name = "abc";
{code}
was:
If there are many queryHistory records,
1) filtering them by query object is very slow;
2) searching them by key(the model name) cannot work.
However, executing query in mysql, the result responses very fast.
{code:java}
select distinct model from kylin_query_history_realization
where project_name = "abc";
select distinct engine_type from kylin_query_history where engine_type <>
'NATIVE' and project_name = "abc";
{code}
> Very slow to filter query history
> -
>
> Key: KYLIN-5763
> URL: https://issues.apache.org/jira/browse/KYLIN-5763
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: 5.0-beta
>Reporter: pengfei.zhan
>Assignee: pengfei.zhan
>Priority: Major
> Fix For: 5.0-beta
>
>
> If there are many queryHistory records,
> 1) filtering them by query object is very slow;
> 2) searching them by key(the model name) cannot work.
>
> However, executing query in mysql, the result responses very fast.
> {code:sql}
> select distinct model from kylin_query_history_realization
> where project_name = "abc";
> select distinct engine_type from kylin_query_history where engine_type <>
> 'NATIVE' and project_name = "abc";
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KYLIN-5763) Very slow to filter query history
[
https://issues.apache.org/jira/browse/KYLIN-5763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
pengfei.zhan updated KYLIN-5763:
Summary: Very slow to filter query history (was: Equivalent aggregations
matches index failed when using the strategy of dimension as measure)
> Very slow to filter query history
> -
>
> Key: KYLIN-5763
> URL: https://issues.apache.org/jira/browse/KYLIN-5763
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: 5.0-beta
>Reporter: pengfei.zhan
>Assignee: pengfei.zhan
>Priority: Major
> Fix For: 5.0-beta
>
>
> The following sql can not match index just has dimension lstg_format_name.
> {code:sql}
> select max(lstg_format_name) max1,
> max( case when 1=1 then lstg_format_nameend ) max2
> from test_kylin_fact{code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KYLIN-5759) Log governance
[ https://issues.apache.org/jira/browse/KYLIN-5759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pengfei.zhan updated KYLIN-5759: Description: Improve the log governance. (was: When a task is scheduled, the task is logged into runningJobs and the current thread is logged into runningJobThreads, which is expected to be the thread that executes the task, but in fact it is the scheduler's thread that is logged, which results in subsequent attempts to interrupt the scheduler FetcherRunner when killing the task.) > Log governance > -- > > Key: KYLIN-5759 > URL: https://issues.apache.org/jira/browse/KYLIN-5759 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Affects Versions: 5.0-beta >Reporter: pengfei.zhan >Assignee: pengfei.zhan >Priority: Major > Fix For: 5.0-beta > > > Improve the log governance. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KYLIN-5759) Log governance
[ https://issues.apache.org/jira/browse/KYLIN-5759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pengfei.zhan updated KYLIN-5759: Issue Type: Improvement (was: Bug) > Log governance > -- > > Key: KYLIN-5759 > URL: https://issues.apache.org/jira/browse/KYLIN-5759 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Affects Versions: 5.0-beta >Reporter: pengfei.zhan >Assignee: pengfei.zhan >Priority: Major > Fix For: 5.0-beta > > > When a task is scheduled, the task is logged into runningJobs and the current > thread is logged into runningJobThreads, which is expected to be the thread > that executes the task, but in fact it is the scheduler's thread that is > logged, which results in subsequent attempts to interrupt the scheduler > FetcherRunner when killing the task. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KYLIN-5759) Log governance
[ https://issues.apache.org/jira/browse/KYLIN-5759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pengfei.zhan updated KYLIN-5759: Summary: Log governance (was: Modify "ExecutableContext#addRunningJob" to avoid schedule job more than once) > Log governance > -- > > Key: KYLIN-5759 > URL: https://issues.apache.org/jira/browse/KYLIN-5759 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: 5.0-beta >Reporter: pengfei.zhan >Assignee: pengfei.zhan >Priority: Major > Fix For: 5.0-beta > > > When a task is scheduled, the task is logged into runningJobs and the current > thread is logged into runningJobThreads, which is expected to be the thread > that executes the task, but in fact it is the scheduler's thread that is > logged, which results in subsequent attempts to interrupt the scheduler > FetcherRunner when killing the task. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KYLIN-5755) Hitting snapshot has a higher priority than answering min/max mesaure with metadata
[ https://issues.apache.org/jira/browse/KYLIN-5755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] pengfei.zhan resolved KYLIN-5755. - Resolution: Duplicate > Hitting snapshot has a higher priority than answering min/max mesaure with > metadata > --- > > Key: KYLIN-5755 > URL: https://issues.apache.org/jira/browse/KYLIN-5755 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Affects Versions: 5.0-beta >Reporter: pengfei.zhan >Assignee: pengfei.zhan >Priority: Major > Fix For: 5.0-beta > > > KYLIN-5578 using metadata to response min/max aggregation queries. When > querying only columns on the dimension table, the query result from the > snapshot does not match the reuslt from metadata, choose the former one. If > the query result of `min/max` aggregation is based on a left/inner join > flat-table, query result from metadata is preferred. -- This message was sent by Atlassian Jira (v8.20.10#820010)
