[jira] [Created] (KYLIN-5847) kylin4.0.1 send mail throw exception
suoli created KYLIN-5847:
Summary: kylin4.0.1 send mail throw exception
Key: KYLIN-5847
URL: https://issues.apache.org/jira/browse/KYLIN-5847
Project: Kylin
Issue Type: Bug
Components: Job Engine
Affects Versions: v4.0.1
Environment: kylin4.0.1-spark2
Reporter: suoli
kylin version is 4.0.1-spark2
*when job is discarded or successed，kylin.log content is:*
2024-04-11 13:46:35,241 ERROR [Scheduler 512464937 Job
fd0b6de7-d474-47a4-9e65-e943366b7805-56] freemarker.runtime:60 : Error
executing FreeMarker template
FreeMarker template error:
The following has evaluated to null or missing:
==> env_name [in template "JOB_DISCARD.ftl" at line 101, column 19]
Tip: If the failing expression is known to be legally refer to something that's
sometimes null or missing, either specify a default value like
myOptionalVar!myDefault, or use <#if
myOptionalVar??>when-present<#else>when-missing. (These only cover the
last step of the expression; to cover the whole expression, use parenthesis:
(myOptionalVar.foo)!myDefault, (myOptionalVar.foo)??
FTL stack trace ("~" means nesting-related):
- Failed at: ${env_name} [in template "JOB_DISCARD.ftl" at line 101, column 17]
Java stack trace (for programmers):
freemarker.core.InvalidReferenceException: [... Exception message was already
printed; see it above ...]
at
freemarker.core.InvalidReferenceException.getInstance(InvalidReferenceException.java:131)
at freemarker.core.EvalUtil.coerceModelToString(EvalUtil.java:355)
at freemarker.core.Expression.evalAndCoerceToString(Expression.java:82)
at freemarker.core.DollarVariable.accept(DollarVariable.java:41)
at freemarker.core.Environment.visit(Environment.java:324)
at freemarker.core.MixedContent.accept(MixedContent.java:54)
at freemarker.core.Environment.visit(Environment.java:324)
at freemarker.core.Environment.process(Environment.java:302)
at freemarker.template.Template.process(Template.java:325)
at
org.apache.kylin.common.util.MailTemplateProvider.buildMailContent(MailTemplateProvider.java:63)
at
org.apache.kylin.job.util.MailNotificationUtil.getMailContent(MailNotificationUtil.java:70)
at org.apache.kylin.engine.mr.CubingJob.formatNotifications(CubingJob.java:251)
at
org.apache.kylin.job.execution.AbstractExecutable.notifyUserStatusChange(AbstractExecutable.java:368)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.onStatusChange(DefaultChainedExecutable.java:179)
at org.apache.kylin.engine.mr.CubingJob.onStatusChange(CubingJob.java:280)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.onExecuteFinished(DefaultChainedExecutable.java:125)
at org.apache.kylin.engine.mr.CubingJob.onExecuteFinished(CubingJob.java:276)
at
org.apache.kylin.job.execution.AbstractExecutable.onExecuteFinishedWithRetry(AbstractExecutable.java:138)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:228)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:113)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2024-04-11 13:46:35,254 ERROR [Scheduler 512464937 Job
fd0b6de7-d474-47a4-9e65-e943366b7805-56] execution.AbstractExecutable:371 :
error send email
java.lang.NullPointerException
at
org.apache.kylin.shaded.com.google.common.base.Preconditions.checkNotNull(Preconditions.java:878)
at
org.apache.kylin.shaded.com.google.common.base.Joiner.toString(Joiner.java:452)
at
org.apache.kylin.shaded.com.google.common.base.Joiner.appendTo(Joiner.java:109)
at
org.apache.kylin.shaded.com.google.common.base.Joiner.appendTo(Joiner.java:152)
at org.apache.kylin.shaded.com.google.common.base.Joiner.join(Joiner.java:195)
at org.apache.kylin.shaded.com.google.common.base.Joiner.join(Joiner.java:185)
at org.apache.kylin.shaded.com.google.common.base.Joiner.join(Joiner.java:203)
at
org.apache.kylin.job.util.MailNotificationUtil.getMailTitle(MailNotificationUtil.java:79)
at org.apache.kylin.engine.mr.CubingJob.formatNotifications(CubingJob.java:252)
at
org.apache.kylin.job.execution.AbstractExecutable.notifyUserStatusChange(AbstractExecutable.java:368)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.onStatusChange(DefaultChainedExecutable.java:179)
at org.apache.kylin.engine.mr.CubingJob.onStatusChange(CubingJob.java:280)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.onExecuteFinished(DefaultChainedExecutable.java:125)
at org.apache.kylin.engine.mr.CubingJob.onExecuteFinished(CubingJob.java:276)
at
org.apache.kylin.job.execution.AbstractExecutable.onExecuteFinishedWithRetry(AbstractExecutable.java:138)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:228)
at
[jira] [Created] (KYLIN-5846) upgrade spark version to 3.2.0-kylin-4.6.16.0.1
Zhimin Wu created KYLIN-5846: Summary: upgrade spark version to 3.2.0-kylin-4.6.16.0.1 Key: KYLIN-5846 URL: https://issues.apache.org/jira/browse/KYLIN-5846 Project: Kylin Issue Type: Bug Components: Spark Engine Affects Versions: 5.0-beta Reporter: Zhimin Wu Assignee: Zhimin Wu Fix For: 5.0.0 upgrade spark version to 3.2.0-kylin-4.6.16.0.1 -- This message was sent by Atlassian Jira (v8.20.10#820010)
来自wxm的邮件
退订
[jira] [Created] (KYLIN-5845) Updating the model metadata when getting the model list with high concurrency may cause the model metadata cache to be abnormal and the model status to become broken
Zhimin Wu created KYLIN-5845: Summary: Updating the model metadata when getting the model list with high concurrency may cause the model metadata cache to be abnormal and the model status to become broken Key: KYLIN-5845 URL: https://issues.apache.org/jira/browse/KYLIN-5845 Project: Kylin Issue Type: Bug Components: Metadata Affects Versions: 5.0-beta Reporter: Zhimin Wu Assignee: Zhimin Wu Fix For: 5.0.0 Fix the issue caused by concurrent initialization by checking if the model has already been initialized. If so, skip the initialization step. Otherwise, perform a copy operation first and then initialize the model to avoid concurrent initialization problems. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5844) Refactor flat table sql ACL
Zhimin Wu created KYLIN-5844: Summary: Refactor flat table sql ACL Key: KYLIN-5844 URL: https://issues.apache.org/jira/browse/KYLIN-5844 Project: Kylin Issue Type: Improvement Components: Job Engine Reporter: Zhimin Wu Assignee: Zhimin Wu Fix For: 5.0.0 Flat-table sql should consider acl, for role_admin ignores acl -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5843) Column access control permissions do not take effect for query-push-down
Zhimin Wu created KYLIN-5843: Summary: Column access control permissions do not take effect for query-push-down Key: KYLIN-5843 URL: https://issues.apache.org/jira/browse/KYLIN-5843 Project: Kylin Issue Type: Improvement Components: Query Engine Affects Versions: 5.0-beta Reporter: Zhimin Wu Assignee: Zhimin Wu Fix For: 5.0.0 *Dev Design* Parse the SQL query to generate a SqlNode, and traverse this SqlNode to find all tables used in the database. Replace each of these tables with a subquery that retrieves only the columns that the user has permission to access. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5842) When an async query hits the cache, it is unable to download the results.
Zhimin Wu created KYLIN-5842: Summary: When an async query hits the cache, it is unable to download the results. Key: KYLIN-5842 URL: https://issues.apache.org/jira/browse/KYLIN-5842 Project: Kylin Issue Type: Improvement Components: Query Engine Affects Versions: 5.0-beta Reporter: Zhimin Wu Assignee: Zhimin Wu Fix For: 5.0.0 When an asynchronous query hits the cache, it is unable to download the results. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5841) Upgrade org.apache.avro:avro to version 1.11.3
Zhimin Wu created KYLIN-5841: Summary: Upgrade org.apache.avro:avro to version 1.11.3 Key: KYLIN-5841 URL: https://issues.apache.org/jira/browse/KYLIN-5841 Project: Kylin Issue Type: Improvement Components: Others Reporter: Zhimin Wu Assignee: Zhimin Wu Fix For: 5.0.0 Upgrade org.apache.avro:avro to version 1.11.3 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5840) Upgrade org.xerial.snappy:snappy-java to version 1.1.10.4
Zhimin Wu created KYLIN-5840: Summary: Upgrade org.xerial.snappy:snappy-java to version 1.1.10.4 Key: KYLIN-5840 URL: https://issues.apache.org/jira/browse/KYLIN-5840 Project: Kylin Issue Type: Improvement Components: Others Reporter: Zhimin Wu Assignee: Zhimin Wu Fix For: 5.0.0 Upgrade org.xerial.snappy:snappy-java to version 1.1.10.4 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5839) Block agg-push-down for non-equal-join
Zhimin Wu created KYLIN-5839:
Summary: Block agg-push-down for non-equal-join
Key: KYLIN-5839
URL: https://issues.apache.org/jira/browse/KYLIN-5839
Project: Kylin
Issue Type: Bug
Components: Query Engine
Affects Versions: 5.0-beta
Reporter: Zhimin Wu
Assignee: Zhimin Wu
Fix For: 5.0.0
The related non-equiv-join sql is as follows. If you want a corresponding
equal-join sql, just remove the condition content `or (t2.fcol_10 is null and
t5.fcol_17 is null)`.
{code:java}
select t2.fcol_7 fcol_7, count(distinct t2.fcol_6) fcol_6
from (select t1.company_code fcol_6, t1.type_name fcol_7,
case when t1.created_date = t0.fcol_1 then 'TRUE'
else 'FALSE' end fcol_10
from ( select company_code, created_date, type_name
from "DEFAULT"."TEST_AGG_PUSH_DOWN" ) t1
join ( select company_code, max(created_date) fcol_1
from "DEFAULT"."TEST_AGG_PUSH_DOWN"
group by company_code
) t0 on t1.company_code = t0.company_code
) t2 join ( select 'TRUE' fcol_17 ) t5
on t2.fcol_10 = t5.fcol_17 or (t2.fcol_10 is null and t5.fcol_17 is null)
group by t2.fcol_7{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KYLIN-5838) Aggregate query without aggregate function may not answered by index when computed column is used as association key
Zhimin Wu created KYLIN-5838: Summary: Aggregate query without aggregate function may not answered by index when computed column is used as association key Key: KYLIN-5838 URL: https://issues.apache.org/jira/browse/KYLIN-5838 Project: Kylin Issue Type: Bug Components: Query Engine Affects Versions: 5.0-beta Reporter: Zhimin Wu Assignee: Zhimin Wu Fix For: 5.0.0 In cases where there are no measures, the join key is not being correctly replaced. The fix is quite simple; when collecting cc replacements in SqlNode, consider the expressions in the condition for cases where there are only group by columns present. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5837) After turning off the computed column function, the query cannot hit the existing recommended index
Zhimin Wu created KYLIN-5837: Summary: After turning off the computed column function, the query cannot hit the existing recommended index Key: KYLIN-5837 URL: https://issues.apache.org/jira/browse/KYLIN-5837 Project: Kylin Issue Type: Bug Components: Query Engine Affects Versions: 5.0-beta Reporter: Zhimin Wu Assignee: Zhimin Wu Fix For: 5.0.0 *Dev Design* * The configuration parameter "kylin.metadata.only-reuse-user-defined-computed-column=false" is designed to enable the usage of user-defined computed columns for reusing only. This decouples the feature of shielding columns from the configuration, which means that the shielding column feature can still be used even with this parameter set to false. * By default, Kylin uses computed columns (CC) for implicit column conversions, which means that user-defined computed columns can replace implicit expressions. However, if the configuration parameter "kylin.query.implicit-computed-column-convert" is set to "false", the user-defined computed columns will not be used for implicit column conversions. * The configuration parameter "kylin.query.agg-computed-column-rewrite" is related to computed columns v2 and it replaces columns at the semantic RelNode level, while "kylin.query.implicit-computed-column-convert" performs column replacements at the SqlNode level. As a general rule, if "kylin.query.implicit-computed-column-convert" is set to false, then "kylin.query.agg-computed-column-rewrite" will not take effect either. However, if "kylin.query.implicit-computed-column-convert" is set to true, then "kylin.query.agg-computed-column-rewrite" can be set to either true or false. * All three configurations can now be modified at the project level. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5836) MapR env Build error
Zhimin Wu created KYLIN-5836: Summary: MapR env Build error Key: KYLIN-5836 URL: https://issues.apache.org/jira/browse/KYLIN-5836 Project: Kylin Issue Type: Bug Components: Job Engine Affects Versions: 5.0-beta Reporter: Zhimin Wu Assignee: Zhimin Wu Fix For: 5.0.0 The MapR environment construction task has failed due to the default Hadoop configuration not being loaded. *RC* The issue is related to the Configuration in HadoopUtil, which by default uses the configuration files located in the KE/hadoop_conf directory. In other environments where the configuration files are complete, there are no issues with retrieving the required parameters. However, in a MapR environment, the default configuration files are located in a different directory, resulting in an incomplete conf object being returned and causing the construction task to fail. *Dev Design* To address this issue, directly loading the default parameters using the Configuration object should provide the complete configuration required for the task to run successfully. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5835) add rules combination to scalar subquery join
Zhimin Wu created KYLIN-5835: Summary: add rules combination to scalar subquery join Key: KYLIN-5835 URL: https://issues.apache.org/jira/browse/KYLIN-5835 Project: Kylin Issue Type: Improvement Affects Versions: 5.0-beta Reporter: Zhimin Wu Assignee: Zhimin Wu Fix For: 5.0.0 add rules combination to scalar subquery join -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5834) The system reports error when the number of models exported exceeds 256 at a time
Zhimin Wu created KYLIN-5834: Summary: The system reports error when the number of models exported exceeds 256 at a time Key: KYLIN-5834 URL: https://issues.apache.org/jira/browse/KYLIN-5834 Project: Kylin Issue Type: Bug Components: Metadata Affects Versions: 5.0-beta Reporter: Zhimin Wu Assignee: Zhimin Wu Fix For: 5.0.0 *Back Ground* Kylin's model export feature generates a compressed package containing all selected models for export. However, when the number of selected models exceeds 256, the export function will fail and display an unfriendly error message to the user on a separate page, indicating an "IndexOutOfBoundsException: Index: 256". *Root Cause* The size of the list received from the frontend has exceeded the maximum limit set for form-urlencoded request parameters in Spring. The default limit for the automatic growth of arrays and collections in Spring's DataBinder class is 256. *Dev Design* A new system configuration item, "kylin.web.data-binder.auto-grow-collection-limit", has been added to "KylinConfigBase.java". This item allows users to set the limit for the size of array lists when handling request parameters. The default value is 256. In the base controller class "NBasicController.java", the parameter "autoGrowCollectionLimit" is set to the value of the system configuration item "kylin.web.data-binder.auto-grow-collection-limit". It should be noted that setting this value too high could potentially lead to out of memory errors if the application is subject to network attacks, such as a request containing "ids[10]". This change will affect all subclasses that inherit from the base controller class. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5833) Refactor QueryContext
Zhimin Wu created KYLIN-5833: Summary: Refactor QueryContext Key: KYLIN-5833 URL: https://issues.apache.org/jira/browse/KYLIN-5833 Project: Kylin Issue Type: Improvement Reporter: Zhimin Wu Assignee: Zhimin Wu Fix For: 5.0.0 Refactor QueryContext -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5832) Add API to determine whether the query hits the index
Zhimin Wu created KYLIN-5832:
Summary: Add API to determine whether the query hits the index
Key: KYLIN-5832
URL: https://issues.apache.org/jira/browse/KYLIN-5832
Project: Kylin
Issue Type: Improvement
Components: Others
Reporter: Zhimin Wu
Assignee: Zhimin Wu
Fix For: 5.0.0
Add a new API to determine if a query hits an index. This API has the following
functions.
# Determine whether the model is hit or not.
# Determine if there is data in the segment.
API : {{POST [http://host:port/kylin/api/query/detection]}}
{{}}
||*Param*||*Datatype*||Required||
|project|string|True|
|sql
|string|
True
|
|limit|int|False|
|offset|int|False|
Sample
{code:java}
curl --location 'http://host:port/kylin/api/query/detection' \
--header 'Accept: application/vnd.apache.kylin-v4-public+json' \
--header 'Accept-Language: cn' \
--header 'Content-Type: application/json;charset=UTF-8' \
--data '{
"project": "project_ch",
"sql": "SELECT LO_PARTKEY, SUM(LO_REVENUE) AS TOTAL_REVENUE\nFROM
SSB.P_LINEORDER\nWHERE LO_ORDERDATE between '\''19920101'\'' AND
'\''19970803'\'' \ngroup by LO_PARTKEY\norder by SUM(LO_REVENUE) DESC\nLIMIT
500"
}' {code}
Response
||*Param*||*Datatype*||
|code|string|
|data|object|
|is_exception|boolean|
|exception_message|string|
|query_id|string|
|is_push_down
|boolean|
|is_post_aggregation
|boolean|
|is_table_index
|boolean|
|is_base_index
|boolean|
|is_cache|boolean|
|is_constants|boolean|
|realizations
|object[]|
|model_id|string|
|model_alias|string|
|layout_id|int|
|index_type|string|
|partial_match_model|boolean|
|valid|boolean|
|is_table_index
|boolean|
|is_base_index
|boolean|
Sample
{code:java}
{
"code": "000",
"data": {
"is_exception": false,
"exception_message": null,
"query_id": "0ef8d8f2-c234-8211-f841-5b3db61ba217",
"is_push_down": false,
"is_post_aggregation": true,
"is_table_index": false,
"is_base_index": true,
"is_cache": false,
"is_constants": false,
"realizations": [
{
"model_id": "aa7d0eab-6f72-3b02-019f-9f2fc84a0ecb",
"model_alias": "AUTO_MODEL_P_LINEORDER_1",
"layout_id": 1,
"index_type": "Agg Index",
"partial_match_model": false,
"valid": true,
"is_table_index": false,
"is_base_index": true
}
]
},
"msg": ""
} {code}
{{}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KYLIN-5831) Enhance SCD2, history table supports left-closed and right-closed time interval
Zhimin Wu created KYLIN-5831:
Summary: Enhance SCD2, history table supports left-closed and
right-closed time interval
Key: KYLIN-5831
URL: https://issues.apache.org/jira/browse/KYLIN-5831
Project: Kylin
Issue Type: Improvement
Components: Spark Engine
Reporter: Zhimin Wu
Assignee: Zhimin Wu
Fix For: 5.0.0
Current SCD2 only support { >= and < } join condition.
In this issue we supported { >=, < }, \{>=, <=}, \{ >, <=} join condition.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KYLIN-5830) When the query connects two or more tables, the query results are inconsistent whether the order by field is in the select field or not
Zhimin Wu created KYLIN-5830:
Summary: When the query connects two or more tables, the query
results are inconsistent whether the order by field is in the select field or
not
Key: KYLIN-5830
URL: https://issues.apache.org/jira/browse/KYLIN-5830
Project: Kylin
Issue Type: New Feature
Components: Query Engine
Affects Versions: 5.0-beta
Reporter: Zhimin Wu
Assignee: Zhimin Wu
Fix For: 5.0.0
Q1:
{{}}
{code:java}
select CUSTOMER.C_PHONE
-- ,LO_PARTKEY
from SSB.CUSTOMER left join SSB.LINEORDER on
CUSTOMER.C_CUSTKEY=LINEORDER.LO_CUSTKEY
order by LINEORDER.LO_PARTKEY NULLS LAST
LIMIT 500{code}
{{}}
Calcite Plan：
{{}}
{code:java}
KapProjectRel(C_PHONE=[$6], ctx=[])
KapLimitRel(ctx=[], fetch=[500])
KapSortRel(sort0=[$12], dir0=[ASC], ctx=[])
KapJoinRel(condition=[=($0, $11)], joinType=[left], ctx=[])
KapTableScan(table=[[SSB, CUSTOMER]], ctx=[1@NDataflow [test]],
fields=[[0, 1, 2, 3, 4, 5, 6, 7, 8]])
KapTableScan(table=[[SSB, LINEORDER]], ctx=[2@NDataflow [test1]],
fields=[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17]]){code}
{{}}
Q2:
{{}}
{code:java}
select CUSTOMER.C_PHONE
,LO_PARTKEY
from SSB.CUSTOMER left join SSB.LINEORDER on
CUSTOMER.C_CUSTKEY=LINEORDER.LO_CUSTKEY
order by LINEORDER.LO_PARTKEY NULLS LAST
LIMIT 500{code}
{{}}
Calcite Plan
{{}}
{code:java}
KapLimitRel(ctx=[], fetch=[500])
KapSortRel(sort0=[$1], dir0=[ASC], ctx=[])
KapProjectRel(C_PHONE=[$6], LO_PARTKEY=[$12], ctx=[])
KapJoinRel(condition=[=($0, $11)], joinType=[left], ctx=[])
KapTableScan(table=[[SSB, CUSTOMER]], ctx=[1@NDataflow [test]],
fields=[[0, 1, 2, 3, 4, 5, 6, 7, 8]])
KapTableScan(table=[[SSB, LINEORDER]], ctx=[2@NDataflow [test1]],
fields=[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17]])
{code}
{{}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KYLIN-5829) Add a tool for generating sql from dumped metadata
Zhimin Wu created KYLIN-5829:
Summary: Add a tool for generating sql from dumped metadata
Key: KYLIN-5829
URL: https://issues.apache.org/jira/browse/KYLIN-5829
Project: Kylin
Issue Type: New Feature
Components: Metadata
Reporter: Zhimin Wu
Assignee: Zhimin Wu
Fix For: 5.0.0
Add a tool for generating sql from dumped metadata
The usage of the command is {{./bin/creata-table.sh
\{absolute_table_directory}}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KYLIN-5828) During multi-jobs concurrent building, the flat table may use inconsistent global dictionaries, resulting in incorrect count distinct query results.
zhimin wu created KYLIN-5828: Summary: During multi-jobs concurrent building, the flat table may use inconsistent global dictionaries, resulting in incorrect count distinct query results. Key: KYLIN-5828 URL: https://issues.apache.org/jira/browse/KYLIN-5828 Project: Kylin Issue Type: Bug Components: Storage - Parquet Reporter: zhimin wu Assignee: zhimin wu *Root Cause* When multiple tasks are concurrently building and using the same global dictionary, the consistency of the dictionary version used in the flat table encoding process is not guaranteed. At the same time, another task expands the dictionary, causing some flat table partitions to mistakenly use the new version of the dictionary partition file. Due to the inconsistent data distribution, the correct dictionary content cannot be obtained, resulting in a flat table encoding column of 0 and ultimately causing an abnormal count distinct value. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5827) cp code from 4.6.18
chunyan.huang created KYLIN-5827: Summary: cp code from 4.6.18 Key: KYLIN-5827 URL: https://issues.apache.org/jira/browse/KYLIN-5827 Project: Kylin Issue Type: Improvement Reporter: chunyan.huang Assignee: chunyan.huang Fix For: 5.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5826) table index diaglog scroll issue
chunyan.huang created KYLIN-5826: Summary: table index diaglog scroll issue Key: KYLIN-5826 URL: https://issues.apache.org/jira/browse/KYLIN-5826 Project: Kylin Issue Type: Improvement Reporter: chunyan.huang Assignee: chunyan.huang Fix For: 5.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5825) clear all mandatory
chunyan.huang created KYLIN-5825: Summary: clear all mandatory Key: KYLIN-5825 URL: https://issues.apache.org/jira/browse/KYLIN-5825 Project: Kylin Issue Type: Improvement Reporter: chunyan.huang Assignee: chunyan.huang Fix For: 5.0.0 Support clearing required dimensions -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5824) bug fix
chunyan.huang created KYLIN-5824: Summary: bug fix Key: KYLIN-5824 URL: https://issues.apache.org/jira/browse/KYLIN-5824 Project: Kylin Issue Type: Bug Reporter: chunyan.huang Assignee: chunyan.huang Fix For: 5.0.0 The system reports error when the number of models exceeds 256 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5823) text Recognize table index issue
chunyan.huang created KYLIN-5823: Summary: text Recognize table index issue Key: KYLIN-5823 URL: https://issues.apache.org/jira/browse/KYLIN-5823 Project: Kylin Issue Type: Bug Reporter: chunyan.huang Assignee: chunyan.huang Fix For: 5.0.0 When adding a table index, the text recognition function cannot be used to identify the columns contained in the measure -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5822) only edit shardby disable save issue
chunyan.huang created KYLIN-5822: Summary: only edit shardby disable save issue Key: KYLIN-5822 URL: https://issues.apache.org/jira/browse/KYLIN-5822 Project: Kylin Issue Type: Bug Reporter: chunyan.huang Assignee: chunyan.huang Fix For: 5.0.0 Unable to save after modifying the Shardby column of the existing table index -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5821) refine setTableInView issue
chunyan.huang created KYLIN-5821: Summary: refine setTableInView issue Key: KYLIN-5821 URL: https://issues.apache.org/jira/browse/KYLIN-5821 Project: Kylin Issue Type: Bug Reporter: chunyan.huang Assignee: chunyan.huang On the model editing page, the positioning of the "Show Table" function in the search bar is not accurate -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5820) batch add measures issue
chunyan.huang created KYLIN-5820: Summary: batch add measures issue Key: KYLIN-5820 URL: https://issues.apache.org/jira/browse/KYLIN-5820 Project: Kylin Issue Type: Improvement Reporter: chunyan.huang Assignee: chunyan.huang h1. Batch adding measure pages supports paging display columns to optimize the page response performance when there are too many columns -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5819) edit cc tips issue
chunyan.huang created KYLIN-5819: Summary: edit cc tips issue Key: KYLIN-5819 URL: https://issues.apache.org/jira/browse/KYLIN-5819 Project: Kylin Issue Type: Improvement Reporter: chunyan.huang Assignee: chunyan.huang h1. Optimize the copywriting tips that will clear the relevant indexes after modifying the expression of the computed column -- This message was sent by Atlassian Jira (v8.20.10#820010)
Community over Code EU 2024: Start planning your trip!
[Note: You're receiving this email because you are subscribed to one or more project dev@ mailing lists at the Apache Software Foundation.] Dear community, We hope you are doing great, are you ready for Community Over Code EU? Check out the featured sessions, get your tickets with special discounts and start planning your trip. Save your spot! Take a look at our lineup of sessions, panelists and featured speakers and make your final choice: * EU policies and regulations affecting open source specialists working in OSPOs The panel will discuss how EU legislation affects the daily work of open source operations. Panelists will cover some recent policy updates, the challenges of staying compliant when managing open source contribution and usage within organizations, and their personal experiences in adapting to the changing European regulatory environment. * Doing for sustainability, what open source did for software In this keynote Asim Hussain will explain the history of Impact Framework, a coalition of hundreds of software practitioners with tangible solutions that directly foster meaningful change by measuring the environmental impacts of a piece of software. Don’t forget that we have special discounts for groups, students and Apache committers. Visit the website to discover more about these rates.[1] It's time for you to start planning your trip. Remember that we have prepared a “How to get there” guide that will be helpful to find out the best transportation, either train, bus, flight or boat to Bratislava from wherever you are coming from. Take a look at the different options and please reach out to us if you have any questions. We have available rooms -with a special rate- at the Radisson Blu Carlton Hotel, where the event will take place and at the Park Inn Hotel which is only 5 minutes walking from the venue. [2] However, you are free to choose any other accommodation options around the city. See you in Bratislava, Community Over Code EU Team [1]: https://eu.communityovercode.org/tickets/ "Register" [2]: https://eu.communityovercode.org/venue/ "Where to stay"
Participate in the ASF 25th Anniversary Campaign
Hi everyone, As part of The ASF’s 25th anniversary campaign[1], we will be celebrating projects and communities in multiple ways. We invite all projects and contributors to participate in the following ways: * Individuals - submit your first contribution: https://news.apache.org/foundation/entry/the-asf-launches-firstasfcontribution-campaign * Projects - share your public good story: https://docs.google.com/forms/d/1vuN-tUnBwpTgOE5xj3Z5AG1hsOoDNLBmGIqQHwQT6k8/viewform?edit_requested=true * Projects - submit a project spotlight for the blog: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=278466116 * Projects - contact the Voice of Apache podcast (formerly Feathercast) to be featured: https://feathercast.apache.org/help/ * Projects - use the 25th anniversary template and the #ASF25Years hashtag on social media: https://docs.google.com/presentation/d/1oDbMol3F_XQuCmttPYxBIOIjRuRBksUjDApjd8Ve3L8/edit#slide=id.g26b0919956e_0_13 If you have questions, email the Marketing & Publicity team at mark...@apache.org. Peace, BKP [1] https://apache.org/asf25years/ [NOTE: You are receiving this message because you are a contributor to an Apache Software Foundation project. The ASF will very occasionally send out messages relating to the Foundation to contributors and members, such as this one.] Brian Proffitt VP, Marketing & Publicity VP, Conferences
Re: How to query the Cube via API and use the dataset for other purpose
Thank you very much for your response, I did ask a pro for help and below
was the sample code on sample SSB project I would like to contribute to
help someone who have same issue like me:
==
import org.apache.spark.sql.types.{StringType, StructField, StructType}
import org.apache.spark.sql.{Row, SparkSession}
import org.json4s.jackson.JsonMethods
import org.json4s.{DefaultFormats, Formats}
import java.io.{BufferedReader, DataOutputStream, InputStreamReader}
import java.net.{HttpURLConnection, URL}
import java.util.Base64
object APIKylinRunSQL {
val KYLIN_QUERY_URL = "http://localhost:7070/kylin/api/query;
val USER_NAME = "x"
val PASSWORD = "y"
val KYLIN_PROJECT = "learn_kylin"
val spark = SparkSession.builder
.master("local")
.appName("Convert JSON to DataFrame")
.getOrCreate()
def main(args: Array[String]): Unit = {
val tablesAndQueries = Map(
"CUSTOMER" -> "select * from SSB.CUSTOMER",
"DATES" -> "SELECT * FROM SSB.DATES",
"PART" -> "SELECT * FROM SSB.PART",
"P_LINEORDER" -> "SELECT * FROM SSB.P_LINEORDER",
"SUPPLIER" -> "SELECT * FROM SSB.SUPPLIER",
"P_LINEORDER" -> "SELECT lo_orderdate, count(1) FROM SSB.P_LINEORDER
GROUP BY lo_orderdate",
"PART" -> "SELECT P_COLOR, count(1) FROM SSB.PART group by P_COLOR"
)
// query times
val numberOfExecutions = 15
// loop query
for (i <- 1 to numberOfExecutions) {
println(s"Executing query $i")
for ((table, query) <- tablesAndQueries) {
println(s"Executing queries for table $table")
println(query)
executeQuery(query)
// wait a seconds
Thread.sleep(1000)
}
}
}
def executeQuery(sqlQuery: String): Unit = {
val queryJson =
s"""
|{
| "project": "$KYLIN_PROJECT",
| "sql": "$sqlQuery"
|}
|""".stripMargin
// Encode the username and password for basic authentication
val encodedAuth =
Base64.getEncoder.encodeToString(s"$USER_NAME:$PASSWORD".getBytes)
val url = new URL(KYLIN_QUERY_URL)
val connection = url.openConnection.asInstanceOf[HttpURLConnection]
connection.setRequestMethod("POST")
connection.setRequestProperty("Authorization", s"Basic $encodedAuth")
connection.setRequestProperty("Content-Type", "application/json")
connection.setRequestProperty("Accept", "application/json")
connection.setDoOutput(true)
val outputStream = connection.getOutputStream
val writer = new DataOutputStream(outputStream)
writer.write(queryJson.getBytes("UTF-8"))
writer.flush()
writer.close()
val responseCode = connection.getResponseCode
if (responseCode == HttpURLConnection.HTTP_OK) {
val inputStream = connection.getInputStream
val reader = new BufferedReader(new InputStreamReader(inputStream))
var inputLine: String = null
val response = new StringBuilder
while ( {
inputLine = reader.readLine;
inputLine != null
}) {
response.append(inputLine)
}
reader.close()
println("Result:")
println(response.toString)
connection.disconnect()
// parse JSON
implicit val formats: Formats = DefaultFormats
val parsedJson = JsonMethods.parse(response.toString)
val columns = (parsedJson \ "columnMetas")
.extract[List[Map[String, Any]]]
// dynamically build the schema based on column name information in
JSON
val schema = StructType(columns.map { col =>
val columnName = col("name").asInstanceOf[String]
StructField(columnName, StringType, nullable = true)
})
schema.printTreeString()
// get data from JSON
val data = (parsedJson \ "results").extract[List[List[Any]]]
// convert data to RDD[Row]
val rowsRDD = spark.sparkContext.parallelize(data.map(row =>
Row.fromSeq(row.map(_.asInstanceOf[AnyRef]
val df = spark.createDataFrame(rowsRDD, schema)
df.show(20, false)
} else {
println(s"Error: $responseCode")
connection.disconnect()
}
}
}
On Sun, Mar 31, 2024 at 8:57 PM Lionel CL wrote:
> Hi Nam,
> You can refer to the spark docs
> https://spark.apache.org/docs/latest/sql-data-sources-jdbc.html
>
> Regards,
> Lu Cao
>
> From: Nam Đỗ Duy
> Date: Sunday, March 31, 2024 at 08:53
> To: dev , u...@kylin.apache.org <
> u...@kylin.apache.org>
> Subject: Re: How to query the Cube via API and use the dataset for other
> purpose
> Dear Sirs/Madames
>
> Could anyone here help me to figureout the way to use scala to query an
> select SQL against kylin cube via API then turn that table result into a
> dataframe in scala for other purpose?
>
> Thank you so much for your time!
>
> Best regards
>
> On Fri, 29 Mar 2024 at 17:52 Nam Đỗ Duy wrote:
>
> > Hi Xiaoxiang,
> > Sir & Madames,
> >
> > I use the following code to query the cube via API but I cannot use the
> > result as a
Re: [External] [jira] [Created] (KYLIN-5767) Calculating total rows abnormal when jdbc datasource is connnected
unscribe From: "pengfei.zhan (Jira)" Date: Fri, Mar 29, 2024, 15:31 Subject: [External] [jira] [Created] (KYLIN-5767) Calculating total rows abnormal when jdbc datasource is connnected To: pengfei.zhan created KYLIN-5767: --- Summary: Calculating total rows abnormal when jdbc datasource is connnected Key: KYLIN-5767 URL: https://issues.apache.org/jira/browse/KYLIN-5767 Project: Kylin Issue Type: Bug Components: Job Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5818) upgrade spark version to 4.6.16.0 GA
zhimin wu created KYLIN-5818: Summary: upgrade spark version to 4.6.16.0 GA Key: KYLIN-5818 URL: https://issues.apache.org/jira/browse/KYLIN-5818 Project: Kylin Issue Type: Improvement Components: Spark Engine Affects Versions: 5.0-beta Reporter: zhimin wu Fix For: 5.0.0 upgrade spark version to 4.6.16.0 GA -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5817) After sending the query, the number of scanRows and scanBytes in response are inaccurate
zhimin wu created KYLIN-5817:
Summary: After sending the query, the number of scanRows and
scanBytes in response are inaccurate
Key: KYLIN-5817
URL: https://issues.apache.org/jira/browse/KYLIN-5817
Project: Kylin
Issue Type: Improvement
Components: Query Engine
Affects Versions: 5.0-beta
Reporter: zhimin wu
Fix For: 5.0.0
h3. Background
When KE calls the query interface, it is found that the scan row count
{{scanRows}} and scan byte count {{scanBytes}} in the response are not accurate.
h3. Root Cause
The query interface internally calls the
{{collectAdaptiveSparkPlanExecMetrics}} method of {{{}QueryMetricUtils{}}}.
When iterating through {{{}exec.children.foreach{}}}, if the {{child}} is a
{{{}SparkPlan{}}}, each time the results of the new {{child}} overwrite the
{{newScanRow}} and {{newScanBytes}} variables without adding them up.
h3. Dev Design
To address the issue, replace the assignment operator {{=}} with the compound
addition assignment operator {{+=}} to ensure that the results are properly
accumulated. This change will allow for the correct aggregation of scan row
count and scan byte count in each iteration.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KYLIN-5816) when turn on the V3 global dictionary and concurrently build Segments, the count query result may be incorrect
zhimin wu created KYLIN-5816: Summary: when turn on the V3 global dictionary and concurrently build Segments, the count query result may be incorrect Key: KYLIN-5816 URL: https://issues.apache.org/jira/browse/KYLIN-5816 Project: Kylin Issue Type: Improvement Components: Others Affects Versions: 5.0-beta Reporter: zhimin wu Fix For: 5.0.0 when turn on the V3 global dictionary and concurrently build Segments, the count query result may be incorrect. *Dev Design* add Distributed locks for global dictionary build -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5815) fix add primary key failed when upgrade concurrently
zhimin wu created KYLIN-5815: Summary: fix add primary key failed when upgrade concurrently Key: KYLIN-5815 URL: https://issues.apache.org/jira/browse/KYLIN-5815 Project: Kylin Issue Type: Improvement Components: Metadata Affects Versions: 5.0-beta Reporter: zhimin wu Fix For: 5.0.0 fix add primary key failed when upgrade concurrently -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5814) query cache support redis sentinel mode
zhimin wu created KYLIN-5814: Summary: query cache support redis sentinel mode Key: KYLIN-5814 URL: https://issues.apache.org/jira/browse/KYLIN-5814 Project: Kylin Issue Type: Improvement Components: Query Engine Reporter: zhimin wu Fix For: 5.0.0 query cache support redis sentinel mode -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5813) history filter issue
chunyan.huang created KYLIN-5813: Summary: history filter issue Key: KYLIN-5813 URL: https://issues.apache.org/jira/browse/KYLIN-5813 Project: Kylin Issue Type: Bug Reporter: chunyan.huang -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5812) add rules combination to scalar subquery join
zhimin wu created KYLIN-5812: Summary: add rules combination to scalar subquery join Key: KYLIN-5812 URL: https://issues.apache.org/jira/browse/KYLIN-5812 Project: Kylin Issue Type: Bug Components: Query Engine Affects Versions: 5.0-beta Reporter: zhimin wu Fix For: 5.0.0 add rules combination to scalar subquery join -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5811) Enhance SCD2, history table supports left-closed and right-closed time interval
zhimin wu created KYLIN-5811: Summary: Enhance SCD2, history table supports left-closed and right-closed time interval Key: KYLIN-5811 URL: https://issues.apache.org/jira/browse/KYLIN-5811 Project: Kylin Issue Type: Bug Reporter: zhimin wu Enhance SCD2, history table supports left-closed and right-closed time interval -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5810) Add the spark.session.isDefined check in queryExec.
zhimin wu created KYLIN-5810: Summary: Add the spark.session.isDefined check in queryExec. Key: KYLIN-5810 URL: https://issues.apache.org/jira/browse/KYLIN-5810 Project: Kylin Issue Type: Bug Reporter: zhimin wu Add the spark.session.isDefined check in queryExec. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5809) Add discarded field for LayoutRecDetailResponse
zhimin wu created KYLIN-5809: Summary: Add discarded field for LayoutRecDetailResponse Key: KYLIN-5809 URL: https://issues.apache.org/jira/browse/KYLIN-5809 Project: Kylin Issue Type: Bug Reporter: zhimin wu Add discarded field for LayoutRecDetailResponse -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5808) Optimizing Performance When Saving Models or Adding Calculated Columns in Kylin
zhimin wu created KYLIN-5808: Summary: Optimizing Performance When Saving Models or Adding Calculated Columns in Kylin Key: KYLIN-5808 URL: https://issues.apache.org/jira/browse/KYLIN-5808 Project: Kylin Issue Type: Improvement Reporter: zhimin wu h3. Dev Design Current Performance Issues in Saving Models and Adding Calculated Columns in Kylin Engine. # Redundant Join Operations in flat table SQL # Optimizing Join Operations by Converting Dataset Join to Logical Plan Join -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5807) fix query execute as user without data query acl
zhimin wu created KYLIN-5807: Summary: fix query execute as user without data query acl Key: KYLIN-5807 URL: https://issues.apache.org/jira/browse/KYLIN-5807 Project: Kylin Issue Type: Bug Reporter: zhimin wu -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5806) fix testContainer image problem
zhimin wu created KYLIN-5806: Summary: fix testContainer image problem Key: KYLIN-5806 URL: https://issues.apache.org/jira/browse/KYLIN-5806 Project: Kylin Issue Type: Bug Reporter: zhimin wu fix testContainer image problem -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5805) Optimize the transaction processing process and improve the response performance of concurrent operations
zhimin wu created KYLIN-5805: Summary: Optimize the transaction processing process and improve the response performance of concurrent operations Key: KYLIN-5805 URL: https://issues.apache.org/jira/browse/KYLIN-5805 Project: Kylin Issue Type: Improvement Reporter: zhimin wu -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5804) Add clickhouse deps as default profile
zhimin wu created KYLIN-5804: Summary: Add clickhouse deps as default profile Key: KYLIN-5804 URL: https://issues.apache.org/jira/browse/KYLIN-5804 Project: Kylin Issue Type: Bug Reporter: zhimin wu Add clickhouse deps as default profile -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5803) add FileSegment
zhimin wu created KYLIN-5803: Summary: add FileSegment Key: KYLIN-5803 URL: https://issues.apache.org/jira/browse/KYLIN-5803 Project: Kylin Issue Type: New Feature Reporter: zhimin wu add FileSegment -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5802) Models with the same name exist when restore metadata
zhimin wu created KYLIN-5802: Summary: Models with the same name exist when restore metadata Key: KYLIN-5802 URL: https://issues.apache.org/jira/browse/KYLIN-5802 Project: Kylin Issue Type: Bug Reporter: zhimin wu -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5801) fix timestampdiff string args constance computation
zhimin wu created KYLIN-5801: Summary: fix timestampdiff string args constance computation Key: KYLIN-5801 URL: https://issues.apache.org/jira/browse/KYLIN-5801 Project: Kylin Issue Type: Bug Reporter: zhimin wu [fix timestampdiff string args constance computation|https://github.com/Kyligence/KAP/pull/30849] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5800) support scalar subquery join aggregate pushdown
zhimin wu created KYLIN-5800: Summary: support scalar subquery join aggregate pushdown Key: KYLIN-5800 URL: https://issues.apache.org/jira/browse/KYLIN-5800 Project: Kylin Issue Type: Improvement Reporter: zhimin wu support scalar subquery join aggregate pushdown -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5799) some support for round, bround, rpad, lpad function
zhimin wu created KYLIN-5799: Summary: some support for round, bround, rpad, lpad function Key: KYLIN-5799 URL: https://issues.apache.org/jira/browse/KYLIN-5799 Project: Kylin Issue Type: Improvement Reporter: zhimin wu ||*function*||*Kylin（sparder query）*||*SPARK*|| |ROUND （dual parameter）|!https://pf-emoji-service--cdn.us-east-1.prod.public.atl-paas.net/standard/caa27a19-fc09-4452-b2b4-a301552fd69c/64x64/2705.png|width=20,height=20!|!https://pf-emoji-service--cdn.us-east-1.prod.public.atl-paas.net/standard/caa27a19-fc09-4452-b2b4-a301552fd69c/64x64/2705.png|width=20,height=20!| |ROUND （Single parameter）|CalciteToSparkPlaner Expection|!https://pf-emoji-service--cdn.us-east-1.prod.public.atl-paas.net/standard/caa27a19-fc09-4452-b2b4-a301552fd69c/64x64/2705.png|width=20,height=20!| |BROUND（dual parameter）|!https://pf-emoji-service--cdn.us-east-1.prod.public.atl-paas.net/standard/caa27a19-fc09-4452-b2b4-a301552fd69c/64x64/2705.png|width=20,height=20!|!https://pf-emoji-service--cdn.us-east-1.prod.public.atl-paas.net/standard/caa27a19-fc09-4452-b2b4-a301552fd69c/64x64/2705.png|width=20,height=20!| |BROUND（Single parameter）|calcite convertToRelNode Expection|!https://pf-emoji-service--cdn.us-east-1.prod.public.atl-paas.net/standard/caa27a19-fc09-4452-b2b4-a301552fd69c/64x64/2705.png|width=20,height=20!| |LPAD（triple parameter）|!https://pf-emoji-service--cdn.us-east-1.prod.public.atl-paas.net/standard/caa27a19-fc09-4452-b2b4-a301552fd69c/64x64/2705.png|width=20,height=20!|!https://pf-emoji-service--cdn.us-east-1.prod.public.atl-paas.net/standard/caa27a19-fc09-4452-b2b4-a301552fd69c/64x64/2705.png|width=20,height=20!| |LPAD（dual parameter）|calcite convertToRelNode Expection|!https://pf-emoji-service--cdn.us-east-1.prod.public.atl-paas.net/standard/caa27a19-fc09-4452-b2b4-a301552fd69c/64x64/2705.png|width=20,height=20!| |RPAD（triple parameter）|!https://pf-emoji-service--cdn.us-east-1.prod.public.atl-paas.net/standard/caa27a19-fc09-4452-b2b4-a301552fd69c/64x64/2705.png|width=20,height=20!|!https://pf-emoji-service--cdn.us-east-1.prod.public.atl-paas.net/standard/caa27a19-fc09-4452-b2b4-a301552fd69c/64x64/2705.png|width=20,height=20!| |RPAD（dual parameter）|calcite convertToRelNode Expection|!https://pf-emoji-service--cdn.us-east-1.prod.public.atl-paas.net/standard/caa27a19-fc09-4452-b2b4-a301552fd69c/64x64/2705.png|width=20,height=20!| we will support above expection -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5798) optimize the loading performance when there are many source tables through caching
zhimin wu created KYLIN-5798:
Summary: optimize the loading performance when there are many
source tables through caching
Key: KYLIN-5798
URL: https://issues.apache.org/jira/browse/KYLIN-5798
Project: Kylin
Issue Type: Improvement
Reporter: zhimin wu
h3. Adding User-Table Permission Cache
Introduce a cache for user-table permissions, with an option to control whether
this feature is enabled or not, with the default setting being disabled. This
feature is utilized when project-level Kerberos authentication is enabled and
Hive table permission filtering is also enabled.
{code:java}
kylin.source.hive.table-access-cache-enabled=TRUE
kylin.source.hive.table-access-cache-size=10
kylin.source.hive.table-access-cache-ttl=7d{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KYLIN-5797) remove nacos jars
zhimin wu created KYLIN-5797: Summary: remove nacos jars Key: KYLIN-5797 URL: https://issues.apache.org/jira/browse/KYLIN-5797 Project: Kylin Issue Type: Improvement Reporter: zhimin wu remove nacos jars -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5796) The system reports error when the number of models exported exceeds 256 at a time
zhimin wu created KYLIN-5796: Summary: The system reports error when the number of models exported exceeds 256 at a time Key: KYLIN-5796 URL: https://issues.apache.org/jira/browse/KYLIN-5796 Project: Kylin Issue Type: Bug Reporter: zhimin wu ### Root Cause The root cause of the issue is that the size of the list passed from the frontend exceeds the maximum limit allowed for form-urlencoded requests in Spring. The default limit for the growth of arrays and collections in Spring's DataBinder class is set to 256: ```java public static final int DEFAULT_AUTO_GROW_COLLECTION_LIMIT = 256; ``` To address this issue, you may need to adjust the configuration related to the form-urlencoded request handling in Spring to accommodate larger list sizes being passed from the frontend. This could involve increasing the maximum limit for auto-growing collections or implementing alternative solutions to handle larger input data efficiently. If you need further assistance or guidance on how to modify the configuration to resolve this issue, please let me know. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5795) Optimizing Startup Speed by remove hadoop.tmp.dir check
zhimin wu created KYLIN-5795: Summary: Optimizing Startup Speed by remove hadoop.tmp.dir check Key: KYLIN-5795 URL: https://issues.apache.org/jira/browse/KYLIN-5795 Project: Kylin Issue Type: Improvement Components: Others Reporter: zhimin wu Optimizing Startup Speed by remove hadoop.tmp.dir check -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: How to query the Cube via API and use the dataset for other purpose
Hi Nam,
You can refer to the spark docs
https://spark.apache.org/docs/latest/sql-data-sources-jdbc.html
Regards,
Lu Cao
From: Nam Đỗ Duy
Date: Sunday, March 31, 2024 at 08:53
To: dev , u...@kylin.apache.org
Subject: Re: How to query the Cube via API and use the dataset for other purpose
Dear Sirs/Madames
Could anyone here help me to figureout the way to use scala to query an
select SQL against kylin cube via API then turn that table result into a
dataframe in scala for other purpose?
Thank you so much for your time!
Best regards
On Fri, 29 Mar 2024 at 17:52 Nam Đỗ Duy wrote:
> Hi Xiaoxiang,
> Sir & Madames,
>
> I use the following code to query the cube via API but I cannot use the
> result as a dataframe, could you suggest a way to do that because it is
> very important for our project.
>
> Thanks and best regards
>
> ===
>
> import org.apache.spark.sql.{DataFrame, SparkSession}
> import org.apache.spark.sql.functions._
>
> object APICaller {
> def main(args: Array[String]): Unit = {
> val spark = SparkSession.builder()
> .appName("APICaller")
> .master("local[*]")
> .getOrCreate()
>
> import spark.implicits._
>
> val username = "namdd"
> val password = "eer123"
> val urlString = "http://localhost:7070/kylin/api/query;
> val project = "learn_kylin"
> val query = "select count(*) from HIVE_DWH_STANDARD.factuserEvent"
>
> val response: String = callAPI(urlString, username, password, project,
> query)
>
> // Convert response to DataFrame
> val df = spark.read.json(Seq(response).toDS())
>
> // Show DataFrame
> df.show()
>
> // Stop Spark session
> spark.stop()
> }
>
> def callAPI(url: String, username: String, password: String, project:
> String, query: String): String = {
> val encodedAuth =
> java.util.Base64.getEncoder.encodeToString(s"$username:$password".getBytes)
>
> val connection = scalaj.http.Http(url)
> .postData(s"""{"project": "$project", "sql": "$query"}""")
> .header("Content-Type", "application/json")
> .header("Accept", "application/json")
> .auth(username, password)
> .asString
>
> if (connection.isError)
> throw new RuntimeException(s"Error calling API: ${connection.body}")
>
> connection.body
> }
> }
>
>
[jira] [Created] (KYLIN-5794) Add Spark parquet footer read cache
zhimin wu created KYLIN-5794: Summary: Add Spark parquet footer read cache Key: KYLIN-5794 URL: https://issues.apache.org/jira/browse/KYLIN-5794 Project: Kylin Issue Type: Improvement Reporter: zhimin wu *Optimization Proposal* In Kylin, the index storage is in Parquet format. When querying, if Spark reads Parquet data, it needs to read the Footer information first. In cases where there are many columns, reading the Footer can consume a considerable amount of time. Therefore, caching the Footer information can improve query performance. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5793) When integrating with LDAP and the configured admin group name is Chinese, system admin users occasionally have abnormal permissions
zhimin wu created KYLIN-5793: Summary: When integrating with LDAP and the configured admin group name is Chinese, system admin users occasionally have abnormal permissions Key: KYLIN-5793 URL: https://issues.apache.org/jira/browse/KYLIN-5793 Project: Kylin Issue Type: Bug Components: Others Reporter: zhimin wu -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5792) LDAP integration, support to obtain the matching relationship between users and user groups through memberUid
zhimin wu created KYLIN-5792: Summary: LDAP integration, support to obtain the matching relationship between users and user groups through memberUid Key: KYLIN-5792 URL: https://issues.apache.org/jira/browse/KYLIN-5792 Project: Kylin Issue Type: New Feature Components: Others Reporter: zhimin wu -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5791) Using CAST function to convert timestapadd to varchar does not return the result in the required time format
zhimin wu created KYLIN-5791:
Summary: Using CAST function to convert timestapadd to varchar
does not return the result in the required time format
Key: KYLIN-5791
URL: https://issues.apache.org/jira/browse/KYLIN-5791
Project: Kylin
Issue Type: Bug
Components: Query Engine
Affects Versions: 5.0-beta
Reporter: zhimin wu
Query statement: SELECT CAST(TIMESTAMPADD(MONTH, -1, TO_DATE('2023-07-31',
'-MM-dd')) AS VARCHAR)
The result of the TIMESTAMPADD function here is 2023-06-30, but when using the
CAST function for conversion, it becomes 2023-06-30 00:00:00.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: How to query the Cube via API and use the dataset for other purpose
Dear Sirs/Madames
Could anyone here help me to figureout the way to use scala to query an
select SQL against kylin cube via API then turn that table result into a
dataframe in scala for other purpose?
Thank you so much for your time!
Best regards
On Fri, 29 Mar 2024 at 17:52 Nam Đỗ Duy wrote:
> Hi Xiaoxiang,
> Sir & Madames,
>
> I use the following code to query the cube via API but I cannot use the
> result as a dataframe, could you suggest a way to do that because it is
> very important for our project.
>
> Thanks and best regards
>
> ===
>
> import org.apache.spark.sql.{DataFrame, SparkSession}
> import org.apache.spark.sql.functions._
>
> object APICaller {
> def main(args: Array[String]): Unit = {
> val spark = SparkSession.builder()
> .appName("APICaller")
> .master("local[*]")
> .getOrCreate()
>
> import spark.implicits._
>
> val username = "namdd"
> val password = "eer123"
> val urlString = "http://localhost:7070/kylin/api/query;
> val project = "learn_kylin"
> val query = "select count(*) from HIVE_DWH_STANDARD.factuserEvent"
>
> val response: String = callAPI(urlString, username, password, project,
> query)
>
> // Convert response to DataFrame
> val df = spark.read.json(Seq(response).toDS())
>
> // Show DataFrame
> df.show()
>
> // Stop Spark session
> spark.stop()
> }
>
> def callAPI(url: String, username: String, password: String, project:
> String, query: String): String = {
> val encodedAuth =
> java.util.Base64.getEncoder.encodeToString(s"$username:$password".getBytes)
>
> val connection = scalaj.http.Http(url)
> .postData(s"""{"project": "$project", "sql": "$query"}""")
> .header("Content-Type", "application/json")
> .header("Accept", "application/json")
> .auth(username, password)
> .asString
>
> if (connection.isError)
> throw new RuntimeException(s"Error calling API: ${connection.body}")
>
> connection.body
> }
> }
>
>
[jira] [Created] (KYLIN-5790) Security of kafka-clients
pengfei.zhan created KYLIN-5790: --- Summary: Security of kafka-clients Key: KYLIN-5790 URL: https://issues.apache.org/jira/browse/KYLIN-5790 Project: Kylin Issue Type: Bug Components: Query Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0-beta -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5789) Clean sparder history and spark history automatically
pengfei.zhan created KYLIN-5789: --- Summary: Clean sparder history and spark history automatically Key: KYLIN-5789 URL: https://issues.apache.org/jira/browse/KYLIN-5789 Project: Kylin Issue Type: Bug Components: Job Engine, Query Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0-beta -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5788) Enhance global dict on flat table encoding stage logging & retry
pengfei.zhan created KYLIN-5788: --- Summary: Enhance global dict on flat table encoding stage logging & retry Key: KYLIN-5788 URL: https://issues.apache.org/jira/browse/KYLIN-5788 Project: Kylin Issue Type: Bug Components: Job Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0-beta -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5787) Use t-digest as spark percentile_approx function
pengfei.zhan created KYLIN-5787: --- Summary: Use t-digest as spark percentile_approx function Key: KYLIN-5787 URL: https://issues.apache.org/jira/browse/KYLIN-5787 Project: Kylin Issue Type: Improvement Components: Job Engine, Query Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0-beta -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5786) Add a write lock when merging v3 dict
pengfei.zhan created KYLIN-5786: --- Summary: Add a write lock when merging v3 dict Key: KYLIN-5786 URL: https://issues.apache.org/jira/browse/KYLIN-5786 Project: Kylin Issue Type: Bug Components: Job Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0-beta -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5785) Support implicit query of split function
pengfei.zhan created KYLIN-5785: --- Summary: Support implicit query of split function Key: KYLIN-5785 URL: https://issues.apache.org/jira/browse/KYLIN-5785 Project: Kylin Issue Type: Bug Components: Query Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0-beta -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5784) Optimization for query history filters
pengfei.zhan created KYLIN-5784: --- Summary: Optimization for query history filters Key: KYLIN-5784 URL: https://issues.apache.org/jira/browse/KYLIN-5784 Project: Kylin Issue Type: Improvement Components: Query Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0-beta KYLIN-5763 related -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5783) Round query result of decimal zero by config
pengfei.zhan created KYLIN-5783: --- Summary: Round query result of decimal zero by config Key: KYLIN-5783 URL: https://issues.apache.org/jira/browse/KYLIN-5783 Project: Kylin Issue Type: Improvement Components: Query Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0-beta -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5782) Duplicated join keys result empty query result
pengfei.zhan created KYLIN-5782: --- Summary: Duplicated join keys result empty query result Key: KYLIN-5782 URL: https://issues.apache.org/jira/browse/KYLIN-5782 Project: Kylin Issue Type: Bug Components: Query Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0-beta -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5781) Adjust the query result of union order
pengfei.zhan created KYLIN-5781: --- Summary: Adjust the query result of union order Key: KYLIN-5781 URL: https://issues.apache.org/jira/browse/KYLIN-5781 Project: Kylin Issue Type: Improvement Components: Query Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0-beta -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5780) Dimension capacity does not work
pengfei.zhan created KYLIN-5780: --- Summary: Dimension capacity does not work Key: KYLIN-5780 URL: https://issues.apache.org/jira/browse/KYLIN-5780 Project: Kylin Issue Type: Bug Components: Modeling Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0-beta -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5779) Without project permission，calling "kylin/api/access/acls" api occurs NPE
pengfei.zhan created KYLIN-5779: --- Summary: Without project permission，calling "kylin/api/access/acls" api occurs NPE Key: KYLIN-5779 URL: https://issues.apache.org/jira/browse/KYLIN-5779 Project: Kylin Issue Type: Bug Components: Query Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0-beta -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5778) Upgrade spark: parquet footer read cache
pengfei.zhan created KYLIN-5778: --- Summary: Upgrade spark: parquet footer read cache Key: KYLIN-5778 URL: https://issues.apache.org/jira/browse/KYLIN-5778 Project: Kylin Issue Type: Improvement Components: Job Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0-beta -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5777) Change spark.yarn.submit.file.replication from 1 to 3
pengfei.zhan created KYLIN-5777: --- Summary: Change spark.yarn.submit.file.replication from 1 to 3 Key: KYLIN-5777 URL: https://issues.apache.org/jira/browse/KYLIN-5777 Project: Kylin Issue Type: Bug Components: Job Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0-beta -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5776) Support parameter type is string for timestampadd
pengfei.zhan created KYLIN-5776: --- Summary: Support parameter type is string for timestampadd Key: KYLIN-5776 URL: https://issues.apache.org/jira/browse/KYLIN-5776 Project: Kylin Issue Type: Bug Components: Query Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0-beta -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5775) Upgrade testcontainers to version 1.16.2
pengfei.zhan created KYLIN-5775: --- Summary: Upgrade testcontainers to version 1.16.2 Key: KYLIN-5775 URL: https://issues.apache.org/jira/browse/KYLIN-5775 Project: Kylin Issue Type: Bug Components: Others Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0-beta -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5774) Optimize Calcite plan to convert spark logical plan
pengfei.zhan created KYLIN-5774: --- Summary: Optimize Calcite plan to convert spark logical plan Key: KYLIN-5774 URL: https://issues.apache.org/jira/browse/KYLIN-5774 Project: Kylin Issue Type: Improvement Components: Query Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0-beta -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5773) Upgrade Spark to 4.6.13.0
pengfei.zhan created KYLIN-5773: --- Summary: Upgrade Spark to 4.6.13.0 Key: KYLIN-5773 URL: https://issues.apache.org/jira/browse/KYLIN-5773 Project: Kylin Issue Type: Bug Components: Spark Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0-beta -- This message was sent by Atlassian Jira (v8.20.10#820010)
How to query the Cube via API and use the dataset for other purpose
Hi Xiaoxiang,
Sir & Madames,
I use the following code to query the cube via API but I cannot use the
result as a dataframe, could you suggest a way to do that because it is
very important for our project.
Thanks and best regards
===
import org.apache.spark.sql.{DataFrame, SparkSession}
import org.apache.spark.sql.functions._
object APICaller {
def main(args: Array[String]): Unit = {
val spark = SparkSession.builder()
.appName("APICaller")
.master("local[*]")
.getOrCreate()
import spark.implicits._
val username = "namdd"
val password = "eer123"
val urlString = "http://localhost:7070/kylin/api/query;
val project = "learn_kylin"
val query = "select count(*) from HIVE_DWH_STANDARD.factuserEvent"
val response: String = callAPI(urlString, username, password, project,
query)
// Convert response to DataFrame
val df = spark.read.json(Seq(response).toDS())
// Show DataFrame
df.show()
// Stop Spark session
spark.stop()
}
def callAPI(url: String, username: String, password: String, project:
String, query: String): String = {
val encodedAuth =
java.util.Base64.getEncoder.encodeToString(s"$username:$password".getBytes)
val connection = scalaj.http.Http(url)
.postData(s"""{"project": "$project", "sql": "$query"}""")
.header("Content-Type", "application/json")
.header("Accept", "application/json")
.auth(username, password)
.asString
if (connection.isError)
throw new RuntimeException(s"Error calling API: ${connection.body}")
connection.body
}
}
[jira] [Created] (KYLIN-5772) Fix vlunerability and excluding ekio
pengfei.zhan created KYLIN-5772: --- Summary: Fix vlunerability and excluding ekio Key: KYLIN-5772 URL: https://issues.apache.org/jira/browse/KYLIN-5772 Project: Kylin Issue Type: Bug Components: Others Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5771) Query cannot be interrupted during Segment pruning phase
pengfei.zhan created KYLIN-5771: --- Summary: Query cannot be interrupted during Segment pruning phase Key: KYLIN-5771 URL: https://issues.apache.org/jira/browse/KYLIN-5771 Project: Kylin Issue Type: Bug Components: Query Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5770) Update spring-security-config from 5.7.8 to 5.7.10
pengfei.zhan created KYLIN-5770: --- Summary: Update spring-security-config from 5.7.8 to 5.7.10 Key: KYLIN-5770 URL: https://issues.apache.org/jira/browse/KYLIN-5770 Project: Kylin Issue Type: Bug Reporter: pengfei.zhan -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5769) Support split function
pengfei.zhan created KYLIN-5769: --- Summary: Support split function Key: KYLIN-5769 URL: https://issues.apache.org/jira/browse/KYLIN-5769 Project: Kylin Issue Type: Improvement Components: Job Engine, Query Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5768) Refine some code for localcache
pengfei.zhan created KYLIN-5768: --- Summary: Refine some code for localcache Key: KYLIN-5768 URL: https://issues.apache.org/jira/browse/KYLIN-5768 Project: Kylin Issue Type: Improvement Components: Query Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5767) Calculating total rows abnormal when jdbc datasource is connnected
pengfei.zhan created KYLIN-5767: --- Summary: Calculating total rows abnormal when jdbc datasource is connnected Key: KYLIN-5767 URL: https://issues.apache.org/jira/browse/KYLIN-5767 Project: Kylin Issue Type: Bug Components: Job Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5766) Normalize query cache key
pengfei.zhan created KYLIN-5766: --- Summary: Normalize query cache key Key: KYLIN-5766 URL: https://issues.apache.org/jira/browse/KYLIN-5766 Project: Kylin Issue Type: Improvement Components: Query Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5765) Optimization of register udf operators
pengfei.zhan created KYLIN-5765: --- Summary: Optimization of register udf operators Key: KYLIN-5765 URL: https://issues.apache.org/jira/browse/KYLIN-5765 Project: Kylin Issue Type: Improvement Components: Query Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5764) Sum expression supports if condition
pengfei.zhan created KYLIN-5764: --- Summary: Sum expression supports if condition Key: KYLIN-5764 URL: https://issues.apache.org/jira/browse/KYLIN-5764 Project: Kylin Issue Type: Improvement Components: Query Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
Community Over Code NA 2024 Travel Assistance Applications now open!
Hello to all users, contributors and Committers! [ You are receiving this email as a subscriber to one or more ASF project dev or user mailing lists and is not being sent to you directly. It is important that we reach all of our users and contributors/committers so that they may get a chance to benefit from this. We apologise in advance if this doesn't interest you but it is on topic for the mailing lists of the Apache Software Foundation; and it is important please that you do not mark this as spam in your email client. Thank You! ] The Travel Assistance Committee (TAC) are pleased to announce that travel assistance applications for Community over Code NA 2024 are now open! We will be supporting Community over Code NA, Denver Colorado in October 7th to the 10th 2024. TAC exists to help those that would like to attend Community over Code events, but are unable to do so for financial reasons. For more info on this years applications and qualifying criteria, please visit the TAC website at < https://tac.apache.org/ >. Applications are already open on https://tac-apply.apache.org/, so don't delay! The Apache Travel Assistance Committee will only be accepting applications from those people that are able to attend the full event. Important: Applications close on Monday 6th May, 2024. Applicants have until the the closing date above to submit their applications (which should contain as much supporting material as required to efficiently and accurately process their request), this will enable TAC to announce successful applications shortly afterwards. As usual, TAC expects to deal with a range of applications from a diverse range of backgrounds; therefore, we encourage (as always) anyone thinking about sending in an application to do so ASAP. For those that will need a Visa to enter the Country - we advise you apply now so that you have enough time in case of interview delays. So do not wait until you know if you have been accepted or not. We look forward to greeting many of you in Denver, Colorado , October 2024! Kind Regards, Gavin (On behalf of the Travel Assistance Committee)
[jira] [Created] (KYLIN-5763) Equivalent aggregations matches index failed when using the strategy of dimension as measure
pengfei.zhan created KYLIN-5763:
---
Summary: Equivalent aggregations matches index failed when using
the strategy of dimension as measure
Key: KYLIN-5763
URL: https://issues.apache.org/jira/browse/KYLIN-5763
Project: Kylin
Issue Type: Bug
Components: Query Engine
Affects Versions: 5.0-beta
Reporter: pengfei.zhan
Assignee: pengfei.zhan
Fix For: 5.0-beta
The following sql can not match index just has dimension lstg_format_name.
{code:sql}
select max(lstg_format_name) max1,
max( case when 1=1 then lstg_format_nameend ) max2
from test_kylin_fact{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KYLIN-5762) Initialize job scheduler encounters NPE
pengfei.zhan created KYLIN-5762: --- Summary: Initialize job scheduler encounters NPE Key: KYLIN-5762 URL: https://issues.apache.org/jira/browse/KYLIN-5762 Project: Kylin Issue Type: Bug Components: Job Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0-beta Given `kylin.job.max-concurrent-jobs=0`, the initialization of job scheduler encounters NPE. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5761) Upgrade tomcat and netty to fix vulnerabilities
pengfei.zhan created KYLIN-5761: --- Summary: Upgrade tomcat and netty to fix vulnerabilities Key: KYLIN-5761 URL: https://issues.apache.org/jira/browse/KYLIN-5761 Project: Kylin Issue Type: Bug Components: Query Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0-beta Upgrade the embedded tomcat and netty to fix vulnerabilities -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5760) The API of get job list supports match model exactly
pengfei.zhan created KYLIN-5760: --- Summary: The API of get job list supports match model exactly Key: KYLIN-5760 URL: https://issues.apache.org/jira/browse/KYLIN-5760 Project: Kylin Issue Type: Bug Components: Job Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0-beta h1. Description At present the API of list jobs (/kylin/api/jobs) will return all job lists that fuzzy match the given key. Sometimes, for better performance, the key is applied to match jobs exactly, however, it's not supported jet. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5759) Modify "ExecutableContext#addRunningJob" to avoid schedule job more than once
pengfei.zhan created KYLIN-5759: --- Summary: Modify "ExecutableContext#addRunningJob" to avoid schedule job more than once Key: KYLIN-5759 URL: https://issues.apache.org/jira/browse/KYLIN-5759 Project: Kylin Issue Type: Bug Components: Job Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0-beta When a task is scheduled, the task is logged into runningJobs and the current thread is logged into runningJobThreads, which is expected to be the thread that executes the task, but in fact it is the scheduler's thread that is logged, which results in subsequent attempts to interrupt the scheduler FetcherRunner when killing the task. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5758) TDS template modify to english
pengfei.zhan created KYLIN-5758: --- Summary: TDS template modify to english Key: KYLIN-5758 URL: https://issues.apache.org/jira/browse/KYLIN-5758 Project: Kylin Issue Type: Bug Components: Query Engine Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0-beta *Steps to reproduce the issue:* # Go to a model and click “Export TDS”. # Open up the file in a text editor and look at the bottom. There are Chinese characters. See attachment. I confirmed this issue is present in 4.5.4 and .11, and likely exists in other versions. The TDS file seems to work fine though. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5757) Optimization of the API of listing models
pengfei.zhan created KYLIN-5757: --- Summary: Optimization of the API of listing models Key: KYLIN-5757 URL: https://issues.apache.org/jira/browse/KYLIN-5757 Project: Kylin Issue Type: Improvement Components: Others Affects Versions: 5.0-beta Reporter: pengfei.zhan Assignee: pengfei.zhan Fix For: 5.0-beta The API of listing models very slow at sometime, improve it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
