[jira] [Updated] (KYLIN-5730) Query dry-run for better modeling
[
https://issues.apache.org/jira/browse/KYLIN-5730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoxiang Yu updated KYLIN-5730:
Description:
h2. Background
When user enable this feature, the query insight page will display helpful
message
for user to understand why model is not match and what to do in next step.
Following messages will display some query analytics, including at least:
# RelNode Tree
# OLAPContext and matched Model for each context
# Spark Physical Plan
Configuration entry is 'kylin.query.dryrun-enabled', at project level.
h2. How to use
# enable this in project configuration
# send a query in Insight page
# read Dry-run message and fix something and retry
h4. Dry-run message sample:
# {{Last Exception :}}
{{{} No realization found for {id = 0, model = not matched, fact table =
TPCH_CN.LINEITEM{
{{ Incapable message : MODEL_UNMATCHED_JOIN, MODEL_UNMATCHED_JOIN,
MODEL_UNMATCHED_JOIN, MODEL_UNMATCHED_JOIN, MODEL_UNMATCHED_JOIN,
MODEL_UNMATCHED_JOIN, MODEL_UNMATCHED_JOIN, MODEL_UNMATCHED_JOIN, }}
# {{OLAPContext(s) and matched model(s) :}}
{{ Ctx=0 is not matched by any model/snapshot, recommend :}}
{{ {}}
{{ "Fact Table" : "TPCH_CN.LINEITEM",}}
{{ "Dimension Tables" : [ "TPCH_CN.REGION", "TPCH_CN.ORDERS",
"TPCH_CN.SUPPLIER", "TPCH_CN.PART", "TPCH_CN.CUSTOMER", "TPCH_CN.NATION"],}}
{{ "Query Columns" : [ "TPCH_CN.NATION.N_NAME", "TPCH_CN.ORDERS.O_ORDERDATE",
"TPCH_CN.REGION.R_NAME", "TPCH_CN.PART.P_TYPE",
"TPCH_CN.LINEITEM.DISC_PRICE"],}}
{{ "Dimension(Group by)" : [ "TPCH_CN.ORDERS.O_ORDERDATE"],}}
{{ "Dimension(Filter cond)" : [ "TPCH_CN.ORDERS.O_ORDERDATE",
"TPCH_CN.REGION.R_NAME", "TPCH_CN.PART.P_TYPE"],}}
{{ "Measure" : [ "SUM(UNKNOWN_ALIAS.$F1)", "SUM(T_1_4CD6601C.DISC_PRICE)"],}}
{{ "Join" : "}}
{{Fact: [T_1_4CD6601C:LINEITEM]}}
{{ Dim: [T_1_4CD6601C:LINEITEM] INNER JOIN [T_2_716C1DB8:PART] ON [L_PARTKEY]
= [P_PARTKEY]}}
{{ Dim: [T_1_4CD6601C:LINEITEM] INNER JOIN [T_3_6D21031C:SUPPLIER] ON
[L_SUPPKEY] = [S_SUPPKEY]}}
{{ Dim: [T_3_6D21031C:SUPPLIER] INNER JOIN [T_7_4F00CD31:NATION] ON
[S_NATIONKEY] = [N_NATIONKEY]}}
{{ Dim: [T_1_4CD6601C:LINEITEM] INNER JOIN [T_4_5CEAD9DF:ORDERS] ON
[L_ORDERKEY] = [O_ORDERKEY]}}
{{ Dim: [T_4_5CEAD9DF:ORDERS] INNER JOIN [T_5_54B979E7:CUSTOMER] ON
[O_CUSTKEY] = [C_CUSTKEY]}}
{{ Dim: [T_5_54B979E7:CUSTOMER] INNER JOIN [T_6_42081A08:NATION] ON
[C_NATIONKEY] = [N_NATIONKEY]}}
{{ Dim: [T_6_42081A08:NATION] INNER JOIN [T_8_B14C00A:REGION] ON
[N_REGIONKEY] = [R_REGIONKEY]",}}
{{ "Index Id" : 0,}}
{{ "Query Column / Index Column" : "5 / 0",}}
{{ "Index Columns" : []}}
{{{} }{{
# {{RelNode(with ctx id) :}}
{{ OLAPToEnumerableConverter}}
{{ LimitRel(ctx=[0@null], fetch=[500])}}
{{ SortRel(sort0=[$0], dir0=[ASC-nulls-first], ctx=[0@null])}}
{{ ProjectRel(O_YEAR=[$0], MKT_SHARE=[/($1, $2)], ctx=[0@null])}}
{{ AggregateRel(group-set=[[0]], groups=[null], agg#0=[SUM($1)],
agg#1=[SUM($2)], ctx=[0@null])}}
{{ ProjectRel(O_YEAR=[EXTRACT(FLAG(YEAR), $54)], $f1=[CASE(=($72,
'BRAZIL'), $16, 0)], VOLUME=[$16], ctx=[0@null])}}
{{ FilterRel(condition=[AND(=($76, 'AMERICA'), >=($54, 1995-01-01),
<=($54, 1996-12-31), =($38, 'ECONOMY ANODIZED STEEL'))], ctx=[0@null])}}
{{ JoinRel(condition=[=($69, $75)], joinType=[inner],
ctx=[0@null])}}
{{ JoinRel(condition=[=($46, $71)], joinType=[inner],
ctx=[0@null])}}
{{ JoinRel(condition=[=($62, $67)], joinType=[inner],
ctx=[0@null])}}
{{ JoinRel(condition=[=($51, $59)], joinType=[inner],
ctx=[0@null])}}
{{ JoinRel(condition=[=($0, $50)], joinType=[inner],
ctx=[0@null])}}
{{ JoinRel(condition=[=($2, $43)], joinType=[inner],
ctx=[0@null])}}
{{ JoinRel(condition=[=($1, $34)], joinType=[inner],
ctx=[0@null])}}
{{ TableScan(table=[[TPCH_CN, LINEITEM]],
ctx=[0@null], fields=[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15,
16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33]])}}
{{ TableScan(table=[[TPCH_CN, PART]], ctx=[0@null],
fields=[[0, 1, 2, 3, 4, 5, 6, 7, 8]])}}
{{ TableScan(table=[[TPCH_CN, SUPPLIER]],
ctx=[0@null], fields=[[0, 1, 2, 3, 4, 5, 6]])}}
{{ TableScan(table=[[TPCH_CN, ORDERS]], ctx=[0@null],
fields=[[0, 1, 2, 3, 4, 5, 6, 7, 8]])}}
{{ TableScan(table=[[TPCH_CN, CUSTOMER]], ctx=[0@null],
fields=[[0, 1, 2, 3, 4, 5, 6, 7]])}}
{{ TableScan(table=[[TPCH_CN, NATION]], ctx=[0@null],
fields=[[0, 1, 2, 3]])}}
{{ TableScan(table=[[TPCH_CN, NATION]], ctx=[0@null],
fields=[[0, 1, 2, 3]])}}
{{ TableScan(table=[[TPCH_CN, REGION]], ctx=[0@null],
fields=[[0, 1, 2]])}}
# {{SQL Text :}}
{{select /*+
[jira] [Updated] (KYLIN-5730) Query dry-run for better modeling
[
https://issues.apache.org/jira/browse/KYLIN-5730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoxiang Yu updated KYLIN-5730:
Description:
h2. Background
When user enable this feature, the query insight page will display helpful
message
for user to understand why model is not match and what to do in next step.
Following messages will display some query analytics, including at least:
# RelNode Tree
# OLAPContext and matched Model for each context
# Spark Physical Plan
Configuration entry is 'kylin.query.dryrun-enabled', at project level.
h2. How to use
# enable this in project configuration
# send a query in Insight page
# read Dry-run message and fix something and retry
h4. Dry-run message sample:
{{1. Last Exception :}}
{{ No realization found for \{id = 0, model = not matched, fact table =
TPCH_CN.LINEITEM}}}
{{ Incapable message : MODEL_UNMATCHED_JOIN, MODEL_UNMATCHED_JOIN,
MODEL_UNMATCHED_JOIN, MODEL_UNMATCHED_JOIN, MODEL_UNMATCHED_JOIN,
MODEL_UNMATCHED_JOIN, MODEL_UNMATCHED_JOIN, MODEL_UNMATCHED_JOIN, }}{{2.
OLAPContext(s) and matched model(s) :}}
{{ Ctx=0 is not matched by any model/snapshot, recommend :}}
{{ {}}
{{ "Fact Table" : "TPCH_CN.LINEITEM",}}
{{ "Dimension Tables" : [ "TPCH_CN.REGION", "TPCH_CN.ORDERS",
"TPCH_CN.SUPPLIER", "TPCH_CN.PART", "TPCH_CN.CUSTOMER", "TPCH_CN.NATION"],}}
{{ "Query Columns" : [ "TPCH_CN.NATION.N_NAME", "TPCH_CN.ORDERS.O_ORDERDATE",
"TPCH_CN.REGION.R_NAME", "TPCH_CN.PART.P_TYPE",
"TPCH_CN.LINEITEM.DISC_PRICE"],}}
{{ "Dimension(Group by)" : [ "TPCH_CN.ORDERS.O_ORDERDATE"],}}
{{ "Dimension(Filter cond)" : [ "TPCH_CN.ORDERS.O_ORDERDATE",
"TPCH_CN.REGION.R_NAME", "TPCH_CN.PART.P_TYPE"],}}
{{ "Measure" : [ "SUM(UNKNOWN_ALIAS.$F1)", "SUM(T_1_4CD6601C.DISC_PRICE)"],}}
{{ "Join" : "}}
{{Fact: [T_1_4CD6601C:LINEITEM]}}
{{ Dim: [T_1_4CD6601C:LINEITEM] INNER JOIN [T_2_716C1DB8:PART] ON [L_PARTKEY]
= [P_PARTKEY]}}
{{ Dim: [T_1_4CD6601C:LINEITEM] INNER JOIN [T_3_6D21031C:SUPPLIER] ON
[L_SUPPKEY] = [S_SUPPKEY]}}
{{ Dim: [T_3_6D21031C:SUPPLIER] INNER JOIN [T_7_4F00CD31:NATION] ON
[S_NATIONKEY] = [N_NATIONKEY]}}
{{ Dim: [T_1_4CD6601C:LINEITEM] INNER JOIN [T_4_5CEAD9DF:ORDERS] ON
[L_ORDERKEY] = [O_ORDERKEY]}}
{{ Dim: [T_4_5CEAD9DF:ORDERS] INNER JOIN [T_5_54B979E7:CUSTOMER] ON
[O_CUSTKEY] = [C_CUSTKEY]}}
{{ Dim: [T_5_54B979E7:CUSTOMER] INNER JOIN [T_6_42081A08:NATION] ON
[C_NATIONKEY] = [N_NATIONKEY]}}
{{ Dim: [T_6_42081A08:NATION] INNER JOIN [T_8_B14C00A:REGION] ON
[N_REGIONKEY] = [R_REGIONKEY]",}}
{{ "Index Id" : 0,}}
{{ "Query Column / Index Column" : "5 / 0",}}
{{ "Index Columns" : []}}
{{{} }{}}}{{{}3. RelNode(with ctx id) :{}}}
{{ OLAPToEnumerableConverter}}
{{ LimitRel(ctx=[0@null], fetch=[500])}}
{{ SortRel(sort0=[$0], dir0=[ASC-nulls-first], ctx=[0@null])}}
{{ ProjectRel(O_YEAR=[$0], MKT_SHARE=[/($1, $2)], ctx=[0@null])}}
{{ AggregateRel(group-set=[[0]], groups=[null], agg#0=[SUM($1)],
agg#1=[SUM($2)], ctx=[0@null])}}
{{ ProjectRel(O_YEAR=[EXTRACT(FLAG(YEAR), $54)], $f1=[CASE(=($72,
'BRAZIL'), $16, 0)], VOLUME=[$16], ctx=[0@null])}}
{{ FilterRel(condition=[AND(=($76, 'AMERICA'), >=($54, 1995-01-01),
<=($54, 1996-12-31), =($38, 'ECONOMY ANODIZED STEEL'))], ctx=[0@null])}}
{{ JoinRel(condition=[=($69, $75)], joinType=[inner],
ctx=[0@null])}}
{{ JoinRel(condition=[=($46, $71)], joinType=[inner],
ctx=[0@null])}}
{{ JoinRel(condition=[=($62, $67)], joinType=[inner],
ctx=[0@null])}}
{{ JoinRel(condition=[=($51, $59)], joinType=[inner],
ctx=[0@null])}}
{{ JoinRel(condition=[=($0, $50)], joinType=[inner],
ctx=[0@null])}}
{{ JoinRel(condition=[=($2, $43)], joinType=[inner],
ctx=[0@null])}}
{{ JoinRel(condition=[=($1, $34)], joinType=[inner],
ctx=[0@null])}}
{{ TableScan(table=[[TPCH_CN, LINEITEM]],
ctx=[0@null], fields=[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15,
16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33]])}}
{{ TableScan(table=[[TPCH_CN, PART]], ctx=[0@null],
fields=[[0, 1, 2, 3, 4, 5, 6, 7, 8]])}}
{{ TableScan(table=[[TPCH_CN, SUPPLIER]],
ctx=[0@null], fields=[[0, 1, 2, 3, 4, 5, 6]])}}
{{ TableScan(table=[[TPCH_CN, ORDERS]], ctx=[0@null],
fields=[[0, 1, 2, 3, 4, 5, 6, 7, 8]])}}
{{ TableScan(table=[[TPCH_CN, CUSTOMER]], ctx=[0@null],
fields=[[0, 1, 2, 3, 4, 5, 6, 7]])}}
{{ TableScan(table=[[TPCH_CN, NATION]], ctx=[0@null],
fields=[[0, 1, 2, 3]])}}
{{ TableScan(table=[[TPCH_CN, NATION]], ctx=[0@null],
fields=[[0, 1, 2, 3]])}}
{{{} TableScan(table=[[TPCH_CN, REGION]], ctx=[0@null],
fields=[[0, 1, 2]]){}}}{{{}4. SQL Text :{}}}
{{select /*+
[jira] [Updated] (KYLIN-5730) Query dry-run for better modeling
[
https://issues.apache.org/jira/browse/KYLIN-5730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoxiang Yu updated KYLIN-5730:
Description:
h2. Background
When user enable this feature, the query insight page will display helpful
message
for user to understand why model is not match and what to do in next step.
Following messages will display some query analytics, including at least:
# RelNode Tree
# OLAPContext and matched Model for each context
# Spark Physical Plan
Configuration entry is 'kylin.query.dryrun-enabled', at project level.
h2. How to use
# enable this in project configuration
# send a query in Insight page
# read Dry-run message and fix something and retry
h4. Dry-run message sample:
1. Last Exception :
No realization found for \{id = 0, model = not matched, fact table =
TPCH_CN.LINEITEM}
Incapable message : MODEL_UNMATCHED_JOIN, MODEL_UNMATCHED_JOIN,
MODEL_UNMATCHED_JOIN, MODEL_UNMATCHED_JOIN, MODEL_UNMATCHED_JOIN,
MODEL_UNMATCHED_JOIN, MODEL_UNMATCHED_JOIN, MODEL_UNMATCHED_JOIN,
2. OLAPContext(s) and matched model(s) :
Ctx=0 is not matched by any model/snapshot, recommend :
{
"Fact Table" : "TPCH_CN.LINEITEM",
"Dimension Tables" : [ "TPCH_CN.REGION", "TPCH_CN.ORDERS",
"TPCH_CN.SUPPLIER", "TPCH_CN.PART", "TPCH_CN.CUSTOMER", "TPCH_CN.NATION"],
"Query Columns" : [ "TPCH_CN.NATION.N_NAME", "TPCH_CN.ORDERS.O_ORDERDATE",
"TPCH_CN.REGION.R_NAME", "TPCH_CN.PART.P_TYPE", "TPCH_CN.LINEITEM.DISC_PRICE"],
"Dimension(Group by)" : [ "TPCH_CN.ORDERS.O_ORDERDATE"],
"Dimension(Filter cond)" : [ "TPCH_CN.ORDERS.O_ORDERDATE",
"TPCH_CN.REGION.R_NAME", "TPCH_CN.PART.P_TYPE"],
"Measure" : [ "SUM(UNKNOWN_ALIAS.$F1)", "SUM(T_1_4CD6601C.DISC_PRICE)"],
"Join" : "
Fact: [T_1_4CD6601C:LINEITEM]
Dim: [T_1_4CD6601C:LINEITEM] INNER JOIN [T_2_716C1DB8:PART] ON [L_PARTKEY] =
[P_PARTKEY]
Dim: [T_1_4CD6601C:LINEITEM] INNER JOIN [T_3_6D21031C:SUPPLIER] ON
[L_SUPPKEY] = [S_SUPPKEY]
Dim: [T_3_6D21031C:SUPPLIER] INNER JOIN [T_7_4F00CD31:NATION] ON
[S_NATIONKEY] = [N_NATIONKEY]
Dim: [T_1_4CD6601C:LINEITEM] INNER JOIN [T_4_5CEAD9DF:ORDERS] ON [L_ORDERKEY]
= [O_ORDERKEY]
Dim: [T_4_5CEAD9DF:ORDERS] INNER JOIN [T_5_54B979E7:CUSTOMER] ON
[O_CUSTKEY] = [C_CUSTKEY]
Dim: [T_5_54B979E7:CUSTOMER] INNER JOIN [T_6_42081A08:NATION] ON
[C_NATIONKEY] = [N_NATIONKEY]
Dim: [T_6_42081A08:NATION] INNER JOIN [T_8_B14C00A:REGION] ON
[N_REGIONKEY] = [R_REGIONKEY]",
"Index Id" : 0,
"Query Column / Index Column" : "5 / 0",
"Index Columns" : []
}
3. RelNode(with ctx id) :
OLAPToEnumerableConverter
LimitRel(ctx=[0@null], fetch=[500])
SortRel(sort0=[$0], dir0=[ASC-nulls-first], ctx=[0@null])
ProjectRel(O_YEAR=[$0], MKT_SHARE=[/($1, $2)], ctx=[0@null])
AggregateRel(group-set=[[0]], groups=[null], agg#0=[SUM($1)],
agg#1=[SUM($2)], ctx=[0@null])
ProjectRel(O_YEAR=[EXTRACT(FLAG(YEAR), $54)], $f1=[CASE(=($72,
'BRAZIL'), $16, 0)], VOLUME=[$16], ctx=[0@null])
FilterRel(condition=[AND(=($76, 'AMERICA'), >=($54, 1995-01-01),
<=($54, 1996-12-31), =($38, 'ECONOMY ANODIZED STEEL'))], ctx=[0@null])
JoinRel(condition=[=($69, $75)], joinType=[inner], ctx=[0@null])
JoinRel(condition=[=($46, $71)], joinType=[inner], ctx=[0@null])
JoinRel(condition=[=($62, $67)], joinType=[inner],
ctx=[0@null])
JoinRel(condition=[=($51, $59)], joinType=[inner],
ctx=[0@null])
JoinRel(condition=[=($0, $50)], joinType=[inner],
ctx=[0@null])
JoinRel(condition=[=($2, $43)], joinType=[inner],
ctx=[0@null])
JoinRel(condition=[=($1, $34)], joinType=[inner],
ctx=[0@null])
TableScan(table=[[TPCH_CN, LINEITEM]],
ctx=[0@null], fields=[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15,
16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33]])
TableScan(table=[[TPCH_CN, PART]], ctx=[0@null],
fields=[[0, 1, 2, 3, 4, 5, 6, 7, 8]])
TableScan(table=[[TPCH_CN, SUPPLIER]], ctx=[0@null],
fields=[[0, 1, 2, 3, 4, 5, 6]])
TableScan(table=[[TPCH_CN, ORDERS]], ctx=[0@null],
fields=[[0, 1, 2, 3, 4, 5, 6, 7, 8]])
TableScan(table=[[TPCH_CN, CUSTOMER]], ctx=[0@null],
fields=[[0, 1, 2, 3, 4, 5, 6, 7]])
TableScan(table=[[TPCH_CN, NATION]], ctx=[0@null],
fields=[[0, 1, 2, 3]])
TableScan(table=[[TPCH_CN, NATION]], ctx=[0@null],
fields=[[0, 1, 2, 3]])
TableScan(table=[[TPCH_CN, REGION]], ctx=[0@null], fields=[[0,
1, 2]])
4. SQL Text :
select /*+ MODEL_PRIORITY(Q8) */ o_year,
sum(case
when nation = 'BRAZIL'
then volume
else 0
end) / sum(volume)
as mkt_share
from
(
select year(o_orderdate) as o_year,
LINEITEM.DISC_PRICE as volume,
n2.n_name as
[jira] [Updated] (KYLIN-5730) Query dry-run for better modeling
[
https://issues.apache.org/jira/browse/KYLIN-5730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoxiang Yu updated KYLIN-5730:
Description:
h2. Background
When user enable this feature, the query insight page will display helpful
message
for user to understand why model is not match and what to do in next step.
Following messages will display some query analytics, including at least:
# RelNode Tree
# OLAPContext and matched Model for each context
# Spark Physical Plan
Configuration entry is 'kylin.query.dryrun-enabled', at project level.
h2. How to use
# enable this in project configuration
# send a query in Insight page
# read Dry-run message and fix something and retry
h4. Dry-run message sample:
1. Last Exception :
No realization found for \{id = 0, model = not matched, fact table =
TPCH_CN.LINEITEM}
Incapable message : MODEL_UNMATCHED_JOIN, MODEL_UNMATCHED_JOIN,
MODEL_UNMATCHED_JOIN, MODEL_UNMATCHED_JOIN, MODEL_UNMATCHED_JOIN,
MODEL_UNMATCHED_JOIN, MODEL_UNMATCHED_JOIN, MODEL_UNMATCHED_JOIN,
2. OLAPContext(s) and matched model(s) :
Ctx=0 is not matched by any model/snapshot, recommend :
{
"Fact Table" : "TPCH_CN.LINEITEM",
"Dimension Tables" : [ "TPCH_CN.REGION", "TPCH_CN.ORDERS",
"TPCH_CN.SUPPLIER", "TPCH_CN.PART", "TPCH_CN.CUSTOMER", "TPCH_CN.NATION"],
"Query Columns" : [ "TPCH_CN.NATION.N_NAME", "TPCH_CN.ORDERS.O_ORDERDATE",
"TPCH_CN.REGION.R_NAME", "TPCH_CN.PART.P_TYPE", "TPCH_CN.LINEITEM.DISC_PRICE"],
"Dimension(Group by)" : [ "TPCH_CN.ORDERS.O_ORDERDATE"],
"Dimension(Filter cond)" : [ "TPCH_CN.ORDERS.O_ORDERDATE",
"TPCH_CN.REGION.R_NAME", "TPCH_CN.PART.P_TYPE"],
"Measure" : [ "SUM(UNKNOWN_ALIAS.$F1)", "SUM(T_1_4CD6601C.DISC_PRICE)"],
"Join" : "
Fact: [T_1_4CD6601C:LINEITEM]
Dim: [T_1_4CD6601C:LINEITEM] INNER JOIN [T_2_716C1DB8:PART] ON [L_PARTKEY] =
[P_PARTKEY]
Dim: [T_1_4CD6601C:LINEITEM] INNER JOIN [T_3_6D21031C:SUPPLIER] ON
[L_SUPPKEY] = [S_SUPPKEY]
Dim: [T_3_6D21031C:SUPPLIER] INNER JOIN [T_7_4F00CD31:NATION] ON
[S_NATIONKEY] = [N_NATIONKEY]
Dim: [T_1_4CD6601C:LINEITEM] INNER JOIN [T_4_5CEAD9DF:ORDERS] ON [L_ORDERKEY]
= [O_ORDERKEY]
Dim: [T_4_5CEAD9DF:ORDERS] INNER JOIN [T_5_54B979E7:CUSTOMER] ON
[O_CUSTKEY] = [C_CUSTKEY]
Dim: [T_5_54B979E7:CUSTOMER] INNER JOIN [T_6_42081A08:NATION] ON
[C_NATIONKEY] = [N_NATIONKEY]
Dim: [T_6_42081A08:NATION] INNER JOIN [T_8_B14C00A:REGION] ON
[N_REGIONKEY] = [R_REGIONKEY]",
"Index Id" : 0,
"Query Column / Index Column" : "5 / 0",
"Index Columns" : []
}
3. RelNode(with ctx id) :
OLAPToEnumerableConverter
LimitRel(ctx=[0@null], fetch=[500])
SortRel(sort0=[$0], dir0=[ASC-nulls-first], ctx=[0@null])
ProjectRel(O_YEAR=[$0], MKT_SHARE=[/($1, $2)], ctx=[0@null])
AggregateRel(group-set=[[0]], groups=[null], agg#0=[SUM($1)],
agg#1=[SUM($2)], ctx=[0@null])
ProjectRel(O_YEAR=[EXTRACT(FLAG(YEAR), $54)], $f1=[CASE(=($72,
'BRAZIL'), $16, 0)], VOLUME=[$16], ctx=[0@null])
FilterRel(condition=[AND(=($76, 'AMERICA'), >=($54, 1995-01-01),
<=($54, 1996-12-31), =($38, 'ECONOMY ANODIZED STEEL'))], ctx=[0@null])
JoinRel(condition=[=($69, $75)], joinType=[inner], ctx=[0@null])
JoinRel(condition=[=($46, $71)], joinType=[inner], ctx=[0@null])
JoinRel(condition=[=($62, $67)], joinType=[inner],
ctx=[0@null])
JoinRel(condition=[=($51, $59)], joinType=[inner],
ctx=[0@null])
JoinRel(condition=[=($0, $50)], joinType=[inner],
ctx=[0@null])
JoinRel(condition=[=($2, $43)], joinType=[inner],
ctx=[0@null])
JoinRel(condition=[=($1, $34)], joinType=[inner],
ctx=[0@null])
TableScan(table=[[TPCH_CN, LINEITEM]],
ctx=[0@null], fields=[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15,
16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33]])
TableScan(table=[[TPCH_CN, PART]], ctx=[0@null],
fields=[[0, 1, 2, 3, 4, 5, 6, 7, 8]])
TableScan(table=[[TPCH_CN, SUPPLIER]], ctx=[0@null],
fields=[[0, 1, 2, 3, 4, 5, 6]])
TableScan(table=[[TPCH_CN, ORDERS]], ctx=[0@null],
fields=[[0, 1, 2, 3, 4, 5, 6, 7, 8]])
TableScan(table=[[TPCH_CN, CUSTOMER]], ctx=[0@null],
fields=[[0, 1, 2, 3, 4, 5, 6, 7]])
TableScan(table=[[TPCH_CN, NATION]], ctx=[0@null],
fields=[[0, 1, 2, 3]])
TableScan(table=[[TPCH_CN, NATION]], ctx=[0@null],
fields=[[0, 1, 2, 3]])
TableScan(table=[[TPCH_CN, REGION]], ctx=[0@null], fields=[[0,
1, 2]])
4. SQL Text :
select /*+ MODEL_PRIORITY(Q8) */ o_year,
sum(case
when nation = 'BRAZIL'
then volume
else 0
end) / sum(volume)
as mkt_share
from
(
select year(o_orderdate) as o_year,
LINEITEM.DISC_PRICE as volume,
n2.n_name as
[jira] [Updated] (KYLIN-5730) Query dry-run for better modeling
[
https://issues.apache.org/jira/browse/KYLIN-5730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoxiang Yu updated KYLIN-5730:
Description:
h2. Background
When user enable this feature, the query insight page will display helpful
message
for user to understand why model is not match.
Following messages will display some query analytics, including at least:
# RelNode Tree
# OLAPContext and matched Model for each context
# Spark Physical Plan
Configuration entry is 'kylin.query.dryrun-enabled', at project level.
h2. How to use
# enable this in project configuration
# send a query in Insight page
# receive and read Dry-run message
h4. Dry-run message sample:
{quote}1. Last Exception :
No realization found for \{id = 0, model = not matched, fact table =
TPCH_CN.LINEITEM}
Incapable message : MODEL_UNMATCHED_JOIN, MODEL_UNMATCHED_JOIN,
MODEL_UNMATCHED_JOIN, MODEL_UNMATCHED_JOIN, MODEL_UNMATCHED_JOIN,
MODEL_UNMATCHED_JOIN, MODEL_UNMATCHED_JOIN, MODEL_UNMATCHED_JOIN,
2. OLAPContext(s) and matched model(s) :
Ctx=0 is not matched by any model/snapshot, recommend :
{
"Fact Table" : "TPCH_CN.LINEITEM",
"Dimension Tables" : [ "TPCH_CN.REGION", "TPCH_CN.ORDERS",
"TPCH_CN.SUPPLIER", "TPCH_CN.PART", "TPCH_CN.CUSTOMER", "TPCH_CN.NATION"],
"Query Columns" : [ "TPCH_CN.NATION.N_NAME", "TPCH_CN.ORDERS.O_ORDERDATE",
"TPCH_CN.REGION.R_NAME", "TPCH_CN.PART.P_TYPE", "TPCH_CN.LINEITEM.DISC_PRICE"],
"Dimension(Group by)" : [ "TPCH_CN.ORDERS.O_ORDERDATE"],
"Dimension(Filter cond)" : [ "TPCH_CN.ORDERS.O_ORDERDATE",
"TPCH_CN.REGION.R_NAME", "TPCH_CN.PART.P_TYPE"],
"Measure" : [ "SUM(UNKNOWN_ALIAS.$F1)", "SUM(T_1_4CD6601C.DISC_PRICE)"],
"Join" : "
Fact: [T_1_4CD6601C:LINEITEM]
Dim: [T_1_4CD6601C:LINEITEM] INNER JOIN [T_2_716C1DB8:PART] ON [L_PARTKEY] =
[P_PARTKEY]
Dim: [T_1_4CD6601C:LINEITEM] INNER JOIN [T_3_6D21031C:SUPPLIER] ON
[L_SUPPKEY] = [S_SUPPKEY]
Dim: [T_3_6D21031C:SUPPLIER] INNER JOIN [T_7_4F00CD31:NATION] ON
[S_NATIONKEY] = [N_NATIONKEY]
Dim: [T_1_4CD6601C:LINEITEM] INNER JOIN [T_4_5CEAD9DF:ORDERS] ON [L_ORDERKEY]
= [O_ORDERKEY]
Dim: [T_4_5CEAD9DF:ORDERS] INNER JOIN [T_5_54B979E7:CUSTOMER] ON
[O_CUSTKEY] = [C_CUSTKEY]
Dim: [T_5_54B979E7:CUSTOMER] INNER JOIN [T_6_42081A08:NATION] ON
[C_NATIONKEY] = [N_NATIONKEY]
Dim: [T_6_42081A08:NATION] INNER JOIN [T_8_B14C00A:REGION] ON
[N_REGIONKEY] = [R_REGIONKEY]",
"Index Id" : 0,
"Query Column / Index Column" : "5 / 0",
"Index Columns" : []
}
3. RelNode(with ctx id) :
OLAPToEnumerableConverter
LimitRel(ctx=[0@null], fetch=[500])
SortRel(sort0=[$0], dir0=[ASC-nulls-first], ctx=[0@null])
ProjectRel(O_YEAR=[$0], MKT_SHARE=[/($1, $2)], ctx=[0@null])
AggregateRel(group-set=[[0]], groups=[null], agg#0=[SUM($1)],
agg#1=[SUM($2)], ctx=[0@null])
ProjectRel(O_YEAR=[EXTRACT(FLAG(YEAR), $54)], $f1=[CASE(=($72,
'BRAZIL'), $16, 0)], VOLUME=[$16], ctx=[0@null])
FilterRel(condition=[AND(=($76, 'AMERICA'), >=($54, 1995-01-01),
<=($54, 1996-12-31), =($38, 'ECONOMY ANODIZED STEEL'))], ctx=[0@null])
JoinRel(condition=[=($69, $75)], joinType=[inner], ctx=[0@null])
JoinRel(condition=[=($46, $71)], joinType=[inner], ctx=[0@null])
JoinRel(condition=[=($62, $67)], joinType=[inner],
ctx=[0@null])
JoinRel(condition=[=($51, $59)], joinType=[inner],
ctx=[0@null])
JoinRel(condition=[=($0, $50)], joinType=[inner],
ctx=[0@null])
JoinRel(condition=[=($2, $43)], joinType=[inner],
ctx=[0@null])
JoinRel(condition=[=($1, $34)], joinType=[inner],
ctx=[0@null])
TableScan(table=[[TPCH_CN, LINEITEM]],
ctx=[0@null], fields=[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15,
16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33]])
TableScan(table=[[TPCH_CN, PART]], ctx=[0@null],
fields=[[0, 1, 2, 3, 4, 5, 6, 7, 8]])
TableScan(table=[[TPCH_CN, SUPPLIER]], ctx=[0@null],
fields=[[0, 1, 2, 3, 4, 5, 6]])
TableScan(table=[[TPCH_CN, ORDERS]], ctx=[0@null],
fields=[[0, 1, 2, 3, 4, 5, 6, 7, 8]])
TableScan(table=[[TPCH_CN, CUSTOMER]], ctx=[0@null],
fields=[[0, 1, 2, 3, 4, 5, 6, 7]])
TableScan(table=[[TPCH_CN, NATION]], ctx=[0@null],
fields=[[0, 1, 2, 3]])
TableScan(table=[[TPCH_CN, NATION]], ctx=[0@null],
fields=[[0, 1, 2, 3]])
TableScan(table=[[TPCH_CN, REGION]], ctx=[0@null], fields=[[0,
1, 2]])
4. SQL Text :
select /*+ MODEL_PRIORITY(Q8) */ o_year,
sum(case
when nation = 'BRAZIL'
then volume
else 0
end) / sum(volume)
as mkt_share
from
(
select year(o_orderdate) as o_year,
LINEITEM.DISC_PRICE as volume,
n2.n_name as nation
from
tpch_cn.lineitem
join
[jira] [Updated] (KYLIN-5730) Query dry-run for better modeling
[ https://issues.apache.org/jira/browse/KYLIN-5730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu updated KYLIN-5730: Description: h2. Background When user enable this feature, the query insight page will display helpful message for user to understand why model is not match. Following messages will display some query analytics, including at least: # RelNode Tree # OLAPContext and matched Model for each context # Spark Physical Plan Configuration entry is 'kylin.query.dryrun-enabled', at project level. h2. How to use # enable this in project configuration # send a query in Insight page # receive and read Dry-run message h2. Screenshots h4. Step1 : enable this in project configuration !KYLIN-5730-01.png! h4. Step2 : send a query in Insight page !KYLIN-5730-02.png! h4. Step3 : Dry-run message in Insight page !KYLIN-5730-03.png! was: When user enable this feature, the query insight page will display helpful message for user to understand why model is not match. Following messages will display some query analytics, including at least: # RelNode Tree # OLAPContext and matched Model for each context # Spark Physical Plan Configuration entry is 'kylin.query.dryrun-enabled', at project level. > Query dry-run for better modeling > - > > Key: KYLIN-5730 > URL: https://issues.apache.org/jira/browse/KYLIN-5730 > Project: Kylin > Issue Type: Improvement >Affects Versions: 5.0-beta >Reporter: Xiaoxiang Yu >Assignee: Xiaoxiang Yu >Priority: Major > Fix For: 5.0.0 > > Attachments: KYLIN-5730-01.png, KYLIN-5730-02.png, KYLIN-5730-03.png > > > h2. Background > When user enable this feature, the query insight page will display helpful > message > for user to understand why model is not match. > > Following messages will display some query analytics, including at least: > # RelNode Tree > # OLAPContext and matched Model for each context > # Spark Physical Plan > Configuration entry is 'kylin.query.dryrun-enabled', at project level. > h2. How to use > # enable this in project configuration > # send a query in Insight page > # receive and read Dry-run message > > > h2. Screenshots > h4. Step1 : enable this in project configuration > !KYLIN-5730-01.png! > h4. Step2 : send a query in Insight page > !KYLIN-5730-02.png! > h4. Step3 : Dry-run message in Insight page > !KYLIN-5730-03.png! > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KYLIN-5730) Query dry-run for better modeling
[ https://issues.apache.org/jira/browse/KYLIN-5730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu updated KYLIN-5730: Attachment: KYLIN-5730-01.png KYLIN-5730-02.png KYLIN-5730-03.png > Query dry-run for better modeling > - > > Key: KYLIN-5730 > URL: https://issues.apache.org/jira/browse/KYLIN-5730 > Project: Kylin > Issue Type: Improvement >Affects Versions: 5.0-beta >Reporter: Xiaoxiang Yu >Assignee: Xiaoxiang Yu >Priority: Major > Fix For: 5.0.0 > > Attachments: KYLIN-5730-01.png, KYLIN-5730-02.png, KYLIN-5730-03.png > > > When user enable this feature, the query insight page will display helpful > message for user to understand why model is not match. > > Following messages will display some query analytics, including at least: > # RelNode Tree > # OLAPContext and matched Model for each context > # Spark Physical Plan > Configuration entry is 'kylin.query.dryrun-enabled', at project level. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KYLIN-5730) Query dry-run for better modeling
Xiaoxiang Yu created KYLIN-5730: --- Summary: Query dry-run for better modeling Key: KYLIN-5730 URL: https://issues.apache.org/jira/browse/KYLIN-5730 Project: Kylin Issue Type: Improvement Affects Versions: 5.0-beta Reporter: Xiaoxiang Yu Assignee: Xiaoxiang Yu Fix For: 5.0.0 When user enable this feature, the query insight page will display helpful message for user to understand why model is not match. Following messages will display some query analytics, including at least: # RelNode Tree # OLAPContext and matched Model for each context # Spark Physical Plan Configuration entry is 'kylin.query.dryrun-enabled', at project level. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KYLIN-5728) Kylin 5 Hive Data source - sync issue with hive view's column type BigInt
[ https://issues.apache.org/jira/browse/KYLIN-5728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17790466#comment-17790466 ] Xiaoxiang Yu commented on KYLIN-5728: - Looks like it is a Spark related issues. Could you please provided your ddl of original table and view? Do you tried to add "cast" in the ddl of the view? > Kylin 5 Hive Data source - sync issue with hive view's column type BigInt > -- > > Key: KYLIN-5728 > URL: https://issues.apache.org/jira/browse/KYLIN-5728 > Project: Kylin > Issue Type: Bug > Components: Spark Engine >Affects Versions: 5.0-beta >Reporter: Sonu Kumar Singh >Priority: Critical > Fix For: 5.0.0 > > > traceId: 4bc0af5c-f138-5507-4c42-9cdb1f17fe03 2023-11-22T06:15:01,594 ERROR > [] [http-nio-9000-exec-6] source.NSparkTableMetaExplorer : Error for > parser view: table > org.apache.spark.sql.AnalysisException: Cannot up cast column1 from bigint to > int. > The type path of the target object is: > You can either add an explicit cast to the input data or choose a higher > precision type of the field in the target object -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KYLIN-5728) Kylin 5 Hive Data source - sync issue with hive view's column type BigInt
[ https://issues.apache.org/jira/browse/KYLIN-5728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17789030#comment-17789030 ] Xiaoxiang Yu commented on KYLIN-5728: - what is the name of this table, looks like the table's name is "table", right? > Kylin 5 Hive Data source - sync issue with hive view's column type BigInt > -- > > Key: KYLIN-5728 > URL: https://issues.apache.org/jira/browse/KYLIN-5728 > Project: Kylin > Issue Type: Bug > Components: Spark Engine >Affects Versions: 5.0-beta >Reporter: Sonu Kumar Singh >Priority: Critical > Fix For: 5.0.0 > > > traceId: 4bc0af5c-f138-5507-4c42-9cdb1f17fe03 2023-11-22T06:15:01,594 ERROR > [] [http-nio-9000-exec-6] source.NSparkTableMetaExplorer : Error for > parser view: table > org.apache.spark.sql.AnalysisException: Cannot up cast column1 from bigint to > int. > The type path of the target object is: > You can either add an explicit cast to the input data or choose a higher > precision type of the field in the target object -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KYLIN-5697) Mirror: Kylin5 request response optimization
[ https://issues.apache.org/jira/browse/KYLIN-5697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu updated KYLIN-5697: Fix Version/s: (was: 5.0-beta) > Mirror: Kylin5 request response optimization > > > Key: KYLIN-5697 > URL: https://issues.apache.org/jira/browse/KYLIN-5697 > Project: Kylin > Issue Type: Improvement > Components: REST Service >Affects Versions: 5.0-alpha >Reporter: Yaguang Jia >Assignee: Yaguang Jia >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KYLIN-5698) A dimension table field with an alias is used for measure, but the field is not added to the Kylin5 model dimension, and the field type cannot be obtained when creating a
[ https://issues.apache.org/jira/browse/KYLIN-5698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu updated KYLIN-5698: Fix Version/s: (was: 5.0-beta) > A dimension table field with an alias is used for measure, but the field is > not added to the Kylin5 model dimension, and the field type cannot be > obtained when creating a tableindex > - > > Key: KYLIN-5698 > URL: https://issues.apache.org/jira/browse/KYLIN-5698 > Project: Kylin > Issue Type: Bug > Components: Modeling >Affects Versions: 5.0-alpha >Reporter: Yaguang Jia >Assignee: Yaguang Jia >Priority: Minor > > *Dev Design* > /kylin/api/models?model_name= In the response of this api, add the return > value to the type in the selected_columns list, and return the field type of > the column. > > > > *Dev Design* > /kylin/api/models?model_name= > 这个api的response里，在selected_columns列表里type加上返回值，返回列的字段类型。 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KYLIN-5694) Concat function supports 0 or 1 parameter
[ https://issues.apache.org/jira/browse/KYLIN-5694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu updated KYLIN-5694: Fix Version/s: (was: 5.0-beta) > Concat function supports 0 or 1 parameter > - > > Key: KYLIN-5694 > URL: https://issues.apache.org/jira/browse/KYLIN-5694 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: 5.0-alpha >Reporter: Yaguang Jia >Assignee: Yaguang Jia >Priority: Minor > > h2. Design > Kylin5 currently does not support UDFs with only one parameter. However, > Spark and Hive do support it. Additionally, it has been verified that Spark > and Hive also support concat with zero parameters. Here, we are adding > support for Kylin5 to handle both zero and one parameter cases. > > > > Kylin5当前自定义的 udf 不支持一个参数的。但是 spark 和 hive 支持。同时还验证了一下 spark 和 hive 还支持 concat > 为0 个参数的情况。这里对增加Kylin5对0个和1个参数的支持。 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KYLIN-5695) Code refactoring : Filter condition -> rewrite
[ https://issues.apache.org/jira/browse/KYLIN-5695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu updated KYLIN-5695: Fix Version/s: (was: 5.0-beta) > Code refactoring : Filter condition -> rewrite > --- > > Key: KYLIN-5695 > URL: https://issues.apache.org/jira/browse/KYLIN-5695 > Project: Kylin > Issue Type: Improvement > Components: Modeling >Affects Versions: 5.0-alpha >Reporter: Yaguang Jia >Assignee: Yaguang Jia >Priority: Minor > > h2. Dev Design > Rewrite the Filter condition functionality for models: > 1. When it is found that the field definition of a model does not comply with > the two-phase definition, a KylinException is generated with necessary error > information. > 2. When it is found that the filter condition of an imported model does not > meet the definition of the model (including case sensitivity issues), a > KylinException is generated with necessary error information. > > > h1. Dev Design > 对模型的Filter condition的rewrite功能重写： > # 当发现模型的字段定义不符合两段式定义时，产生KylinException，并附带必要的错误信息； > # 当发现导入的模型的filter condition，不满足模型的定义时（包括大小写匹配问题），产生KylinException，并附带必要的错误信息； -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KYLIN-5535) when the creating user name is the suffix part of the existing user name, creating user fails and reports 'duplicate name'
[ https://issues.apache.org/jira/browse/KYLIN-5535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu resolved KYLIN-5535. - Resolution: Fixed > when the creating user name is the suffix part of the existing user name, > creating user fails and reports 'duplicate name' > -- > > Key: KYLIN-5535 > URL: https://issues.apache.org/jira/browse/KYLIN-5535 > Project: Kylin > Issue Type: Bug >Affects Versions: 5.0-alpha >Reporter: Yaguang Jia >Assignee: Yaguang Jia >Priority: Minor > Fix For: 5.0-beta > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KYLIN-5687) Remove the log information added abnormally during job execution
[ https://issues.apache.org/jira/browse/KYLIN-5687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu updated KYLIN-5687: Fix Version/s: (was: 5.0-beta) > Remove the log information added abnormally during job execution > > > Key: KYLIN-5687 > URL: https://issues.apache.org/jira/browse/KYLIN-5687 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: 5.0-alpha >Reporter: Yaguang Jia >Assignee: sibing.zhang >Priority: Minor > > Remove the log information added abnormally during job execution -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KYLIN-5691) Kylin5 datediff function supports cases where the parameter is a string.
[
https://issues.apache.org/jira/browse/KYLIN-5691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoxiang Yu updated KYLIN-5691:
Fix Version/s: (was: 5.0-beta)
> Kylin5 datediff function supports cases where the parameter is a string.
>
>
> Key: KYLIN-5691
> URL: https://issues.apache.org/jira/browse/KYLIN-5691
> Project: Kylin
> Issue Type: Improvement
> Components: Query Engine
>Affects Versions: 5.0-alpha
>Reporter: Yaguang Jia
>Priority: Minor
>
> h2. Design
> Previously datediff only supported calls where both parameters were of type
> date, such as select datediff(date'2023-05-01', date'2023-05-02')
> Now it has been extended to also support date strings where both arguments
> are in ''-MM-dd'' format, such as select datediff('2023-05-01',
> '2023-05-02' )
>
>
>
> 之前datediff只支持两个参数都为date类型的调用，如 select datediff(date'2023-05-01',
> date'2023-05-02')
> 现在做了扩展，也可以支持两个参数都为‘‘-MM-dd’’格式的日期字符串，如select datediff('2023-05-01',
> '2023-05-02')
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KYLIN-5534) When normal users export TDS, due to the lack of duplicate name verification, which is inconsistent with the behavior of Admin users
[
https://issues.apache.org/jira/browse/KYLIN-5534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoxiang Yu resolved KYLIN-5534.
-
Resolution: Fixed
> When normal users export TDS, due to the lack of duplicate name verification,
> which is inconsistent with the behavior of Admin users
>
>
> Key: KYLIN-5534
> URL: https://issues.apache.org/jira/browse/KYLIN-5534
> Project: Kylin
> Issue Type: Bug
> Components: Modeling
>Affects Versions: 5.0-alpha
>Reporter: Yaguang Jia
>Assignee: Yaguang Jia
>Priority: Minor
> Fix For: 5.0-beta
>
>
> The front-end exports TDS files, and will verify various duplicate names
> through this interface /kylin/api/models/validate_export.
> Before verification, all the columns of the model table will be obtained and
> stored in a map such as MapString, Column Def model Cols Map. The
> Column Def in this map will have an is Hidden attribute, which is true by
> default, which means to hide the column in TDS.
> However, when verifying the duplicate names of metrics, table columns, and
> dimension names, the model Cols Map will be obtained to get the columns from
> it and perform column Def.is Hidden() filtering. Obviously all columns will
> be filtered out. Caused the mistake to avoid the duplicate name check.
> Therefore, an error will be reported when importing tableau.
> Fix Design
> When verifying duplicate names, remove the filtering of the model Cols Map.
>
> 前端导出TDS文件，会通过此接口{{{}/kylin/api/models/validate_export{}}}校验各种重名问题。
> 在校验前会获取到模型表的所有列，存储在{{{}Map
> modelColsMap{}}}这样一个map中。这个map中的{{{}ColumnDef{}}}会有一个{{{}isHidden{}}}属性，默认为true，含义就是在TDS中隐藏该列。
> 但是后续在进行度量与表列以及维度名重名校验时，会获取到{{{}modelColsMap{}}}这个map从中拿到列，进行{{{}columnDef.isHidden(){}}}过滤。显然所有列都会被过滤掉。导致错误的避开了重名校验。因此，在导入tableau时会报错。
> h4. *Fix Design*
> 重名校验时，去掉对{{{}modelColsMap{}}} 的过滤。
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KYLIN-5696) Logical view with Chinese name sampling or building exception
[ https://issues.apache.org/jira/browse/KYLIN-5696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu updated KYLIN-5696: Fix Version/s: (was: 5.0-beta) > Logical view with Chinese name sampling or building exception > - > > Key: KYLIN-5696 > URL: https://issues.apache.org/jira/browse/KYLIN-5696 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: 5.0-alpha >Reporter: Yaguang Jia >Assignee: Yaguang Jia >Priority: Major > > *Root Cause* > The Logical View did not handle special characters (such as Chinese table > names) that appear in the database table name when dropping a table. > *Dev Design* > Enhance the handling of special characters in DDL statements within the > Logical View by using backticks for annotation. > > > > *Root Cause* > Logical View 在 drop table 时没有对库表名称中出现的特殊字符（比如中文表名）做兼容处理 > *Dev Design* > 增加 Logical View 中的 DDL 对特殊字符的处理，用反引号标注 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KYLIN-5689) Kylin5 local debugging and debugging Kylin5 using the metadata in the diagnostic package
[ https://issues.apache.org/jira/browse/KYLIN-5689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu updated KYLIN-5689: Fix Version/s: (was: 5.0-beta) > Kylin5 local debugging and debugging Kylin5 using the metadata in the > diagnostic package > > > Key: KYLIN-5689 > URL: https://issues.apache.org/jira/browse/KYLIN-5689 > Project: Kylin > Issue Type: New Feature > Components: Metadata >Affects Versions: 5.0-alpha >Reporter: Yaguang Jia >Assignee: Yaguang Jia >Priority: Major > Attachments: KYLIN-5689 Kylin5 local debugging and debugging Kylin5 > using the metadata in the diagnostic package.pdf, KYLIN-5689 Kylin5本地调试 和 > 使用诊断包中的元数据进行调试.pdf > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KYLIN-5692) Modify nested computed column expression, save model failed
[ https://issues.apache.org/jira/browse/KYLIN-5692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu updated KYLIN-5692: Fix Version/s: (was: 5.0-beta) > Modify nested computed column expression, save model failed > --- > > Key: KYLIN-5692 > URL: https://issues.apache.org/jira/browse/KYLIN-5692 > Project: Kylin > Issue Type: Bug > Components: Modeling >Affects Versions: 5.0-alpha >Reporter: Yaguang Jia >Assignee: Yaguang Jia >Priority: Major > > h2. Root Cause > The problem lies in the method discardInvalidColsAndMeasForBrokenModel. The > expression of cc is not expanded, while innerExpression is expanded. This > method only collects the original columns in the table. When there is nested > computed column, some computed columns will not be collected, resulting in > this method mistakenly judging certain cc columns as invalid. > h2. Fix Design > When parsing cc, still use innerExpression, but innerExpression is compatible > with Spark syntax. Directly handing it over to Calcite for SQL parsing and > obtaining tables and columns may encounter syntax issues such as unsupported > functions. Therefore, simply call the QueryUtil.adaptCalciteSyntax method. > > > h2. Root Cause > 问题出在discardInvalidColsAndMeasForBrokenModel这个方法, cc 的 expression 是不展开的，而 > innerExpression 是展开的。这个方法中只收集了表中的原始列，当存在可计算列的嵌套是，那么有些可计算列是不会被收集的，从而这个方法就误判了某些 > cc 列是非法的。 > h2. Fix Design > 在解析 cc 的时候依然使用 innerExpression，但 innerExpression 是 spark 语法兼容的，直接交给 Calcite > 执行 SQL 解析获取表和列可能会存在函数不支持等语法问题，因此调用一下 QueryUtil.adaptCalciteSyntax 这个方法即可。 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KYLIN-5474) Kylin5 throw [null Exception] org.h2.Driver in local debug mode
[ https://issues.apache.org/jira/browse/KYLIN-5474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu updated KYLIN-5474: Fix Version/s: (was: 5.0-beta) > Kylin5 throw [null Exception] org.h2.Driver in local debug mode > - > > Key: KYLIN-5474 > URL: https://issues.apache.org/jira/browse/KYLIN-5474 > Project: Kylin > Issue Type: Bug > Components: Environment >Affects Versions: 5.0-alpha >Reporter: Yaguang Jia >Assignee: Yaguang Jia >Priority: Minor > Attachments: image (1).png, image (2).png > > > For Kyln5 local debug mode, please refer -> > https://issues.apache.org/jira/browse/KYLIN-5269 > > kylin5 throw [null Exception] org.h2.Driver when run sql in local debug mode > !image (1).png|width=1017,height=417! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KYLIN-5649) When query contains computed columns, fail to guarantee the priority of using the aggregate index to answer the aggregate query
[ https://issues.apache.org/jira/browse/KYLIN-5649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu resolved KYLIN-5649. - Resolution: Fixed > When query contains computed columns, fail to guarantee the priority of using > the aggregate index to answer the aggregate query > --- > > Key: KYLIN-5649 > URL: https://issues.apache.org/jira/browse/KYLIN-5649 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: 5.0-alpha >Reporter: Zhiting Guo >Assignee: Zhiting Guo >Priority: Major > Fix For: 5.0-beta > > Attachments: When query contains computed columns, fail to guarantee > the priority of using the aggregate index to answer the aggregate query.pdf > > > After enabling the "kylin.query.use-tableindex-answer-non-raw-query = true" & > & "kylin.query.layout.prefer-aggindex = true" parameter, the aggregate query > can match the aggregate index and the basic detail index, but the final hit > is the basic detail index. What is puzzling is why the aggregate query is > answered using the basic detail index? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KYLIN-5688) After a system admin user without data permission is added to a user group with data permission for a project, the system admin still has no data permission for this proj
[ https://issues.apache.org/jira/browse/KYLIN-5688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu updated KYLIN-5688: Fix Version/s: (was: 5.0-beta) > After a system admin user without data permission is added to a user group > with data permission for a project, the system admin still has no data > permission for this project > - > > Key: KYLIN-5688 > URL: https://issues.apache.org/jira/browse/KYLIN-5688 > Project: Kylin > Issue Type: Bug > Components: Security >Affects Versions: 5.0-alpha >Reporter: Yaguang Jia >Assignee: Zhiting Guo >Priority: Minor > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KYLIN-5650) In the cloud environment, there is a probability that the dictionary metadata file will be read abnormally during building job, resulting in incorrect query results.
[
https://issues.apache.org/jira/browse/KYLIN-5650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoxiang Yu resolved KYLIN-5650.
-
Resolution: Fixed
> In the cloud environment, there is a probability that the dictionary metadata
> file will be read abnormally during building job, resulting in incorrect
> query results.
> -
>
> Key: KYLIN-5650
> URL: https://issues.apache.org/jira/browse/KYLIN-5650
> Project: Kylin
> Issue Type: Bug
> Components: Tools, Build and Test
>Affects Versions: 5.0-alpha
>Reporter: Zhiting Guo
>Assignee: Zhiting Guo
>Priority: Major
> Fix For: 5.0-beta
>
> Attachments: In the cloud environment, there is a probability that
> the dictionary metadata file will be read abnormally during building job,
> resulting in incorrect query results..pdf
>
>
> Checked the dictionary, there are no duplicate values. Checked the execution
> plan of the build dictionary step, there is no problem. Checked the steps of
> building a flat table and found that there was a problem in the step of flat
> table encoding dictionary.
> The reason for the error is that the encoding is not performed after
> repartition according to the dictionary column. As shown in the figure, there
> is no repartition, and the encode column appears in the plan.
> There are also the following logs:
> {code:java}
> 2023-03-26T20:26:30,868 INFO [logger-thread-0] dict.NGlobalDictHDFSStore :
> Commit from
> s3a://datalake-kc-s3-prd-bj/kylin/kcprodYcHG_kylin/datalake_kylin/dict/global_dict/GDT.GDT_CMPLYA_FCT_DIST_RESLT/IS_STAT/working
> to
> s3a://datalake-kc-s3-prd-bj/kylin/kcprodYcHG_kylin/datalake_kylin/dict/global_dict/GDT.GDT_CMPLYA_FCT_DIST_RESLT/IS_STAT/version_1679862387539
> 2023-03-26T20:31:14,501 INFO [logger-thread-0] dict.NGlobalDictionaryV2 :
> getMetaInfo versions.length is 12
> 2023-03-26T20:31:14,547 INFO [logger-thread-0] dict.NGlobalDictHDFSStore :
> because metaFiles.length is 0, metaInfo is null
> 2023-03-26T20:31:14,547 INFO [logger-thread-0] dict.NGlobalDictionaryV2 :
> getMetaInfo metadata is null : [true]{code}
> This is on s3, after renaming the dictionary directory, no metadata file is
> queried. However, if the meta is not obtained in the code and no error is
> reported, it is not reasonable to encode directly without repartition. In
> short, the result is that the encoding of the dictionary column on the flat
> table fails.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KYLIN-5651) supports obtaining table comment from Hive
[ https://issues.apache.org/jira/browse/KYLIN-5651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu resolved KYLIN-5651. - Resolution: Fixed > supports obtaining table comment from Hive > -- > > Key: KYLIN-5651 > URL: https://issues.apache.org/jira/browse/KYLIN-5651 > Project: Kylin > Issue Type: Improvement > Components: REST Service >Affects Versions: 5.0-alpha >Reporter: Zhiting Guo >Assignee: Zhiting Guo >Priority: Major > Fix For: 5.0-beta > > Attachments: supports obtaining table comment from Hive.pdf > > > The API to get the table cannot get the table comment > Get /kylin/api/tables -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KYLIN-5652) Network anomalies or metabase anomalies cause Project Epoch to change frequently, which may cause the job in pending status for a long time and no longer to be executed
[ https://issues.apache.org/jira/browse/KYLIN-5652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu resolved KYLIN-5652. - Resolution: Fixed > Network anomalies or metabase anomalies cause Project Epoch to change > frequently, which may cause the job in pending status for a long time and no > longer to be executed > > > Key: KYLIN-5652 > URL: https://issues.apache.org/jira/browse/KYLIN-5652 > Project: Kylin > Issue Type: Bug > Components: Metadata >Affects Versions: 5.0-alpha >Reporter: Zhiting Guo >Assignee: Zhiting Guo >Priority: Major > Fix For: 5.0-beta > > Attachments: Network anomalies or metabase anomalies cause Project > Epoch to change frequently, which may cause the job in pending status for a > long time and no longer to be executed.pdf > > > The scheduling of the project is not shut down during the shutdown period. At > this time, the start operation is triggered. When starting, it is found that > the scheduling is still in the start state in the if judgment, and it is > returned. > Then the shutdown operation ends, the task scheduling is turned off, and the > task of the project has been in the pending state since then. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KYLIN-5653) Enable Spark Parquet Page Index by default to improve query performance
[ https://issues.apache.org/jira/browse/KYLIN-5653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu resolved KYLIN-5653. - Resolution: Fixed > Enable Spark Parquet Page Index by default to improve query performance > --- > > Key: KYLIN-5653 > URL: https://issues.apache.org/jira/browse/KYLIN-5653 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: 5.0-alpha >Reporter: Zhiting Guo >Assignee: Zhiting Guo >Priority: Major > Fix For: 5.0-beta > > > Spark has officially supported Parquet PageIndex support, so kylin can enable > the PageIndex function by default -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KYLIN-5647) upload hive_1_2_2 jars to HDFS before kylin start
[ https://issues.apache.org/jira/browse/KYLIN-5647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu resolved KYLIN-5647. - Resolution: Fixed > upload hive_1_2_2 jars to HDFS before kylin start > - > > Key: KYLIN-5647 > URL: https://issues.apache.org/jira/browse/KYLIN-5647 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Affects Versions: 5.0-alpha >Reporter: Zhiting Guo >Assignee: Zhiting Guo >Priority: Major > Fix For: 5.0-beta > > Attachments: Auto upload hive jars.pdf > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KYLIN-5648) Add log for sparder init user
[
https://issues.apache.org/jira/browse/KYLIN-5648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoxiang Yu resolved KYLIN-5648.
-
Resolution: Fixed
> Add log for sparder init user
> -
>
> Key: KYLIN-5648
> URL: https://issues.apache.org/jira/browse/KYLIN-5648
> Project: Kylin
> Issue Type: Improvement
> Components: Others
>Affects Versions: 5.0-alpha
>Reporter: Zhiting Guo
>Assignee: Zhiting Guo
>Priority: Major
> Fix For: 5.0-beta
>
>
> The user name used when spark initialization needs to be explicitly printed
> in the log.
> *dev design*
> Add a line of log printing when spader starts
> {code:java}
> logInfo(s"sparder init
> user:${UserGroupInformation.getCurrentUser.getUserName}"){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KYLIN-5646) The build job reports an error at the step of detecting time partition columns in the Yarn Cluster mode
[ https://issues.apache.org/jira/browse/KYLIN-5646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu resolved KYLIN-5646. - Resolution: Fixed > The build job reports an error at the step of detecting time partition > columns in the Yarn Cluster mode > --- > > Key: KYLIN-5646 > URL: https://issues.apache.org/jira/browse/KYLIN-5646 > Project: Kylin > Issue Type: Bug > Components: Tools, Build and Test >Affects Versions: 5.0-alpha >Reporter: Zhiting Guo >Assignee: Zhiting Guo >Priority: Major > Fix For: 5.0-beta > > > When building Spark YARN-Cluster mode, when detecting incremental time > partition columns, initializing KylinConfig reports an error Didn't find > KYLIN_HOME or KYLIN_HOME > *Reproduce method* > Build the partition table model incrementally using Spark YARN_Cluster mode, > and set kylin.engine.check-partition-col-enabled=true (the default value is > true) > *Root Cause* > Modified the autoSetShufflePartitions of the pushdown query in [KYLIN-5571], > no need to execute when the pre-modification build task detects the delta > time column format (only the pushdown query is executed) > After modification, autoSetShufflePartitions is executed asynchronously, the > following two methods will get KylinConfig through > KylinConfig.getInstanceFromEnv, > At this time, the asynchronous execution of the new thread cannot use the > built KylinConfig, so the KylinConfig will be initialized, > However, the build task jvm and the KE main process are not the same machine, > and KYLIN_CONF and KYLIN_HOME cannot be obtained, so the build task fails to > run > * ResourceDetectUtils.getResourceSizeWithTimeoutByConcurrency > * ResourceDetectUtils.getResourceSizBySerial > *fix design* > In all the logic of newly opened threads, if KylinConfig is used, this method > KylinConfig.getInstanceFromEnv() is not used. Unified is obtained by an > external thread and passed to the place where it needs to be used -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KYLIN-5645) add response params for model list api
[
https://issues.apache.org/jira/browse/KYLIN-5645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoxiang Yu resolved KYLIN-5645.
-
Resolution: Fixed
> add response params for model list api
> --
>
> Key: KYLIN-5645
> URL: https://issues.apache.org/jira/browse/KYLIN-5645
> Project: Kylin
> Issue Type: Bug
> Components: REST Service
>Affects Versions: 5.0-alpha
>Reporter: Zhiting Guo
>Assignee: Zhiting Guo
>Priority: Major
> Fix For: 5.0-beta
>
>
> For api GET /kylin/api/models, if set lite=false, the response will not
> contain partition_column_in_dims and empty_model, which are expected by
> frontend.
> *fix design*
> Regardless of the value of lite，add partition_column_in_dims and empty_model
> to the response.{*}{*}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KYLIN-5644) fix diag api security, encryption changed from base64 to AES
[ https://issues.apache.org/jira/browse/KYLIN-5644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu resolved KYLIN-5644. - Resolution: Fixed > fix diag api security, encryption changed from base64 to AES > > > Key: KYLIN-5644 > URL: https://issues.apache.org/jira/browse/KYLIN-5644 > Project: Kylin > Issue Type: Bug > Components: REST Service, Security >Affects Versions: 5.0-alpha >Reporter: Zhiting Guo >Assignee: Zhiting Guo >Priority: Major > Fix For: 5.0-beta > > > *dev design* > Continue to develop along the existing logic, adjust Base64 encryption to AES > encryption, encryption & decryption algorithm multiplexing has been > implemented as follows: > Encryption: org.apache.kylin.common.util.EncryptUtil#encrypt(String > strToEncrypt) > Decryption: org.apache.kylin.common.util.EncryptUtil#decrypt(String > strToDecrypt) > Because there will be special characters after AES encryption, such as: +, > when API parameters are passed, they will be recognized as spaces, resulting > in subsequent errors. > So here is the adjustment, the encryption algorithm is changed to: first > encrypt with EncryptUtil#encrypt and then encrypt twice with Base64, and the > decryption algorithm is the same: first decrypt with Base64 and then decrypt > twice with EncryptUtil#decrypt. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KYLIN-5643) Add public api for batch delete index
[ https://issues.apache.org/jira/browse/KYLIN-5643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu resolved KYLIN-5643. - Resolution: Fixed > Add public api for batch delete index > - > > Key: KYLIN-5643 > URL: https://issues.apache.org/jira/browse/KYLIN-5643 > Project: Kylin > Issue Type: Improvement > Components: Metadata >Affects Versions: 5.0-alpha >Reporter: Zhiting Guo >Assignee: Zhiting Guo >Priority: Major > Fix For: 5.0-beta > > > Move the "kylin/api/index_plans/index" DELETE API from NIndexPlanController > to OpenIndexPlanController as a public api. > demo: > curl --location --request DELETE > 'http://127.0.0.1:9099/kylin/api/index_plans/index?project=project1_name=abc_ids=201' > \ --header 'Accept: application/vnd.apache.kylin-v4-public+json' \ --header > 'Authorization: Basic YWRtaW46S1lMSU4=' > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KYLIN-5642) Align the default value of the parameter kylin.metadata.audit-log.max-size with the product manual
[ https://issues.apache.org/jira/browse/KYLIN-5642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu resolved KYLIN-5642. - Resolution: Fixed > Align the default value of the parameter kylin.metadata.audit-log.max-size > with the product manual > -- > > Key: KYLIN-5642 > URL: https://issues.apache.org/jira/browse/KYLIN-5642 > Project: Kylin > Issue Type: Improvement > Components: Documentation, Metadata >Affects Versions: 5.0-alpha >Reporter: Zhiting Guo >Assignee: Zhiting Guo >Priority: Major > Fix For: 5.0-beta > > > Set the default value as 50 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KYLIN-5641) fix set spark conf in serverless mod
[ https://issues.apache.org/jira/browse/KYLIN-5641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu resolved KYLIN-5641. - Resolution: Fixed > fix set spark conf in serverless mod > > > Key: KYLIN-5641 > URL: https://issues.apache.org/jira/browse/KYLIN-5641 > Project: Kylin > Issue Type: Bug > Components: Tools, Build and Test >Affects Versions: 5.0-alpha >Reporter: Zhiting Guo >Assignee: Zhiting Guo >Priority: Major > Fix For: 5.0-beta > > > In serverless mode, it will cause a NoSuchMethodException when build a model. > To fix it, just remove the set of > spark.sql.sources.repartitionWritingDataSource -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KYLIN-5270) Contribute eBay's authentication mechanism
[ https://issues.apache.org/jira/browse/KYLIN-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu updated KYLIN-5270: Fix Version/s: (was: 5.0-beta) > Contribute eBay's authentication mechanism > -- > > Key: KYLIN-5270 > URL: https://issues.apache.org/jira/browse/KYLIN-5270 > Project: Kylin > Issue Type: Improvement > Components: Others >Affects Versions: 5.0-alpha >Reporter: Li Can >Priority: Minor > > Contribute eBay's authentication mechanism from DI team -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KYLIN-5684) Optimize the copywriting tips that will clear the relevant indexes after modifying the expression of the computed column
[ https://issues.apache.org/jira/browse/KYLIN-5684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu updated KYLIN-5684: Fix Version/s: (was: 5.0-beta) > Optimize the copywriting tips that will clear the relevant indexes after > modifying the expression of the computed column > > > Key: KYLIN-5684 > URL: https://issues.apache.org/jira/browse/KYLIN-5684 > Project: Kylin > Issue Type: Bug >Reporter: Laura Xia >Assignee: Laura Xia >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KYLIN-5463) How to estimate the size of the cuboid for the KYLIN5.
[ https://issues.apache.org/jira/browse/KYLIN-5463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu updated KYLIN-5463: Fix Version/s: (was: 5.0-beta) > How to estimate the size of the cuboid for the KYLIN5. > -- > > Key: KYLIN-5463 > URL: https://issues.apache.org/jira/browse/KYLIN-5463 > Project: Kylin > Issue Type: Sub-task >Reporter: Kun Liu >Assignee: Kun Liu >Priority: Minor > > Currently, we can't estimate the row size for the cuboid, can use the row > count as the weight temporarily -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KYLIN-5683) There is a pagination problem when table data is in integer columns
[ https://issues.apache.org/jira/browse/KYLIN-5683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu updated KYLIN-5683: Fix Version/s: (was: 5.0-beta) > There is a pagination problem when table data is in integer columns > --- > > Key: KYLIN-5683 > URL: https://issues.apache.org/jira/browse/KYLIN-5683 > Project: Kylin > Issue Type: Bug >Reporter: Laura Xia >Assignee: Laura Xia >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KYLIN-5682) ER graph optimizes performance
[ https://issues.apache.org/jira/browse/KYLIN-5682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu updated KYLIN-5682: Fix Version/s: (was: 5.0-beta) > ER graph optimizes performance > -- > > Key: KYLIN-5682 > URL: https://issues.apache.org/jira/browse/KYLIN-5682 > Project: Kylin > Issue Type: Improvement >Reporter: Laura Xia >Assignee: Laura Xia >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KYLIN-5638) kylin create spark history dir auto in SparkApplication
[ https://issues.apache.org/jira/browse/KYLIN-5638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu resolved KYLIN-5638. - Resolution: Fixed > kylin create spark history dir auto in SparkApplication > --- > > Key: KYLIN-5638 > URL: https://issues.apache.org/jira/browse/KYLIN-5638 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Affects Versions: 5.0-alpha >Reporter: Zhiting Guo >Assignee: Zhiting Guo >Priority: Major > Fix For: 5.0-beta > > > *dev design* > Before creating a spark session when building a job, check the configuration > of the event log directory and find the directory. If the directory does not > exist, create it. This will prevent different spark history directories from > being configured for different projects. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KYLIN-5640) Support to automatically adjust the Bloom Filter based on data distribution
[
https://issues.apache.org/jira/browse/KYLIN-5640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoxiang Yu resolved KYLIN-5640.
-
Resolution: Fixed
> Support to automatically adjust the Bloom Filter based on data distribution
> ---
>
> Key: KYLIN-5640
> URL: https://issues.apache.org/jira/browse/KYLIN-5640
> Project: Kylin
> Issue Type: Improvement
> Components: Query Engine
>Affects Versions: 5.0-alpha
>Reporter: Zhiting Guo
>Assignee: Zhiting Guo
>Priority: Major
> Fix For: 5.0-beta
>
>
> h3. Why are the changes needed?
> Now the usage of bloom filter is to specify the NDV(number of distinct
> values), and then build BloomFilter. In general scenarios, it is actually not
> sure how much the distinct value is.
> If BloomFilter can be automatically generated according to the data, the file
> size can be reduced and the reading efficiency can also be improved.
> h3. What changes were proposed in this pull request?
> {{DynamicBlockBloomFilter}} contains multiple {{BlockSplitBloomFilter}} as
> candidates and inserts values in the candidates at the same time. Use the
> largest bloom filter as an approximate deduplication counter, and then remove
> incapable bloom filter candidates during data insertion.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KYLIN-5639) Refine kylin-it dependency
[ https://issues.apache.org/jira/browse/KYLIN-5639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu resolved KYLIN-5639. - Resolution: Fixed > Refine kylin-it dependency > -- > > Key: KYLIN-5639 > URL: https://issues.apache.org/jira/browse/KYLIN-5639 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Affects Versions: 5.0-alpha >Reporter: Zhiting Guo >Assignee: Zhiting Guo >Priority: Major > Fix For: 5.0-beta > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KYLIN-5636) automatically clean up dependent files after the build task
[ https://issues.apache.org/jira/browse/KYLIN-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu resolved KYLIN-5636. - Resolution: Fixed > automatically clean up dependent files after the build task > --- > > Key: KYLIN-5636 > URL: https://issues.apache.org/jira/browse/KYLIN-5636 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Affects Versions: 5.0-alpha >Reporter: Zhiting Guo >Assignee: Zhiting Guo >Priority: Major > Fix For: 5.0-beta > > > *question:* > The files uploaded under the path spark.kubernetes.file.upload.path are not > automatically deleted > 1: When spark creates a driverPod, it uploads dependencies to the specified > path. The build task is in cluster mode and needs to create a driverPod. > Running the build task multiple times results in a large path file. > 2: At present, the upload.path path we configured (s3a://kylin/spark-on-k8s) > is a fixed path, and spark will create a subdirectory in this directory, the > spark-upload-uuid directory, and then store the dependencies in it. > *dev design* > Core idea, add dynamic subdirectory under the original upload.path path, > delete the entire subdirectory when the task is over > Build task: upload.path + jobId (e.g. s3a://kylin/spark-on-k8s/uuid) > Delete the dependency directory when the build task is finished > > Automatically delete dependent function is called, kill-9 situation will lead > to the deletion function is not called, garbage cleaning function needs to be > added to the bottom of the policy, such as greater than three months before > the directory is automatically deleted -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KYLIN-5637) minor fix get delta table ddl
[ https://issues.apache.org/jira/browse/KYLIN-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu resolved KYLIN-5637. - Resolution: Fixed > minor fix get delta table ddl > - > > Key: KYLIN-5637 > URL: https://issues.apache.org/jira/browse/KYLIN-5637 > Project: Kylin > Issue Type: Improvement > Components: RDBMS Source >Affects Versions: 5.0-alpha >Reporter: Zhiting Guo >Assignee: Zhiting Guo >Priority: Major > Fix For: 5.0-beta > > > The delta data source does not support operations such as msck partition, > show create table, etc. Need to do some processing -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KYLIN-5635) Adapt for delta table
[ https://issues.apache.org/jira/browse/KYLIN-5635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu resolved KYLIN-5635. - Fix Version/s: 5.0-beta Resolution: Fixed > Adapt for delta table > - > > Key: KYLIN-5635 > URL: https://issues.apache.org/jira/browse/KYLIN-5635 > Project: Kylin > Issue Type: Improvement > Components: RDBMS Source >Affects Versions: 5.0-alpha >Reporter: Zhiting Guo >Assignee: Zhiting Guo >Priority: Major > Fix For: 5.0-beta > > > When obtaining delta column information, because the column information is > not saved in the catalog (due to the use of DeltaCatalog), it cannot be > obtained directly from the catalog. We first determine whether the table is a > delta table. The Delta SDK provides a function. If so, read the table through > spark.table to get the schema. Here, spark scans Metadata to get the schema > information under the back path > > Also due to the use of DeltaCatalog, delta table does not support the show > create table statement, this is because deltaCatalog does some checks, does > not support this SQL , here by judging whether it is delta in advance, if it > is directly through location and table spell a ddl return. > > Limit: The partition column is not processed here, so the partition column is > not recognized, and delta does not manage its own partition through the > catalog. It is obtained in real time by scanning the Metadata under the > confidant path, so it does not affect the reading of data. The only place > that has an impact is the function of snapshot partition construction. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KYLIN-5634) Support query executor expansion and contraction
[ https://issues.apache.org/jira/browse/KYLIN-5634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu resolved KYLIN-5634. - Fix Version/s: 5.0-beta Resolution: Fixed > Support query executor expansion and contraction > > > Key: KYLIN-5634 > URL: https://issues.apache.org/jira/browse/KYLIN-5634 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Affects Versions: 5.0-alpha >Reporter: Zhiting Guo >Assignee: Zhiting Guo >Priority: Major > Fix For: 5.0-beta > > Attachments: Support query executor expansion and contraction.pdf > > > Details are included in the attachment. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KYLIN-5633) The query can answered with the existing data of the current model (accepting index or segment data is not uniform), and there should be no query failure or push-down
[ https://issues.apache.org/jira/browse/KYLIN-5633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu resolved KYLIN-5633. - Resolution: Fixed > The query can answered with the existing data of the current model (accepting > index or segment data is not uniform), and there should be no query failure > or push-down > -- > > Key: KYLIN-5633 > URL: https://issues.apache.org/jira/browse/KYLIN-5633 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Affects Versions: 5.0-alpha >Reporter: Zhiting Guo >Assignee: Zhiting Guo >Priority: Major > Fix For: 5.0-beta > > Attachments: Segment heterogeneous query behavior (1).pdf > > > Details are included in the attachment. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KYLIN-5634) Support query executor expansion and contraction
[ https://issues.apache.org/jira/browse/KYLIN-5634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu updated KYLIN-5634: Fix Version/s: (was: 5.0-beta) > Support query executor expansion and contraction > > > Key: KYLIN-5634 > URL: https://issues.apache.org/jira/browse/KYLIN-5634 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Affects Versions: 5.0-alpha >Reporter: Zhiting Guo >Assignee: Zhiting Guo >Priority: Major > Attachments: Support query executor expansion and contraction.pdf > > > Details are included in the attachment. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KYLIN-5635) Adapt for delta table
[ https://issues.apache.org/jira/browse/KYLIN-5635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu updated KYLIN-5635: Fix Version/s: (was: 5.0-beta) > Adapt for delta table > - > > Key: KYLIN-5635 > URL: https://issues.apache.org/jira/browse/KYLIN-5635 > Project: Kylin > Issue Type: Improvement > Components: RDBMS Source >Affects Versions: 5.0-alpha >Reporter: Zhiting Guo >Assignee: Zhiting Guo >Priority: Major > > When obtaining delta column information, because the column information is > not saved in the catalog (due to the use of DeltaCatalog), it cannot be > obtained directly from the catalog. We first determine whether the table is a > delta table. The Delta SDK provides a function. If so, read the table through > spark.table to get the schema. Here, spark scans Metadata to get the schema > information under the back path > > Also due to the use of DeltaCatalog, delta table does not support the show > create table statement, this is because deltaCatalog does some checks, does > not support this SQL , here by judging whether it is delta in advance, if it > is directly through location and table spell a ddl return. > > Limit: The partition column is not processed here, so the partition column is > not recognized, and delta does not manage its own partition through the > catalog. It is obtained in real time by scanning the Metadata under the > confidant path, so it does not affect the reading of data. The only place > that has an impact is the function of snapshot partition construction. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KYLIN-5533) The model list page loads slowly when selecting 100 items per page
[
https://issues.apache.org/jira/browse/KYLIN-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoxiang Yu resolved KYLIN-5533.
-
Resolution: Fixed
> The model list page loads slowly when selecting 100 items per page
> --
>
> Key: KYLIN-5533
> URL: https://issues.apache.org/jira/browse/KYLIN-5533
> Project: Kylin
> Issue Type: Bug
>Affects Versions: 5.0-alpha
>Reporter: Yaguang Jia
>Assignee: Yaguang Jia
>Priority: Major
> Fix For: 5.0-beta
>
>
> Backend Dev design:
> Based on the original API kylin/api/model, a new parameter lite is added to
> do branch control, while true returns lightweight data, which defaults to
> false and does not change the original behavior.
> Expected response, deleted
> segments,selected_columns,simplified_dimensions,all_named_columns,simplified_tables,all_measures,simplified_measures
> field
> The new Boolean value empty _ model indicates whether there are no other
> dimensions and metrics except the count_all metric. Determine success as
> true, default false,partition_column_in_dims indicates whether the time
> partition column is in the dimension, judge success as true, and default
> false.
>
> 在原有接口kylin/api/model基础上新增参数lite做分支控制，true则返回轻量级数据，默认false，不改变原有行为。
> 预期response，删除了segments，selected_columns，simplified_dimensions，all_named_columns，simplified_tables，all_measures，simplified_measures
> 字段
> 新增boolean值{{{}empty_model{}}}表示是否除了count_all度量外没有其他的维度和度量
> ，判断成功为true，默认false，{{{}partition_column_in_dims{}}}
> 表示时间分区列是否在维度中，判断成功为true，默认false。
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KYLIN-5536) Kylin query optimization, by limiting the data range of max query, improve query efficiency
[
https://issues.apache.org/jira/browse/KYLIN-5536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoxiang Yu resolved KYLIN-5536.
-
Resolution: Fixed
> Kylin query optimization, by limiting the data range of max query, improve
> query efficiency
> ---
>
> Key: KYLIN-5536
> URL: https://issues.apache.org/jira/browse/KYLIN-5536
> Project: Kylin
> Issue Type: Improvement
> Components: Query Engine
>Affects Versions: 5.0-alpha
>Reporter: Yaguang Jia
>Assignee: Yaguang Jia
>Priority: Major
> Fix For: 5.0-beta
>
>
> h2. Dev design
> 1、Add configuration kylin.query.max-measure-segment-pruner-before-days
> Limit the time range of the query. The default value is -1, which is
> equivalent to turning off this optimization. When configured to 0, no data is
> scanned. When the configuration parameter is incorrect (e.g. 0.1), the effect
> is to not turn on the switch. Includes three levels: model, project, and
> system, in decreasing order of priority.
> 2、Where will the optimization be done?
> segment pruner at:
> org.apache.kylin.query.routing.RealizationPruner#pruneSegments
> 3、What kind of queries will be optimized?
> select from T [where xxx]
> The query must be max(time partitioned column; where condition is optional;
> no group by column
> 4、When configuration parameters are specified, which segment is selected to
> answer the query?
> From the last (new) segment, the segment is selected according to the
> configuration time.
> h3. dev design
> h4. 1、新增配置 {{kylin.query.max-measure-segment-pruner-before-days}}
> 用于限定查询时的时间范围。默认值为-1，相当于关闭此优化。当配置为0时，不扫描数据。当配置参数不对（比如0.1）时，效果为不打开开关。包括模型、项目、系统三个级别，优先级依次降低。
> *2、将优化做在哪？*
> segment pruner处：org.apache.kylin.query.routing.RealizationPruner#pruneSegments
> *3、什么样的查询会被优化？*
> select from T [where xxx]
> 查询必须是max(时间分区列；where 条件可有可无；不能有group by 列
> *4、当指定了配置参数时，选择哪些segment来回答查询？*
> 从最后（新）一个segment起，按照配置时间选择segment。
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KYLIN-5632) Optimize and clean up some useless code in the query
[ https://issues.apache.org/jira/browse/KYLIN-5632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu resolved KYLIN-5632. - Resolution: Fixed > Optimize and clean up some useless code in the query > > > Key: KYLIN-5632 > URL: https://issues.apache.org/jira/browse/KYLIN-5632 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Affects Versions: 5.0-alpha >Reporter: Zhiting Guo >Assignee: Zhiting Guo >Priority: Major > Fix For: 5.0-beta > > > this issue will includes: > 1. Refactoring the selection logic of realizations > 2. Rename, move package or drop some useless class > 3. fix some unstable ut, add some ignored ut, move some ut to the module of > kylin-it > 4. move index matchers to ChooserContext > 5. Move candidate sorting method to the QueryRouter -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KYLIN-5537) High-risk vulnerability: maven:com.fasterxml.woodstox:woodstox-core@5.3.0/CVE-2022-40152
[ https://issues.apache.org/jira/browse/KYLIN-5537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu resolved KYLIN-5537. - Resolution: Fixed > High-risk vulnerability: > maven:com.fasterxml.woodstox:woodstox-core@5.3.0/CVE-2022-40152 > > > Key: KYLIN-5537 > URL: https://issues.apache.org/jira/browse/KYLIN-5537 > Project: Kylin > Issue Type: Bug >Affects Versions: 5.0-alpha >Reporter: Yaguang Jia >Priority: Major > Fix For: 5.0-beta > > > maven:com.fasterxml.woodstox:woodstox-core@5.3.0 > [CVE-2022-40152|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2022-40152] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KYLIN-5532) Complex case when queries get stuck in the query parsing phase, resulting in service exceptions, high memory and the queries cannot be canceled
[ https://issues.apache.org/jira/browse/KYLIN-5532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu resolved KYLIN-5532. - Resolution: Fixed > Complex case when queries get stuck in the query parsing phase, resulting in > service exceptions, high memory and the queries cannot be canceled > --- > > Key: KYLIN-5532 > URL: https://issues.apache.org/jira/browse/KYLIN-5532 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: 5.0-alpha >Reporter: Yaguang Jia >Assignee: Yaguang Jia >Priority: Major > Fix For: 5.0-beta > > > Fix: Unable to cancel calcite parsing stage queries -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KYLIN-5531) [Job Engine] simplify FlatTableAndDictBase and SegmentFlatTable to remove redundant code
[ https://issues.apache.org/jira/browse/KYLIN-5531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu resolved KYLIN-5531. - Resolution: Fixed > [Job Engine] simplify FlatTableAndDictBase and SegmentFlatTable to remove > redundant code > > > Key: KYLIN-5531 > URL: https://issues.apache.org/jira/browse/KYLIN-5531 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: 5.0-alpha >Reporter: Yaguang Jia >Assignee: Yaguang Jia >Priority: Major > Fix For: 5.0-beta > > > FlatTableAndDictBase taken from SegmentFlatTable, contains a lot of duplicate > code, need to be streamlined -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KYLIN-5530) Build Performance Optimization
[ https://issues.apache.org/jira/browse/KYLIN-5530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu resolved KYLIN-5530. - Resolution: Fixed > Build Performance Optimization > -- > > Key: KYLIN-5530 > URL: https://issues.apache.org/jira/browse/KYLIN-5530 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Affects Versions: 5.0-alpha >Reporter: Yaguang Jia >Assignee: Yaguang Jia >Priority: Major > Fix For: 5.0-beta > > Attachments: (Chinese) KYLIN-5530 Build Performance Optimization.pdf, > (English) KYLIN-5530 Build Performance Optimization.pdf > > > 1. remove the repartitionWriter method for building indexes > Background: repartition this behavior on the cloud due to the read and write > IO problems of object storage, the implementation costs are too high, which > brings more significant problems. > The current index construction needs to write index data to temp directory > first, and then read and repartition into new data files for storage. This > method of wasting a lot of IO needs to be removed and modified to directly > repartition write into the final index file, transforming spark's > repartition, which needs to achieve the following goals: > - Solve the scenario of skew > - solve the problem of a large number of small files > 2. When building a Flat Table, the dimension table directly reads the > Snapshot file > The reasons are as follows: > - If the dimension table is a view, the view will be calculated once when > building a snapshot, and once when building a flat table, so once building a > dimension table view, it will be calculated twice. > - There are uncertainties in the data format of the source data, etc. > Optimization direction: When building a flat table, the dimension table does > not read from the source data, but directly reads the Snapshot file data > > --- > > 1. 去除构建索引的repartitionWriter方法 > 背景：repartition这个行为在云上由于对象存储的读写IO问题，实现成本太高，带来的问题就比较显著。 > 当前索引的构建需要先将索引数据写到temp目录，再读取之后repartition成新的数据文件存储。需要去除这种浪费大量IO的方法，修改为直接repartition写成最终的索引文件，改造spark的repartition，需要达成以下目标： > - 解决skew的场景 > - 解决大量小文件的问题 > > 2. 构建Flat Table时维表直接读取Snapshot的文件 > 原因如下： > - 如果维表为view，构建snapshot时会计算一次view，构建Flat Table时会计算一次，所以一次构建维表view会计算两次。 > - 源数据的数据格式等存在不确定性 > 优化方向：构建平表时，维表不从源数据读取，直接读取Snapshot文件数据 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KYLIN-5526) After setting the upper limit of concurrent asynchronous queries, submitting multiple asynchronous queries at the same time may cause the number of asynchronous queries
[ https://issues.apache.org/jira/browse/KYLIN-5526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu resolved KYLIN-5526. - Resolution: Fixed > After setting the upper limit of concurrent asynchronous queries, submitting > multiple asynchronous queries at the same time may cause the number of > asynchronous queries to be greater than the set upper limit > > > Key: KYLIN-5526 > URL: https://issues.apache.org/jira/browse/KYLIN-5526 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: 5.0-alpha >Reporter: Yaguang Jia >Assignee: Yaguang Jia >Priority: Major > Fix For: 5.0-beta > > > When encountering a large number of concurrent asynchronous query requests, > the number of asynchronous query jobs will exceed the parameter limit, which > needs to be fixed > > 在遇到并发异步查询请求大的时候，异步查询job的数目会大于参数限制，需要修复 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KYLIN-5529) Support adding base indexes by selecting "Base Aggregate Index" or "Base Detail Index".
[ https://issues.apache.org/jira/browse/KYLIN-5529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu resolved KYLIN-5529. - Resolution: Fixed > Support adding base indexes by selecting "Base Aggregate Index" or "Base > Detail Index". > --- > > Key: KYLIN-5529 > URL: https://issues.apache.org/jira/browse/KYLIN-5529 > Project: Kylin > Issue Type: New Feature > Components: Job Engine >Affects Versions: 5.0-alpha >Reporter: Yaguang Jia >Assignee: Yaguang Jia >Priority: Major > Fix For: 5.0-beta > > Attachments: (Chinese) KYLIN-5529 Support adding base indexes by > selecting Base Aggregate Index or Base Detail Index.pdf, (English) > KYLIN-5529 Support adding base indexes by selecting _Base Aggregate Index_ or > _Base Detail Index_..pdf > > > It is hoped that Basic Aggregated Index can be single-selected when adding a > basic index, and that both pages and APIs are supported. > > 希望在添加基础索引时可单选“基础聚合索引” ， 且同时支持页面和API的添加方式。 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KYLIN-5524) When there are more than 2 parameters of concat function in cc column, SQL modeling fails.
[
https://issues.apache.org/jira/browse/KYLIN-5524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoxiang Yu resolved KYLIN-5524.
-
Resolution: Fixed
> When there are more than 2 parameters of concat function in cc column, SQL
> modeling fails.
> --
>
> Key: KYLIN-5524
> URL: https://issues.apache.org/jira/browse/KYLIN-5524
> Project: Kylin
> Issue Type: Bug
> Components: Modeling
>Affects Versions: 5.0-alpha
>Reporter: Yaguang Jia
>Assignee: Yaguang Jia
>Priority: Major
> Fix For: 5.0-beta
>
> Attachments: image-2023-04-25-14-48-51-840.png
>
>
> When there are more than 2 parameters of concat function listed in cc in SQL,
> SQL modeling fails and an error of \{{no match found for function signature}}
> is reported.
>
> 在 SQL 中 cc 列有concat函数参数大于 2 个的情况下，SQL 建模失败，报\{{no match found for function
> signature}}的错误
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KYLIN-5521) JoinsGraph optimization: Query SQL association order change causes the model to fail to hit
[ https://issues.apache.org/jira/browse/KYLIN-5521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu resolved KYLIN-5521. - Resolution: Fixed > JoinsGraph optimization: Query SQL association order change causes the model > to fail to hit > --- > > Key: KYLIN-5521 > URL: https://issues.apache.org/jira/browse/KYLIN-5521 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: 5.0-alpha >Reporter: Yaguang Jia >Assignee: Yaguang Jia >Priority: Major > Fix For: 5.0-beta > > Attachments: (Chinese)KYLIN-5521 JoinsGraph optimization_ Query SQL > association order change causes the model to fail to hit.pdf, > (English)KYLIN-5521 JoinsGraph optimization_ Query SQL association order > change causes the model to fail to hit.pdf > > > In the log, the following two query statements, only the order of the > association of the first 2 tables is changed (the actual logical relationship > has not changed), resulting in the query statement can not be hit by the same > index. When there are only 2 tables after from, the order of the replacement > table association can hit the same index. > > 日志中，以下两条查询语句，只是前面2张表的关联先后顺序换了一下（实际逻辑关系并未改变），导致查询语句不能被同一个索引击中。而当from后面只有2张表时，替换表关联的先后顺序是可以击中同一个索引的。 > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KYLIN-5523) Kylin5 supports using computable columns as Join Key and partition columns
[ https://issues.apache.org/jira/browse/KYLIN-5523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu resolved KYLIN-5523. - Resolution: Fixed > Kylin5 supports using computable columns as Join Key and partition columns > -- > > Key: KYLIN-5523 > URL: https://issues.apache.org/jira/browse/KYLIN-5523 > Project: Kylin > Issue Type: New Feature >Affects Versions: 5.0-alpha >Reporter: Yaguang Jia >Assignee: Yaguang Jia >Priority: Major > Fix For: 5.0-beta > > Attachments: (Chinese)Kylin5 supports using computable columns as > Join Key and partition columns.pdf, (English)Kylin5 supports using computable > columns as Join Key and partition columns (1).pdf > > > As a modeler, the formats of the associated fields on our different Hive > tables are not exactly the same, and we hope to convert one of them with the > help of computable columns, so as to facilitate modeling on KE. [Currently, > the Hive view processing method is used, but once it needs to be adjusted, > there are many places involved, which is inconvenient, and the production > environment does not allow too many views to be built] > > 作为一个建模人员，我们不同的Hive表上的 关联字段 > 格式不完全一样，希望借助可计算列将其中1个转换，从而方便的在KE上建模。【当前采用Hive视图处理的方法，但一旦需要调整涉及到的地方比较多，不方便，生产环境也不让建太多视图】 > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KYLIN-5404) Prefer to use common standard dependencies rather than self-maintained ones
[ https://issues.apache.org/jira/browse/KYLIN-5404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu updated KYLIN-5404: Fix Version/s: (was: 5.0-beta) > Prefer to use common standard dependencies rather than self-maintained ones > --- > > Key: KYLIN-5404 > URL: https://issues.apache.org/jira/browse/KYLIN-5404 > Project: Kylin > Issue Type: Task >Reporter: Zhong Yanghong >Priority: Major > > Better to use common standard dependencies rather than self-maintained ones: > - Spark > - Calcite > - Google guava > - Spring session -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KYLIN-5328) Query match join error
[
https://issues.apache.org/jira/browse/KYLIN-5328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoxiang Yu updated KYLIN-5328:
Fix Version/s: (was: 5.0-beta)
> Query match join error
> --
>
> Key: KYLIN-5328
> URL: https://issues.apache.org/jira/browse/KYLIN-5328
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: 5.0-alpha
>Reporter: pengfei.zhan
>Assignee: pengfei.zhan
>Priority: Major
>
> Setting `kylin.query.join-match-optimization-enabled=true`, we can use the
> left join model to answer the inner join query statement. But it encounters a
> bug.
> For example:
> The model is as follows,
> {code:sql}
> lineorder
> inner join customer on lineorder.lo_custkey = customer.c_custkey
> left join dates on lineorder.lo_orderdate = dates.d_datekey
> {code}
> The query statement is as follows can match this model,
> {code:sql}
> select lineorder.lo_linenumber
> from ssb.lineorder
> inner join ssb.customer on lineorder.lo_custkey = customer.c_custkey
> left join ssb.dates on lineorder.lo_orderdate = dates.d_datekey
> {code}
> However, the query statement is as follows cannot match this model,
> {code:sql}
> select lineorder.lo_linenumber
> from ssb.customer
> inner join ssb.lineorder as lineorder on lineorder.lo_custkey =
> customer.c_custkey
> left join ssb.dates as dates on lineorder.lo_orderdate = dates.d_datekey
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KYLIN-5464) Index Planner for Kylin 5.0
[ https://issues.apache.org/jira/browse/KYLIN-5464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu updated KYLIN-5464: Fix Version/s: (was: 5.0-beta) > Index Planner for Kylin 5.0 > --- > > Key: KYLIN-5464 > URL: https://issues.apache.org/jira/browse/KYLIN-5464 > Project: Kylin > Issue Type: New Feature > Components: Job Engine >Affects Versions: 5.0-alpha >Reporter: Xiaoxiang Yu >Assignee: Kun Liu >Priority: Major > > This is a umbrella issue for Index Planner(aka Cube Planner) for Kylin 5.0 > codebase. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KYLIN-5465) kylin5 Embedded Dashboard for Query and Job
[ https://issues.apache.org/jira/browse/KYLIN-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu resolved KYLIN-5465. - Resolution: Fixed > kylin5 Embedded Dashboard for Query and Job > --- > > Key: KYLIN-5465 > URL: https://issues.apache.org/jira/browse/KYLIN-5465 > Project: Kylin > Issue Type: New Feature > Components: Metrics >Reporter: Li Can >Assignee: Li Can >Priority: Major > Fix For: 5.0-beta > > Attachments: dashboard.png > > > 1.display job, cube statistics data. > 2.there have two query tables, if the query sql is error, then realization > will not store the data, so we use the realization table to display > statistics, because the error query will not include latency results. > 3.base on quote 2, we add some columns to realization table to collect data -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KYLIN-5388) Add a rest api to support to update database name and table name without rebuilding data
[
https://issues.apache.org/jira/browse/KYLIN-5388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoxiang Yu updated KYLIN-5388:
Fix Version/s: (was: 5.0-beta)
> Add a rest api to support to update database name and table name without
> rebuilding data
>
>
> Key: KYLIN-5388
> URL: https://issues.apache.org/jira/browse/KYLIN-5388
> Project: Kylin
> Issue Type: Improvement
> Components: Metadata
>Affects Versions: 5.0-alpha
>Reporter: Zhenning Zhang
>Assignee: Zhenning Zhang
>Priority: Major
>
> 1) add root table alias for model in kylin5.0
> 2) add a table mapping updater
> influenced metadata：
> {code:java}
> - table
> - database name
> - table name
> - model
> - root fact table
> - join table
> - computed column
> - dataflow
> - segments
> - snapshot
> - column source bytes{code}
>
> 3) add a api for table name update request
> here is an example:
> {code:java}
> curl -kXPOST 'http://localhost:7070/kylin/api/tables/default/update' -H
> 'Authorization: Basic XX' \
> -H 'Content-Type: application/json' \
> -d '{ "mapping":{ "DEFAULT.KYLIN_SALES":
> { "database": "TEST", "tableName": "KYLIN_FACT" }
> , "DEFAULT.KYLIN_CAL_DT":
> { "tableName": "CAL_DT" }
> , "DEFAULT.KYLIN_CATEGORY_GROUPINGS":
> { "database": "TEST" }
> }, "isUseExisting":true}'
> {code}
> 4) add UT and json files for test.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KYLIN-5302) Update Kylin related dependencies to community stable version
[ https://issues.apache.org/jira/browse/KYLIN-5302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu updated KYLIN-5302: Fix Version/s: (was: 5.0-beta) > Update Kylin related dependencies to community stable version > - > > Key: KYLIN-5302 > URL: https://issues.apache.org/jira/browse/KYLIN-5302 > Project: Kylin > Issue Type: Improvement > Components: Others >Affects Versions: 5.0-alpha >Reporter: mukvin >Assignee: mukvin >Priority: Major > > Kylin needs to update some dependencies version to the community version. > Such as Curator, Spring Session. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KYLIN-5303) Customize static resource for kylin
[ https://issues.apache.org/jira/browse/KYLIN-5303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu updated KYLIN-5303: Fix Version/s: (was: 5.0-beta) > Customize static resource for kylin > --- > > Key: KYLIN-5303 > URL: https://issues.apache.org/jira/browse/KYLIN-5303 > Project: Kylin > Issue Type: Improvement > Components: Web >Affects Versions: 5.0-alpha >Reporter: mukvin >Assignee: mukvin >Priority: Major > > The user wants the Kylin web URL suffix can be customized. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KYLIN-5700) Command line injection vulnerability when generating diagnostic packages via scripts
[
https://issues.apache.org/jira/browse/KYLIN-5700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoxiang Yu updated KYLIN-5700:
Fix Version/s: (was: 5.0-beta)
> Command line injection vulnerability when generating diagnostic packages via
> scripts
>
>
> Key: KYLIN-5700
> URL: https://issues.apache.org/jira/browse/KYLIN-5700
> Project: Kylin
> Issue Type: Bug
> Components: Tools, Build and Test
>Affects Versions: 5.0-alpha
>Reporter: Yaguang Jia
>Assignee: Yaguang Jia
>Priority: Critical
>
> h2. Background
> In the current code, there are many scenarios where a cmd needs to be spliced
> and then executed by ProcessBuilder. The parameters of the spliced cmd may
> come from the interface, and there is a lack of parameter legitimacy
> checking, which may be vulnerable to malicious attacks.
> When splicing spark commands, the {{checkCommandInjection}} method is used to
> avoid injection attacks, but it only avoids injection attacks caused by
> backquotes and $(), such as {{{}`rm -rf /` $(rm -rf /){}}}, but not other
> scenarios, such as {{cat nohup.out2 && echo success || echo failed echo
> failed}}
> h2. Fix Design
> Parameter checking when splicing cmd commands, including the following four
> scenarios:
> 1. diagnostic package, it will splice the parameters of the diag.sh script,
> such as project, jobId, path, etc. It will check each parameter in turn, and
> if it matches{{ ^[a-zA-Z0-9_. /-]+$ }}is enough
> 2. When exporting influxDB data, it will splice the database address and
> database name as the parameter of influx command, the former meets{{
> [a-zA-Z0-9._-](:[0-9])?}} and {{^[0-9a-zA-Z_-]+$}} for the latter.
> 3. When fetching yarn's stats, the url of yarn is spliced as an argument to
> the curl command, conforming to{{ ^(http(s)? ://)? [a-zA-Z0-9._-](:[0-9])?
> (/[a-zA-Z0-9._-]+)*/? $}} That's it.
> 4. When executing the beeline command, the beeline-params in the
> configuration will be spliced into the command. The composition of the
> beeline-params is more complicated, forcing each parameter value to be
> converted to a string by wrapping it with ', such as {{abc → 'abc', ab'c →
> 'ab'\''c'}}
> h2. Background
> 在当前代码中，有众多场景需要拼接出一条 cmd, 然后通过 {{ProcessBuilder}}
> 来执行，拼接cmd的参数可能会来自接口，并且缺少参数合法性的检验，有被恶意攻击的可能。
> 当拼接 spark 命令时，使用了 {{checkCommandInjection}} 方法来避免注入攻击，但是该方法仅规避了 反引号 和 $()
> 导致的注入攻击，如 {{`rm -rf /`}} {{{}$(rm -rf /){}}}，无法规避其他场景，如 {{cat nohup.out2 &&
> echo success || echo failed}}
> h2. Fix Design
> 在拼接cmd命令时对参数进行检查，包括以下四种场景：
> # 打诊断包时，会拼接 diag.sh 脚本的参数，如项目、jobId、路径等，依次检查每一个参数，符合 {{^[a-zA-Z0-9_./-]+$}}
> 即可
> # 导出influxDB 数据时，会在命令里拼接 *数据库地址* 以及 {*}数据库名称{*}作为 influx命令的参数，前者符合
> {{[a-zA-Z0-9._-]{+}(:[0-9]{+})?}} 即可，后者符合{{{}^[0-9a-zA-Z_-]+${}}} 即可
> # 获取yarn的统计指标时，会拼接yarn 的url地址作为 curl 命令的参数，符合
> {{^(http(s)?://)?[a-zA-Z0-9._-]{+}(:[0-9]{+})?(/[a-zA-Z0-9._-]+)*/?$}} 即可
> # 执行 beeline 命令时，会将配置中的 beeline-params
> 拼接到命令中，beeline-params的构成较为复杂，强制将每一个参数值使用{{{}'{}}}包起来转为字符串，如 abc → ‘abc'，ab’c
> → ‘ab’\''c'
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KYLIN-5699) Support special character for bitmap measure column & computed column
[
https://issues.apache.org/jira/browse/KYLIN-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoxiang Yu updated KYLIN-5699:
Fix Version/s: (was: 5.0-beta)
> Support special character for bitmap measure column & computed column
> -
>
> Key: KYLIN-5699
> URL: https://issues.apache.org/jira/browse/KYLIN-5699
> Project: Kylin
> Issue Type: Improvement
> Components: Modeling
>Affects Versions: 5.0-alpha
>Reporter: Yaguang Jia
>Assignee: Yaguang Jia
>Priority: Critical
>
> *Root Cause*
> The Spark org.apache.spark.sql.functions.expr function cannot recognize
> special characters such as Chinese characters and needs to be enclosed in
> backticks.
> *Dev Design*
> Identify the places where expr is called and add compatibility handling for
> special characters:
> Build the computable column
> {code:java}
> FlatTableAndDictBase#concatCCs{code}
> Build dictionary
> {code:java}
> DFBuilderHelper#selectColumnsInTable, DFBuilderHelper#chooseSuitableCols{code}
> Build dictionary
> {code:java}
> DFDictionaryBuilder#wrapCol{code}
>
>
>
> *Root Cause*
> Spark {{org.apache.spark.sql.functions.expr}} 函数不能识别中文等特殊字符，需要用反引号包裹。
> *Dev Design*
> 找出 expr 调用处，增加对特殊字符的兼容处理：
> # 构建可计算列 {{FlatTableAndDictBase#concatCCs}}
> # 构建字典
> {{{}DFBuilderHelper#selectColumnsInTable{}}}、{{{}DFBuilderHelper#chooseSuitableCols{}}}
> # 构建字典 {{DFDictionaryBuilder#wrapCol}}
> {{}}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KYLIN-5690) Optimize the Kylin read metadata for maxId
[ https://issues.apache.org/jira/browse/KYLIN-5690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu resolved KYLIN-5690. - Resolution: Fixed > Optimize the Kylin read metadata for maxId > -- > > Key: KYLIN-5690 > URL: https://issues.apache.org/jira/browse/KYLIN-5690 > Project: Kylin > Issue Type: Improvement > Components: Metadata >Affects Versions: 5.0-alpha >Reporter: Yaguang Jia >Assignee: Yaguang Jia >Priority: Critical > Fix For: 5.0-beta > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KYLIN-5693) Reduce the number of times Spark reads Parquet Footer to improve query performance
[ https://issues.apache.org/jira/browse/KYLIN-5693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu updated KYLIN-5693: Fix Version/s: (was: 5.0-beta) > Reduce the number of times Spark reads Parquet Footer to improve query > performance > -- > > Key: KYLIN-5693 > URL: https://issues.apache.org/jira/browse/KYLIN-5693 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Affects Versions: 5.0-alpha >Reporter: Yaguang Jia >Assignee: Yaguang Jia >Priority: Critical > > h2. Dev Design > Parquet footer metadata is now always read twice in vectorized parquet reader. > When the NameNode is under high pressure, it will cost time to read twice. > Actually we can avoid reading the footer twice by reading all row groups in > advance and filter row groups according to filters that require push down (no > need to read the footer metadata again the second time). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KYLIN-5528) Add Sparder (Query Spark) Executor's GC information to the diagnostic package
[ https://issues.apache.org/jira/browse/KYLIN-5528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu resolved KYLIN-5528. - Resolution: Fixed > Add Sparder (Query Spark) Executor's GC information to the diagnostic package > - > > Key: KYLIN-5528 > URL: https://issues.apache.org/jira/browse/KYLIN-5528 > Project: Kylin > Issue Type: Improvement > Components: Spark Engine, Tools, Build and Test >Affects Versions: 5.0-alpha >Reporter: Yaguang Jia >Assignee: Yaguang Jia >Priority: Critical > Fix For: 5.0-beta > > Attachments: (Chinese) KYLIN-5528 Add Sparder (Query Spark) > Executor's GC information to the diagnostic package.pdf, (English) KYLIN-5528 > Add Sparder (Query Spark) Executor's GC information to the diagnostic > package.pdf > > > Add gc information on Executor side to facilitate query performance diagnosis > > 添加Executor端的gc信息，便于查询性能诊断 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KYLIN-5525) When the excluded column is enabled and the query hits multiple identical models at the same time, the query may push down
[ https://issues.apache.org/jira/browse/KYLIN-5525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu resolved KYLIN-5525. - Resolution: Fixed > When the excluded column is enabled and the query hits multiple identical > models at the same time, the query may push down > -- > > Key: KYLIN-5525 > URL: https://issues.apache.org/jira/browse/KYLIN-5525 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: 5.0-alpha >Reporter: Yaguang Jia >Assignee: Yaguang Jia >Priority: Critical > Fix For: 5.0-beta > > Attachments: image-2023-04-25-15-15-41-563.png > > > When there is only a table index, union all and its two subqueries can > respond quickly. When the aggregate index is added, the subquery can hit the > model, but the union all statement is pushdown > > 当只有明细索引时，union all及其两个子查询都能很快响应，当增加了聚合索引后，子查询可以击中模型，但是union all语句走了下压 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KYLIN-5522) Fix build issues in local mode and use lang3 package
[ https://issues.apache.org/jira/browse/KYLIN-5522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu resolved KYLIN-5522. - Resolution: Fixed > Fix build issues in local mode and use lang3 package > > > Key: KYLIN-5522 > URL: https://issues.apache.org/jira/browse/KYLIN-5522 > Project: Kylin > Issue Type: Task >Affects Versions: 5.0-alpha >Reporter: Yaguang Jia >Assignee: Yaguang Jia >Priority: Critical > Fix For: 5.0-beta > > > 1. local mode cannot build > 2. All use the lang3 package -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KYLIN-5595) [kylin 5.0] Launch Job Node not initialize spark session issue
[ https://issues.apache.org/jira/browse/KYLIN-5595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu resolved KYLIN-5595. - Resolution: Fixed > [kylin 5.0] Launch Job Node not initialize spark session issue > -- > > Key: KYLIN-5595 > URL: https://issues.apache.org/jira/browse/KYLIN-5595 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Affects Versions: 5.0-beta >Reporter: Li Can >Assignee: Li Can >Priority: Major > Fix For: 5.0-beta > > Attachments: image (87).png, image (88).png > > > Saving model will execute 'checkFlatTableSql' method at job node, and it > will not skip the step by default. When execute 'checkFlatTableSql' method, > it will initialize spark session if the job node just started, the process of > getting spark session costs too much time. > The pic 87 shows that get spark session costs more than 63s, and the > execution of checking sql costs more than 2s, it is not friendly for saving > model first time after node launched, and it is also unreasonable. > So I suggest that the job node's process of spark session initialization > should be consistent with the query node, it means that the spark session > should be initialized as the node just started. And the spark session is a > singleton model, just need once initialization, as the pic 88 display. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KYLIN-5596) kylin5 dashboard test units error issue
[ https://issues.apache.org/jira/browse/KYLIN-5596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu resolved KYLIN-5596. - Resolution: Fixed > kylin5 dashboard test units error issue > --- > > Key: KYLIN-5596 > URL: https://issues.apache.org/jira/browse/KYLIN-5596 > Project: Kylin > Issue Type: Test >Affects Versions: 5.0-beta >Reporter: Li Can >Assignee: Li Can >Priority: Major > Fix For: 5.0-beta > > > main branch code updated, but some code of dashboard is not been consistent > with the main branch, the test units need to fix. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KYLIN-5576) Listing files using WHERE conditions with subquery on partition columns will lead to the failure of building model
[ https://issues.apache.org/jira/browse/KYLIN-5576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu updated KYLIN-5576: Labels: Minor (was: ) > Listing files using WHERE conditions with subquery on partition columns will > lead to the failure of building model > -- > > Key: KYLIN-5576 > URL: https://issues.apache.org/jira/browse/KYLIN-5576 > Project: Kylin > Issue Type: Bug > Components: Modeling >Affects Versions: 5.0-alpha >Reporter: Guangyuan Feng >Assignee: Guangyuan Feng >Priority: Major > Labels: Minor > Fix For: 5.0-beta > > > KE should not using the filters containing subquery on partition columns to > detecting resources. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KYLIN-5562) Building job will be scheduled and executed repeatedly
[ https://issues.apache.org/jira/browse/KYLIN-5562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu updated KYLIN-5562: Fix Version/s: (was: 5.0-alpha) > Building job will be scheduled and executed repeatedly > -- > > Key: KYLIN-5562 > URL: https://issues.apache.org/jira/browse/KYLIN-5562 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: 5.0-alpha >Reporter: Guangyuan Feng >Assignee: sibing.zhang >Priority: Major > Attachments: job-repeatedly-scheduled-logs.png > > > This issue will happen in the recent versions, because of the previous > changes on the logic of appending *RUNNING* job, so we need to revert the > related code. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KYLIN-5519) DateFormatTest shouldn't rely on user's timezone
[
https://issues.apache.org/jira/browse/KYLIN-5519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoxiang Yu resolved KYLIN-5519.
-
Resolution: Fixed
> DateFormatTest shouldn't rely on user's timezone
>
>
> Key: KYLIN-5519
> URL: https://issues.apache.org/jira/browse/KYLIN-5519
> Project: Kylin
> Issue Type: Improvement
> Components: Tools, Build and Test
>Affects Versions: 5.0-alpha
>Reporter: Shao Feng Shi
>Assignee: Shao Feng Shi
>Priority: Major
> Fix For: 5.0-alpha
>
>
> When I run "mvn clean test", I got the following error:
>
> {code:java}
> [INFO] Results:
> [INFO]
> [ERROR] Failures:
> [ERROR] DateFormatTest.testStringToMillis:200 expected:<166982400> but
> was:<166985280>
> [ERROR] DateFormatTest.testStringToMillisSupplement:220
> expected:<166982400> but was:<166985280> {code}
> When look into the "DateFormatTest.testStringToMillis", it has no timezone
> specified when converting a Date time to epoch time. In the
> DateFormat.getDateFormat(), it uses the default timezone:
> {code:java}
> public static FastDateFormat getDateFormat(String datePattern) {
> FastDateFormat r = formatMap.get(datePattern);
> if (r == null) {
> r = FastDateFormat.getInstance(datePattern, TimeZone.getDefault());
> formatMap.put(datePattern, r);
> }
> return r;
> } {code}
>
> The function and test case shouldn't rely on user's default timezone. Which
> caused it may fail on another timezone.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KYLIN-5284) kylin5 any users login with request the apis includes "/admin" path
[ https://issues.apache.org/jira/browse/KYLIN-5284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu resolved KYLIN-5284. - Fix Version/s: 5.0-alpha Resolution: Invalid > kylin5 any users login with request the apis includes "/admin" path > > > Key: KYLIN-5284 > URL: https://issues.apache.org/jira/browse/KYLIN-5284 > Project: Kylin > Issue Type: Improvement > Components: REST Service >Affects Versions: 5.0-alpha >Reporter: Li Can >Priority: Minor > Fix For: 5.0-alpha > > Attachments: image-2022-10-27-17-48-38-138.png, > image-2022-10-27-17-52-04-877.png, image-2022-10-27-17-54-02-381.png > > > !image-2022-10-27-17-48-38-138.png|width=537,height=214! > !image-2022-10-27-17-52-04-877.png|width=536,height=207! > when any users login, the routerguard request the apis include "/admin" path, > is this reasonable? why these apis designed include "/admin" path? as I have > tested, if the user without admin permission he/she will login failed, hence > I use interceptor to change this perssion level. > looking forward reply and explanations. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KYLIN-5220) New integration testing module
[ https://issues.apache.org/jira/browse/KYLIN-5220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu resolved KYLIN-5220. - Assignee: Xiaoxiang Yu Resolution: Duplicate > New integration testing module > -- > > Key: KYLIN-5220 > URL: https://issues.apache.org/jira/browse/KYLIN-5220 > Project: Kylin > Issue Type: Sub-task >Reporter: Xiaoxiang Yu >Assignee: Xiaoxiang Yu >Priority: Major > Fix For: 5.0-alpha > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KYLIN-5331) When the where conditions are mutually exclusive, the aggregation query returns "The current query exceeds the data range of the model service. Please add a segment
[
https://issues.apache.org/jira/browse/KYLIN-5331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoxiang Yu resolved KYLIN-5331.
-
Resolution: Fixed
> When the where conditions are mutually exclusive, the aggregation query
> returns "The current query exceeds the data range of the model service.
> Please add a segment
>
>
> Key: KYLIN-5331
> URL: https://issues.apache.org/jira/browse/KYLIN-5331
> Project: Kylin
> Issue Type: Bug
>Affects Versions: 5.0-alpha
>Reporter: sibing.zhang
>Assignee: sibing.zhang
>Priority: Major
> Fix For: 5.0-alpha
>
> Attachments: 9ba667a7-13a9-4c0d-9c58-a8bde6d9b591.png
>
>
> The conditions in the query are mutually exclusive: CUSTOMER.C_NATION in
> ('INDONESIA') and CUSTOMER.C_NATION in ('KENYA'), the result of the
> aggregation query is empty and return "The current query exceeds the data
> range of the model service.Please add a segment."
> *root cause*
> When the query is in the select realization and prune segment steps, all
> segments are returned directly in the case of a full build. However, in the
> case of incrementally build, if the filter conditions are mutually exclusive,
> an empty list will be returned, which will cause the subsequent logic to find
> that the segment is not hit, and set the returned result to be empty. The two
> behave inconsistently.
> *fix design*
> Added property olapContext.storageContext.isFilterCondAlwaysFalse. When
> org.apache.kylin.query.routing.RealizationPruner#pruneSegments judges that
> the filter condition is false, set isFilterCondAlwaysFalse to true. When
> filling sqlResponse, set isFilterCondAlwaysFalse to true and indexType to
> enumeration "Filter Conflict". For the front end, when the indexType is
> "Filter Conflict", no text will be prompted.
> !9ba667a7-13a9-4c0d-9c58-a8bde6d9b591.png|width=911,height=377!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KYLIN-5284) kylin5 any users login with request the apis includes "/admin" path
[ https://issues.apache.org/jira/browse/KYLIN-5284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu updated KYLIN-5284: Fix Version/s: (was: 5.0-alpha) > kylin5 any users login with request the apis includes "/admin" path > > > Key: KYLIN-5284 > URL: https://issues.apache.org/jira/browse/KYLIN-5284 > Project: Kylin > Issue Type: Improvement > Components: REST Service >Affects Versions: 5.0-alpha >Reporter: Li Can >Priority: Minor > Attachments: image-2022-10-27-17-48-38-138.png, > image-2022-10-27-17-52-04-877.png, image-2022-10-27-17-54-02-381.png > > > !image-2022-10-27-17-48-38-138.png|width=537,height=214! > !image-2022-10-27-17-52-04-877.png|width=536,height=207! > when any users login, the routerguard request the apis include "/admin" path, > is this reasonable? why these apis designed include "/admin" path? as I have > tested, if the user without admin permission he/she will login failed, hence > I use interceptor to change this perssion level. > looking forward reply and explanations. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KYLIN-5331) When the where conditions are mutually exclusive, the aggregation query returns "The current query exceeds the data range of the model service. Please add a segment
[
https://issues.apache.org/jira/browse/KYLIN-5331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoxiang Yu reassigned KYLIN-5331:
---
Assignee: sibing.zhang
> When the where conditions are mutually exclusive, the aggregation query
> returns "The current query exceeds the data range of the model service.
> Please add a segment
>
>
> Key: KYLIN-5331
> URL: https://issues.apache.org/jira/browse/KYLIN-5331
> Project: Kylin
> Issue Type: Bug
>Affects Versions: 5.0-alpha
>Reporter: sibing.zhang
>Assignee: sibing.zhang
>Priority: Major
> Fix For: 5.0-alpha
>
> Attachments: 9ba667a7-13a9-4c0d-9c58-a8bde6d9b591.png
>
>
> The conditions in the query are mutually exclusive: CUSTOMER.C_NATION in
> ('INDONESIA') and CUSTOMER.C_NATION in ('KENYA'), the result of the
> aggregation query is empty and return "The current query exceeds the data
> range of the model service.Please add a segment."
> *root cause*
> When the query is in the select realization and prune segment steps, all
> segments are returned directly in the case of a full build. However, in the
> case of incrementally build, if the filter conditions are mutually exclusive,
> an empty list will be returned, which will cause the subsequent logic to find
> that the segment is not hit, and set the returned result to be empty. The two
> behave inconsistently.
> *fix design*
> Added property olapContext.storageContext.isFilterCondAlwaysFalse. When
> org.apache.kylin.query.routing.RealizationPruner#pruneSegments judges that
> the filter condition is false, set isFilterCondAlwaysFalse to true. When
> filling sqlResponse, set isFilterCondAlwaysFalse to true and indexType to
> enumeration "Filter Conflict". For the front end, when the indexType is
> "Filter Conflict", no text will be prompted.
> !9ba667a7-13a9-4c0d-9c58-a8bde6d9b591.png|width=911,height=377!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KYLIN-5332) After turning on the switch "kylin.query.use-tableindex-answer-select-star.enabled", select * only answers the dimension in the cube, but the table index contains more c
[ https://issues.apache.org/jira/browse/KYLIN-5332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu resolved KYLIN-5332. - Resolution: Fixed > After turning on the switch > "kylin.query.use-tableindex-answer-select-star.enabled", select * only > answers the dimension in the cube, but the table index contains more columns > --- > > Key: KYLIN-5332 > URL: https://issues.apache.org/jira/browse/KYLIN-5332 > Project: Kylin > Issue Type: Bug >Affects Versions: 5.0-alpha >Reporter: sibing.zhang >Assignee: sibing.zhang >Priority: Major > Fix For: 5.0-alpha > > > 该现象表现为dimension中有5个列，但是tableindex中有10个列。通过select * > 查询出来的结果仅有5个列。预期是需要查询出全部的10个列。 > *RootCause:* > 开启开关 > “kylin.query.use-tableindex-answer-select-star.enabled”后，olaptable在注册sourceColumn的时候，会根据当前table在model中查询上线的模型。如果没有找到则返回所有列，如果找到，则返回model的effectDimensions作为表的暴露列。 > 当前逻辑中dimension和measure会被一起写入table index，但是measure写入table > index时并没有把该列设置为dimension，导致获取不到该列。查询select *的时候，则会缺少列。 > *Fix Design：* > 直接通过dataflow获取allColumn，这个column来源索引中的列。 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KYLIN-5332) After turning on the switch "kylin.query.use-tableindex-answer-select-star.enabled", select * only answers the dimension in the cube, but the table index contains more c
[ https://issues.apache.org/jira/browse/KYLIN-5332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu reassigned KYLIN-5332: --- Assignee: sibing.zhang > After turning on the switch > "kylin.query.use-tableindex-answer-select-star.enabled", select * only > answers the dimension in the cube, but the table index contains more columns > --- > > Key: KYLIN-5332 > URL: https://issues.apache.org/jira/browse/KYLIN-5332 > Project: Kylin > Issue Type: Bug >Affects Versions: 5.0-alpha >Reporter: sibing.zhang >Assignee: sibing.zhang >Priority: Major > Fix For: 5.0-alpha > > > 该现象表现为dimension中有5个列，但是tableindex中有10个列。通过select * > 查询出来的结果仅有5个列。预期是需要查询出全部的10个列。 > *RootCause:* > 开启开关 > “kylin.query.use-tableindex-answer-select-star.enabled”后，olaptable在注册sourceColumn的时候，会根据当前table在model中查询上线的模型。如果没有找到则返回所有列，如果找到，则返回model的effectDimensions作为表的暴露列。 > 当前逻辑中dimension和measure会被一起写入table index，但是measure写入table > index时并没有把该列设置为dimension，导致获取不到该列。查询select *的时候，则会缺少列。 > *Fix Design：* > 直接通过dataflow获取allColumn，这个column来源索引中的列。 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KYLIN-5333) V2 version api is forward compatible
[ https://issues.apache.org/jira/browse/KYLIN-5333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu resolved KYLIN-5333. - Resolution: Fixed > V2 version api is forward compatible > > > Key: KYLIN-5333 > URL: https://issues.apache.org/jira/browse/KYLIN-5333 > Project: Kylin > Issue Type: Bug >Affects Versions: 5.0-alpha >Reporter: sibing.zhang >Assignee: sibing.zhang >Priority: Major > Fix For: 5.0-alpha > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KYLIN-5334) When the optional parameter "page_offset" of the job list API is an abnormal value, the returned error message is unclear
[
https://issues.apache.org/jira/browse/KYLIN-5334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoxiang Yu resolved KYLIN-5334.
-
Resolution: Fixed
> When the optional parameter "page_offset" of the job list API is an abnormal
> value, the returned error message is unclear
> -
>
> Key: KYLIN-5334
> URL: https://issues.apache.org/jira/browse/KYLIN-5334
> Project: Kylin
> Issue Type: Bug
>Affects Versions: 5.0-alpha
>Reporter: sibing.zhang
>Assignee: sibing.zhang
>Priority: Major
> Fix For: 5.0-alpha
>
> Attachments: image-2022-12-05-19-59-13-386.png
>
>
> 返回任务列表API（{{{}/kylin/api/jobs）的参数page_offset为非负整数外的异常值时，接口报错不明确。比如，下图中将{}}}{{{}page_offset置为abc，接口返回的错误信息不友好。{}}}
> !image-2022-12-05-19-59-13-386.png|width=1047,height=181!
> *root cause:*
> 获取任务接口未对参数{{{}page_offset{}}}、{{{}page_size{}}}做校验导致。
> *fix design:*
> 在接口中增加{{{}page_offset{}}} {{page_size}} 参数的校验。
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (KYLIN-5333) V2 version api is forward compatible
[ https://issues.apache.org/jira/browse/KYLIN-5333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoxiang Yu reassigned KYLIN-5333: --- Assignee: sibing.zhang > V2 version api is forward compatible > > > Key: KYLIN-5333 > URL: https://issues.apache.org/jira/browse/KYLIN-5333 > Project: Kylin > Issue Type: Bug >Affects Versions: 5.0-alpha >Reporter: sibing.zhang >Assignee: sibing.zhang >Priority: Major > Fix For: 5.0-alpha > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KYLIN-5334) When the optional parameter "page_offset" of the job list API is an abnormal value, the returned error message is unclear
[
https://issues.apache.org/jira/browse/KYLIN-5334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoxiang Yu reassigned KYLIN-5334:
---
Assignee: sibing.zhang
> When the optional parameter "page_offset" of the job list API is an abnormal
> value, the returned error message is unclear
> -
>
> Key: KYLIN-5334
> URL: https://issues.apache.org/jira/browse/KYLIN-5334
> Project: Kylin
> Issue Type: Bug
>Affects Versions: 5.0-alpha
>Reporter: sibing.zhang
>Assignee: sibing.zhang
>Priority: Major
> Fix For: 5.0-alpha
>
> Attachments: image-2022-12-05-19-59-13-386.png
>
>
> 返回任务列表API（{{{}/kylin/api/jobs）的参数page_offset为非负整数外的异常值时，接口报错不明确。比如，下图中将{}}}{{{}page_offset置为abc，接口返回的错误信息不友好。{}}}
> !image-2022-12-05-19-59-13-386.png|width=1047,height=181!
> *root cause:*
> 获取任务接口未对参数{{{}page_offset{}}}、{{{}page_size{}}}做校验导致。
> *fix design:*
> 在接口中增加{{{}page_offset{}}} {{page_size}} 参数的校验。
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KYLIN-5336) Using the SQL modeling API, the model can be online without a segment
[
https://issues.apache.org/jira/browse/KYLIN-5336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoxiang Yu resolved KYLIN-5336.
-
Resolution: Invalid
> Using the SQL modeling API, the model can be online without a segment
> -
>
> Key: KYLIN-5336
> URL: https://issues.apache.org/jira/browse/KYLIN-5336
> Project: Kylin
> Issue Type: Bug
>Affects Versions: 5.0-alpha
>Reporter: sibing.zhang
>Assignee: sibing.zhang
>Priority: Major
> Fix For: 5.0-alpha
>
>
> *复现方式*
> # 创建项目，加载数据
> # 调用sql建模api：/kylin/api/models/model_suggestion，传入以下参数：
> {
> "project":"***",
> "sqls":["***"],
> "with_segment":false,
> "with_model_online":true}
> # 此时，创建了一个没有segment，但上线了的模型
> # 前端页面上线模型，必须有segment（空segment也可以）
> *期望结果*
> 调用sql建模api行为须和前端保持一致
> *root cause*
> 对上线的模型，未判断segment是否存在。
> *fix design*
>
> 从接口参数处加以校验。当传入的参数"with_model_online"为true时，传入的”with_segment“参数必须为true，为false或者非boolean类型时则返回对应的错误信息。
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KYLIN-5337) The accept_recommendation parameter in the model optimization API has not been verified
[
https://issues.apache.org/jira/browse/KYLIN-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoxiang Yu resolved KYLIN-5337.
-
Resolution: Invalid
> The accept_recommendation parameter in the model optimization API has not
> been verified
> ---
>
> Key: KYLIN-5337
> URL: https://issues.apache.org/jira/browse/KYLIN-5337
> Project: Kylin
> Issue Type: Bug
>Affects Versions: 5.0-alpha
>Reporter: sibing.zhang
>Assignee: sibing.zhang
>Priority: Major
> Fix For: 5.0-alpha
>
>
> 模型优化API：{{{}POST
> http://host:port/kylin/api/models/model_optimization所使用的参数accept_recommendation为boolean类型。实际传入非boolean的值不会报错，而是当作默认值false来处理。{}}}
> {{*预期*}}
> {{{}对模型优化API中的{}}}{{{}accept_recommendation参数进行校验{}}}
> *fix design*
> 增加参数类型校验{{{}{}}}
> {{{}{}}}{{{}{}}}
> {{}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KYLIN-5338) The password change API is not friendly to the required parameter verification information
[
https://issues.apache.org/jira/browse/KYLIN-5338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoxiang Yu resolved KYLIN-5338.
-
Resolution: Fixed
> The password change API is not friendly to the required parameter
> verification information
> --
>
> Key: KYLIN-5338
> URL: https://issues.apache.org/jira/browse/KYLIN-5338
> Project: Kylin
> Issue Type: Bug
>Affects Versions: 5.0-alpha
>Reporter: sibing.zhang
>Assignee: sibing.zhang
>Priority: Major
> Fix For: 5.0-alpha
>
>
> 修改密码API：{{{}PUT [http://host:port/kylin/api/user/password]的以下两个参数：{}}}
> * {{password}} - {{必填}} {{{}string{}}}，用户旧密码。
> * {{new_password}} - {{必填}} {{{}string{}}}，用户新密码。
> {{{}没有在不同场景下进行非空校验。如，当admin用户修改普通用户密码时，{}}}{{{}password{}}}{{{}参数为非必填。{}}}{{{}new_password{}}}{{{}在任何用户修改密码的情况下都应该进行非空校验。{}}}
>
> *fix design*
> * 增加{{{}new_password{}}}的必填校验。
> * 如果是admin或者系统管理员用户修改普通用户或者其他系统管理员(非默认admin账户)时，可以不用输入{{{}password{}}}参数。
> * 如果是普通账户修改自己密码，没有输入{{{}password{}}}参数时，提示"KE-010043201: 请求参数 “password”
> 为空或值为空。请检查请求参数是否正确填写。"
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (KYLIN-5337) The accept_recommendation parameter in the model optimization API has not been verified
[
https://issues.apache.org/jira/browse/KYLIN-5337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoxiang Yu reassigned KYLIN-5337:
---
Assignee: sibing.zhang
> The accept_recommendation parameter in the model optimization API has not
> been verified
> ---
>
> Key: KYLIN-5337
> URL: https://issues.apache.org/jira/browse/KYLIN-5337
> Project: Kylin
> Issue Type: Bug
>Affects Versions: 5.0-alpha
>Reporter: sibing.zhang
>Assignee: sibing.zhang
>Priority: Major
> Fix For: 5.0-alpha
>
>
> 模型优化API：{{{}POST
> http://host:port/kylin/api/models/model_optimization所使用的参数accept_recommendation为boolean类型。实际传入非boolean的值不会报错，而是当作默认值false来处理。{}}}
> {{*预期*}}
> {{{}对模型优化API中的{}}}{{{}accept_recommendation参数进行校验{}}}
> *fix design*
> 增加参数类型校验{{{}{}}}
> {{{}{}}}{{{}{}}}
> {{}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (KYLIN-5336) Using the SQL modeling API, the model can be online without a segment
[
https://issues.apache.org/jira/browse/KYLIN-5336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoxiang Yu reassigned KYLIN-5336:
---
Assignee: sibing.zhang
> Using the SQL modeling API, the model can be online without a segment
> -
>
> Key: KYLIN-5336
> URL: https://issues.apache.org/jira/browse/KYLIN-5336
> Project: Kylin
> Issue Type: Bug
>Affects Versions: 5.0-alpha
>Reporter: sibing.zhang
>Assignee: sibing.zhang
>Priority: Major
> Fix For: 5.0-alpha
>
>
> *复现方式*
> # 创建项目，加载数据
> # 调用sql建模api：/kylin/api/models/model_suggestion，传入以下参数：
> {
> "project":"***",
> "sqls":["***"],
> "with_segment":false,
> "with_model_online":true}
> # 此时，创建了一个没有segment，但上线了的模型
> # 前端页面上线模型，必须有segment（空segment也可以）
> *期望结果*
> 调用sql建模api行为须和前端保持一致
> *root cause*
> 对上线的模型，未判断segment是否存在。
> *fix design*
>
> 从接口参数处加以校验。当传入的参数"with_model_online"为true时，传入的”with_segment“参数必须为true，为false或者非boolean类型时则返回对应的错误信息。
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (KYLIN-5338) The password change API is not friendly to the required parameter verification information
[
https://issues.apache.org/jira/browse/KYLIN-5338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoxiang Yu reassigned KYLIN-5338:
---
Assignee: sibing.zhang
> The password change API is not friendly to the required parameter
> verification information
> --
>
> Key: KYLIN-5338
> URL: https://issues.apache.org/jira/browse/KYLIN-5338
> Project: Kylin
> Issue Type: Bug
>Affects Versions: 5.0-alpha
>Reporter: sibing.zhang
>Assignee: sibing.zhang
>Priority: Major
> Fix For: 5.0-alpha
>
>
> 修改密码API：{{{}PUT [http://host:port/kylin/api/user/password]的以下两个参数：{}}}
> * {{password}} - {{必填}} {{{}string{}}}，用户旧密码。
> * {{new_password}} - {{必填}} {{{}string{}}}，用户新密码。
> {{{}没有在不同场景下进行非空校验。如，当admin用户修改普通用户密码时，{}}}{{{}password{}}}{{{}参数为非必填。{}}}{{{}new_password{}}}{{{}在任何用户修改密码的情况下都应该进行非空校验。{}}}
>
> *fix design*
> * 增加{{{}new_password{}}}的必填校验。
> * 如果是admin或者系统管理员用户修改普通用户或者其他系统管理员(非默认admin账户)时，可以不用输入{{{}password{}}}参数。
> * 如果是普通账户修改自己密码，没有输入{{{}password{}}}参数时，提示"KE-010043201: 请求参数 “password”
> 为空或值为空。请检查请求参数是否正确填写。"
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KYLIN-5339) Renew Epoch Retry did not interrupt the old thread in time, and the new thread failed to write data, resulting in kylin losing epoch
[
https://issues.apache.org/jira/browse/KYLIN-5339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoxiang Yu resolved KYLIN-5339.
-
Resolution: Fixed
> Renew Epoch Retry did not interrupt the old thread in time, and the new
> thread failed to write data, resulting in kylin losing epoch
>
>
> Key: KYLIN-5339
> URL: https://issues.apache.org/jira/browse/KYLIN-5339
> Project: Kylin
> Issue Type: Bug
>Affects Versions: 5.0-alpha
>Reporter: sibing.zhang
>Priority: Major
> Fix For: 5.0-alpha
>
> Attachments: 31c439f4-0a2b-4616-949d-415f4b417f2e.png,
> 602360ee-fa81-4c8c-a7d2-4fdd73d284ff.png
>
>
> epoch renew时有两次retry，每次有超时60s的机制。renew时使用线程池来执行。这个线程池容量由开关
> kylin.server.renew-epoch-pool-size=3决定。这里存在的问题是：renew线程超时60s后没有终止该线程，又拉起了另一个renew线程，对同样的数据进行了更新。此时第一个线程由于没有终止，最后renew成功了，并把数据的MVCC+1。后面renew的线程renew时，会判断MVCC：
> !31c439f4-0a2b-4616-949d-415f4b417f2e.png|width=583,height=64!
> 此时，发现没有满足条件的数据，导致return的update affectedRows = 0。最终，造成了当前节点丢失了所有项目的控制权。流程可见下图：
>
> *!602360ee-fa81-4c8c-a7d2-4fdd73d284ff.png|width=560,height=574!*
> *fix design*
> Epoch
> Renew有超时失败的重试机制({{{}kylin.server.leader-race.heart-beat-timeout=60s{}}})。重试时，原有的事务没有停止，新开事务进行了数据库更新。由于Epoch
>
> 更新时，会校验mvcc的值，所以这里导致第二个事务被第一个事务冲突了。鉴于此，增加事务Timeout机制，Timeout={{{}kylin.server.leader-race.heart-beat-timeout=60s{}}}-1s。事务超时自动回滚，避免了Renew重试时事务冲突的问题。
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
