[jira] [Created] (KYLIN-4497) PreparedStatement query do not support "limit ? offset ?"
Chao Long created KYLIN-4497:
Summary: PreparedStatement query do not support "limit ? offset ?"
Key: KYLIN-4497
URL: https://issues.apache.org/jira/browse/KYLIN-4497
Project: Kylin
Issue Type: Bug
Components: Driver - JDBC
Reporter: Chao Long
Assignee: Chao Long
Fix For: v3.1.0
Query
{code:java}
String sql = "select part_dt, sum(price) from KYLIN_SALES group by part_dt
order by part_dt limit ? offset ?";
conn = getConnection();
statement = conn.prepareStatement(sql);
statement.setInt(1, 10);
statement.setInt(2, 0);
{code}
Error log
{code:java}
2020-05-15 09:32:36,062 ERROR [Query b87327f4-7299-32f2-4a2e-c19795a42288-44]
service.QueryService:542 : Exception while executing query
java.sql.SQLException: Error while preparing statement [select part_dt,
sum(price) from KYLIN_SALES group by part_dt order by part_dt limit ? offset ?]
at org.apache.calcite.avatica.Helper.createException(Helper.java:56)
at org.apache.calcite.avatica.Helper.createException(Helper.java:41)
at
org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement_(CalciteConnectionImpl.java:210)
at
org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:192)
at
org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:89)
at
org.apache.calcite.avatica.AvaticaConnection.prepareStatement(AvaticaConnection.java:175)
at
org.apache.kylin.rest.service.QueryService.createPreparedContext(QueryService.java:1262)
at
org.apache.kylin.rest.service.QueryService.access$600(QueryService.java:141)
at
org.apache.kylin.rest.service.QueryService$PreparedContextFactory.create(QueryService.java:1301)
at
org.apache.kylin.rest.service.QueryService$PreparedContextFactory.create(QueryService.java:1296)
at
org.apache.commons.pool2.BaseKeyedPooledObjectFactory.makeObject(BaseKeyedPooledObjectFactory.java:60)
at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.create(GenericKeyedObjectPool.java:1064)
at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:358)
at
org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:281)
at
org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:680)
at
org.apache.kylin.rest.service.QueryService.query(QueryService.java:219)
at
org.apache.kylin.rest.service.QueryService.queryAndUpdateCache(QueryService.java:480)
at
org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:440)
at
org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:378)
at
org.apache.kylin.rest.controller.QueryController.query(QueryController.java:93)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at
org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:209)
at
com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:244)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:492)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:165)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:1025)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:452)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1195)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:654)
at
[jira] [Commented] (KYLIN-4464) Query ... row_number over(order by c1) ... order by c2 ... get wrong order result
[
https://issues.apache.org/jira/browse/KYLIN-4464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17101321#comment-17101321
]
Chao Long commented on KYLIN-4464:
--
*Test Evidence*
*After fix*
*Kylin result (order by lstg_format_name)***
*!image-2020-05-07-10-32-39-983.png!*
*Calcite execution plan***
{code:java}
EXECUTION PLAN BEFORE REWRITE
OLAPToEnumerableConverter
OLAPLimitRel(ctx=[], fetch=[5])
OLAPSortRel(sort0=[$1], dir0=[DESC], ctx=[])
OLAPWindowRel(window#0=[window(partition {} order by [0] rows between
UNBOUNDED PRECEDING and CURRENT ROW aggs [ROW_NUMBER()])], ctx=[],
groups=[[window(partition {} order by [0] rows between UNBOUNDED PRECEDING and
CURRENT ROW aggs [ROW_NUMBER()])]])
OLAPAggregateRel(group=[{0}], GMV=[SUM($1)], ctx=[])
OLAPProjectRel(LSTG_FORMAT_NAME=[$2], PRICE=[$5], ctx=[])
OLAPFilterRel(condition=[IS NOT NULL($2)], ctx=[])
OLAPTableScan(table=[[DEFAULT, KYLIN_SALES]], ctx=[], fields=[[0,
1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]])
{code}
> Query ... row_number over(order by c1) ... order by c2 ... get wrong order
> result
> -
>
> Key: KYLIN-4464
> URL: https://issues.apache.org/jira/browse/KYLIN-4464
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Reporter: Chao Long
>Assignee: Chao Long
>Priority: Major
> Fix For: v3.0.2, v2.6.6
>
> Attachments: image-2020-04-20-18-19-40-925.png,
> image-2020-04-20-18-22-34-765.png, image-2020-05-07-10-32-39-983.png
>
>
> *SQL*
> {code:java}
> select t.*, row_number() over (order by t.lstg_format_name) as row_num from (
> select lstg_format_name, sum(price) as GMV from
> KYLIN_SALES
> where lstg_format_name is not null
> group by lstg_format_name
> ) as t
> order by t.GMV desc limit 5{quote}
> {code}
>
> *Kylin result (order by lstg_format_name)*
> !image-2020-04-20-18-19-40-925.png!
>
> *Calcite execution plan*
> {code:java}
> EXECUTION PLAN BEFORE REWRITE
> OLAPToEnumerableConverter
> OLAPWindowRel(window#0=[window(partition {} order by [0 DESC] rows between
> UNBOUNDED PRECEDING and CURRENT ROW aggs [ROW_NUMBER()])], ctx=[],
> groups=[[window(partition {} order by [0 DESC] rows between UNBOUNDED
> PRECEDING and CURRENT ROW aggs [ROW_NUMBER()])]])
> OLAPLimitRel(ctx=[], fetch=[5])
> OLAPSortRel(sort0=[$1], dir0=[DESC], ctx=[])
> OLAPAggregateRel(group=[{0}], GMV=[SUM($1)], ctx=[])
> OLAPProjectRel(LSTG_FORMAT_NAME=[$2], PRICE=[$5], ctx=[])
> OLAPFilterRel(condition=[IS NOT NULL($2)], ctx=[])
> OLAPTableScan(table=[[DEFAULT, KYLIN_SALES]], ctx=[],
> fields=[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]])
> {code}
>
> *Hive result(order by GMV)*
> !image-2020-04-20-18-22-34-765.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KYLIN-4464) Query ... row_number over(order by c1) ... order by c2 ... get wrong order result
[
https://issues.apache.org/jira/browse/KYLIN-4464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Long updated KYLIN-4464:
-
Attachment: image-2020-05-07-10-32-39-983.png
> Query ... row_number over(order by c1) ... order by c2 ... get wrong order
> result
> -
>
> Key: KYLIN-4464
> URL: https://issues.apache.org/jira/browse/KYLIN-4464
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Reporter: Chao Long
>Assignee: Chao Long
>Priority: Major
> Fix For: v3.0.2, v2.6.6
>
> Attachments: image-2020-04-20-18-19-40-925.png,
> image-2020-04-20-18-22-34-765.png, image-2020-05-07-10-32-39-983.png
>
>
> *SQL*
> {code:java}
> select t.*, row_number() over (order by t.lstg_format_name) as row_num from (
> select lstg_format_name, sum(price) as GMV from
> KYLIN_SALES
> where lstg_format_name is not null
> group by lstg_format_name
> ) as t
> order by t.GMV desc limit 5{quote}
> {code}
>
> *Kylin result (order by lstg_format_name)*
> !image-2020-04-20-18-19-40-925.png!
>
> *Calcite execution plan*
> {code:java}
> EXECUTION PLAN BEFORE REWRITE
> OLAPToEnumerableConverter
> OLAPWindowRel(window#0=[window(partition {} order by [0 DESC] rows between
> UNBOUNDED PRECEDING and CURRENT ROW aggs [ROW_NUMBER()])], ctx=[],
> groups=[[window(partition {} order by [0 DESC] rows between UNBOUNDED
> PRECEDING and CURRENT ROW aggs [ROW_NUMBER()])]])
> OLAPLimitRel(ctx=[], fetch=[5])
> OLAPSortRel(sort0=[$1], dir0=[DESC], ctx=[])
> OLAPAggregateRel(group=[{0}], GMV=[SUM($1)], ctx=[])
> OLAPProjectRel(LSTG_FORMAT_NAME=[$2], PRICE=[$5], ctx=[])
> OLAPFilterRel(condition=[IS NOT NULL($2)], ctx=[])
> OLAPTableScan(table=[[DEFAULT, KYLIN_SALES]], ctx=[],
> fields=[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]])
> {code}
>
> *Hive result(order by GMV)*
> !image-2020-04-20-18-22-34-765.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KYLIN-4464) Query ... row_number over(order by c1) ... order by c2 ... get wrong order result
[
https://issues.apache.org/jira/browse/KYLIN-4464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Long updated KYLIN-4464:
-
Fix Version/s: (was: Future)
v2.6.6
v3.0.2
> Query ... row_number over(order by c1) ... order by c2 ... get wrong order
> result
> -
>
> Key: KYLIN-4464
> URL: https://issues.apache.org/jira/browse/KYLIN-4464
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Reporter: Chao Long
>Assignee: Chao Long
>Priority: Major
> Fix For: v3.0.2, v2.6.6
>
> Attachments: image-2020-04-20-18-19-40-925.png,
> image-2020-04-20-18-22-34-765.png
>
>
> *SQL*
> {code:java}
> select t.*, row_number() over (order by t.lstg_format_name) as row_num from (
> select lstg_format_name, sum(price) as GMV from
> KYLIN_SALES
> where lstg_format_name is not null
> group by lstg_format_name
> ) as t
> order by t.GMV desc limit 5{quote}
> {code}
>
> *Kylin result (order by lstg_format_name)*
> !image-2020-04-20-18-19-40-925.png!
>
> *Calcite execution plan*
> {code:java}
> EXECUTION PLAN BEFORE REWRITE
> OLAPToEnumerableConverter
> OLAPWindowRel(window#0=[window(partition {} order by [0 DESC] rows between
> UNBOUNDED PRECEDING and CURRENT ROW aggs [ROW_NUMBER()])], ctx=[],
> groups=[[window(partition {} order by [0 DESC] rows between UNBOUNDED
> PRECEDING and CURRENT ROW aggs [ROW_NUMBER()])]])
> OLAPLimitRel(ctx=[], fetch=[5])
> OLAPSortRel(sort0=[$1], dir0=[DESC], ctx=[])
> OLAPAggregateRel(group=[{0}], GMV=[SUM($1)], ctx=[])
> OLAPProjectRel(LSTG_FORMAT_NAME=[$2], PRICE=[$5], ctx=[])
> OLAPFilterRel(condition=[IS NOT NULL($2)], ctx=[])
> OLAPTableScan(table=[[DEFAULT, KYLIN_SALES]], ctx=[],
> fields=[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]])
> {code}
>
> *Hive result(order by GMV)*
> !image-2020-04-20-18-22-34-765.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KYLIN-4464) Query ... row_number over(order by c1) ... order by c2 ... get wrong order result
Chao Long created KYLIN-4464:
Summary: Query ... row_number over(order by c1) ... order by c2
... get wrong order result
Key: KYLIN-4464
URL: https://issues.apache.org/jira/browse/KYLIN-4464
Project: Kylin
Issue Type: Bug
Components: Query Engine
Reporter: Chao Long
Assignee: Chao Long
Fix For: Future
Attachments: image-2020-04-20-18-19-40-925.png,
image-2020-04-20-18-22-34-765.png
*SQL*
{code:java}
select t.*, row_number() over (order by t.lstg_format_name) as row_num from (
select lstg_format_name, sum(price) as GMV from
KYLIN_SALES
where lstg_format_name is not null
group by lstg_format_name
) as t
order by t.GMV desc limit 5{quote}
{code}
*Kylin result (order by lstg_format_name)*
!image-2020-04-20-18-19-40-925.png!
*Calcite execution plan*
{code:java}
EXECUTION PLAN BEFORE REWRITE
OLAPToEnumerableConverter
OLAPWindowRel(window#0=[window(partition {} order by [0 DESC] rows between
UNBOUNDED PRECEDING and CURRENT ROW aggs [ROW_NUMBER()])], ctx=[],
groups=[[window(partition {} order by [0 DESC] rows between UNBOUNDED PRECEDING
and CURRENT ROW aggs [ROW_NUMBER()])]])
OLAPLimitRel(ctx=[], fetch=[5])
OLAPSortRel(sort0=[$1], dir0=[DESC], ctx=[])
OLAPAggregateRel(group=[{0}], GMV=[SUM($1)], ctx=[])
OLAPProjectRel(LSTG_FORMAT_NAME=[$2], PRICE=[$5], ctx=[])
OLAPFilterRel(condition=[IS NOT NULL($2)], ctx=[])
OLAPTableScan(table=[[DEFAULT, KYLIN_SALES]], ctx=[], fields=[[0,
1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]])
{code}
*Hive result(order by GMV)*
!image-2020-04-20-18-22-34-765.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-4262) pid in GC filename inconsistent with real pid
[ https://issues.apache.org/jira/browse/KYLIN-4262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16977376#comment-16977376 ] Chao Long commented on KYLIN-4262: -- Test evidence !image-2019-11-19-18-55-18-113.png! > pid in GC filename inconsistent with real pid > - > > Key: KYLIN-4262 > URL: https://issues.apache.org/jira/browse/KYLIN-4262 > Project: Kylin > Issue Type: Bug >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v3.0.0, v2.6.5 > > Attachments: image-2019-11-18-17-19-49-059.png, > image-2019-11-18-17-19-56-990.png, image-2019-11-19-18-55-18-113.png > > > pid in GC filename > !image-2019-11-18-17-19-49-059.png! > > real pid > !image-2019-11-18-17-19-56-990.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4262) pid in GC filename inconsistent with real pid
[ https://issues.apache.org/jira/browse/KYLIN-4262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long updated KYLIN-4262: - Attachment: image-2019-11-19-18-55-18-113.png > pid in GC filename inconsistent with real pid > - > > Key: KYLIN-4262 > URL: https://issues.apache.org/jira/browse/KYLIN-4262 > Project: Kylin > Issue Type: Bug >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v3.0.0, v2.6.5 > > Attachments: image-2019-11-18-17-19-49-059.png, > image-2019-11-18-17-19-56-990.png, image-2019-11-19-18-55-18-113.png > > > pid in GC filename > !image-2019-11-18-17-19-49-059.png! > > real pid > !image-2019-11-18-17-19-56-990.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4262) pid in GC filename inconsistent with real pid
[ https://issues.apache.org/jira/browse/KYLIN-4262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long updated KYLIN-4262: - Fix Version/s: v2.6.5 v3.0.0 > pid in GC filename inconsistent with real pid > - > > Key: KYLIN-4262 > URL: https://issues.apache.org/jira/browse/KYLIN-4262 > Project: Kylin > Issue Type: Bug >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v3.0.0, v2.6.5 > > Attachments: image-2019-11-18-17-19-49-059.png, > image-2019-11-18-17-19-56-990.png > > > pid in GC filename > !image-2019-11-18-17-19-49-059.png! > > real pid > !image-2019-11-18-17-19-56-990.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4262) pid in GC filename inconsistent with real pid
Chao Long created KYLIN-4262: Summary: pid in GC filename inconsistent with real pid Key: KYLIN-4262 URL: https://issues.apache.org/jira/browse/KYLIN-4262 Project: Kylin Issue Type: Bug Reporter: Chao Long Assignee: Chao Long Attachments: image-2019-11-18-17-19-49-059.png, image-2019-11-18-17-19-56-990.png pid in GC filename !image-2019-11-18-17-19-49-059.png! real pid !image-2019-11-18-17-19-56-990.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KYLIN-3625) Query engine for Parquet
[ https://issues.apache.org/jira/browse/KYLIN-3625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long resolved KYLIN-3625. -- Resolution: Resolved > Query engine for Parquet > > > Key: KYLIN-3625 > URL: https://issues.apache.org/jira/browse/KYLIN-3625 > Project: Kylin > Issue Type: Sub-task > Components: Query Engine, Storage - Parquet >Reporter: Shaofeng SHI >Assignee: Chao Long >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Resolved] (KYLIN-3626) Allow customization for Parquet path
[ https://issues.apache.org/jira/browse/KYLIN-3626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long resolved KYLIN-3626. -- Resolution: Resolved > Allow customization for Parquet path > > > Key: KYLIN-3626 > URL: https://issues.apache.org/jira/browse/KYLIN-3626 > Project: Kylin > Issue Type: Sub-task > Components: Job Engine, Metadata, Storage - Parquet >Reporter: Shaofeng SHI >Assignee: Chao Long >Priority: Major > > Due to the difference of HDFS and object store like S3/blob store, Kylin > needs to allow different path patterns for the files when putting to the > underlying storage. > > For example, to HDFS, this pattern is good: > /kylin/metadata_name/cube_name/segment_name/file1.parquet > > But, the above pattern is not good for S3, since S3 uses the first couple > bytes of the path to do the hashing, so we'd better add salt ahead of the > original pathname: > /salt/kylin/metadata_name/cube_name/segment_name/file1.parquet > or do a reverse order: > /segment_name/cube_name/metadata_name/kylin/file1.parquet > > To get the flexibility, we should make this extensible. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Resolved] (KYLIN-3777) Remove dict from query
[ https://issues.apache.org/jira/browse/KYLIN-3777?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long resolved KYLIN-3777. -- Resolution: Resolved > Remove dict from query > -- > > Key: KYLIN-3777 > URL: https://issues.apache.org/jira/browse/KYLIN-3777 > Project: Kylin > Issue Type: Sub-task >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > > To improve query performance, we should remove dict from query side. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Resolved] (KYLIN-4085) Segment parallel building may cause segment not found
[ https://issues.apache.org/jira/browse/KYLIN-4085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long resolved KYLIN-4085. -- Resolution: Fixed > Segment parallel building may cause segment not found > - > > Key: KYLIN-4085 > URL: https://issues.apache.org/jira/browse/KYLIN-4085 > Project: Kylin > Issue Type: Bug > Components: Metadata >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v3.0.0-beta > > > In the case of multi-node and parallel building one same cube, the > JDBCResourceStore split the update metadata step in two sql, which can't > guarantee the atomicity. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Resolved] (KYLIN-4089) Integration test failed with JDBCMetastore
[ https://issues.apache.org/jira/browse/KYLIN-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long resolved KYLIN-4089. -- Resolution: Fixed > Integration test failed with JDBCMetastore > -- > > Key: KYLIN-4089 > URL: https://issues.apache.org/jira/browse/KYLIN-4089 > Project: Kylin > Issue Type: Bug > Components: Integration, Tools, Build and Test >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v3.0.0-beta > > > [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.822 > s <<< FAILURE! - in > org.apache.kylin.storage.hbase.ITAclTableMigrationToolTest[ERROR] > testBasic(org.apache.kylin.storage.hbase.ITAclTableMigrationToolTest) Time > elapsed: 0.812 s <<< ERROR!java.lang.NullPointerException > at > org.apache.kylin.storage.hbase.ITAclTableMigrationToolTest.testBasic(ITAclTableMigrationToolTest.java:95) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (KYLIN-4154) Metadata inconsistency between multi Kylin server caused by Broadcaster closing
Chao Long created KYLIN-4154: Summary: Metadata inconsistency between multi Kylin server caused by Broadcaster closing Key: KYLIN-4154 URL: https://issues.apache.org/jira/browse/KYLIN-4154 Project: Kylin Issue Type: Bug Components: Metadata Reporter: Chao Long Assignee: Chao Long Fix For: v3.0.0 To avoid Broadcaster memory leak, KYLIN-4131 close the metadata sync thread after receiving "Sync All" event, but there may be some events in the waiting queue, which haven't been synced yet. So we should sync all events before closing sync thread. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (KYLIN-4131) Broadcaster memory leak
[
https://issues.apache.org/jira/browse/KYLIN-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Long updated KYLIN-4131:
-
Description:
h3. *Test evidence*
Use MAT(Memory Analyzer) to analyze this issue.
# start Kylin, dump a jvm heap info snapshot and import into MAT. We can see
there is only one Broadcaster and ProjectManager object.
!image-2019-08-12-17-22-15-783.png|width=721,height=520!
# Call 'list cube' api and 'reload metadata' api many times. We can see there
are many BroadCaster and Manager objects in the memory, which should be
recycled by GC, but they are not.
!image-2019-08-12-17-22-23-477.png|width=724,height=498!
*Root cause*
Broadcaster hold the ref of Listener object defined in Manager, and the
Listener object hold the ref of Manager, because the impl class of Listener is
an inner class of Manager.
Broadcaster.java
{code:java}
private Map> listenerMap = Maps.newConcurrentMap();
{code}
ProjectManager.java -> ProjectSyncListener
{code:java}
private class ProjectSyncListener extends Broadcaster.Listener {
@Override
public void onEntityChange(Broadcaster broadcaster, String entity, Event event,
String cacheKey)
throws IOException {
String project = cacheKey;
if (event == Event.DROP) {
removeProjectLocal(project);
return;
}
reloadProjectQuietly(project);
broadcaster.notifyProjectSchemaUpdate(project);
broadcaster.notifyProjectDataUpdate(project);
}
}
{code}
And Broadcaster can't be recycled by GC because it holds a running thread,
which cause the Manager object also not being recycled.
*After Fix*
The Broadcaster objects and Manager objects are recycled by GC.
!image-2019-08-12-17-16-05-532.png!
was:
h3. *Test evidence*
Use MAT(Memory Analyzer) to analyze this issue.
# start Kylin, dump a jvm heap info snapshot and import into MAT. We can see
there is only one Broadcaster and ProjectManager object.
!image-2019-08-12-17-22-15-783.png|width=721,height=520!
# Call 'list cube' api and 'reload metadata' api many times. We can see there
are many BroadCaster and Manager objects in the memory, which should be
recycled by GC, but they are not.
!image-2019-08-12-17-22-23-477.png!
*Root cause*
Broadcaster hold the ref of Listener object defined in Manager, and the
Listener object hold the ref of Manager, because the impl class of Listener is
an inner class of Manager.
Broadcaster.java
{code:java}
private Map> listenerMap = Maps.newConcurrentMap();
{code}
ProjectManager.java -> ProjectSyncListener
{code:java}
private class ProjectSyncListener extends Broadcaster.Listener {
@Override
public void onEntityChange(Broadcaster broadcaster, String entity, Event event,
String cacheKey)
throws IOException {
String project = cacheKey;
if (event == Event.DROP) {
removeProjectLocal(project);
return;
}
reloadProjectQuietly(project);
broadcaster.notifyProjectSchemaUpdate(project);
broadcaster.notifyProjectDataUpdate(project);
}
}
{code}
And Broadcaster can't be recycled by GC because it holds a running thread,
which cause the Manager object also not being recycled.
*After Fix*
The Broadcaster objects and Manager objects are recycled by GC.
!image-2019-08-12-17-16-05-532.png!
> Broadcaster memory leak
> ---
>
> Key: KYLIN-4131
> URL: https://issues.apache.org/jira/browse/KYLIN-4131
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
>Reporter: Chao Long
>Assignee: Chao Long
>Priority: Critical
> Fix For: v3.0.0
>
> Attachments: image-2019-08-12-16-39-46-262.png,
> image-2019-08-12-16-39-52-327.png, image-2019-08-12-17-16-05-532.png,
> image-2019-08-12-17-22-15-783.png, image-2019-08-12-17-22-23-477.png
>
>
> h3. *Test evidence*
> Use MAT(Memory Analyzer) to analyze this issue.
> # start Kylin, dump a jvm heap info snapshot and import into MAT. We can
> see there is only one Broadcaster and ProjectManager object.
> !image-2019-08-12-17-22-15-783.png|width=721,height=520!
> # Call 'list cube' api and 'reload metadata' api many times. We can see
> there are many BroadCaster and Manager objects in the memory, which should be
> recycled by GC, but they are not.
> !image-2019-08-12-17-22-23-477.png|width=724,height=498!
>
> *Root cause*
> Broadcaster hold the ref of Listener object defined in Manager, and the
> Listener object hold the ref of Manager, because the impl class of Listener
> is an inner class of Manager.
> Broadcaster.java
> {code:java}
> private Map> listenerMap = Maps.newConcurrentMap();
> {code}
> ProjectManager.java -> ProjectSyncListener
> {code:java}
> private class ProjectSyncListener extends Broadcaster.Listener {
> @Override
> public void onEntityChange(Broadcaster broadcaster, String entity, Event
> event, String cacheKey)
> throws IOException {
> String project = cacheKey;
> if (event == Event.DROP) {
>
[jira] [Updated] (KYLIN-4131) Broadcaster memory leak
[
https://issues.apache.org/jira/browse/KYLIN-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Long updated KYLIN-4131:
-
Description:
h3. *Test evidence*
Use MAT(Memory Analyzer) to analyze this issue.
# start Kylin, dump a jvm heap info snapshot and import into MAT. We can see
there is only one Broadcaster and ProjectManager object.
!image-2019-08-12-17-22-15-783.png|width=721,height=520!
# Call 'list cube' api and 'reload metadata' api many times. We can see there
are many BroadCaster and Manager objects in the memory, which should be
recycled by GC, but they are not.
!image-2019-08-12-17-22-23-477.png!
*Root cause*
Broadcaster hold the ref of Listener object defined in Manager, and the
Listener object hold the ref of Manager, because the impl class of Listener is
an inner class of Manager.
Broadcaster.java
{code:java}
private Map> listenerMap = Maps.newConcurrentMap();
{code}
ProjectManager.java -> ProjectSyncListener
{code:java}
private class ProjectSyncListener extends Broadcaster.Listener {
@Override
public void onEntityChange(Broadcaster broadcaster, String entity, Event event,
String cacheKey)
throws IOException {
String project = cacheKey;
if (event == Event.DROP) {
removeProjectLocal(project);
return;
}
reloadProjectQuietly(project);
broadcaster.notifyProjectSchemaUpdate(project);
broadcaster.notifyProjectDataUpdate(project);
}
}
{code}
And Broadcaster can't be recycled by GC because it holds a running thread,
which cause the Manager object also not being recycled.
*After Fix*
The Broadcaster objects and Manager objects are recycled by GC.
!image-2019-08-12-17-16-05-532.png!
was:
h3. *Test evidence*
Use MAT(Memory Analyzer) to analyze this issue.
# start Kylin, dump a jvm heap info snapshot and import into MAT. We can see
there is only one Broadcaster and ProjectManager object.
!image-2019-08-12-17-22-15-783.png!
# Call 'list cube' api and 'reload metadata' api many times. We can see there
are many BroadCaster and Manager objects in the memory, which should be
recycled by GC, but they are not.
!image-2019-08-12-17-22-23-477.png!
*Root cause*
Broadcaster hold the ref of Listener object defined in Manager, and the
Listener object hold the ref of Manager, because the impl class of Listener is
an inner class of Manager.
Broadcaster.java
{code:java}
private Map> listenerMap = Maps.newConcurrentMap();
{code}
ProjectManager.java -> ProjectSyncListener
{code:java}
private class ProjectSyncListener extends Broadcaster.Listener {
@Override
public void onEntityChange(Broadcaster broadcaster, String entity, Event event,
String cacheKey)
throws IOException {
String project = cacheKey;
if (event == Event.DROP) {
removeProjectLocal(project);
return;
}
reloadProjectQuietly(project);
broadcaster.notifyProjectSchemaUpdate(project);
broadcaster.notifyProjectDataUpdate(project);
}
}
{code}
And Broadcaster can't be recycled by GC because it holds a running thread,
which cause the Manager object also not being recycled.
*After Fix*
The Broadcaster objects and Manager objects are recycled by GC.
!image-2019-08-12-17-16-05-532.png!
> Broadcaster memory leak
> ---
>
> Key: KYLIN-4131
> URL: https://issues.apache.org/jira/browse/KYLIN-4131
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
>Reporter: Chao Long
>Assignee: Chao Long
>Priority: Critical
> Fix For: v3.0.0
>
> Attachments: image-2019-08-12-16-39-46-262.png,
> image-2019-08-12-16-39-52-327.png, image-2019-08-12-17-16-05-532.png,
> image-2019-08-12-17-22-15-783.png, image-2019-08-12-17-22-23-477.png
>
>
> h3. *Test evidence*
> Use MAT(Memory Analyzer) to analyze this issue.
> # start Kylin, dump a jvm heap info snapshot and import into MAT. We can
> see there is only one Broadcaster and ProjectManager object.
> !image-2019-08-12-17-22-15-783.png|width=721,height=520!
> # Call 'list cube' api and 'reload metadata' api many times. We can see
> there are many BroadCaster and Manager objects in the memory, which should be
> recycled by GC, but they are not.
> !image-2019-08-12-17-22-23-477.png!
>
> *Root cause*
> Broadcaster hold the ref of Listener object defined in Manager, and the
> Listener object hold the ref of Manager, because the impl class of Listener
> is an inner class of Manager.
> Broadcaster.java
> {code:java}
> private Map> listenerMap = Maps.newConcurrentMap();
> {code}
> ProjectManager.java -> ProjectSyncListener
> {code:java}
> private class ProjectSyncListener extends Broadcaster.Listener {
> @Override
> public void onEntityChange(Broadcaster broadcaster, String entity, Event
> event, String cacheKey)
> throws IOException {
> String project = cacheKey;
> if (event == Event.DROP) {
> removeProjectLocal(project);
> return;
> }
>
[jira] [Created] (KYLIN-4131) Broadcaster memory leak
Chao Long created KYLIN-4131:
Summary: Broadcaster memory leak
Key: KYLIN-4131
URL: https://issues.apache.org/jira/browse/KYLIN-4131
Project: Kylin
Issue Type: Bug
Components: Metadata
Reporter: Chao Long
Assignee: Chao Long
Fix For: v3.0.0
Attachments: image-2019-08-12-16-39-46-262.png,
image-2019-08-12-16-39-52-327.png, image-2019-08-12-17-16-05-532.png,
image-2019-08-12-17-22-15-783.png, image-2019-08-12-17-22-23-477.png
h3. *Test evidence*
Use MAT(Memory Analyzer) to analyze this issue.
# start Kylin, dump a jvm heap info snapshot and import into MAT. We can see
there is only one Broadcaster and ProjectManager object.
!image-2019-08-12-17-22-15-783.png!
# Call 'list cube' api and 'reload metadata' api many times. We can see there
are many BroadCaster and Manager objects in the memory, which should be
recycled by GC, but they are not.
!image-2019-08-12-17-22-23-477.png!
*Root cause*
Broadcaster hold the ref of Listener object defined in Manager, and the
Listener object hold the ref of Manager, because the impl class of Listener is
an inner class of Manager.
Broadcaster.java
{code:java}
private Map> listenerMap = Maps.newConcurrentMap();
{code}
ProjectManager.java -> ProjectSyncListener
{code:java}
private class ProjectSyncListener extends Broadcaster.Listener {
@Override
public void onEntityChange(Broadcaster broadcaster, String entity, Event event,
String cacheKey)
throws IOException {
String project = cacheKey;
if (event == Event.DROP) {
removeProjectLocal(project);
return;
}
reloadProjectQuietly(project);
broadcaster.notifyProjectSchemaUpdate(project);
broadcaster.notifyProjectDataUpdate(project);
}
}
{code}
And Broadcaster can't be recycled by GC because it holds a running thread,
which cause the Manager object also not being recycled.
*After Fix*
The Broadcaster objects and Manager objects are recycled by GC.
!image-2019-08-12-17-16-05-532.png!
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (KYLIN-4130) Coordinator->StreamingBuildJobStatusChecker thread always hold a old CubeManager
Chao Long created KYLIN-4130:
Summary: Coordinator->StreamingBuildJobStatusChecker thread always
hold a old CubeManager
Key: KYLIN-4130
URL: https://issues.apache.org/jira/browse/KYLIN-4130
Project: Kylin
Issue Type: Improvement
Components: Real-time Streaming
Reporter: Chao Long
Assignee: Chao Long
Fix For: v3.0.0
{code}
private class StreamingBuildJobStatusChecker implements Runnable {
private int maxJobTryCnt = 5;
private CubeManager cubeManager =
CubeManager.getInstance(KylinConfig.getInstanceFromEnv());
private ConcurrentMap> segmentBuildJobMap = Maps
.newConcurrentMap();
private CopyOnWriteArrayList pendingCubeName =
Lists.newCopyOnWriteArrayList();
{code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (KYLIN-4108) Show slow query hit cube in slow query page
Chao Long created KYLIN-4108: Summary: Show slow query hit cube in slow query page Key: KYLIN-4108 URL: https://issues.apache.org/jira/browse/KYLIN-4108 Project: Kylin Issue Type: Improvement Components: Metrics Reporter: Chao Long Assignee: Chao Long Fix For: v3.0.0 -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (KYLIN-4085) Segment parallel building may cause segment not found
[ https://issues.apache.org/jira/browse/KYLIN-4085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long updated KYLIN-4085: - Fix Version/s: (was: v3.0.0) v3.0.0-beta > Segment parallel building may cause segment not found > - > > Key: KYLIN-4085 > URL: https://issues.apache.org/jira/browse/KYLIN-4085 > Project: Kylin > Issue Type: Bug > Components: Metadata >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v3.0.0-beta > > > In the case of multi-node and parallel building one same cube, the > JDBCResourceStore split the update metadata step in two sql, which can't > guarantee the atomicity. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (KYLIN-4089) Integration test failed with JDBCMetastore
[ https://issues.apache.org/jira/browse/KYLIN-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long updated KYLIN-4089: - Fix Version/s: (was: v3.0.0) v3.0.0-beta > Integration test failed with JDBCMetastore > -- > > Key: KYLIN-4089 > URL: https://issues.apache.org/jira/browse/KYLIN-4089 > Project: Kylin > Issue Type: Bug > Components: Integration, Tools, Build and Test >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v3.0.0-beta > > > [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.822 > s <<< FAILURE! - in > org.apache.kylin.storage.hbase.ITAclTableMigrationToolTest[ERROR] > testBasic(org.apache.kylin.storage.hbase.ITAclTableMigrationToolTest) Time > elapsed: 0.812 s <<< ERROR!java.lang.NullPointerException > at > org.apache.kylin.storage.hbase.ITAclTableMigrationToolTest.testBasic(ITAclTableMigrationToolTest.java:95) -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (KYLIN-4085) Segment parallel building may cause segment not found
[ https://issues.apache.org/jira/browse/KYLIN-4085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long updated KYLIN-4085: - Fix Version/s: (was: v3.0.0-alpha2) v3.0.0 > Segment parallel building may cause segment not found > - > > Key: KYLIN-4085 > URL: https://issues.apache.org/jira/browse/KYLIN-4085 > Project: Kylin > Issue Type: Bug > Components: Metadata >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v3.0.0 > > > In the case of multi-node and parallel building one same cube, the > JDBCResourceStore split the update metadata step in two sql, which can't > guarantee the atomicity. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (KYLIN-4089) Integration test failed with JDBCMetastore
[ https://issues.apache.org/jira/browse/KYLIN-4089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long updated KYLIN-4089: - Fix Version/s: (was: v3.0.0-alpha2) v3.0.0 > Integration test failed with JDBCMetastore > -- > > Key: KYLIN-4089 > URL: https://issues.apache.org/jira/browse/KYLIN-4089 > Project: Kylin > Issue Type: Bug > Components: Integration, Tools, Build and Test >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v3.0.0 > > > [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.822 > s <<< FAILURE! - in > org.apache.kylin.storage.hbase.ITAclTableMigrationToolTest[ERROR] > testBasic(org.apache.kylin.storage.hbase.ITAclTableMigrationToolTest) Time > elapsed: 0.812 s <<< ERROR!java.lang.NullPointerException > at > org.apache.kylin.storage.hbase.ITAclTableMigrationToolTest.testBasic(ITAclTableMigrationToolTest.java:95) -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (KYLIN-4085) Segment parallel building may cause segment not found
[ https://issues.apache.org/jira/browse/KYLIN-4085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long updated KYLIN-4085: - Fix Version/s: v3.0.0-alpha2 > Segment parallel building may cause segment not found > - > > Key: KYLIN-4085 > URL: https://issues.apache.org/jira/browse/KYLIN-4085 > Project: Kylin > Issue Type: Bug > Components: Metadata >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v3.0.0-alpha2 > > > In the case of multi-node and parallel building one same cube, the > JDBCResourceStore split the update metadata step in two sql, which can't > guarantee the atomicity. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (KYLIN-4089) Integration test failed with JDBCMetastore
Chao Long created KYLIN-4089: Summary: Integration test failed with JDBCMetastore Key: KYLIN-4089 URL: https://issues.apache.org/jira/browse/KYLIN-4089 Project: Kylin Issue Type: Bug Components: Integration, Tools, Build and Test Reporter: Chao Long Assignee: Chao Long Fix For: v3.0.0-alpha2 [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.822 s <<< FAILURE! - in org.apache.kylin.storage.hbase.ITAclTableMigrationToolTest[ERROR] testBasic(org.apache.kylin.storage.hbase.ITAclTableMigrationToolTest) Time elapsed: 0.812 s <<< ERROR!java.lang.NullPointerException at org.apache.kylin.storage.hbase.ITAclTableMigrationToolTest.testBasic(ITAclTableMigrationToolTest.java:95) -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (KYLIN-4085) Segment parallel building may cause segment not found
[ https://issues.apache.org/jira/browse/KYLIN-4085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long updated KYLIN-4085: - Description: In the case of multi-node and parallel building one same cube, the JDBCResourceStore split the update metadata step in two sql, which can't guarantee the atomicity. > Segment parallel building may cause segment not found > - > > Key: KYLIN-4085 > URL: https://issues.apache.org/jira/browse/KYLIN-4085 > Project: Kylin > Issue Type: Bug > Components: Metadata >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > > In the case of multi-node and parallel building one same cube, the > JDBCResourceStore split the update metadata step in two sql, which can't > guarantee the atomicity. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (KYLIN-4085) Segment parallel building may cause segment not found
Chao Long created KYLIN-4085: Summary: Segment parallel building may cause segment not found Key: KYLIN-4085 URL: https://issues.apache.org/jira/browse/KYLIN-4085 Project: Kylin Issue Type: Bug Components: Metadata Reporter: Chao Long Assignee: Chao Long -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (KYLIN-4064) parameter 'engineType' is not working when running integration test
Chao Long created KYLIN-4064: Summary: parameter 'engineType' is not working when running integration test Key: KYLIN-4064 URL: https://issues.apache.org/jira/browse/KYLIN-4064 Project: Kylin Issue Type: Bug Components: Integration Reporter: Chao Long Assignee: Chao Long Fix For: v3.0.0-beta -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-4049) Refresh segment job will always delete old segment storage
[ https://issues.apache.org/jira/browse/KYLIN-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long updated KYLIN-4049: - Affects Version/s: v3.0.0-alpha v2.6.2 > Refresh segment job will always delete old segment storage > -- > > Key: KYLIN-4049 > URL: https://issues.apache.org/jira/browse/KYLIN-4049 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.6.2, v3.0.0-alpha >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v3.0.0, v2.6.3 > > > KYLIN-3839 give a config to allow cleaning the storage after refreshing or > deleting a segment, but it doesn't cover refreshing job. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-4049) Refresh segment job will always delete old segment storage
[ https://issues.apache.org/jira/browse/KYLIN-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long resolved KYLIN-4049. -- Resolution: Fixed > Refresh segment job will always delete old segment storage > -- > > Key: KYLIN-4049 > URL: https://issues.apache.org/jira/browse/KYLIN-4049 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v3.0.0, v2.6.3 > > > KYLIN-3839 give a config to allow cleaning the storage after refreshing or > deleting a segment, but it doesn't cover refreshing job. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-4049) Refresh segment job will always delete old segment storage
[ https://issues.apache.org/jira/browse/KYLIN-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long updated KYLIN-4049: - Description: KYLIN-3839 give a config to allow cleaning the storage after refreshing or deleting a segment, but it doesn't cover refreshing job. > Refresh segment job will always delete old segment storage > -- > > Key: KYLIN-4049 > URL: https://issues.apache.org/jira/browse/KYLIN-4049 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v3.0.0, v2.6.3 > > > KYLIN-3839 give a config to allow cleaning the storage after refreshing or > deleting a segment, but it doesn't cover refreshing job. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-4049) Refresh segment job will always delete old segment storage
Chao Long created KYLIN-4049: Summary: Refresh segment job will always delete old segment storage Key: KYLIN-4049 URL: https://issues.apache.org/jira/browse/KYLIN-4049 Project: Kylin Issue Type: Bug Components: Job Engine Reporter: Chao Long Assignee: Chao Long Fix For: v3.0.0, v2.6.3 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-4045) ParquetMROutput miss megering cuboid data step
[ https://issues.apache.org/jira/browse/KYLIN-4045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long updated KYLIN-4045: - Component/s: Storage - Parquet > ParquetMROutput miss megering cuboid data step > -- > > Key: KYLIN-4045 > URL: https://issues.apache.org/jira/browse/KYLIN-4045 > Project: Kylin > Issue Type: Bug > Components: Storage - Parquet >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: Future > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-4045) ParquetMROutput miss megering cuboid data step
Chao Long created KYLIN-4045: Summary: ParquetMROutput miss megering cuboid data step Key: KYLIN-4045 URL: https://issues.apache.org/jira/browse/KYLIN-4045 Project: Kylin Issue Type: Bug Reporter: Chao Long Assignee: Chao Long Fix For: Future -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3998) Make "bpus-min-benefit-ratio" configurable in cube planner phase 1
[ https://issues.apache.org/jira/browse/KYLIN-3998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long updated KYLIN-3998: - Description: When I was testing the impact of parameter “bpus-min-benefit-ratio” on cube-planner phase 1, I found the parameter always uses the default value and can't be overwritten. > Make "bpus-min-benefit-ratio" configurable in cube planner phase 1 > -- > > Key: KYLIN-3998 > URL: https://issues.apache.org/jira/browse/KYLIN-3998 > Project: Kylin > Issue Type: Improvement >Reporter: Chao Long >Assignee: Chao Long >Priority: Minor > Fix For: Future > > > When I was testing the impact of parameter “bpus-min-benefit-ratio” on > cube-planner phase 1, I found the parameter always uses the default value and > can't be overwritten. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3998) Make "bpus-min-benefit-ratio" configurable in cube planner phase 1
Chao Long created KYLIN-3998: Summary: Make "bpus-min-benefit-ratio" configurable in cube planner phase 1 Key: KYLIN-3998 URL: https://issues.apache.org/jira/browse/KYLIN-3998 Project: Kylin Issue Type: Improvement Reporter: Chao Long Assignee: Chao Long Fix For: Future -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3993) Sum over a case statement fails with integer data type
[
https://issues.apache.org/jira/browse/KYLIN-3993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16829898#comment-16829898
]
Chao Long commented on KYLIN-3993:
--
Hi [~anoop.krishnaswamy],
Is the return type of sum measure the same as the real type of column
'accountmonthsopen'?
If not, you can take a look at this issue KYLIN-3957, it's similar to yours.
> Sum over a case statement fails with integer data type
> --
>
> Key: KYLIN-3993
> URL: https://issues.apache.org/jira/browse/KYLIN-3993
> Project: Kylin
> Issue Type: Bug
> Components: Driver - ODBC
>Affects Versions: v2.6.1
>Reporter: Anoop Krishnaswamy
>Priority: Major
>
> Hi,
> When we use sum over a case statement, it works when the field is of a
> certain type like decimal(19,3). But the same fails when used with integer
> field:
> This works:
> select sum(case when balance_indicator='1' then cyc_xxx_balanceadbavg else 0
> end) , balance_indicator from fct_profit_table_quarterly
> group by balance_indicator
> Whereas the below sql fails
> select sum(case when balance_indicator='1' then accountmonthsopen else 0 end)
> , balance_indicator from fct_profit_table_quarterly
> group by balance_indicator
>
> with the error message:
> {color:#b94a48}java.math.BigDecimal cannot be cast to java.lang.Integer while
> executing SQL: "select sum(case when balance_indicator='1' then
> accountmonthsopen else 0 end) , balance_indicator from
> fct_profit_table_quarterly group by balance_indicator LIMIT 5"{color}
>
> {color:#b94a48}The difference between 1st sql and 2nd sql being - in the
> first query the metric used in the case statement is of decimal(19,4) and the
> metric used in 2nd statement is of type int.{color}
>
> {color:#b94a48}Kindly help{color}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3957) Query system_cube get exception Cannot cast "java.math.BigDecimal" to "java.lang.Double"
[
https://issues.apache.org/jira/browse/KYLIN-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Long updated KYLIN-3957:
-
Affects Version/s: v2.6.1
> Query system_cube get exception Cannot cast "java.math.BigDecimal" to
> "java.lang.Double"
>
>
> Key: KYLIN-3957
> URL: https://issues.apache.org/jira/browse/KYLIN-3957
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v2.6.1
>Reporter: Chao Long
>Assignee: Chao Long
>Priority: Major
> Fix For: v3.0.0, v2.6.2
>
>
> In system cube, the return dataType of column whose real dataType is Double
> will be converted to Decimal in SUM measure.
> {code:java}
> FunctionDesc function = new FunctionDesc();
> function.setExpression(FunctionDesc.FUNC_SUM);
> function.setParameter(parameterDesc);
>
> function.setReturnType(dataType.equals(HiveTableCreator.HiveTypeEnum.HDOUBLE.toString())
> ? HiveTableCreator.HiveTypeEnum.HDECIMAL.toString()
> : dataType);
> {code}
> but query with that measure will get exception:
> {code}
> Caused by: org.codehaus.commons.compiler.CompileException: Line 108, Column
> 44: Cannot cast "java.math.BigDecimal" to "java.lang.Double"
> at
> org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:10092)
> at org.codehaus.janino.UnitCompiler.compileGet2(UnitCompiler.java:3839)
> at org.codehaus.janino.UnitCompiler.access$6400(UnitCompiler.java:183)
> at org.codehaus.janino.UnitCompiler$10.visitCast(UnitCompiler.java:3246)
> at org.codehaus.janino.Java$Cast.accept(Java.java:3802)
> at org.codehaus.janino.UnitCompiler.compileGet(UnitCompiler.java:3278)
> at org.codehaus.janino.UnitCompiler.compileGet2(UnitCompiler.java:3845)
> at org.codehaus.janino.UnitCompiler.access$8600(UnitCompiler.java:183)
> at
> org.codehaus.janino.UnitCompiler$10.visitParenthesizedExpression(UnitCompiler.java:3274)
> at
> org.codehaus.janino.Java$ParenthesizedExpression.accept(Java.java:3830)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3987) Give more reducer for UHC column in fact distinct job
[ https://issues.apache.org/jira/browse/KYLIN-3987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long updated KYLIN-3987: - Description: The defualt value of the reducer number for UHC column in fact distinct job is too small(default 1), the fact distinct job may be very slow, even throw OOM exception. We should give it a large value by default. > Give more reducer for UHC column in fact distinct job > - > > Key: KYLIN-3987 > URL: https://issues.apache.org/jira/browse/KYLIN-3987 > Project: Kylin > Issue Type: Bug >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v2.6.2 > > > The defualt value of the reducer number for UHC column in fact distinct job > is too small(default 1), the fact distinct job may be very slow, even throw > OOM exception. We should give it a large value by default. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3987) Give more reducer for UHC column in fact distinct job
Chao Long created KYLIN-3987: Summary: Give more reducer for UHC column in fact distinct job Key: KYLIN-3987 URL: https://issues.apache.org/jira/browse/KYLIN-3987 Project: Kylin Issue Type: Bug Reporter: Chao Long Assignee: Chao Long Fix For: v2.6.2 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3976) Failed to find metadata store by url: kylin_metadata@hbase
[ https://issues.apache.org/jira/browse/KYLIN-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16824752#comment-16824752 ] Chao Long commented on KYLIN-3976: -- Hi, Kylin-2.3.2 does not support hadoop3, and you can download Kylin binary package(2.5.2 or above) for hadoop3 on the website. > Failed to find metadata store by url: kylin_metadata@hbase > -- > > Key: KYLIN-3976 > URL: https://issues.apache.org/jira/browse/KYLIN-3976 > Project: Kylin > Issue Type: Bug > Components: Metadata > Environment: > kylin2.3.2+hadoop3.1.2+hive2.3.4+hbase1.2.0+zookeeper3.4.5 >Reporter: ZhangXihao >Priority: Blocker > Original Estimate: 48h > Remaining Estimate: 48h > > 2019-04-24 17:30:01,894 DEBUG [main] hbase.HBaseConnection:337 : Creating > HTable 'kylin_metadata' > 2019-04-24 17:30:01,936 INFO [main] Configuration.deprecation:1174 : > hadoop.native.lib is deprecated. Instead, use io.native.lib.available > 2019-04-24 17:30:02,089 DEBUG [main] util.ZookeeperDistributedLock:223 : > 7116@ambari02 trying to unlock > /kylin/kylin_metadata/create_htable/kylin_metadata/lock > 2019-04-24 17:30:02,099 INFO [main] util.ZookeeperDistributedLock:234 : > 7116@ambari02 released lock at > /kylin/kylin_metadata/create_htable/kylin_metadata/lock > Exception in thread "main" java.lang.IllegalArgumentException: Failed to find > metadata store by url: kylin_metadata@hbase > at > org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:96) > at > org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:108) > at > org.apache.kylin.rest.service.AclTableMigrationTool.checkIfNeedMigrate(AclTableMigrationTool.java:94) > at > org.apache.kylin.tool.AclTableMigrationCLI.main(AclTableMigrationCLI.java:41) > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at > org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:90) > ... 3 more > Caused by: java.lang.NoSuchMethodError: > org.apache.hadoop.hbase.HTableDescriptor.addFamily(Lorg/apache/hadoop/hbase/HColumnDescriptor;)Lorg/apache/hadoop/hbase/HTableDescriptor; > at > org.apache.kylin.storage.hbase.HBaseConnection.createHTableIfNeeded(HBaseConnection.java:344) > at > org.apache.kylin.storage.hbase.HBaseResourceStore.createHTableIfNeeded(HBaseResourceStore.java:111) > at > org.apache.kylin.storage.hbase.HBaseResourceStore.(HBaseResourceStore.java:92) > ... 8 more > 2019-04-24 17:30:02,105 INFO [close-hbase-conn] hbase.HBaseConnection:137 : > Closing HBase connections... > 2019-04-24 17:30:02,107 INFO [close-hbase-conn] > client.ConnectionManager$HConnectionImplementation:1830 : Closing zookeeper > sessionid=0x16a4c1bd0c80008 > 2019-04-24 17:30:02,111 INFO [close-hbase-conn] zookeeper.ZooKeeper:684 : > Session: 0x16a4c1bd0c80008 closed > 2019-04-24 17:30:02,111 INFO [main-EventThread] zookeeper.ClientCnxn:512 : > EventThread shut down > 2019-04-24 17:30:02,114 INFO [Thread-7] zookeeper.ZooKeeper:684 : Session: > 0x16a4c1bd0c80009 closed > 2019-04-24 17:30:02,114 INFO [main-EventThread] zookeeper.ClientCnxn:512 : > EventThread shut down -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3948) org.apache.hadoop.hive.metastore.HiveMetaStoreClint.(Lorg/apache/hadoop/conf/configuration;)V
[ https://issues.apache.org/jira/browse/KYLIN-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16820645#comment-16820645 ] Chao Long commented on KYLIN-3948: -- Hi [~praveenece], kylin-2.5.2 for hadoop3 not support spark engine yet. > org.apache.hadoop.hive.metastore.HiveMetaStoreClint.(Lorg/apache/hadoop/conf/configuration;)V > --- > > Key: KYLIN-3948 > URL: https://issues.apache.org/jira/browse/KYLIN-3948 > Project: Kylin > Issue Type: Test > Components: Environment >Affects Versions: v2.5.2 > Environment: hadoop-3.1.1 > apache-hive-2.3.4-bin > hbase-2.1.1 > apache-kylin-2.5.2-bin-hadoop3 >Reporter: praveenece >Priority: Critical > > hadoop-3.1.1 > apache-hive-2.3.4-bin > hbase-2.1.1 > apache-kylin-2.5.2-bin-hadoop3 > Now I tried with above environment, kylin is stated but i unable to load > metadata from hbase to kylin getting this below error in kylin UI. > org.apache.hadoop.hive.metastore.HiveMetaStoreClint.(Lorg/apache/hadoop/conf/configuration;)V > 2019-04-09 15:02:44,979 DEBUG [http-nio-7070-exec-9] common.KylinConfig:327 : > KYLIN_CONF property was not set, will seek KYLIN_HOME env variable > 2019-04-09 15:02:50,071 ERROR [http-nio-7070-exec-6] > controller.TableController:190 : > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(Lorg/apache/hadoop/conf/Configuration;)V > java.lang.NoSuchMethodError: > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(Lorg/apache/hadoop/conf/Configuration;)V > at > org.apache.kylin.source.hive.CLIHiveClient.getMetaStoreClient(CLIHiveClient.java:132) > at > org.apache.kylin.source.hive.CLIHiveClient.getHiveDbNames(CLIHiveClient.java:116) > at > org.apache.kylin.source.hive.HiveMetadataExplorer.listDatabases(HiveMetadataExplorer.java:47) > at > org.apache.kylin.rest.service.TableService.getSourceDbNames(TableService.java:277) > at > org.apache.kylin.rest.controller.TableController.showHiveDatabases(TableController.java:188) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) > at > org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3957) Query system_cube get exception Cannot cast "java.math.BigDecimal" to "java.lang.Double"
[
https://issues.apache.org/jira/browse/KYLIN-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Long resolved KYLIN-3957.
--
Resolution: Fixed
Fix Version/s: Future
> Query system_cube get exception Cannot cast "java.math.BigDecimal" to
> "java.lang.Double"
>
>
> Key: KYLIN-3957
> URL: https://issues.apache.org/jira/browse/KYLIN-3957
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Reporter: Chao Long
>Assignee: Chao Long
>Priority: Major
> Fix For: Future
>
>
> In system cube, the return dataType of column whose real dataType is Double
> will be converted to Decimal in SUM measure.
> {code:java}
> FunctionDesc function = new FunctionDesc();
> function.setExpression(FunctionDesc.FUNC_SUM);
> function.setParameter(parameterDesc);
>
> function.setReturnType(dataType.equals(HiveTableCreator.HiveTypeEnum.HDOUBLE.toString())
> ? HiveTableCreator.HiveTypeEnum.HDECIMAL.toString()
> : dataType);
> {code}
> but query with that measure will get exception:
> {code}
> Caused by: org.codehaus.commons.compiler.CompileException: Line 108, Column
> 44: Cannot cast "java.math.BigDecimal" to "java.lang.Double"
> at
> org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:10092)
> at org.codehaus.janino.UnitCompiler.compileGet2(UnitCompiler.java:3839)
> at org.codehaus.janino.UnitCompiler.access$6400(UnitCompiler.java:183)
> at org.codehaus.janino.UnitCompiler$10.visitCast(UnitCompiler.java:3246)
> at org.codehaus.janino.Java$Cast.accept(Java.java:3802)
> at org.codehaus.janino.UnitCompiler.compileGet(UnitCompiler.java:3278)
> at org.codehaus.janino.UnitCompiler.compileGet2(UnitCompiler.java:3845)
> at org.codehaus.janino.UnitCompiler.access$8600(UnitCompiler.java:183)
> at
> org.codehaus.janino.UnitCompiler$10.visitParenthesizedExpression(UnitCompiler.java:3274)
> at
> org.codehaus.janino.Java$ParenthesizedExpression.accept(Java.java:3830)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3948) org.apache.hadoop.hive.metastore.HiveMetaStoreClint.(Lorg/apache/hadoop/conf/configuration;)V
[ https://issues.apache.org/jira/browse/KYLIN-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818761#comment-16818761 ] Chao Long commented on KYLIN-3948: -- Hi [~praveenece], you can try to use the latest binary for hadoop3(apache-kylin-2.6.1-bin-hadoop3.tar.gz). > org.apache.hadoop.hive.metastore.HiveMetaStoreClint.(Lorg/apache/hadoop/conf/configuration;)V > --- > > Key: KYLIN-3948 > URL: https://issues.apache.org/jira/browse/KYLIN-3948 > Project: Kylin > Issue Type: Test > Components: Environment >Affects Versions: v2.5.2 > Environment: hadoop-3.1.1 > apache-hive-2.3.4-bin > hbase-2.1.1 > apache-kylin-2.5.2-bin-hadoop3 >Reporter: praveenece >Priority: Critical > > hadoop-3.1.1 > apache-hive-2.3.4-bin > hbase-2.1.1 > apache-kylin-2.5.2-bin-hadoop3 > Now I tried with above environment, kylin is stated but i unable to load > metadata from hbase to kylin getting this below error in kylin UI. > org.apache.hadoop.hive.metastore.HiveMetaStoreClint.(Lorg/apache/hadoop/conf/configuration;)V > 2019-04-09 15:02:44,979 DEBUG [http-nio-7070-exec-9] common.KylinConfig:327 : > KYLIN_CONF property was not set, will seek KYLIN_HOME env variable > 2019-04-09 15:02:50,071 ERROR [http-nio-7070-exec-6] > controller.TableController:190 : > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(Lorg/apache/hadoop/conf/Configuration;)V > java.lang.NoSuchMethodError: > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(Lorg/apache/hadoop/conf/Configuration;)V > at > org.apache.kylin.source.hive.CLIHiveClient.getMetaStoreClient(CLIHiveClient.java:132) > at > org.apache.kylin.source.hive.CLIHiveClient.getHiveDbNames(CLIHiveClient.java:116) > at > org.apache.kylin.source.hive.HiveMetadataExplorer.listDatabases(HiveMetadataExplorer.java:47) > at > org.apache.kylin.rest.service.TableService.getSourceDbNames(TableService.java:277) > at > org.apache.kylin.rest.controller.TableController.showHiveDatabases(TableController.java:188) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) > at > org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3948) org.apache.hadoop.hive.metastore.HiveMetaStoreClint.(Lorg/apache/hadoop/conf/configuration;)V
[
https://issues.apache.org/jira/browse/KYLIN-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16818561#comment-16818561
]
Chao Long commented on KYLIN-3948:
--
Hi [~praveenece], Kylin Hadoop3 branch use the API of Hive3.1.0 to get
_HiveMetaStoreClint,_ and in Hive2.3.4 has no such API, so it throw
_java.lang.NoSuchMethodError_ exception.
{code:java}
3.1.0
{code}
{code:java}
private HiveMetaStoreClient getMetaStoreClient() throws Exception {
if (metaStoreClient == null) {
metaStoreClient = new HiveMetaStoreClient(conf);
}
return metaStoreClient;
}
{code}
I think you also need to upgrade the Hive to Hive3.1.0.
> org.apache.hadoop.hive.metastore.HiveMetaStoreClint.(Lorg/apache/hadoop/conf/configuration;)V
> ---
>
> Key: KYLIN-3948
> URL: https://issues.apache.org/jira/browse/KYLIN-3948
> Project: Kylin
> Issue Type: Test
> Components: Environment
>Affects Versions: v2.5.2
> Environment: hadoop-3.1.1
> apache-hive-2.3.4-bin
> hbase-2.1.1
> apache-kylin-2.5.2-bin-hadoop3
>Reporter: praveenece
>Priority: Critical
>
> hadoop-3.1.1
> apache-hive-2.3.4-bin
> hbase-2.1.1
> apache-kylin-2.5.2-bin-hadoop3
> Now I tried with above environment, kylin is stated but i unable to load
> metadata from hbase to kylin getting this below error in kylin UI.
> org.apache.hadoop.hive.metastore.HiveMetaStoreClint.(Lorg/apache/hadoop/conf/configuration;)V
> 2019-04-09 15:02:44,979 DEBUG [http-nio-7070-exec-9] common.KylinConfig:327 :
> KYLIN_CONF property was not set, will seek KYLIN_HOME env variable
> 2019-04-09 15:02:50,071 ERROR [http-nio-7070-exec-6]
> controller.TableController:190 :
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(Lorg/apache/hadoop/conf/Configuration;)V
> java.lang.NoSuchMethodError:
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(Lorg/apache/hadoop/conf/Configuration;)V
> at
> org.apache.kylin.source.hive.CLIHiveClient.getMetaStoreClient(CLIHiveClient.java:132)
> at
> org.apache.kylin.source.hive.CLIHiveClient.getHiveDbNames(CLIHiveClient.java:116)
> at
> org.apache.kylin.source.hive.HiveMetadataExplorer.listDatabases(HiveMetadataExplorer.java:47)
> at
> org.apache.kylin.rest.service.TableService.getSourceDbNames(TableService.java:277)
> at
> org.apache.kylin.rest.controller.TableController.showHiveDatabases(TableController.java:188)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
> at
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3957) Query system_cube get exception Cannot cast "java.math.BigDecimal" to "java.lang.Double"
[
https://issues.apache.org/jira/browse/KYLIN-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Long updated KYLIN-3957:
-
Description:
In system cube, the return dataType of column whose real dataType is Double
will be converted to Decimal in SUM measure.
{code:java}
FunctionDesc function = new FunctionDesc();
function.setExpression(FunctionDesc.FUNC_SUM);
function.setParameter(parameterDesc);
function.setReturnType(dataType.equals(HiveTableCreator.HiveTypeEnum.HDOUBLE.toString())
? HiveTableCreator.HiveTypeEnum.HDECIMAL.toString()
: dataType);
{code}
but query with that measure will get exception:
{code}
Caused by: org.codehaus.commons.compiler.CompileException: Line 108, Column 44:
Cannot cast "java.math.BigDecimal" to "java.lang.Double"
at
org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:10092)
at org.codehaus.janino.UnitCompiler.compileGet2(UnitCompiler.java:3839)
at org.codehaus.janino.UnitCompiler.access$6400(UnitCompiler.java:183)
at org.codehaus.janino.UnitCompiler$10.visitCast(UnitCompiler.java:3246)
at org.codehaus.janino.Java$Cast.accept(Java.java:3802)
at org.codehaus.janino.UnitCompiler.compileGet(UnitCompiler.java:3278)
at org.codehaus.janino.UnitCompiler.compileGet2(UnitCompiler.java:3845)
at org.codehaus.janino.UnitCompiler.access$8600(UnitCompiler.java:183)
at
org.codehaus.janino.UnitCompiler$10.visitParenthesizedExpression(UnitCompiler.java:3274)
at
org.codehaus.janino.Java$ParenthesizedExpression.accept(Java.java:3830)
{code}
> Query system_cube get exception Cannot cast "java.math.BigDecimal" to
> "java.lang.Double"
>
>
> Key: KYLIN-3957
> URL: https://issues.apache.org/jira/browse/KYLIN-3957
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Reporter: Chao Long
>Assignee: Chao Long
>Priority: Major
>
> In system cube, the return dataType of column whose real dataType is Double
> will be converted to Decimal in SUM measure.
> {code:java}
> FunctionDesc function = new FunctionDesc();
> function.setExpression(FunctionDesc.FUNC_SUM);
> function.setParameter(parameterDesc);
>
> function.setReturnType(dataType.equals(HiveTableCreator.HiveTypeEnum.HDOUBLE.toString())
> ? HiveTableCreator.HiveTypeEnum.HDECIMAL.toString()
> : dataType);
> {code}
> but query with that measure will get exception:
> {code}
> Caused by: org.codehaus.commons.compiler.CompileException: Line 108, Column
> 44: Cannot cast "java.math.BigDecimal" to "java.lang.Double"
> at
> org.codehaus.janino.UnitCompiler.compileError(UnitCompiler.java:10092)
> at org.codehaus.janino.UnitCompiler.compileGet2(UnitCompiler.java:3839)
> at org.codehaus.janino.UnitCompiler.access$6400(UnitCompiler.java:183)
> at org.codehaus.janino.UnitCompiler$10.visitCast(UnitCompiler.java:3246)
> at org.codehaus.janino.Java$Cast.accept(Java.java:3802)
> at org.codehaus.janino.UnitCompiler.compileGet(UnitCompiler.java:3278)
> at org.codehaus.janino.UnitCompiler.compileGet2(UnitCompiler.java:3845)
> at org.codehaus.janino.UnitCompiler.access$8600(UnitCompiler.java:183)
> at
> org.codehaus.janino.UnitCompiler$10.visitParenthesizedExpression(UnitCompiler.java:3274)
> at
> org.codehaus.janino.Java$ParenthesizedExpression.accept(Java.java:3830)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KYLIN-3950) Cube planner optimize job only use inmem algorithm
[ https://issues.apache.org/jira/browse/KYLIN-3950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long resolved KYLIN-3950. -- Resolution: Resolved Fix Version/s: (was: v2.6.2) Future > Cube planner optimize job only use inmem algorithm > -- > > Key: KYLIN-3950 > URL: https://issues.apache.org/jira/browse/KYLIN-3950 > Project: Kylin > Issue Type: Bug >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: Future > > > Currently, cube planner optimize job only use inmem algorithm to optimize > cube, which should be configurable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3938) can't delete job OPTIMIZE CHECKPOINT
[ https://issues.apache.org/jira/browse/KYLIN-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long updated KYLIN-3938: - Fix Version/s: (was: v2.6.2) Future > can't delete job OPTIMIZE CHECKPOINT > > > Key: KYLIN-3938 > URL: https://issues.apache.org/jira/browse/KYLIN-3938 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.6.1 >Reporter: kaishen >Assignee: Chao Long >Priority: Major > Fix For: Future > > > I'm using kylin-2.6.1-cdh57 ，when I use cube planner , it will generate tow > jobs one is : OPTIMIZE CUBE another is : OPTIMIZE CHECKPOINT which depend on > job OPTIMIZE CUBE . > now the first job OPTIMIZE CUBE was failed , and I want to discard the > second job , I got error : > java.lang.ClassCastException: > org.apache.kylin.job.execution.CheckpointExecutable cannot be cast to > org.apache.kylin.engine.mr.CubingJob -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3946) No cube for AVG measure after include count column
[ https://issues.apache.org/jira/browse/KYLIN-3946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long resolved KYLIN-3946. -- Resolution: Fixed Fix Version/s: (was: v2.6.2) Future > No cube for AVG measure after include count column > -- > > Key: KYLIN-3946 > URL: https://issues.apache.org/jira/browse/KYLIN-3946 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: Future > > > Before, avg(col) will use sum(col)/count(1) to calculate. > After including count column aggregation KYLIN-3883, avg(col) will use > sum(col)/count(col) to calculate. > If there is no predefined count(col) measure, query with avg(col) will get > exception "NoRealizationFoundException: No realization found for > OLAPContext", which will effect the query on old cubes. So we should consider > compatibility. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3938) can't delete job OPTIMIZE CHECKPOINT
[ https://issues.apache.org/jira/browse/KYLIN-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long updated KYLIN-3938: - Fix Version/s: (was: v2.6.1) v2.6.2 > can't delete job OPTIMIZE CHECKPOINT > > > Key: KYLIN-3938 > URL: https://issues.apache.org/jira/browse/KYLIN-3938 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.6.1 >Reporter: kaishen >Assignee: Chao Long >Priority: Major > Fix For: v2.6.2 > > > I'm using kylin-2.6.1-cdh57 ，when I use cube planner , it will generate tow > jobs one is : OPTIMIZE CUBE another is : OPTIMIZE CHECKPOINT which depend on > job OPTIMIZE CUBE . > now the first job OPTIMIZE CUBE was failed , and I want to discard the > second job , I got error : > java.lang.ClassCastException: > org.apache.kylin.job.execution.CheckpointExecutable cannot be cast to > org.apache.kylin.engine.mr.CubingJob -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-3951) JDBC source Table name forced to uppercase
[ https://issues.apache.org/jira/browse/KYLIN-3951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long closed KYLIN-3951. Resolution: Duplicate > JDBC source Table name forced to uppercase > -- > > Key: KYLIN-3951 > URL: https://issues.apache.org/jira/browse/KYLIN-3951 > Project: Kylin > Issue Type: Bug > Components: Driver - JDBC >Affects Versions: v2.6.0 >Reporter: zhangxingzhi >Priority: Minor > Attachments: image-2019-04-12-09-34-28-478.png > > Original Estimate: 168h > Remaining Estimate: 168h > > When I use mysql jdbc source, build cube, get a bug, I check the sqoop > command and find that kylin converts the database name in my mysql to > uppercase, and my Database name is pub_dev. > !image-2019-04-12-09-34-28-478.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3951) JDBC source Table name forced to uppercase
[ https://issues.apache.org/jira/browse/KYLIN-3951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16815899#comment-16815899 ] Chao Long commented on KYLIN-3951: -- Duplicate with KYLIN-3779 There is a workaround in the comments under KYLIN-3779. > JDBC source Table name forced to uppercase > -- > > Key: KYLIN-3951 > URL: https://issues.apache.org/jira/browse/KYLIN-3951 > Project: Kylin > Issue Type: Bug > Components: Driver - JDBC >Affects Versions: v2.6.0 >Reporter: zhangxingzhi >Priority: Minor > Attachments: image-2019-04-12-09-34-28-478.png > > Original Estimate: 168h > Remaining Estimate: 168h > > When I use mysql jdbc source, build cube, get a bug, I check the sqoop > command and find that kylin converts the database name in my mysql to > uppercase, and my Database name is pub_dev. > !image-2019-04-12-09-34-28-478.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3950) Cube planner optimize job only use inmem algorithm
[ https://issues.apache.org/jira/browse/KYLIN-3950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long updated KYLIN-3950: - Description: Currently, cube planner optimize job only use inmem algorithm to optimize cube, which should be configurable. > Cube planner optimize job only use inmem algorithm > -- > > Key: KYLIN-3950 > URL: https://issues.apache.org/jira/browse/KYLIN-3950 > Project: Kylin > Issue Type: Bug >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v2.6.2 > > > Currently, cube planner optimize job only use inmem algorithm to optimize > cube, which should be configurable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3950) Cube planner optimize job only use inmem algorithm
Chao Long created KYLIN-3950: Summary: Cube planner optimize job only use inmem algorithm Key: KYLIN-3950 URL: https://issues.apache.org/jira/browse/KYLIN-3950 Project: Kylin Issue Type: Bug Reporter: Chao Long Assignee: Chao Long Fix For: v2.6.2 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3938) can't delete job OPTIMIZE CHECKPOINT
[ https://issues.apache.org/jira/browse/KYLIN-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long resolved KYLIN-3938. -- Resolution: Fixed > can't delete job OPTIMIZE CHECKPOINT > > > Key: KYLIN-3938 > URL: https://issues.apache.org/jira/browse/KYLIN-3938 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.6.1 >Reporter: kaishen >Assignee: Chao Long >Priority: Major > Fix For: v2.6.1 > > > I'm using kylin-2.6.1-cdh57 ，when I use cube planner , it will generate tow > jobs one is : OPTIMIZE CUBE another is : OPTIMIZE CHECKPOINT which depend on > job OPTIMIZE CUBE . > now the first job OPTIMIZE CUBE was failed , and I want to discard the > second job , I got error : > java.lang.ClassCastException: > org.apache.kylin.job.execution.CheckpointExecutable cannot be cast to > org.apache.kylin.engine.mr.CubingJob -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3946) No cube for AVG measure after include count column
[ https://issues.apache.org/jira/browse/KYLIN-3946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long updated KYLIN-3946: - Description: Before, avg(col) will use sum(col)/count(1) to calculate. After including count column aggregation KYLIN-3883, avg(col) will use sum(col)/count(col) to calculate. If there is no predefined count(col) measure, query with avg(col) will get exception "NoRealizationFoundException: No realization found for OLAPContext", which will effect the query on old cubes. So we should consider compatibility. > No cube for AVG measure after include count column > -- > > Key: KYLIN-3946 > URL: https://issues.apache.org/jira/browse/KYLIN-3946 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v2.6.2 > > > Before, avg(col) will use sum(col)/count(1) to calculate. > After including count column aggregation KYLIN-3883, avg(col) will use > sum(col)/count(col) to calculate. > If there is no predefined count(col) measure, query with avg(col) will get > exception "NoRealizationFoundException: No realization found for > OLAPContext", which will effect the query on old cubes. So we should consider > compatibility. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3946) No cube for AVG measure after include count column
Chao Long created KYLIN-3946: Summary: No cube for AVG measure after include count column Key: KYLIN-3946 URL: https://issues.apache.org/jira/browse/KYLIN-3946 Project: Kylin Issue Type: Bug Components: Query Engine Reporter: Chao Long Assignee: Chao Long Fix For: v2.6.2 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-3945) Debug in local get error "Invalid 'log4jConfigLocation' parameter"
[
https://issues.apache.org/jira/browse/KYLIN-3945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Long closed KYLIN-3945.
Resolution: Won't Fix
Fix Version/s: v2.6.1
root cause:
When debug in local, the env property(kylin.env) should be "DEV".
> Debug in local get error "Invalid 'log4jConfigLocation' parameter"
> --
>
> Key: KYLIN-3945
> URL: https://issues.apache.org/jira/browse/KYLIN-3945
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.6.2
>Reporter: Chao Long
>Assignee: Chao Long
>Priority: Major
> Fix For: v2.6.1
>
>
> {code}
> 严重: Exception sending context initialized event to listener instance of class
> org.apache.kylin.rest.util.Log4jConfigListener
> java.lang.IllegalArgumentException: Invalid 'log4jConfigLocation' parameter:
> class path resource [kylin-server-log4j.properties] cannot be resolved to URL
> because it does not exist
> at
> org.springframework.web.util.Log4jWebConfigurer.initLogging(Log4jWebConfigurer.java:158)
> at
> org.springframework.web.util.Log4jConfigListener.contextInitialized(Log4jConfigListener.java:49)
> at
> org.apache.kylin.rest.util.Log4jConfigListener.contextInitialized(Log4jConfigListener.java:36)
> at
> org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:5157)
> at
> org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5680)
> at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
> at
> org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1707)
> at
> org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1697)
> at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
> at java.util.concurrent.FutureTask.run(FutureTask.java)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3945) Debug in local get error "Invalid 'log4jConfigLocation' parameter"
[
https://issues.apache.org/jira/browse/KYLIN-3945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Long updated KYLIN-3945:
-
Description:
{code}
严重: Exception sending context initialized event to listener instance of class
org.apache.kylin.rest.util.Log4jConfigListener
java.lang.IllegalArgumentException: Invalid 'log4jConfigLocation' parameter:
class path resource [kylin-server-log4j.properties] cannot be resolved to URL
because it does not exist
at
org.springframework.web.util.Log4jWebConfigurer.initLogging(Log4jWebConfigurer.java:158)
at
org.springframework.web.util.Log4jConfigListener.contextInitialized(Log4jConfigListener.java:49)
at
org.apache.kylin.rest.util.Log4jConfigListener.contextInitialized(Log4jConfigListener.java:36)
at
org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:5157)
at
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5680)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
at
org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1707)
at
org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1697)
at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
at java.util.concurrent.FutureTask.run(FutureTask.java)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
was:
严重: Exception sending context initialized event to listener instance of class
org.apache.kylin.rest.util.Log4jConfigListener
java.lang.IllegalArgumentException: Invalid 'log4jConfigLocation' parameter:
class path resource [kylin-server-log4j.properties] cannot be resolved to URL
because it does not exist
at
org.springframework.web.util.Log4jWebConfigurer.initLogging(Log4jWebConfigurer.java:158)
at
org.springframework.web.util.Log4jConfigListener.contextInitialized(Log4jConfigListener.java:49)
at
org.apache.kylin.rest.util.Log4jConfigListener.contextInitialized(Log4jConfigListener.java:36)
at
org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:5157)
at
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5680)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
at
org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1707)
at
org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1697)
at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
at java.util.concurrent.FutureTask.run(FutureTask.java)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
> Debug in local get error "Invalid 'log4jConfigLocation' parameter"
> --
>
> Key: KYLIN-3945
> URL: https://issues.apache.org/jira/browse/KYLIN-3945
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.6.2
>Reporter: Chao Long
>Assignee: Chao Long
>Priority: Major
>
> {code}
> 严重: Exception sending context initialized event to listener instance of class
> org.apache.kylin.rest.util.Log4jConfigListener
> java.lang.IllegalArgumentException: Invalid 'log4jConfigLocation' parameter:
> class path resource [kylin-server-log4j.properties] cannot be resolved to URL
> because it does not exist
> at
> org.springframework.web.util.Log4jWebConfigurer.initLogging(Log4jWebConfigurer.java:158)
> at
> org.springframework.web.util.Log4jConfigListener.contextInitialized(Log4jConfigListener.java:49)
> at
> org.apache.kylin.rest.util.Log4jConfigListener.contextInitialized(Log4jConfigListener.java:36)
> at
> org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:5157)
> at
> org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5680)
> at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
> at
> org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1707)
> at
> org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1697)
> at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
> at java.util.concurrent.FutureTask.run(FutureTask.java)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was
[jira] [Created] (KYLIN-3945) Debug in local get error "Invalid 'log4jConfigLocation' parameter"
Chao Long created KYLIN-3945: Summary: Debug in local get error "Invalid 'log4jConfigLocation' parameter" Key: KYLIN-3945 URL: https://issues.apache.org/jira/browse/KYLIN-3945 Project: Kylin Issue Type: Bug Affects Versions: v2.6.2 Reporter: Chao Long Assignee: Chao Long 严重: Exception sending context initialized event to listener instance of class org.apache.kylin.rest.util.Log4jConfigListener java.lang.IllegalArgumentException: Invalid 'log4jConfigLocation' parameter: class path resource [kylin-server-log4j.properties] cannot be resolved to URL because it does not exist at org.springframework.web.util.Log4jWebConfigurer.initLogging(Log4jWebConfigurer.java:158) at org.springframework.web.util.Log4jConfigListener.contextInitialized(Log4jConfigListener.java:49) at org.apache.kylin.rest.util.Log4jConfigListener.contextInitialized(Log4jConfigListener.java:36) at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:5157) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5680) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1707) at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1697) at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266) at java.util.concurrent.FutureTask.run(FutureTask.java) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3943) system-cube.sh has some problems
[ https://issues.apache.org/jira/browse/KYLIN-3943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long updated KYLIN-3943: - Attachment: image-2019-04-09-23-07-19-229.png > system-cube.sh has some problems > > > Key: KYLIN-3943 > URL: https://issues.apache.org/jira/browse/KYLIN-3943 > Project: Kylin > Issue Type: Bug >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v2.6.2 > > Attachments: image-2019-04-09-18-50-35-111.png, > image-2019-04-09-23-07-19-229.png > > > 1, The name of system cube should not hard code. the suffix should get from > config. > !image-2019-04-09-18-50-35-111.png! > 2, server port should not hard code > !image-2019-04-09-23-07-19-229.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3943) system-cube.sh has some problems
[ https://issues.apache.org/jira/browse/KYLIN-3943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long updated KYLIN-3943: - Description: 1, The name of system cube should not hard code. the suffix should get from config. !image-2019-04-09-18-50-35-111.png! 2, server port should not hard code !image-2019-04-09-23-07-19-229.png! was: 1, The name of system cube should not hard code. the suffix should get from config. !image-2019-04-09-18-50-35-111.png! 2, Use CubeBuildingCLI instead of rest api to build cube. > system-cube.sh has some problems > > > Key: KYLIN-3943 > URL: https://issues.apache.org/jira/browse/KYLIN-3943 > Project: Kylin > Issue Type: Bug >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v2.6.2 > > Attachments: image-2019-04-09-18-50-35-111.png, > image-2019-04-09-23-07-19-229.png > > > 1, The name of system cube should not hard code. the suffix should get from > config. > !image-2019-04-09-18-50-35-111.png! > 2, server port should not hard code > !image-2019-04-09-23-07-19-229.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3943) system-cube.sh has some problems
[ https://issues.apache.org/jira/browse/KYLIN-3943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long updated KYLIN-3943: - Attachment: (was: image-2019-04-09-23-05-51-460.png) > system-cube.sh has some problems > > > Key: KYLIN-3943 > URL: https://issues.apache.org/jira/browse/KYLIN-3943 > Project: Kylin > Issue Type: Bug >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v2.6.2 > > Attachments: image-2019-04-09-18-50-35-111.png > > > 1, The name of system cube should not hard code. the suffix should get from > config. > !image-2019-04-09-18-50-35-111.png! > 2, Use CubeBuildingCLI instead of rest api to build cube. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3943) system-cube.sh has some problems
[ https://issues.apache.org/jira/browse/KYLIN-3943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long updated KYLIN-3943: - Attachment: image-2019-04-09-23-06-24-784.png > system-cube.sh has some problems > > > Key: KYLIN-3943 > URL: https://issues.apache.org/jira/browse/KYLIN-3943 > Project: Kylin > Issue Type: Bug >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v2.6.2 > > Attachments: image-2019-04-09-18-50-35-111.png > > > 1, The name of system cube should not hard code. the suffix should get from > config. > !image-2019-04-09-18-50-35-111.png! > 2, Use CubeBuildingCLI instead of rest api to build cube. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3943) system-cube.sh has some problems
[ https://issues.apache.org/jira/browse/KYLIN-3943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long updated KYLIN-3943: - Attachment: (was: image-2019-04-09-23-06-24-784.png) > system-cube.sh has some problems > > > Key: KYLIN-3943 > URL: https://issues.apache.org/jira/browse/KYLIN-3943 > Project: Kylin > Issue Type: Bug >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v2.6.2 > > Attachments: image-2019-04-09-18-50-35-111.png > > > 1, The name of system cube should not hard code. the suffix should get from > config. > !image-2019-04-09-18-50-35-111.png! > 2, Use CubeBuildingCLI instead of rest api to build cube. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3943) system-cube.sh has some problems
[ https://issues.apache.org/jira/browse/KYLIN-3943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long updated KYLIN-3943: - Attachment: image-2019-04-09-23-06-09-663.png > system-cube.sh has some problems > > > Key: KYLIN-3943 > URL: https://issues.apache.org/jira/browse/KYLIN-3943 > Project: Kylin > Issue Type: Bug >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v2.6.2 > > Attachments: image-2019-04-09-18-50-35-111.png > > > 1, The name of system cube should not hard code. the suffix should get from > config. > !image-2019-04-09-18-50-35-111.png! > 2, Use CubeBuildingCLI instead of rest api to build cube. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3943) system-cube.sh has some problems
[ https://issues.apache.org/jira/browse/KYLIN-3943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long updated KYLIN-3943: - Attachment: image-2019-04-09-23-05-51-460.png > system-cube.sh has some problems > > > Key: KYLIN-3943 > URL: https://issues.apache.org/jira/browse/KYLIN-3943 > Project: Kylin > Issue Type: Bug >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v2.6.2 > > Attachments: image-2019-04-09-18-50-35-111.png, > image-2019-04-09-23-05-51-460.png > > > 1, The name of system cube should not hard code. the suffix should get from > config. > !image-2019-04-09-18-50-35-111.png! > 2, Use CubeBuildingCLI instead of rest api to build cube. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3943) system-cube.sh has some problems
[ https://issues.apache.org/jira/browse/KYLIN-3943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long updated KYLIN-3943: - Attachment: (was: screenshot-1.png) > system-cube.sh has some problems > > > Key: KYLIN-3943 > URL: https://issues.apache.org/jira/browse/KYLIN-3943 > Project: Kylin > Issue Type: Bug >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v2.6.2 > > Attachments: image-2019-04-09-18-50-35-111.png > > > 1, The name of system cube should not hard code. the suffix should get from > config. > !image-2019-04-09-18-50-35-111.png! > 2, Use CubeBuildingCLI instead of rest api to build cube. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3943) system-cube.sh has some problems
[ https://issues.apache.org/jira/browse/KYLIN-3943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long updated KYLIN-3943: - Attachment: screenshot-1.png > system-cube.sh has some problems > > > Key: KYLIN-3943 > URL: https://issues.apache.org/jira/browse/KYLIN-3943 > Project: Kylin > Issue Type: Bug >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v2.6.2 > > Attachments: image-2019-04-09-18-50-35-111.png, screenshot-1.png > > > 1, The name of system cube should not hard code. the suffix should get from > config. > !image-2019-04-09-18-50-35-111.png! > 2, Use CubeBuildingCLI instead of rest api to build cube. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3943) system-cube.sh has some problems
[ https://issues.apache.org/jira/browse/KYLIN-3943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long updated KYLIN-3943: - Description: 1, The name of system cube should not hard code. the suffix should get from config. !image-2019-04-09-18-50-35-111.png! 2, Use CubeBuildingCLI instead of rest api to build cube. > system-cube.sh has some problems > > > Key: KYLIN-3943 > URL: https://issues.apache.org/jira/browse/KYLIN-3943 > Project: Kylin > Issue Type: Bug >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v2.6.2 > > Attachments: image-2019-04-09-18-50-35-111.png > > > 1, The name of system cube should not hard code. the suffix should get from > config. > !image-2019-04-09-18-50-35-111.png! > 2, Use CubeBuildingCLI instead of rest api to build cube. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3943) system-cube.sh has some problems
[ https://issues.apache.org/jira/browse/KYLIN-3943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long updated KYLIN-3943: - Attachment: image-2019-04-09-18-50-35-111.png > system-cube.sh has some problems > > > Key: KYLIN-3943 > URL: https://issues.apache.org/jira/browse/KYLIN-3943 > Project: Kylin > Issue Type: Bug >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v2.6.2 > > Attachments: image-2019-04-09-18-50-35-111.png > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3943) system-cube.sh has some problems
Chao Long created KYLIN-3943: Summary: system-cube.sh has some problems Key: KYLIN-3943 URL: https://issues.apache.org/jira/browse/KYLIN-3943 Project: Kylin Issue Type: Bug Reporter: Chao Long Assignee: Chao Long Fix For: v2.6.2 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3938) can't delete job OPTIMIZE CHECKPOINT
[ https://issues.apache.org/jira/browse/KYLIN-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16813040#comment-16813040 ] Chao Long commented on KYLIN-3938: -- I will check it. > can't delete job OPTIMIZE CHECKPOINT > > > Key: KYLIN-3938 > URL: https://issues.apache.org/jira/browse/KYLIN-3938 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.6.1 >Reporter: kaishen >Assignee: Chao Long >Priority: Major > Fix For: v2.6.1 > > > I'm using kylin-2.6.1-cdh57 ，when I use cube planner , it will generate tow > jobs one is : OPTIMIZE CUBE another is : OPTIMIZE CHECKPOINT which depend on > job OPTIMIZE CUBE . > now the first job OPTIMIZE CUBE was failed , and I want to discard the > second job , I got error : > java.lang.ClassCastException: > org.apache.kylin.job.execution.CheckpointExecutable cannot be cast to > org.apache.kylin.engine.mr.CubingJob -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KYLIN-3938) can't delete job OPTIMIZE CHECKPOINT
[ https://issues.apache.org/jira/browse/KYLIN-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long reassigned KYLIN-3938: Assignee: Chao Long > can't delete job OPTIMIZE CHECKPOINT > > > Key: KYLIN-3938 > URL: https://issues.apache.org/jira/browse/KYLIN-3938 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.6.1 >Reporter: kaishen >Assignee: Chao Long >Priority: Major > Fix For: v2.6.1 > > > I'm using kylin-2.6.1-cdh57 ，when I use cube planner , it will generate tow > jobs one is : OPTIMIZE CUBE another is : OPTIMIZE CHECKPOINT which depend on > job OPTIMIZE CUBE . > now the first job OPTIMIZE CUBE was failed , and I want to discard the > second job , I got error : > java.lang.ClassCastException: > org.apache.kylin.job.execution.CheckpointExecutable cannot be cast to > org.apache.kylin.engine.mr.CubingJob -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Issue Comment Deleted] (KYLIN-3924) Build cube on step 'build Dimension Dictionary ', the contents of non-dimensional columns are also checked and built
[ https://issues.apache.org/jira/browse/KYLIN-3924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long updated KYLIN-3924: - Comment: was deleted (was: Hi, Kylin only build dictionary for dict-encoding columns and some columns in special measure(precisely count distinct, topN) which need global dictionary.) > Build cube on step 'build Dimension Dictionary ', the contents of > non-dimensional columns are also checked and built > > > Key: KYLIN-3924 > URL: https://issues.apache.org/jira/browse/KYLIN-3924 > Project: Kylin > Issue Type: Bug > Components: Job Engine, Tools, Build and Test >Affects Versions: v2.6.1 >Reporter: wangxiaojing >Priority: Major > Labels: build, patch > Attachments: builderr.log > > > Hi all, > I use version 2.6.1 , when build cube on step 'build Dimension Dictionary' > for lookup tables ,it throws some exceptions,it seems checks the content size > of each field in the table while building Dimension Dictionary, and build all > fields in the table with a dictionary. > Suggestion: > 1. Large content fields may not be dimension columns, in fact, it is not > necessary to check the value size of non-dimension columns，only check > dimension columns. > 2. If the value of some fields in the dimension table is too long, the > exception is not clear, can not be located the problem column quickly. > 3. There is no need to build dictionaries for non-dimensional columns -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3930) ArrayIndexOutOfBoundsException when building
[
https://issues.apache.org/jira/browse/KYLIN-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16812316#comment-16812316
]
Chao Long commented on KYLIN-3930:
--
Hi [~jackyWoo], We know, sharded storage type was set as default storage type
since Kylin-1.5.0, And I see the Kylin you are using is v2.5.0. Could you tell
me what scenario make you still using non-sharded storage type.
> ArrayIndexOutOfBoundsException when building
>
>
> Key: KYLIN-3930
> URL: https://issues.apache.org/jira/browse/KYLIN-3930
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: all
>Reporter: Jacky Woo
>Priority: Major
> Fix For: v2.6.2
>
> Attachments: KYLIN-3930.master.01.patch
>
>
> h2. ArrayIndexOutOfBoundsException when building.
> I hive a cube building error with kylin-2.5.0:
> {code:java}
> 2019-03-31 02:45:18,460 ERROR [main] org.apache.kylin.engine.mr.KylinMapper:
> java.lang.ArrayIndexOutOfBoundsException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.kylin.engine.mr.common.NDCuboidBuilder.buildKeyInternal(NDCuboidBuilder.java:106)
> at
> org.apache.kylin.engine.mr.common.NDCuboidBuilder.buildKey(NDCuboidBuilder.java:71)
> at
> org.apache.kylin.engine.mr.steps.NDCuboidMapper.doMap(NDCuboidMapper.java:112)
> at
> org.apache.kylin.engine.mr.steps.NDCuboidMapper.doMap(NDCuboidMapper.java:47)
> at org.apache.kylin.engine.mr.KylinMapper.map(KylinMapper.java:77)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:796)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> {code}
> I checked the code of "NDCuboidBuilder.buildKeyInternal" method
> {code:java}
> private void buildKeyInternal(Cuboid parentCuboid, Cuboid childCuboid,
> ByteArray[] splitBuffers, ByteArray newKeyBodyBuf) {
> RowKeyEncoder rowkeyEncoder =
> rowKeyEncoderProvider.getRowkeyEncoder(childCuboid);
> // rowkey columns
> long mask = Long.highestOneBit(parentCuboid.getId());
> long parentCuboidId = parentCuboid.getId();
> long childCuboidId = childCuboid.getId();
> long parentCuboidIdActualLength = (long)Long.SIZE -
> Long.numberOfLeadingZeros(parentCuboid.getId());
> int index = rowKeySplitter.getBodySplitOffset(); // skip shard and
> cuboidId
> int offset = RowConstants.ROWKEY_SHARDID_LEN +
> RowConstants.ROWKEY_CUBOIDID_LEN; // skip shard and cuboidId
> for (int i = 0; i < parentCuboidIdActualLength; i++) {
> if ((mask & parentCuboidId) > 0) {// if the this bit position
> equals
> // 1
> if ((mask & childCuboidId) > 0) {// if the child cuboid has
> this
> // column
> System.arraycopy(splitBuffers[index].array(),
> splitBuffers[index].offset(), newKeyBodyBuf.array(), offset,
> splitBuffers[index].length());
> offset += splitBuffers[index].length();
> }
> index++;
> }
> mask = mask >> 1;
> }
> rowkeyEncoder.fillHeader(newKeyBodyBuf.array());
> }
> {code}
> Found that "offset = SHARDID_LEN + CUBOIDID_LEN" , which is wrong when cube
> is not sharding. In my case my cube's storage type is 0, which means it is
> not sharding.
> So, I set offset according to cube sharding, like below:
> {code:java}
> int offset = rowKeySplitter.getHeaderLength(); // skip shard and cuboidId
> {code}
> After modifying building succeeds in my environment.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (KYLIN-3903) Fix dependence issue when debug in local
[ https://issues.apache.org/jira/browse/KYLIN-3903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long closed KYLIN-3903. Resolution: Won't Fix > Fix dependence issue when debug in local > > > Key: KYLIN-3903 > URL: https://issues.apache.org/jira/browse/KYLIN-3903 > Project: Kylin > Issue Type: Sub-task >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KYLIN-3768) Save streaming metadata a standard kylin path in zookeeper
[
https://issues.apache.org/jira/browse/KYLIN-3768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Long reassigned KYLIN-3768:
Assignee: Chao Long (was: Ma Gang)
> Save streaming metadata a standard kylin path in zookeeper
> --
>
> Key: KYLIN-3768
> URL: https://issues.apache.org/jira/browse/KYLIN-3768
> Project: Kylin
> Issue Type: Sub-task
> Components: Real-time Streaming
>Reporter: Ma Gang
>Assignee: Chao Long
>Priority: Major
> Fix For: v3.0.0-alpha
>
>
> Currently we save streaming metadata in the zk path:
> /kylin/stream/${DEPLOY_ENV}, it should be changed to use a standard path like
> the distribute lock path: /kylin/\{METADATA_URL_PREFIX}/stream
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3929) Check satisfaction before execute cubeplanner algorithm
[
https://issues.apache.org/jira/browse/KYLIN-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Long updated KYLIN-3929:
-
Fix Version/s: (was: v2.6.2)
> Check satisfaction before execute cubeplanner algorithm
> ---
>
> Key: KYLIN-3929
> URL: https://issues.apache.org/jira/browse/KYLIN-3929
> Project: Kylin
> Issue Type: Improvement
>Reporter: Chao Long
>Assignee: Chao Long
>Priority: Minor
> Fix For: v3.0.0-alpha
>
>
> {code:java}
> if (!ifForceRecommend && allCuboidCount <= threshold1) {
> return null;
> }
> {code}
> In class CuboidRecommender.java, this check after the selection of cube
> planner algorithm.
> When ifForceRecommend=false and allCuboidCount <= threshold1, cube planner
> algorithm should not be triggered.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3929) Check satisfaction before execute cubeplanner algorithm
[
https://issues.apache.org/jira/browse/KYLIN-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Long updated KYLIN-3929:
-
Fix Version/s: v3.0.0-alpha
> Check satisfaction before execute cubeplanner algorithm
> ---
>
> Key: KYLIN-3929
> URL: https://issues.apache.org/jira/browse/KYLIN-3929
> Project: Kylin
> Issue Type: Improvement
>Reporter: Chao Long
>Assignee: Chao Long
>Priority: Minor
> Fix For: v2.6.2, v3.0.0-alpha
>
>
> {code:java}
> if (!ifForceRecommend && allCuboidCount <= threshold1) {
> return null;
> }
> {code}
> In class CuboidRecommender.java, this check after the selection of cube
> planner algorithm.
> When ifForceRecommend=false and allCuboidCount <= threshold1, cube planner
> algorithm should not be triggered.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KYLIN-3922) Fail to update coprocessor when run DeployCoprocessorCLI
[
https://issues.apache.org/jira/browse/KYLIN-3922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Long resolved KYLIN-3922.
--
Resolution: Resolved
> Fail to update coprocessor when run DeployCoprocessorCLI
>
>
> Key: KYLIN-3922
> URL: https://issues.apache.org/jira/browse/KYLIN-3922
> Project: Kylin
> Issue Type: Bug
> Components: Client - CLI
>Affects Versions: v2.6.0
>Reporter: nichunen
>Assignee: nichunen
>Priority: Major
> Fix For: v2.6.2
>
>
> Update coprocessor by run
> {code:java}
> $KYLIN_HOME/bin/kylin.sh
> org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI
> $KYLIN_HOME/lib/kylin-coprocessor-*.jar all
> {code}
> ArrayIndexOutOfBoundsException thrown with log:
> {code:java}
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 2 at
> org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI.main(DeployCoprocessorCLI.java:118)
>
> 2019-03-27 20:08:02,105 INFO [close-hbase-conn] hbase.HBaseConnection:137 :
> Closing HBase connections...
> 2019-03-27 20:08:02,106 INFO [close-hbase-conn]
> client.ConnectionManager$HConnectionImplementation:1847 : Closing zookeeper
> sessionid=0x169a4aff61f4457
> 2019-03-27 20:08:02,111 INFO [close-hbase-conn] zookeeper.ZooKeeper:684 :
> Session: 0x169a4aff61f4457 closed
> 2019-03-27 20:08:02,111 INFO [main-EventThread] zookeeper.ClientCnxn:512 :
> EventThread shut down 2019-03-27 20:08:02,112 INFO [close-hbase-conn]
> client.ConnectionManager$HConnectionImplementation:2297 : Closing master
> protocol: MasterService 2019-03-27 20:08:02,124 INFO [close-hbase-conn]
> client.ConnectionManager$HConnectionImplementation:1847 : Closing zookeeper
> sessionid=0x469a4aff7c42d72
> 2019-03-27 20:08:02,126 INFO [close-hbase-conn] zookeeper.ZooKeeper:684 :
> Session: 0x469a4aff7c42d72 closed 2019-03-27 20:08:02,126 INFO
> [main-EventThread] zookeeper.ClientCnxn:512 : EventThread shut down
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KYLIN-3923) UT GeneralColumnDataTest fail
[
https://issues.apache.org/jira/browse/KYLIN-3923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Long resolved KYLIN-3923.
--
Resolution: Resolved
> UT GeneralColumnDataTest fail
> -
>
> Key: KYLIN-3923
> URL: https://issues.apache.org/jira/browse/KYLIN-3923
> Project: Kylin
> Issue Type: Bug
> Components: Real-time Streaming
>Reporter: Chao Long
>Assignee: Chao Long
>Priority: Major
> Fix For: v3.0.0-alpha
>
>
> {quote}org.apache.kylin.common.KylinConfigCannotInitException: Didn't find
> KYLIN_CONF or KYLIN_HOME, please set one of them
> at
> org.apache.kylin.common.KylinConfig.getSitePropertiesFile(KylinConfig.java:334)
> at
> org.apache.kylin.common.KylinConfig.buildSiteOrderedProps(KylinConfig.java:376)
> at
> org.apache.kylin.common.KylinConfig.buildSiteProperties(KylinConfig.java:356)
> at
> org.apache.kylin.common.KylinConfig.getInstanceFromEnv(KylinConfig.java:136)
> at org.apache.kylin.metadata.datatype.DataType.(DataType.java:233)
> at
> org.apache.kylin.metadata.datatype.DataType.getType(DataType.java:147)
> at
> org.apache.kylin.stream.core.storage.columnar.GeneralColumnDataTest.testWriteRead(GeneralColumnDataTest.java:46)
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3924) Build cube on step 'build Dimension Dictionary ', the contents of non-dimensional columns are also checked and built
[ https://issues.apache.org/jira/browse/KYLIN-3924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809597#comment-16809597 ] Chao Long commented on KYLIN-3924: -- Hi, Kylin only build dictionary for dict-encoding columns and some columns in special measure(precisely count distinct, topN) which need global dictionary. > Build cube on step 'build Dimension Dictionary ', the contents of > non-dimensional columns are also checked and built > > > Key: KYLIN-3924 > URL: https://issues.apache.org/jira/browse/KYLIN-3924 > Project: Kylin > Issue Type: Bug > Components: Job Engine, Tools, Build and Test >Affects Versions: v2.6.1 >Reporter: wangxiaojing >Priority: Major > Labels: build, patch > Attachments: builderr.log > > > Hi all, > I use version 2.6.1 , when build cube on step 'build Dimension Dictionary' > for lookup tables ,it throws some exceptions,it seems checks the content size > of each field in the table while building Dimension Dictionary, and build all > fields in the table with a dictionary. > Suggestion: > 1. Large content fields may not be dimension columns, in fact, it is not > necessary to check the value size of non-dimension columns，only check > dimension columns. > 2. If the value of some fields in the dimension table is too long, the > exception is not clear, can not be located the problem column quickly. > 3. There is no need to build dictionaries for non-dimensional columns -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-3927) pushdown only return one column
[ https://issues.apache.org/jira/browse/KYLIN-3927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long closed KYLIN-3927. Resolution: Duplicate > pushdown only return one column > --- > > Key: KYLIN-3927 > URL: https://issues.apache.org/jira/browse/KYLIN-3927 > Project: Kylin > Issue Type: Improvement >Reporter: jinguowei >Priority: Major > > 下推查询，仅仅只返回一列，应该是返回查询的世界列 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3929) Check satisfaction before execute cubeplanner algorithm
[
https://issues.apache.org/jira/browse/KYLIN-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Long resolved KYLIN-3929.
--
Resolution: Resolved
> Check satisfaction before execute cubeplanner algorithm
> ---
>
> Key: KYLIN-3929
> URL: https://issues.apache.org/jira/browse/KYLIN-3929
> Project: Kylin
> Issue Type: Improvement
>Reporter: Chao Long
>Assignee: Chao Long
>Priority: Minor
> Fix For: v2.6.2
>
>
> {code:java}
> if (!ifForceRecommend && allCuboidCount <= threshold1) {
> return null;
> }
> {code}
> In class CuboidRecommender.java, this check after the selection of cube
> planner algorithm.
> When ifForceRecommend=false and allCuboidCount <= threshold1, cube planner
> algorithm should not be triggered.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3931) Caused by: java.lang.RuntimeException: Could not create interface org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactory Is the hadoop compatibility jar
[ https://issues.apache.org/jira/browse/KYLIN-3931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16809576#comment-16809576 ] Chao Long commented on KYLIN-3931: -- Hi, you need download the Kylin binary package for hadoop3. > Caused by: java.lang.RuntimeException: Could not create interface > org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactory Is the > hadoop compatibility jar on the classpath? > -- > > Key: KYLIN-3931 > URL: https://issues.apache.org/jira/browse/KYLIN-3931 > Project: Kylin > Issue Type: Test > Components: Environment >Affects Versions: v2.5.2 > Environment: apache-hive-2.3.4-bin > hbase-1.4.9 > hadoop-3.1.0 > apache-kylin-2.5.2-bin-hbase1x.tar.gz >Reporter: praveenece >Priority: Critical > > I am getting the below error. please find the environment details. > apache-hive-2.3.4-bin > hbase-1.4.9 > hadoop-3.1.0 > apache-kylin-2.5.2-bin-hbase1x.tar.gz > Engine Type: Spark > Kylin 63% completed after that i am getting this below error. > WARN TaskSetManager: Lost task 0.0 in stage 1.0 (TID 2, server.com, executor > 1): java.lang.ExceptionInInitializerError > at > org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.getNewWriter(HFileOutputFormat2.java:305) > at > org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:229) > at > org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:167) > at > org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply$mcV$sp(PairRDDFunctions.scala:1125) > at > org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123) > at > org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123) > at > org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1353) > at > org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1131) > at > org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1102) > at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) > at org.apache.spark.scheduler.Task.run(Task.scala:99) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.lang.RuntimeException: Could not create interface > org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactory Is the > hadoop compatibility jar on the classpath? > at > org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:73) > at org.apache.hadoop.hbase.io.MetricsIO.(MetricsIO.java:31) > at org.apache.hadoop.hbase.io.hfile.HFile.(HFile.java:191) > ... 15 more > Caused by: java.util.NoSuchElementException > at > java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:365) > at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) > at java.util.ServiceLoader$1.next(ServiceLoader.java:480) > at > org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:59) > ... 17 more > 19/04/01 11:47:55 INFO TaskSetManager: Starting task 0.1 in stage 1.0 (TID 3, > server.com, executor 1, partition 0, NODE_LOCAL, 5842 bytes) > 19/04/01 11:47:55 WARN TaskSetManager: Lost task 0.1 in stage 1.0 (TID 3, > server.com, executor 1): java.lang.NoClassDefFoundError: Could not initialize > class org.apache.hadoop.hbase.io.hfile.HFile -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3933) Currently replica set related operation need refresh current front-end page
[ https://issues.apache.org/jira/browse/KYLIN-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long updated KYLIN-3933: - Summary: Currently replica set related operation need refresh current front-end page (was: Now replica set related operation need refresh current front-end page) > Currently replica set related operation need refresh current front-end page > --- > > Key: KYLIN-3933 > URL: https://issues.apache.org/jira/browse/KYLIN-3933 > Project: Kylin > Issue Type: Bug > Components: Real-time Streaming, Web >Reporter: Chao Long >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3933) Now replica set related operation need refresh current front-end page
Chao Long created KYLIN-3933: Summary: Now replica set related operation need refresh current front-end page Key: KYLIN-3933 URL: https://issues.apache.org/jira/browse/KYLIN-3933 Project: Kylin Issue Type: Bug Components: Real-time Streaming, Web Reporter: Chao Long -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3929) Check satisfaction before execute cubeplanner algorithm
[
https://issues.apache.org/jira/browse/KYLIN-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Long updated KYLIN-3929:
-
Description:
{code:java}
if (!ifForceRecommend && allCuboidCount <= threshold1) {
return null;
}
{code}
In class CuboidRecommender.java, this check after the selection of cube planner
algorithm.
When ifForceRecommend=false and allCuboidCount <= threshold1, cube planner
algorithm should not be triggered.
> Check satisfaction before execute cubeplanner algorithm
> ---
>
> Key: KYLIN-3929
> URL: https://issues.apache.org/jira/browse/KYLIN-3929
> Project: Kylin
> Issue Type: Improvement
>Reporter: Chao Long
>Assignee: Chao Long
>Priority: Minor
> Fix For: v2.6.2
>
>
> {code:java}
> if (!ifForceRecommend && allCuboidCount <= threshold1) {
> return null;
> }
> {code}
> In class CuboidRecommender.java, this check after the selection of cube
> planner algorithm.
> When ifForceRecommend=false and allCuboidCount <= threshold1, cube planner
> algorithm should not be triggered.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3929) Check satisfaction before execute cubeplanner algorithm
Chao Long created KYLIN-3929: Summary: Check satisfaction before execute cubeplanner algorithm Key: KYLIN-3929 URL: https://issues.apache.org/jira/browse/KYLIN-3929 Project: Kylin Issue Type: Improvement Reporter: Chao Long Assignee: Chao Long Fix For: v2.6.2 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3923) UT GeneralColumnDataTest fail
[
https://issues.apache.org/jira/browse/KYLIN-3923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Long updated KYLIN-3923:
-
Description:
{quote}org.apache.kylin.common.KylinConfigCannotInitException: Didn't find
KYLIN_CONF or KYLIN_HOME, please set one of them
at
org.apache.kylin.common.KylinConfig.getSitePropertiesFile(KylinConfig.java:334)
at
org.apache.kylin.common.KylinConfig.buildSiteOrderedProps(KylinConfig.java:376)
at
org.apache.kylin.common.KylinConfig.buildSiteProperties(KylinConfig.java:356)
at
org.apache.kylin.common.KylinConfig.getInstanceFromEnv(KylinConfig.java:136)
at org.apache.kylin.metadata.datatype.DataType.(DataType.java:233)
at
org.apache.kylin.metadata.datatype.DataType.getType(DataType.java:147)
at
org.apache.kylin.stream.core.storage.columnar.GeneralColumnDataTest.testWriteRead(GeneralColumnDataTest.java:46)
{quote}
> UT GeneralColumnDataTest fail
> -
>
> Key: KYLIN-3923
> URL: https://issues.apache.org/jira/browse/KYLIN-3923
> Project: Kylin
> Issue Type: Bug
> Components: Real-time Streaming
>Reporter: Chao Long
>Assignee: Chao Long
>Priority: Major
> Fix For: v3.0.0-alpha
>
>
> {quote}org.apache.kylin.common.KylinConfigCannotInitException: Didn't find
> KYLIN_CONF or KYLIN_HOME, please set one of them
> at
> org.apache.kylin.common.KylinConfig.getSitePropertiesFile(KylinConfig.java:334)
> at
> org.apache.kylin.common.KylinConfig.buildSiteOrderedProps(KylinConfig.java:376)
> at
> org.apache.kylin.common.KylinConfig.buildSiteProperties(KylinConfig.java:356)
> at
> org.apache.kylin.common.KylinConfig.getInstanceFromEnv(KylinConfig.java:136)
> at org.apache.kylin.metadata.datatype.DataType.(DataType.java:233)
> at
> org.apache.kylin.metadata.datatype.DataType.getType(DataType.java:147)
> at
> org.apache.kylin.stream.core.storage.columnar.GeneralColumnDataTest.testWriteRead(GeneralColumnDataTest.java:46)
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3923) UT GeneralColumnDataTest fail
Chao Long created KYLIN-3923: Summary: UT GeneralColumnDataTest fail Key: KYLIN-3923 URL: https://issues.apache.org/jira/browse/KYLIN-3923 Project: Kylin Issue Type: Bug Components: Real-time Streaming Reporter: Chao Long Assignee: Chao Long Fix For: v3.0.0-alpha -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3873) Fix inappropriate use of memory in SparkFactDistinct.java
[ https://issues.apache.org/jira/browse/KYLIN-3873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long updated KYLIN-3873: - Description: Class SparkFactDistinct.java has some inappropriate use of memory (was: Class SparkFactDistinct.java has some ) > Fix inappropriate use of memory in SparkFactDistinct.java > -- > > Key: KYLIN-3873 > URL: https://issues.apache.org/jira/browse/KYLIN-3873 > Project: Kylin > Issue Type: Improvement >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v2.6.2 > > > Class SparkFactDistinct.java has some inappropriate use of memory -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3873) Fix inappropriate use of memory in SparkFactDistinct.java
[ https://issues.apache.org/jira/browse/KYLIN-3873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long updated KYLIN-3873: - Summary: Fix inappropriate use of memory in SparkFactDistinct.java (was: Improve SparkFactDistinct performance) > Fix inappropriate use of memory in SparkFactDistinct.java > -- > > Key: KYLIN-3873 > URL: https://issues.apache.org/jira/browse/KYLIN-3873 > Project: Kylin > Issue Type: Improvement >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v2.6.2 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3873) Fix inappropriate use of memory in SparkFactDistinct.java
[ https://issues.apache.org/jira/browse/KYLIN-3873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Long updated KYLIN-3873: - Description: Class SparkFactDistinct.java has some > Fix inappropriate use of memory in SparkFactDistinct.java > -- > > Key: KYLIN-3873 > URL: https://issues.apache.org/jira/browse/KYLIN-3873 > Project: Kylin > Issue Type: Improvement >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v2.6.2 > > > Class SparkFactDistinct.java has some -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3903) Fix dependence issue when debug in local
Chao Long created KYLIN-3903: Summary: Fix dependence issue when debug in local Key: KYLIN-3903 URL: https://issues.apache.org/jira/browse/KYLIN-3903 Project: Kylin Issue Type: Sub-task Reporter: Chao Long Assignee: Chao Long -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3872) Kylin web UI http 404 error
[
https://issues.apache.org/jira/browse/KYLIN-3872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795161#comment-16795161
]
Chao Long commented on KYLIN-3872:
--
Hi [~mimimiracle], I'm glad to hear your problem has been resolved. It seems a
spark package version issue.
Btw, the spark version of Kylin2.6.1 is 2.3.2 and use
$KYLIN_HOME/bin/download-spark.sh will download the spark binary of right
version.
> Kylin web UI http 404 error
>
>
> Key: KYLIN-3872
> URL: https://issues.apache.org/jira/browse/KYLIN-3872
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.6.1
> Environment: CDH6.1
> Hadoop3.0.0+cdh6.1.0
> HDFS 3.0.0+cdh6.1.0
> YARN 3.0.0+cdh6.1.0
> HBase 2.1.0+cdh6.1.0
> Hive 2.1.1+cdh6.1.0
> spark 2.4.0+cdh6.1.0
> ZooKeeper 3.4.5+cdh6.1.0
>Reporter: mimimiracle
>Priority: Blocker
> Labels: Kylin, tomcat
> Fix For: v2.6.1
>
>
> kylin server is started but the web cannot be access
> {code:java}
> [root@server1 logs]# netstat -anp| grep 7070
> tcp 0 0 0.0.0.0:7070 0.0.0.0:* LISTEN 11487/java
> {code}
> below is some of log records
> {code:java}
> 2019-03-13 18:28:18,697 WARN [localhost-startStop-1]
> scan.StandardJarScanner:146 : Failed to scan
> [file:/opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/jars/derbyLocale_ja_JP.jar]
> from classloader hierarchy
> java.io.FileNotFoundException:
> /opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/jars/derbyLocale_ja_JP.jar
> (No such file or directory)
> at java.util.zip.ZipFile.open(Native Method)
> at java.util.zip.ZipFile.(ZipFile.java:225)
> at java.util.zip.ZipFile.(ZipFile.java:155)
> at java.util.jar.JarFile.(JarFile.java:166)
> at java.util.jar.JarFile.(JarFile.java:130)
> at
> org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:196)
> at
> org.apache.tomcat.util.scan.JarFileUrlJar.(JarFileUrlJar.java:65)
> at
> org.apache.tomcat.util.scan.JarFactory.newInstance(JarFactory.java:49)
> at
> org.apache.tomcat.util.scan.StandardJarScanner.process(StandardJarScanner.java:374)
> at
> org.apache.tomcat.util.scan.StandardJarScanner.processURLs(StandardJarScanner.java:309)
> at
> org.apache.tomcat.util.scan.StandardJarScanner.doScanClassPath(StandardJarScanner.java:266)
> at
> org.apache.tomcat.util.scan.StandardJarScanner.scan(StandardJarScanner.java:229)
> at
> org.apache.catalina.startup.ContextConfig.processJarsForWebFragments(ContextConfig.java:1888)
> at
> org.apache.catalina.startup.ContextConfig.webConfig(ContextConfig.java:1116)
> at
> org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:765)
> at
> org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:299)
> at
> org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:94)
> at
> org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5154)
> at
> org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
> at
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:754)
> at
> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:730)
> java.io.FileNotFoundException:
> /opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/jars/derbyLocale_pt_BR.jar
> (No such file or directory)
> at java.util.zip.ZipFile.open(Native Method)
> at java.util.zip.ZipFile.(ZipFile.java:225)
> at java.util.zip.ZipFile.(ZipFile.java:155)
> at java.util.jar.JarFile.(JarFile.java:166)
> at java.util.jar.JarFile.(JarFile.java:130)
> at
> org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:196)
> at
> org.apache.tomcat.util.scan.JarFileUrlJar.(JarFileUrlJar.java:65)
> at
> org.apache.tomcat.util.scan.JarFactory.newInstance(JarFactory.java:49)
> at
> org.apache.tomcat.util.scan.StandardJarScanner.process(StandardJarScanner.java:374)
> at
> org.apache.tomcat.util.scan.StandardJarScanner.processURLs(StandardJarScanner.java:309)
> at
> org.apache.tomcat.util.scan.StandardJarScanner.doScanClassPath(StandardJarScanner.java:266)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
> at
> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
> at
>
[jira] [Created] (KYLIN-3873) Improve SparkFactDistinct performance
Chao Long created KYLIN-3873: Summary: Improve SparkFactDistinct performance Key: KYLIN-3873 URL: https://issues.apache.org/jira/browse/KYLIN-3873 Project: Kylin Issue Type: Improvement Reporter: Chao Long Assignee: Chao Long Fix For: v2.6.2 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3872) Kylin web UI http 404 error
[
https://issues.apache.org/jira/browse/KYLIN-3872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16791664#comment-16791664
]
Chao Long edited comment on KYLIN-3872 at 3/13/19 12:50 PM:
Hi [~mimimiracle], did you use kylin binary package for cdh6?
was (Author: wayne0101):
Hi [~mimimiracle], did you use the binary package for cdh6?
> Kylin web UI http 404 error
>
>
> Key: KYLIN-3872
> URL: https://issues.apache.org/jira/browse/KYLIN-3872
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.6.1
> Environment: CDH6.1
> Hadoop3.0.0+cdh6.1.0
> HDFS 3.0.0+cdh6.1.0
> YARN 3.0.0+cdh6.1.0
> HBase 2.1.0+cdh6.1.0
> Hive 2.1.1+cdh6.1.0
> spark 2.4.0+cdh6.1.0
> ZooKeeper 3.4.5+cdh6.1.0
>Reporter: mimimiracle
>Priority: Blocker
> Labels: Kylin, tomcat
>
> kylin server is started but the web cannot be access
> {code:java}
> [root@server1 logs]# netstat -anp| grep 7070
> tcp 0 0 0.0.0.0:7070 0.0.0.0:* LISTEN 11487/java
> {code}
> below is some of log records
> {code:java}
> 2019-03-13 18:28:18,697 WARN [localhost-startStop-1]
> scan.StandardJarScanner:146 : Failed to scan
> [file:/opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/jars/derbyLocale_ja_JP.jar]
> from classloader hierarchy
> java.io.FileNotFoundException:
> /opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/jars/derbyLocale_ja_JP.jar
> (No such file or directory)
> at java.util.zip.ZipFile.open(Native Method)
> at java.util.zip.ZipFile.(ZipFile.java:225)
> at java.util.zip.ZipFile.(ZipFile.java:155)
> at java.util.jar.JarFile.(JarFile.java:166)
> at java.util.jar.JarFile.(JarFile.java:130)
> at
> org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:196)
> at
> org.apache.tomcat.util.scan.JarFileUrlJar.(JarFileUrlJar.java:65)
> at
> org.apache.tomcat.util.scan.JarFactory.newInstance(JarFactory.java:49)
> at
> org.apache.tomcat.util.scan.StandardJarScanner.process(StandardJarScanner.java:374)
> at
> org.apache.tomcat.util.scan.StandardJarScanner.processURLs(StandardJarScanner.java:309)
> at
> org.apache.tomcat.util.scan.StandardJarScanner.doScanClassPath(StandardJarScanner.java:266)
> at
> org.apache.tomcat.util.scan.StandardJarScanner.scan(StandardJarScanner.java:229)
> at
> org.apache.catalina.startup.ContextConfig.processJarsForWebFragments(ContextConfig.java:1888)
> at
> org.apache.catalina.startup.ContextConfig.webConfig(ContextConfig.java:1116)
> at
> org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:765)
> at
> org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:299)
> at
> org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:94)
> at
> org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5154)
> at
> org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
> at
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:754)
> at
> org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:730)
> java.io.FileNotFoundException:
> /opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/jars/derbyLocale_pt_BR.jar
> (No such file or directory)
> at java.util.zip.ZipFile.open(Native Method)
> at java.util.zip.ZipFile.(ZipFile.java:225)
> at java.util.zip.ZipFile.(ZipFile.java:155)
> at java.util.jar.JarFile.(JarFile.java:166)
> at java.util.jar.JarFile.(JarFile.java:130)
> at
> org.apache.tomcat.util.compat.JreCompat.jarFileNewInstance(JreCompat.java:196)
> at
> org.apache.tomcat.util.scan.JarFileUrlJar.(JarFileUrlJar.java:65)
> at
> org.apache.tomcat.util.scan.JarFactory.newInstance(JarFactory.java:49)
> at
> org.apache.tomcat.util.scan.StandardJarScanner.process(StandardJarScanner.java:374)
> at
> org.apache.tomcat.util.scan.StandardJarScanner.processURLs(StandardJarScanner.java:309)
> at
> org.apache.tomcat.util.scan.StandardJarScanner.doScanClassPath(StandardJarScanner.java:266)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
> at
> org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
> at
> org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
> at
>
