[jira] [Created] (KYLIN-3351) Cube Planner not working in apche kylin 2.3.0(open Source)
praveenece created KYLIN-3351: - Summary: Cube Planner not working in apche kylin 2.3.0(open Source) Key: KYLIN-3351 URL: https://issues.apache.org/jira/browse/KYLIN-3351 Project: Kylin Issue Type: Task Reporter: praveenece Hi Team i want test Cube planner in apache-kylin(2.3.0),So i created cube with segment, and i hit query to cube many times more than thousands but the cube planner there is no change in cuboid level like color changing and not get COUNT in exactly row count and other items.can u please guide me. Note: Could u tell me? Cube Planner Working only old Cube (like 3 month before) or new Cube also. Configuration(kylin.Properties) kylin.cube.cubeplanner.enabled=true Kylin Version=2.3.0(SINGLE NODE) MyCube creation =10days before I referred: http://kylin.apache.org/docs23/howto/howto_use_cube_planner.html -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3347) QueryService Exception when using calcite function ex : {fn CURRENT_TIMESTAMP(0)}
[ https://issues.apache.org/jira/browse/KYLIN-3347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16453561#comment-16453561 ] ASF GitHub Bot commented on KYLIN-3347: --- rogercloud commented on a change in pull request #135: KYLIN-3347 : calcite function QueryService Exception URL: https://github.com/apache/kylin/pull/135#discussion_r184280761 ## File path: query/src/main/java/org/apache/kylin/query/util/DefaultQueryTransformer.java ## @@ -50,6 +50,8 @@ private static final Pattern PIN_SUM_OF_FN_CONVERT = Pattern .compile(S0 + "SUM" + S0 + "\\(" + S0 + "\\{\\s*fn" + SM + "convert" + S0 + "\\(" + S0 + "([^\\s,]+)" + S0 + "," + S0 + "SQL_DOUBLE" + S0 + "\\)" + S0 + "\\}" + S0 + "\\)", Pattern.CASE_INSENSITIVE); +private static final Pattern PTN_HAVING_FUNCTION = Pattern.compile("\\{fn" + "(.*?)" + "\\}", Review comment: If TIMESTAMPADD and SQL_TSI_HOUR are also included, it should be added as separated patterns by following the current design. Maybe the design is not the best, but follow it can avoid unexpected issues. Unless you can tell why the "general" matcher is better. Kylin can support calcite reserved functions, and it's welcome to contribute this part. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > QueryService Exception when using calcite function ex : {fn > CURRENT_TIMESTAMP(0)} > - > > Key: KYLIN-3347 > URL: https://issues.apache.org/jira/browse/KYLIN-3347 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: v2.3.0, v2.3.1 >Reporter: Hokyung Song >Priority: Major > > Hello. :) > I'm trying to upgrade kylin version to v2.3.x > But, when I use cacite function query, it occured error. (v2.2.0 is ok) > In my opinion, the reason is that regular expression matcher of > *DefaultQueryTransformer.java* has changed at v2.3.x > I will contribute a path on this problem soon. > thanks > (ps. I'm using Kylin with Tableau 10) > > stack trace as follows (v2.3.0) > {code:java} > 2018-04-23 14:22:12,475 INFO [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > service.QueryService:429 : The original query: SELECT {fn > CURRENT_TIMESTAMP(0)} AS "COL" > 2018-04-23 14:22:12,526 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > runtime.CalciteException:59 : > org.apache.calcite.sql.validate.SqlValidatorException: Function '{fn > CURRENT_TIMESTAMP}' is not defined > 2018-04-23 14:22:12,535 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > runtime.CalciteException:61 : > org.apache.calcite.runtime.CalciteContextException: From line 1, column 8 to > line 1, column 32: Function '{fn CURRENT_TIMESTAMP}' is not defined > 2018-04-23 14:22:12,535 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > service.QueryService:536 : Exception while executing query > java.sql.SQLException: Error while preparing statement [SELECT {fn > CURRENT_TIMESTAMP(0)} AS "COL"] > at org.apache.calcite.avatica.Helper.createException(Helper.java:56) > at org.apache.calcite.avatica.Helper.createException(Helper.java:41) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement_(CalciteConnectionImpl.java:203) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:185) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:86) > at > org.apache.calcite.avatica.AvaticaConnection.prepareStatement(AvaticaConnection.java:168) > at > org.apache.kylin.rest.service.QueryService.getPrepareOnlySqlResponse(QueryService.java:994) > at org.apache.kylin.rest.service.QueryService.execute(QueryService.java:916) > at > org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:661) > at org.apache.kylin.rest.service.QueryService.query(QueryService.java:188) > at > org.apache.kylin.rest.service.QueryService.queryAndUpdateCache(QueryService.java:496) > at > org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:455) > at > org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:390) > at > org.apache.kylin.rest.controller.QueryController.prepareQuery(QueryController.java:99) > at sun.reflect.GeneratedMethodAccessor138.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.springframework.web.method
[jira] [Commented] (KYLIN-3347) QueryService Exception when using calcite function ex : {fn CURRENT_TIMESTAMP(0)}
[ https://issues.apache.org/jira/browse/KYLIN-3347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16453552#comment-16453552 ] ASF GitHub Bot commented on KYLIN-3347: --- imsong commented on a change in pull request #135: KYLIN-3347 : calcite function QueryService Exception URL: https://github.com/apache/kylin/pull/135#discussion_r184268450 ## File path: query/src/main/java/org/apache/kylin/query/util/DefaultQueryTransformer.java ## @@ -50,6 +50,8 @@ private static final Pattern PIN_SUM_OF_FN_CONVERT = Pattern .compile(S0 + "SUM" + S0 + "\\(" + S0 + "\\{\\s*fn" + SM + "convert" + S0 + "\\(" + S0 + "([^\\s,]+)" + S0 + "," + S0 + "SQL_DOUBLE" + S0 + "\\)" + S0 + "\\}" + S0 + "\\)", Pattern.CASE_INSENSITIVE); +private static final Pattern PTN_HAVING_FUNCTION = Pattern.compile("\\{fn" + "(.*?)" + "\\}", Review comment: @rogercloud Hello Roger I also need to use TIMESTAMPADD and SQL_TSI_HOUR functions. Because, Tableau + kylin odbc generate automatically query using this functions. I don't think that is great idea... I hope that Kylin can support calcite reserved functions. https://calcite.apache.org/docs/reference.html thank you. :) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > QueryService Exception when using calcite function ex : {fn > CURRENT_TIMESTAMP(0)} > - > > Key: KYLIN-3347 > URL: https://issues.apache.org/jira/browse/KYLIN-3347 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: v2.3.0, v2.3.1 >Reporter: Hokyung Song >Priority: Major > > Hello. :) > I'm trying to upgrade kylin version to v2.3.x > But, when I use cacite function query, it occured error. (v2.2.0 is ok) > In my opinion, the reason is that regular expression matcher of > *DefaultQueryTransformer.java* has changed at v2.3.x > I will contribute a path on this problem soon. > thanks > (ps. I'm using Kylin with Tableau 10) > > stack trace as follows (v2.3.0) > {code:java} > 2018-04-23 14:22:12,475 INFO [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > service.QueryService:429 : The original query: SELECT {fn > CURRENT_TIMESTAMP(0)} AS "COL" > 2018-04-23 14:22:12,526 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > runtime.CalciteException:59 : > org.apache.calcite.sql.validate.SqlValidatorException: Function '{fn > CURRENT_TIMESTAMP}' is not defined > 2018-04-23 14:22:12,535 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > runtime.CalciteException:61 : > org.apache.calcite.runtime.CalciteContextException: From line 1, column 8 to > line 1, column 32: Function '{fn CURRENT_TIMESTAMP}' is not defined > 2018-04-23 14:22:12,535 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > service.QueryService:536 : Exception while executing query > java.sql.SQLException: Error while preparing statement [SELECT {fn > CURRENT_TIMESTAMP(0)} AS "COL"] > at org.apache.calcite.avatica.Helper.createException(Helper.java:56) > at org.apache.calcite.avatica.Helper.createException(Helper.java:41) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement_(CalciteConnectionImpl.java:203) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:185) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:86) > at > org.apache.calcite.avatica.AvaticaConnection.prepareStatement(AvaticaConnection.java:168) > at > org.apache.kylin.rest.service.QueryService.getPrepareOnlySqlResponse(QueryService.java:994) > at org.apache.kylin.rest.service.QueryService.execute(QueryService.java:916) > at > org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:661) > at org.apache.kylin.rest.service.QueryService.query(QueryService.java:188) > at > org.apache.kylin.rest.service.QueryService.queryAndUpdateCache(QueryService.java:496) > at > org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:455) > at > org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:390) > at > org.apache.kylin.rest.controller.QueryController.prepareQuery(QueryController.java:99) > at sun.reflect.GeneratedMethodAccessor138.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.springframework.web.method.support.InvocableHandler
[jira] [Commented] (KYLIN-3347) QueryService Exception when using calcite function ex : {fn CURRENT_TIMESTAMP(0)}
[ https://issues.apache.org/jira/browse/KYLIN-3347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16453490#comment-16453490 ] ASF GitHub Bot commented on KYLIN-3347: --- imsong commented on a change in pull request #135: KYLIN-3347 : calcite function QueryService Exception URL: https://github.com/apache/kylin/pull/135#discussion_r184268450 ## File path: query/src/main/java/org/apache/kylin/query/util/DefaultQueryTransformer.java ## @@ -50,6 +50,8 @@ private static final Pattern PIN_SUM_OF_FN_CONVERT = Pattern .compile(S0 + "SUM" + S0 + "\\(" + S0 + "\\{\\s*fn" + SM + "convert" + S0 + "\\(" + S0 + "([^\\s,]+)" + S0 + "," + S0 + "SQL_DOUBLE" + S0 + "\\)" + S0 + "\\}" + S0 + "\\)", Pattern.CASE_INSENSITIVE); +private static final Pattern PTN_HAVING_FUNCTION = Pattern.compile("\\{fn" + "(.*?)" + "\\}", Review comment: I also need to use TIMESTAMPADD and SQL_TSI_HOUR functions. Because, Tableau + kylin odbc generate automatically query using this functions. I don't think that is great idea... I hope that Kylin can support calcite reserved functions. https://calcite.apache.org/docs/reference.html thank you. :) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > QueryService Exception when using calcite function ex : {fn > CURRENT_TIMESTAMP(0)} > - > > Key: KYLIN-3347 > URL: https://issues.apache.org/jira/browse/KYLIN-3347 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: v2.3.0, v2.3.1 >Reporter: Hokyung Song >Priority: Major > > Hello. :) > I'm trying to upgrade kylin version to v2.3.x > But, when I use cacite function query, it occured error. (v2.2.0 is ok) > In my opinion, the reason is that regular expression matcher of > *DefaultQueryTransformer.java* has changed at v2.3.x > I will contribute a path on this problem soon. > thanks > (ps. I'm using Kylin with Tableau 10) > > stack trace as follows (v2.3.0) > {code:java} > 2018-04-23 14:22:12,475 INFO [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > service.QueryService:429 : The original query: SELECT {fn > CURRENT_TIMESTAMP(0)} AS "COL" > 2018-04-23 14:22:12,526 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > runtime.CalciteException:59 : > org.apache.calcite.sql.validate.SqlValidatorException: Function '{fn > CURRENT_TIMESTAMP}' is not defined > 2018-04-23 14:22:12,535 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > runtime.CalciteException:61 : > org.apache.calcite.runtime.CalciteContextException: From line 1, column 8 to > line 1, column 32: Function '{fn CURRENT_TIMESTAMP}' is not defined > 2018-04-23 14:22:12,535 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > service.QueryService:536 : Exception while executing query > java.sql.SQLException: Error while preparing statement [SELECT {fn > CURRENT_TIMESTAMP(0)} AS "COL"] > at org.apache.calcite.avatica.Helper.createException(Helper.java:56) > at org.apache.calcite.avatica.Helper.createException(Helper.java:41) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement_(CalciteConnectionImpl.java:203) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:185) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:86) > at > org.apache.calcite.avatica.AvaticaConnection.prepareStatement(AvaticaConnection.java:168) > at > org.apache.kylin.rest.service.QueryService.getPrepareOnlySqlResponse(QueryService.java:994) > at org.apache.kylin.rest.service.QueryService.execute(QueryService.java:916) > at > org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:661) > at org.apache.kylin.rest.service.QueryService.query(QueryService.java:188) > at > org.apache.kylin.rest.service.QueryService.queryAndUpdateCache(QueryService.java:496) > at > org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:455) > at > org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:390) > at > org.apache.kylin.rest.controller.QueryController.prepareQuery(QueryController.java:99) > at sun.reflect.GeneratedMethodAccessor138.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod
[jira] [Commented] (KYLIN-3347) QueryService Exception when using calcite function ex : {fn CURRENT_TIMESTAMP(0)}
[ https://issues.apache.org/jira/browse/KYLIN-3347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16453488#comment-16453488 ] ASF GitHub Bot commented on KYLIN-3347: --- imsong commented on a change in pull request #135: KYLIN-3347 : calcite function QueryService Exception URL: https://github.com/apache/kylin/pull/135#discussion_r184268450 ## File path: query/src/main/java/org/apache/kylin/query/util/DefaultQueryTransformer.java ## @@ -50,6 +50,8 @@ private static final Pattern PIN_SUM_OF_FN_CONVERT = Pattern .compile(S0 + "SUM" + S0 + "\\(" + S0 + "\\{\\s*fn" + SM + "convert" + S0 + "\\(" + S0 + "([^\\s,]+)" + S0 + "," + S0 + "SQL_DOUBLE" + S0 + "\\)" + S0 + "\\}" + S0 + "\\)", Pattern.CASE_INSENSITIVE); +private static final Pattern PTN_HAVING_FUNCTION = Pattern.compile("\\{fn" + "(.*?)" + "\\}", Review comment: I also need to use TIMESTAMPADD and SQL_TSI_HOUR functions. Because, Tableau + kylin odbc generate automatically query using this functions. I don't think that is great idea... I hope that Kylin can support calcite reserved functions. https://calcite.apache.org/docs/reference.html thank you. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > QueryService Exception when using calcite function ex : {fn > CURRENT_TIMESTAMP(0)} > - > > Key: KYLIN-3347 > URL: https://issues.apache.org/jira/browse/KYLIN-3347 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: v2.3.0, v2.3.1 >Reporter: Hokyung Song >Priority: Major > > Hello. :) > I'm trying to upgrade kylin version to v2.3.x > But, when I use cacite function query, it occured error. (v2.2.0 is ok) > In my opinion, the reason is that regular expression matcher of > *DefaultQueryTransformer.java* has changed at v2.3.x > I will contribute a path on this problem soon. > thanks > (ps. I'm using Kylin with Tableau 10) > > stack trace as follows (v2.3.0) > {code:java} > 2018-04-23 14:22:12,475 INFO [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > service.QueryService:429 : The original query: SELECT {fn > CURRENT_TIMESTAMP(0)} AS "COL" > 2018-04-23 14:22:12,526 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > runtime.CalciteException:59 : > org.apache.calcite.sql.validate.SqlValidatorException: Function '{fn > CURRENT_TIMESTAMP}' is not defined > 2018-04-23 14:22:12,535 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > runtime.CalciteException:61 : > org.apache.calcite.runtime.CalciteContextException: From line 1, column 8 to > line 1, column 32: Function '{fn CURRENT_TIMESTAMP}' is not defined > 2018-04-23 14:22:12,535 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > service.QueryService:536 : Exception while executing query > java.sql.SQLException: Error while preparing statement [SELECT {fn > CURRENT_TIMESTAMP(0)} AS "COL"] > at org.apache.calcite.avatica.Helper.createException(Helper.java:56) > at org.apache.calcite.avatica.Helper.createException(Helper.java:41) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement_(CalciteConnectionImpl.java:203) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:185) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:86) > at > org.apache.calcite.avatica.AvaticaConnection.prepareStatement(AvaticaConnection.java:168) > at > org.apache.kylin.rest.service.QueryService.getPrepareOnlySqlResponse(QueryService.java:994) > at org.apache.kylin.rest.service.QueryService.execute(QueryService.java:916) > at > org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:661) > at org.apache.kylin.rest.service.QueryService.query(QueryService.java:188) > at > org.apache.kylin.rest.service.QueryService.queryAndUpdateCache(QueryService.java:496) > at > org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:455) > at > org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:390) > at > org.apache.kylin.rest.controller.QueryController.prepareQuery(QueryController.java:99) > at sun.reflect.GeneratedMethodAccessor138.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMetho
[jira] [Commented] (KYLIN-3221) Some improvements for lookup table
[ https://issues.apache.org/jira/browse/KYLIN-3221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16453408#comment-16453408 ] Ma Gang commented on KYLIN-3221: Also add a new action button 'Lookup Refresh' for each cube, when click the button, a dialog will popup, let user choose which lookup table need to be refreshed, and if the table is not set to global, user can choose some or all segments that the related snapshot need to be refreshed, then user can click 'submit' to submit a new job to build the table snapshot independently. > Some improvements for lookup table > --- > > Key: KYLIN-3221 > URL: https://issues.apache.org/jira/browse/KYLIN-3221 > Project: Kylin > Issue Type: Improvement > Components: Job Engine, Metadata, Query Engine >Reporter: Ma Gang >Assignee: Ma Gang >Priority: Major > > There are two limitations for current look table design: > # lookup table size is limited, because table snapshot need to be cached in > Kylin server, too large snapshot table will break the server. > # lookup table snapshot references are stored in all segments of the cube, > cannot support global snapshot table, the global snapshot table means when > the lookup table is updated, it will take effective for all segments. > To resolve the above limitations, we decide to do some improvements for the > existing lookup table design, below is the initial document, any comments and > suggestions are welcome. > h2. Metadata > Will add a new property in CubeDesc to describe how lookup tables will be > snapshot, it can be defined during the cube design > |{{@JsonProperty}}{{(}}{{"snapshot_table_desc_list"}}{{)}} > {{private}} {{List snapshotTableDescList = > Collections.emptyList();}}| > SnapshotTableDesc defines how table is stored and whether it is global or > not, currently we can support two types of store: > # "metaStore", table snapshot is stored in the metadata store, it is the > same as current design, and this is the default option. > # "hbaseStore', table snapshot is stored in an additional hbase table. > |{{@JsonProperty}}{{(}}{{"table_name"}}{{)}} > {{private}} {{String tableName;}} > > {{@JsonProperty}}{{(}}{{"store_type"}}{{)}} > {{private}} {{String snapshotStorageType = }}{{"metaStore"}}{{;}} > > @JsonProperty("local_cache_enable") > private boolean enableLocalCache = true; > > {{@JsonProperty}}{{(}}{{"global"}}{{)}} > {{private}} {{boolean}} {{global = }}{{false}}{{;}}| > > Add 'snapshots' property in CubeInstance, to store snapshots resource path > for each table, when the table snapshot is set to global in cube design: > |{{@JsonProperty}}{{(}}{{"snapshots"}}{{)}} > {{private}} {{Map snapshots; }}{{// tableName -> > tableResoucePath mapping}}| > > Add new meta model ExtTableSnapshot to describe the extended table snapshot > information, the information is stored in a new metastore path: > /ext_table_snapshot/\{tableName}/\{uuid}.snapshot, the metadata including > following info: > |{{@JsonProperty}}{{(}}{{"tableName"}}{{)}} > {{private}} {{String tableName;}} > > {{@JsonProperty}}{{(}}{{"signature"}}{{)}} > {{private}} {{TableSignature signature;}} > > {{@JsonProperty}}{{(}}{{"storage_location_identifier"}}{{)}} > {{private}} {{String storageLocationIdentifier;}} > > @JsonProperty("key_columns") > private String[] keyColumns; // the key columns of the table > > @JsonProperty("storage_type") > private String storageType; > > {{@JsonProperty}}{{(}}{{"size"}}{{)}} > {{private}} {{long}} {{size;}} > > {{@JsonProperty}}{{(}}{{"row_cnt"}}{{)}} > {{private}} {{long}} {{rowCnt;}}| > > Add new section in 'Advance Setting' tab when do cube design, user can set > table snapshot properties for each table, and by default, it is segment level > and store to metadata store > h2. Build > If user specify 'hbaseStore' storageType for any lookup table, will use > MapReduce job convert the hive source table to hfiles, and then bulk load > hfiles to HTable. So it will add two job steps to do the lookup table > materialization. > h2. HBase Lookup Table Schema > all data are stored in raw value > suppose the lookup table has primary keys: key1,key2 > rowkey will be: > ||2bytes||2 bytes||len1 bytes||2 bytes||len2 bytes|| > |shard|key1 value length(len1)|key1 value|key 2 value length(len2)|key2 value| > the first 2 bytes is shard number, HBase table can be pre-split, the shard > size is configurable through Kylin's properties: > "kylin.snapshot.ext.shard-mb", default size is 500MB. > 1 column family c, multiple columns which column name is the index of the > column in the table definition > |c| > |1|2|...| > > h2. Query > For key lookup query, directly call hbase get api to get entire row according > to key (call local cache if t
[jira] [Updated] (KYLIN-3221) Some improvements for lookup table
[ https://issues.apache.org/jira/browse/KYLIN-3221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ma Gang updated KYLIN-3221: --- Description: There are two limitations for current look table design: # lookup table size is limited, because table snapshot need to be cached in Kylin server, too large snapshot table will break the server. # lookup table snapshot references are stored in all segments of the cube, cannot support global snapshot table, the global snapshot table means when the lookup table is updated, it will take effective for all segments. To resolve the above limitations, we decide to do some improvements for the existing lookup table design, below is the initial document, any comments and suggestions are welcome. h2. Metadata Will add a new property in CubeDesc to describe how lookup tables will be snapshot, it can be defined during the cube design |{{@JsonProperty}}{{(}}{{"snapshot_table_desc_list"}}{{)}} {{private}} {{List snapshotTableDescList = Collections.emptyList();}}| SnapshotTableDesc defines how table is stored and whether it is global or not, currently we can support two types of store: # "metaStore", table snapshot is stored in the metadata store, it is the same as current design, and this is the default option. # "hbaseStore', table snapshot is stored in an additional hbase table. |{{@JsonProperty}}{{(}}{{"table_name"}}{{)}} {{private}} {{String tableName;}} {{@JsonProperty}}{{(}}{{"store_type"}}{{)}} {{private}} {{String snapshotStorageType = }}{{"metaStore"}}{{;}} @JsonProperty("local_cache_enable") private boolean enableLocalCache = true; {{@JsonProperty}}{{(}}{{"global"}}{{)}} {{private}} {{boolean}} {{global = }}{{false}}{{;}}| Add 'snapshots' property in CubeInstance, to store snapshots resource path for each table, when the table snapshot is set to global in cube design: |{{@JsonProperty}}{{(}}{{"snapshots"}}{{)}} {{private}} {{Map snapshots; }}{{// tableName -> tableResoucePath mapping}}| Add new meta model ExtTableSnapshot to describe the extended table snapshot information, the information is stored in a new metastore path: /ext_table_snapshot/\{tableName}/\{uuid}.snapshot, the metadata including following info: |{{@JsonProperty}}{{(}}{{"tableName"}}{{)}} {{private}} {{String tableName;}} {{@JsonProperty}}{{(}}{{"signature"}}{{)}} {{private}} {{TableSignature signature;}} {{@JsonProperty}}{{(}}{{"storage_location_identifier"}}{{)}} {{private}} {{String storageLocationIdentifier;}} @JsonProperty("key_columns") private String[] keyColumns; // the key columns of the table @JsonProperty("storage_type") private String storageType; {{@JsonProperty}}{{(}}{{"size"}}{{)}} {{private}} {{long}} {{size;}} {{@JsonProperty}}{{(}}{{"row_cnt"}}{{)}} {{private}} {{long}} {{rowCnt;}}| Add new section in 'Advance Setting' tab when do cube design, user can set table snapshot properties for each table, and by default, it is segment level and store to metadata store h2. Build If user specify 'hbaseStore' storageType for any lookup table, will use MapReduce job convert the hive source table to hfiles, and then bulk load hfiles to HTable. So it will add two job steps to do the lookup table materialization. h2. HBase Lookup Table Schema all data are stored in raw value suppose the lookup table has primary keys: key1,key2 rowkey will be: ||2bytes||2 bytes||len1 bytes||2 bytes||len2 bytes|| |shard|key1 value length(len1)|key1 value|key 2 value length(len2)|key2 value| the first 2 bytes is shard number, HBase table can be pre-split, the shard size is configurable through Kylin's properties: "kylin.snapshot.ext.shard-mb", default size is 500MB. 1 column family c, multiple columns which column name is the index of the column in the table definition |c| |1|2|...| h2. Query For key lookup query, directly call hbase get api to get entire row according to key (call local cache if there is local cache enable) For queries that need fetch keys according to the derived columns, iterate all rows to get related keys. (call local cache if there is local cache enable) For queries that only hit the lookup table, iterate all rows and let calcite to do aggregation and filter. (call local cache if there is local cache enable) h2. Management For each lookup table, admin can view how many snapshots it has in Kylin, and can view each snapshot type/size information and which cube/segments the snapshot is referenced, the snapshot tables that have no reference can be deleted. Add a new action button 'Lookup Refresh' for each cube, when click the button, a dialog will popup, let user choose which lookup table need to refresh, and if the table is not set to global, user can choose some or all segments that the related snapshot need to be refresh, then user can click 'submit' to submit a new job to build the table snapshot independently. h2. Cleanup
[jira] [Commented] (KYLIN-3347) QueryService Exception when using calcite function ex : {fn CURRENT_TIMESTAMP(0)}
[ https://issues.apache.org/jira/browse/KYLIN-3347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16453384#comment-16453384 ] ASF GitHub Bot commented on KYLIN-3347: --- rogercloud commented on a change in pull request #135: KYLIN-3347 : calcite function QueryService Exception URL: https://github.com/apache/kylin/pull/135#discussion_r184256367 ## File path: query/src/main/java/org/apache/kylin/query/util/DefaultQueryTransformer.java ## @@ -50,6 +50,8 @@ private static final Pattern PIN_SUM_OF_FN_CONVERT = Pattern .compile(S0 + "SUM" + S0 + "\\(" + S0 + "\\{\\s*fn" + SM + "convert" + S0 + "\\(" + S0 + "([^\\s,]+)" + S0 + "," + S0 + "SQL_DOUBLE" + S0 + "\\)" + S0 + "\\}" + S0 + "\\)", Pattern.CASE_INSENSITIVE); +private static final Pattern PTN_HAVING_FUNCTION = Pattern.compile("\\{fn" + "(.*?)" + "\\}", Review comment: It's too general as it matches all functions. How about adding a CURRENT_TIMESTAMP function like PTN_MIN_1 instead? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > QueryService Exception when using calcite function ex : {fn > CURRENT_TIMESTAMP(0)} > - > > Key: KYLIN-3347 > URL: https://issues.apache.org/jira/browse/KYLIN-3347 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: v2.3.0, v2.3.1 >Reporter: Hokyung Song >Priority: Major > > Hello. :) > I'm trying to upgrade kylin version to v2.3.x > But, when I use cacite function query, it occured error. (v2.2.0 is ok) > In my opinion, the reason is that regular expression matcher of > *DefaultQueryTransformer.java* has changed at v2.3.x > I will contribute a path on this problem soon. > thanks > (ps. I'm using Kylin with Tableau 10) > > stack trace as follows (v2.3.0) > {code:java} > 2018-04-23 14:22:12,475 INFO [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > service.QueryService:429 : The original query: SELECT {fn > CURRENT_TIMESTAMP(0)} AS "COL" > 2018-04-23 14:22:12,526 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > runtime.CalciteException:59 : > org.apache.calcite.sql.validate.SqlValidatorException: Function '{fn > CURRENT_TIMESTAMP}' is not defined > 2018-04-23 14:22:12,535 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > runtime.CalciteException:61 : > org.apache.calcite.runtime.CalciteContextException: From line 1, column 8 to > line 1, column 32: Function '{fn CURRENT_TIMESTAMP}' is not defined > 2018-04-23 14:22:12,535 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > service.QueryService:536 : Exception while executing query > java.sql.SQLException: Error while preparing statement [SELECT {fn > CURRENT_TIMESTAMP(0)} AS "COL"] > at org.apache.calcite.avatica.Helper.createException(Helper.java:56) > at org.apache.calcite.avatica.Helper.createException(Helper.java:41) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement_(CalciteConnectionImpl.java:203) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:185) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:86) > at > org.apache.calcite.avatica.AvaticaConnection.prepareStatement(AvaticaConnection.java:168) > at > org.apache.kylin.rest.service.QueryService.getPrepareOnlySqlResponse(QueryService.java:994) > at org.apache.kylin.rest.service.QueryService.execute(QueryService.java:916) > at > org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:661) > at org.apache.kylin.rest.service.QueryService.query(QueryService.java:188) > at > org.apache.kylin.rest.service.QueryService.queryAndUpdateCache(QueryService.java:496) > at > org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:455) > at > org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:390) > at > org.apache.kylin.rest.controller.QueryController.prepareQuery(QueryController.java:99) > at sun.reflect.GeneratedMethodAccessor138.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) > at > org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) > at > org.springframework.web.servlet.mvc.method.annotation.S
[jira] [Commented] (KYLIN-3347) QueryService Exception when using calcite function ex : {fn CURRENT_TIMESTAMP(0)}
[ https://issues.apache.org/jira/browse/KYLIN-3347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16453383#comment-16453383 ] ASF GitHub Bot commented on KYLIN-3347: --- rogercloud commented on a change in pull request #135: KYLIN-3347 : calcite function QueryService Exception URL: https://github.com/apache/kylin/pull/135#discussion_r184256469 ## File path: query/src/main/java/org/apache/kylin/query/util/DefaultQueryTransformer.java ## @@ -138,6 +140,12 @@ public String transform(String sql, String project, String defaultSchema) { sql = sql.substring(0, m.start(1)) + "'" + value + "'" + sql.substring(m.end(3)); } +while (true) { Review comment: Please add some comments to explain how the pattern is used. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > QueryService Exception when using calcite function ex : {fn > CURRENT_TIMESTAMP(0)} > - > > Key: KYLIN-3347 > URL: https://issues.apache.org/jira/browse/KYLIN-3347 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: v2.3.0, v2.3.1 >Reporter: Hokyung Song >Priority: Major > > Hello. :) > I'm trying to upgrade kylin version to v2.3.x > But, when I use cacite function query, it occured error. (v2.2.0 is ok) > In my opinion, the reason is that regular expression matcher of > *DefaultQueryTransformer.java* has changed at v2.3.x > I will contribute a path on this problem soon. > thanks > (ps. I'm using Kylin with Tableau 10) > > stack trace as follows (v2.3.0) > {code:java} > 2018-04-23 14:22:12,475 INFO [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > service.QueryService:429 : The original query: SELECT {fn > CURRENT_TIMESTAMP(0)} AS "COL" > 2018-04-23 14:22:12,526 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > runtime.CalciteException:59 : > org.apache.calcite.sql.validate.SqlValidatorException: Function '{fn > CURRENT_TIMESTAMP}' is not defined > 2018-04-23 14:22:12,535 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > runtime.CalciteException:61 : > org.apache.calcite.runtime.CalciteContextException: From line 1, column 8 to > line 1, column 32: Function '{fn CURRENT_TIMESTAMP}' is not defined > 2018-04-23 14:22:12,535 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > service.QueryService:536 : Exception while executing query > java.sql.SQLException: Error while preparing statement [SELECT {fn > CURRENT_TIMESTAMP(0)} AS "COL"] > at org.apache.calcite.avatica.Helper.createException(Helper.java:56) > at org.apache.calcite.avatica.Helper.createException(Helper.java:41) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement_(CalciteConnectionImpl.java:203) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:185) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:86) > at > org.apache.calcite.avatica.AvaticaConnection.prepareStatement(AvaticaConnection.java:168) > at > org.apache.kylin.rest.service.QueryService.getPrepareOnlySqlResponse(QueryService.java:994) > at org.apache.kylin.rest.service.QueryService.execute(QueryService.java:916) > at > org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:661) > at org.apache.kylin.rest.service.QueryService.query(QueryService.java:188) > at > org.apache.kylin.rest.service.QueryService.queryAndUpdateCache(QueryService.java:496) > at > org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:455) > at > org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:390) > at > org.apache.kylin.rest.controller.QueryController.prepareQuery(QueryController.java:99) > at sun.reflect.GeneratedMethodAccessor138.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) > at > org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) > at > org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) > at > org
[jira] [Commented] (KYLIN-3221) Some improvements for lookup table
[ https://issues.apache.org/jira/browse/KYLIN-3221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16453381#comment-16453381 ] Ma Gang commented on KYLIN-3221: According to performance test, when save big lookup table in hbase, and wants to get lots of keys from htable, it will take lots of time, per test, get about 110k random keys from htable will take about 200s. So I add a local LRU disk cache in query server to improve the lookup join performance, currently I use RocksDB as the disk cache, it is configurable and can be disable in the cube configuration. If the local cache is enable, will add another step in build job to warm up the cache. Per test, the rocksdb cache performance is good, in my vm with 4 core, 10G ram, HDD disk, it take about 4 seconds to randomly get 110k keys, and scan performance is also very good. > Some improvements for lookup table > --- > > Key: KYLIN-3221 > URL: https://issues.apache.org/jira/browse/KYLIN-3221 > Project: Kylin > Issue Type: Improvement > Components: Job Engine, Metadata, Query Engine >Reporter: Ma Gang >Assignee: Ma Gang >Priority: Major > > There are two limitations for current look table design: > # lookup table size is limited, because table snapshot need to be cached in > Kylin server, too large snapshot table will break the server. > # lookup table snapshot references are stored in all segments of the cube, > cannot support global snapshot table, the global snapshot table means when > the lookup table is updated, it will take effective for all segments. > To resolve the above limitations, we decide to do some improvements for the > existing lookup table design, below is the initial document, any comments and > suggestions are welcome. > h2. Metadata > Will add a new property in CubeDesc to describe how lookup tables will be > snapshot, it can be defined during the cube design > |{{@JsonProperty}}{{(}}{{"snapshot_table_desc_list"}}{{)}} > {{private}} {{List snapshotTableDescList = > Collections.emptyList();}}| > SnapshotTableDesc defines how table is stored and whether it is global or > not, currently we can support two types of store: > # "metaStore", table snapshot is stored in the metadata store, it is the > same as current design, and this is the default option. > # "hbaseStore', table snapshot is stored in an additional hbase table. > |{{@JsonProperty}}{{(}}{{"table_name"}}{{)}} > {{private}} {{String tableName;}} > > {{@JsonProperty}}{{(}}{{"store_type"}}{{)}} > {{private}} {{String snapshotStorageType = }}{{"metaStore"}}{{;}} > > @JsonProperty("local_cache_enable") > private boolean enableLocalCache = true; > > {{@JsonProperty}}{{(}}{{"global"}}{{)}} > {{private}} {{boolean}} {{global = }}{{false}}{{;}}| > > Add 'snapshots' property in CubeInstance, to store snapshots resource path > for each table, when the table snapshot is set to global in cube design: > |{{@JsonProperty}}{{(}}{{"snapshots"}}{{)}} > {{private}} {{Map snapshots; }}{{// tableName -> > tableResoucePath mapping}}| > > Add new meta model ExtTableSnapshot to describe the extended table snapshot > information, the information is stored in a new metastore path: > /ext_table_snapshot/\{tableName}/\{uuid}.snapshot, the metadata including > following info: > |{{@JsonProperty}}{{(}}{{"tableName"}}{{)}} > {{private}} {{String tableName;}} > > {{@JsonProperty}}{{(}}{{"signature"}}{{)}} > {{private}} {{TableSignature signature;}} > > {{@JsonProperty}}{{(}}{{"storage_location_identifier"}}{{)}} > {{private}} {{String storageLocationIdentifier;}} > > @JsonProperty("key_columns") > private String[] keyColumns; // the key columns of the table > > @JsonProperty("storage_type") > private String storageType; > > {{@JsonProperty}}{{(}}{{"size"}}{{)}} > {{private}} {{long}} {{size;}} > > {{@JsonProperty}}{{(}}{{"row_cnt"}}{{)}} > {{private}} {{long}} {{rowCnt;}}| > > Add new section in 'Advance Setting' tab when do cube design, user can set > table snapshot properties for each table, and by default, it is segment level > and store to metadata store > h2. Build > If user specify 'hbaseStore' storageType for any lookup table, will use > MapReduce job convert the hive source table to hfiles, and then bulk load > hfiles to HTable. So it will add two job steps to do the lookup table > materialization. > h2. HBase Lookup Table Schema > all data are stored in raw value > suppose the lookup table has primary keys: key1,key2 > rowkey will be: > ||2bytes||2 bytes||len1 bytes||2 bytes||len2 bytes|| > |shard|key1 value length(len1)|key1 value|key 2 value length(len2)|key2 value| > the first 2 bytes is shard number, HBase table can be pre-split, the shard > size is configurable through Kylin's properties: > "kylin.snapsh
[jira] [Updated] (KYLIN-3221) Some improvements for lookup table
[ https://issues.apache.org/jira/browse/KYLIN-3221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ma Gang updated KYLIN-3221: --- Description: There are two limitations for current look table design: # lookup table size is limited, because table snapshot need to be cached in Kylin server, too large snapshot table will break the server. # lookup table snapshot references are stored in all segments of the cube, cannot support global snapshot table, the global snapshot table means when the lookup table is updated, it will take effective for all segments. To resolve the above limitations, we decide to do some improvements for the existing lookup table design, below is the initial document, any comments and suggestions are welcome. h2. Metadata Will add a new property in CubeDesc to describe how lookup tables will be snapshot, it can be defined during the cube design |{{@JsonProperty}}{{(}}{{"snapshot_table_desc_list"}}{{)}} {{private}} {{List snapshotTableDescList = Collections.emptyList();}}| SnapshotTableDesc defines how table is stored and whether it is global or not, currently we can support two types of store: # "metaStore", table snapshot is stored in the metadata store, it is the same as current design, and this is the default option. # "hbaseStore', table snapshot is stored in an additional hbase table. |{{@JsonProperty}}{{(}}{{"table_name"}}{{)}} {{private}} {{String tableName;}} {{@JsonProperty}}{{(}}{{"store_type"}}{{)}} {{private}} {{String snapshotStorageType = }}{{"metaStore"}}{{;}} @JsonProperty("local_cache_enable") private boolean enableLocalCache = true; {{@JsonProperty}}{{(}}{{"global"}}{{)}} {{private}} {{boolean}} {{global = }}{{false}}{{;}}| Add 'snapshots' property in CubeInstance, to store snapshots resource path for each table, when the table snapshot is set to global in cube design: |{{@JsonProperty}}{{(}}{{"snapshots"}}{{)}} {{private}} {{Map snapshots; }}{{// tableName -> tableResoucePath mapping}}| Add new meta model ExtTableSnapshot to describe the extended table snapshot information, the information is stored in a new metastore path: /ext_table_snapshot/\{tableName}/\{uuid}.snapshot, the metadata including following info: |{{@JsonProperty}}{{(}}{{"tableName"}}{{)}} {{private}} {{String tableName;}} {{@JsonProperty}}{{(}}{{"signature"}}{{)}} {{private}} {{TableSignature signature;}} {{@JsonProperty}}{{(}}{{"storage_location_identifier"}}{{)}} {{private}} {{String storageLocationIdentifier;}} @JsonProperty("key_columns") private String[] keyColumns; // the key columns of the table @JsonProperty("storage_type") private String storageType; {{@JsonProperty}}{{(}}{{"size"}}{{)}} {{private}} {{long}} {{size;}} {{@JsonProperty}}{{(}}{{"row_cnt"}}{{)}} {{private}} {{long}} {{rowCnt;}}| Add new section in 'Advance Setting' tab when do cube design, user can set table snapshot properties for each table, and by default, it is segment level and store to metadata store h2. Build If user specify 'hbaseStore' storageType for any lookup table, will use MapReduce job convert the hive source table to hfiles, and then bulk load hfiles to HTable. So it will add two job steps to do the lookup table materialization. h2. HBase Lookup Table Schema all data are stored in raw value suppose the lookup table has primary keys: key1,key2 rowkey will be: ||2bytes||2 bytes||len1 bytes||2 bytes||len2 bytes|| |shard|key1 value length(len1)|key1 value|key 2 value length(len2)|key2 value| the first 2 bytes is shard number, HBase table can be pre-split, the shard size is configurable through Kylin's properties: "kylin.snapshot.ext.shard-mb", default size is 500MB. 1 column family c, multiple columns which column name is the index of the column in the table definition |c| |1|2|...| h2. Query For key lookup query, directly call hbase get api to get entire row according to key (call local cache if there is local cache enable) For queries that need fetch keys according to the derived columns, iterate all rows to get related keys. (call local cache if there is local cache enable) For queries that only hit the lookup table, iterate all rows and let calcite to do aggregation and filter. (call local cache if there is local cache enable) h2. Management For each lookup table, admin can view how many snapshots it has in Kylin, and can view each snapshot type/size information and which cube/segments the snapshot is referenced, the snapshot tables that have no reference can be deleted. h2. Cleanup When clean up metadata store, need to remove snapshot stored in HBase. And need to clean up metadata store periodically by cronjob. h2. Future # Add coprocessor for lookup table, to improve the performance of lookup table query, and queries that filter by derived columns. # Add secondly index support for external snapshot table. was: There are two limitations for
[jira] [Updated] (KYLIN-3221) Some improvements for lookup table
[ https://issues.apache.org/jira/browse/KYLIN-3221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ma Gang updated KYLIN-3221: --- Description: There are two limitations for current look table design: # lookup table size is limited, because table snapshot need to be cached in Kylin server, too large snapshot table will break the server. # lookup table snapshot references are stored in all segments of the cube, cannot support global snapshot table, the global snapshot table means when the lookup table is updated, it will take effective for all segments. To resolve the above limitations, we decide to do some improvements for the existing lookup table design, below is the initial document, any comments and suggestions are welcome. h2. Metadata Will add a new property in CubeDesc to describe how lookup tables will be snapshot, it can be defined during the cube design |{{@JsonProperty}}{{(}}{{"snapshot_table_desc_list"}}{{)}} {{private}} {{List snapshotTableDescList = Collections.emptyList();}}| SnapshotTableDesc defines how table is stored and whether it is global or not, currently we can support two types of store: # "metaStore", table snapshot is stored in the metadata store, it is the same as current design, and this is the default option. # "hbaseStore', table snapshot is stored in an additional hbase table. |{{@JsonProperty}}{{(}}{{"table_name"}}{{)}} {{private}} {{String tableName;}} {{@JsonProperty}}{{(}}{{"store_type"}}{{)}} {{private}} {{String snapshotStorageType = }}{{"metaStore"}}{{;}} {{@JsonProperty}}{{(}}{{"global"}}{{)}} {{private}} {{boolean}} {{global = }}{{false}}{{;}}| Add 'snapshots' property in CubeInstance, to store snapshots resource path for each table, when the table snapshot is set to global in cube design: |{{@JsonProperty}}{{(}}{{"snapshots"}}{{)}} {{private}} {{Map snapshots; }}{{// tableName -> tableResoucePath mapping}}| Add new meta model ExtTableSnapshot to describe the extended table snapshot information, the information is stored in a new metastore path: /ext_table_snapshot/\{tableName}/\{uuid}.snapshot, the metadata including following info: |{{@JsonProperty}}{{(}}{{"tableName"}}{{)}} {{private}} {{String tableName;}} {{@JsonProperty}}{{(}}{{"signature"}}{{)}} {{private}} {{TableSignature signature;}} {{@JsonProperty}}{{(}}{{"storage_location_identifier"}}{{)}} {{private}} {{String storageLocationIdentifier;}} @JsonProperty("key_columns") private String[] keyColumns; // the key columns of the table @JsonProperty("storage_type") private String storageType; {{@JsonProperty}}{{(}}{{"size"}}{{)}} {{private}} {{long}} {{size;}} {{@JsonProperty}}{{(}}{{"row_cnt"}}{{)}} {{private}} {{long}} {{rowCnt;}}| Add new section in 'Advance Setting' tab when do cube design, user can set table snapshot properties for each table, and by default, it is segment level and store to metadata store h2. Build If user specify 'hbaseStore' storageType for any lookup table, will use MapReduce job convert the hive source table to hfiles, and then bulk load hfiles to HTable. So it will add two job steps to do the lookup table materialization. h2. HBase Lookup Table Schema all data are stored in raw value suppose the lookup table has primary keys: key1,key2 rowkey will be: ||2bytes||2 bytes||len1 bytes||2 bytes||len2 bytes|| |shard|key1 value length(len1)|key1 value|key 2 value length(len2)|key2 value| the first 2 bytes is shard number, HBase table can be pre-split, the shard size is configurable through Kylin's properties: "kylin.snapshot.ext.shard-mb", default size is 500MB. 1 column family c, multiple columns which column name is the index of the column in the table definition |c| |1|2|...| h2. Query For key lookup query, directly call hbase get api to get entire row according to key. For queries that need fetch keys according to the derived columns, iterate all rows to get related keys. For queries that only hit the lookup table, iterate all rows and let calcite to do aggregation and filter. h2. Management For each lookup table, admin can view how many snapshots it has in Kylin, and can view each snapshot type/size information and which cube/segments the snapshot is referenced, the snapshot tables that have no reference can be deleted. h2. Cleanup When clean up metadata store, need to remove snapshot stored in HBase. And need to clean up metadata store periodically by cronjob. h2. Future # Add coprocessor for lookup table, to improve the performance of lookup table query, and queries that filter by derived columns. # Add secondly index support for external snapshot table. was: There are two limitations for current look table design: # lookup table size is limited, because table snapshot need to be cached in Kylin server, too large snapshot table will break the server. # lookup table snapshot references are stored in all segment
[jira] [Commented] (KYLIN-3347) QueryService Exception when using calcite function ex : {fn CURRENT_TIMESTAMP(0)}
[ https://issues.apache.org/jira/browse/KYLIN-3347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16453354#comment-16453354 ] ASF GitHub Bot commented on KYLIN-3347: --- rogercloud commented on issue #135: KYLIN-3347 : calcite function QueryService Exception URL: https://github.com/apache/kylin/pull/135#issuecomment-384488172 @imsong Some UT failes, please have a check. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > QueryService Exception when using calcite function ex : {fn > CURRENT_TIMESTAMP(0)} > - > > Key: KYLIN-3347 > URL: https://issues.apache.org/jira/browse/KYLIN-3347 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: v2.3.0, v2.3.1 >Reporter: Hokyung Song >Priority: Major > > Hello. :) > I'm trying to upgrade kylin version to v2.3.x > But, when I use cacite function query, it occured error. (v2.2.0 is ok) > In my opinion, the reason is that regular expression matcher of > *DefaultQueryTransformer.java* has changed at v2.3.x > I will contribute a path on this problem soon. > thanks > (ps. I'm using Kylin with Tableau 10) > > stack trace as follows (v2.3.0) > {code:java} > 2018-04-23 14:22:12,475 INFO [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > service.QueryService:429 : The original query: SELECT {fn > CURRENT_TIMESTAMP(0)} AS "COL" > 2018-04-23 14:22:12,526 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > runtime.CalciteException:59 : > org.apache.calcite.sql.validate.SqlValidatorException: Function '{fn > CURRENT_TIMESTAMP}' is not defined > 2018-04-23 14:22:12,535 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > runtime.CalciteException:61 : > org.apache.calcite.runtime.CalciteContextException: From line 1, column 8 to > line 1, column 32: Function '{fn CURRENT_TIMESTAMP}' is not defined > 2018-04-23 14:22:12,535 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > service.QueryService:536 : Exception while executing query > java.sql.SQLException: Error while preparing statement [SELECT {fn > CURRENT_TIMESTAMP(0)} AS "COL"] > at org.apache.calcite.avatica.Helper.createException(Helper.java:56) > at org.apache.calcite.avatica.Helper.createException(Helper.java:41) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement_(CalciteConnectionImpl.java:203) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:185) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:86) > at > org.apache.calcite.avatica.AvaticaConnection.prepareStatement(AvaticaConnection.java:168) > at > org.apache.kylin.rest.service.QueryService.getPrepareOnlySqlResponse(QueryService.java:994) > at org.apache.kylin.rest.service.QueryService.execute(QueryService.java:916) > at > org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:661) > at org.apache.kylin.rest.service.QueryService.query(QueryService.java:188) > at > org.apache.kylin.rest.service.QueryService.queryAndUpdateCache(QueryService.java:496) > at > org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:455) > at > org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:390) > at > org.apache.kylin.rest.controller.QueryController.prepareQuery(QueryController.java:99) > at sun.reflect.GeneratedMethodAccessor138.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) > at > org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) > at > org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) > at > org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) > at > org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967) > at > org.springframework.web.servlet.Disp
[jira] [Created] (KYLIN-3350) Build cube incrementally from a file list
Shaofeng SHI created KYLIN-3350: --- Summary: Build cube incrementally from a file list Key: KYLIN-3350 URL: https://issues.apache.org/jira/browse/KYLIN-3350 Project: Kylin Issue Type: New Feature Components: Job Engine Reporter: Shaofeng SHI This a new requirement. In some cases, upstream components dump the new data into files, and then expect Kylin can consume data from a given file list, instead of from Hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3347) QueryService Exception when using calcite function ex : {fn CURRENT_TIMESTAMP(0)}
[ https://issues.apache.org/jira/browse/KYLIN-3347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16452281#comment-16452281 ] Shaofeng SHI commented on KYLIN-3347: - [~rogershi] Roger, could you please review the PR [https://github.com/apache/kylin/pull/135] ? > QueryService Exception when using calcite function ex : {fn > CURRENT_TIMESTAMP(0)} > - > > Key: KYLIN-3347 > URL: https://issues.apache.org/jira/browse/KYLIN-3347 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: v2.3.0, v2.3.1 >Reporter: Hokyung Song >Priority: Major > > Hello. :) > I'm trying to upgrade kylin version to v2.3.x > But, when I use cacite function query, it occured error. (v2.2.0 is ok) > In my opinion, the reason is that regular expression matcher of > *DefaultQueryTransformer.java* has changed at v2.3.x > I will contribute a path on this problem soon. > thanks > (ps. I'm using Kylin with Tableau 10) > > stack trace as follows (v2.3.0) > {code:java} > 2018-04-23 14:22:12,475 INFO [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > service.QueryService:429 : The original query: SELECT {fn > CURRENT_TIMESTAMP(0)} AS "COL" > 2018-04-23 14:22:12,526 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > runtime.CalciteException:59 : > org.apache.calcite.sql.validate.SqlValidatorException: Function '{fn > CURRENT_TIMESTAMP}' is not defined > 2018-04-23 14:22:12,535 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > runtime.CalciteException:61 : > org.apache.calcite.runtime.CalciteContextException: From line 1, column 8 to > line 1, column 32: Function '{fn CURRENT_TIMESTAMP}' is not defined > 2018-04-23 14:22:12,535 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > service.QueryService:536 : Exception while executing query > java.sql.SQLException: Error while preparing statement [SELECT {fn > CURRENT_TIMESTAMP(0)} AS "COL"] > at org.apache.calcite.avatica.Helper.createException(Helper.java:56) > at org.apache.calcite.avatica.Helper.createException(Helper.java:41) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement_(CalciteConnectionImpl.java:203) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:185) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:86) > at > org.apache.calcite.avatica.AvaticaConnection.prepareStatement(AvaticaConnection.java:168) > at > org.apache.kylin.rest.service.QueryService.getPrepareOnlySqlResponse(QueryService.java:994) > at org.apache.kylin.rest.service.QueryService.execute(QueryService.java:916) > at > org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:661) > at org.apache.kylin.rest.service.QueryService.query(QueryService.java:188) > at > org.apache.kylin.rest.service.QueryService.queryAndUpdateCache(QueryService.java:496) > at > org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:455) > at > org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:390) > at > org.apache.kylin.rest.controller.QueryController.prepareQuery(QueryController.java:99) > at sun.reflect.GeneratedMethodAccessor138.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) > at > org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) > at > org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) > at > org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) > at > org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967) > at > org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) > at > org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) > at > org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:650) > at > org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) > at javax.servlet.http.HttpServlet.
[jira] [Commented] (KYLIN-3349) Cube Build NumberFormatException when using Spark
[ https://issues.apache.org/jira/browse/KYLIN-3349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16452259#comment-16452259 ] Shaofeng SHI commented on KYLIN-3349: - Hello Hokyung, I think this is a data quality issue: the column is declared as "BigInt", while its value (like '0.000') is not. To ensure the Cube data is accurate, Kylin won't tolerate such error I think. The correct behavior is to fix the source data. > Cube Build NumberFormatException when using Spark > - > > Key: KYLIN-3349 > URL: https://issues.apache.org/jira/browse/KYLIN-3349 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.2.0, v2.3.0, v2.3.1 >Reporter: Hokyung Song >Priority: Major > > When I use spark engine to build cube, I have this error in spark when > building cube. > In my opinion, data has 0.00 as string, it cannot cast to long or double. > stack trace as follows > {code:java} > 2018-04-24 12:54:11,685 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : 18/04/24 > 12:54:11 WARN TaskSetManager: Lost task 193.0 in stage 0.0 (TID 1, hadoop, > executor 1): java.lang.NumberFormatException: For input string: "0." > 2018-04-24 12:54:11,686 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > 2018-04-24 12:54:11,686 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > java.lang.Long.parseLong(Long.java:589) > 2018-04-24 12:54:11,686 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > java.lang.Long.valueOf(Long.java:803) > 2018-04-24 12:54:11,686 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > org.apache.kylin.measure.basic.LongIngester.valueOf(LongIngester.java:38) > 2018-04-24 12:54:11,686 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > org.apache.kylin.measure.basic.LongIngester.valueOf(LongIngester.java:28) > 2018-04-24 12:54:11,686 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > org.apache.kylin.engine.mr.common.BaseCuboidBuilder.buildValueOf(BaseCuboidBuilder.java:163) > 2018-04-24 12:54:11,686 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > org.apache.kylin.engine.mr.common.BaseCuboidBuilder.buildValueObjects(BaseCuboidBuilder.java:128) > 2018-04-24 12:54:11,687 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > org.apache.kylin.engine.spark.SparkCubingByLayer$EncodeBaseCuboid.call(SparkCubingByLayer.java:309) > 2018-04-24 12:54:11,687 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > org.apache.kylin.engine.spark.SparkCubingByLayer$EncodeBaseCuboid.call(SparkCubingByLayer.java:271) > 2018-04-24 12:54:11,687 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1043) > 2018-04-24 12:54:11,687 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1043) > 2018-04-24 12:54:11,687 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > scala.collection.Iterator$$anon$11.next(Iterator.scala:409) > 2018-04-24 12:54:11,687 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:193) > 2018-04-24 12:54:11,687 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63) > 2018-04-24 12:54:11,687 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96) > 2018-04-24 12:54:11,687 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53) > 2018-04-24 12:54:11,688 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > org.apac
[jira] [Commented] (KYLIN-3349) Cube Build NumberFormatException when using Spark
[ https://issues.apache.org/jira/browse/KYLIN-3349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16451948#comment-16451948 ] ASF GitHub Bot commented on KYLIN-3349: --- codecov-io commented on issue #136: KYLIN-3349 : Spark Numberformat Exception solved URL: https://github.com/apache/kylin/pull/136#issuecomment-384223551 # [Codecov](https://codecov.io/gh/apache/kylin/pull/136?src=pr&el=h1) Report > Merging [#136](https://codecov.io/gh/apache/kylin/pull/136?src=pr&el=desc) into [master](https://codecov.io/gh/apache/kylin/commit/b1c8944471558afe63d61b6578c494decbae53cc?src=pr&el=desc) will **decrease** coverage by `0.01%`. > The diff coverage is `0%`. [![Impacted file tree graph](https://codecov.io/gh/apache/kylin/pull/136/graphs/tree.svg?height=150&width=650&token=JawVgbgsVo&src=pr)](https://codecov.io/gh/apache/kylin/pull/136?src=pr&el=tree) ```diff @@ Coverage Diff @@ ## master #136 +/- ## - Coverage 21.95% 21.93% -0.02% + Complexity 3898 3896 -2 Files 995 995 Lines 5998359983 Branches 8641 8641 - Hits 1316713157 -10 - Misses4559445607 +13 + Partials 1222 1219 -3 ``` | [Impacted Files](https://codecov.io/gh/apache/kylin/pull/136?src=pr&el=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...org/apache/kylin/measure/basic/DoubleIngester.java](https://codecov.io/gh/apache/kylin/pull/136/diff?src=pr&el=tree#diff-Y29yZS1tZXRhZGF0YS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUva3lsaW4vbWVhc3VyZS9iYXNpYy9Eb3VibGVJbmdlc3Rlci5qYXZh) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...a/org/apache/kylin/measure/basic/LongIngester.java](https://codecov.io/gh/apache/kylin/pull/136/diff?src=pr&el=tree#diff-Y29yZS1tZXRhZGF0YS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUva3lsaW4vbWVhc3VyZS9iYXNpYy9Mb25nSW5nZXN0ZXIuamF2YQ==) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [...rg/apache/kylin/storage/hbase/HBaseConnection.java](https://codecov.io/gh/apache/kylin/pull/136/diff?src=pr&el=tree#diff-c3RvcmFnZS1oYmFzZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUva3lsaW4vc3RvcmFnZS9oYmFzZS9IQmFzZUNvbm5lY3Rpb24uamF2YQ==) | `11.51% <0%> (-6.29%)` | `4% <0%> (-2%)` | | | [...g/apache/kylin/source/datagen/ColumnGenerator.java](https://codecov.io/gh/apache/kylin/pull/136/diff?src=pr&el=tree#diff-Y29yZS1tZXRhZGF0YS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUva3lsaW4vc291cmNlL2RhdGFnZW4vQ29sdW1uR2VuZXJhdG9yLmphdmE=) | `72.29% <0%> (+1.35%)` | `8% <0%> (ø)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/136?src=pr&el=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/kylin/pull/136?src=pr&el=footer). Last update [b1c8944...8e81ebb](https://codecov.io/gh/apache/kylin/pull/136?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Cube Build NumberFormatException when using Spark > - > > Key: KYLIN-3349 > URL: https://issues.apache.org/jira/browse/KYLIN-3349 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.2.0, v2.3.0, v2.3.1 >Reporter: Hokyung Song >Priority: Major > > When I use spark engine to build cube, I have this error in spark when > building cube. > In my opinion, data has 0.00 as string, it cannot cast to long or double. > stack trace as follows > {code:java} > 2018-04-24 12:54:11,685 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : 18/04/24 > 12:54:11 WARN TaskSetManager: Lost task 193.0 in stage 0.0 (TID 1, hadoop, > executor 1): java.lang.NumberFormatException: For input string: "0." > 2018-04-24 12:54:11,686 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > 2018-04-24 12:54:11,686 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-
[jira] [Commented] (KYLIN-3349) Cube Build NumberFormatException when using Spark
[ https://issues.apache.org/jira/browse/KYLIN-3349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16451937#comment-16451937 ] ASF GitHub Bot commented on KYLIN-3349: --- coveralls commented on issue #136: KYLIN-3349 : Spark Numberformat Exception solved URL: https://github.com/apache/kylin/pull/136#issuecomment-384218761 ## Pull Request Test Coverage Report for [Build 3123](https://coveralls.io/builds/16686243) * **0** of **2** **(0.0%)** changed or added relevant lines in **2** files are covered. * **14** unchanged lines in **1** file lost coverage. * Overall coverage decreased (**-0.02%**) to **23.97%** --- | Changes Missing Coverage | Covered Lines | Changed/Added Lines | % | | :-|--||---: | | [core-metadata/src/main/java/org/apache/kylin/measure/basic/DoubleIngester.java](https://coveralls.io/builds/16686243/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmeasure%2Fbasic%2FDoubleIngester.java#L39) | 0 | 1 | 0.0% | [core-metadata/src/main/java/org/apache/kylin/measure/basic/LongIngester.java](https://coveralls.io/builds/16686243/source?filename=core-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fmeasure%2Fbasic%2FLongIngester.java#L39) | 0 | 1 | 0.0% | Files with Coverage Reduction | New Missed Lines | % | | :-|--|--: | | [storage-hbase/src/main/java/org/apache/kylin/storage/hbase/HBaseConnection.java](https://coveralls.io/builds/16686243/source?filename=storage-hbase%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fstorage%2Fhbase%2FHBaseConnection.java#L80) | 14 | 15.18% | | Totals | [![Coverage Status](https://coveralls.io/builds/16686243/badge)](https://coveralls.io/builds/16686243) | | :-- | --: | | Change from base [Build 3118](https://coveralls.io/builds/16663108): | -0.02% | | Covered Lines: | 14378 | | Relevant Lines: | 59983 | --- # 💛 - [Coveralls](https://coveralls.io) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Cube Build NumberFormatException when using Spark > - > > Key: KYLIN-3349 > URL: https://issues.apache.org/jira/browse/KYLIN-3349 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.2.0, v2.3.0, v2.3.1 >Reporter: Hokyung Song >Priority: Major > > When I use spark engine to build cube, I have this error in spark when > building cube. > In my opinion, data has 0.00 as string, it cannot cast to long or double. > stack trace as follows > {code:java} > 2018-04-24 12:54:11,685 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : 18/04/24 > 12:54:11 WARN TaskSetManager: Lost task 193.0 in stage 0.0 (TID 1, hadoop, > executor 1): java.lang.NumberFormatException: For input string: "0." > 2018-04-24 12:54:11,686 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > 2018-04-24 12:54:11,686 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > java.lang.Long.parseLong(Long.java:589) > 2018-04-24 12:54:11,686 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > java.lang.Long.valueOf(Long.java:803) > 2018-04-24 12:54:11,686 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > org.apache.kylin.measure.basic.LongIngester.valueOf(LongIngester.java:38) > 2018-04-24 12:54:11,686 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > org.apache.kylin.measure.basic.LongIngester.valueOf(LongIngester.java:28) > 2018-04-24 12:54:11,686 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > org.apache.kylin.engine.mr.common.BaseCuboidBuilder.buildValueOf(BaseCuboidBuilder.java:163) > 2018-04-24 12:54:11,686 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > org.apache.kylin.engine.mr.common.BaseCuboidBuilder.buildValueObjects(BaseCuboidBuilder.java:128) > 2018-04-24 12:54:11,687 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > org.apache.kylin.engine.spark.SparkCubingByLayer$EncodeBaseCuboid.call(SparkCubingByLayer.java:309) > 20
[jira] [Commented] (KYLIN-3349) Cube Build NumberFormatException when using Spark
[ https://issues.apache.org/jira/browse/KYLIN-3349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16451912#comment-16451912 ] ASF GitHub Bot commented on KYLIN-3349: --- asfgit commented on issue #136: KYLIN-3349 : Spark Numberformat Exception solved URL: https://github.com/apache/kylin/pull/136#issuecomment-384212932 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Cube Build NumberFormatException when using Spark > - > > Key: KYLIN-3349 > URL: https://issues.apache.org/jira/browse/KYLIN-3349 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.2.0, v2.3.0, v2.3.1 >Reporter: Hokyung Song >Priority: Major > > When I use spark engine to build cube, I have this error in spark when > building cube. > In my opinion, data has 0.00 as string, it cannot cast to long or double. > stack trace as follows > {code:java} > 2018-04-24 12:54:11,685 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : 18/04/24 > 12:54:11 WARN TaskSetManager: Lost task 193.0 in stage 0.0 (TID 1, hadoop, > executor 1): java.lang.NumberFormatException: For input string: "0." > 2018-04-24 12:54:11,686 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > 2018-04-24 12:54:11,686 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > java.lang.Long.parseLong(Long.java:589) > 2018-04-24 12:54:11,686 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > java.lang.Long.valueOf(Long.java:803) > 2018-04-24 12:54:11,686 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > org.apache.kylin.measure.basic.LongIngester.valueOf(LongIngester.java:38) > 2018-04-24 12:54:11,686 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > org.apache.kylin.measure.basic.LongIngester.valueOf(LongIngester.java:28) > 2018-04-24 12:54:11,686 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > org.apache.kylin.engine.mr.common.BaseCuboidBuilder.buildValueOf(BaseCuboidBuilder.java:163) > 2018-04-24 12:54:11,686 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > org.apache.kylin.engine.mr.common.BaseCuboidBuilder.buildValueObjects(BaseCuboidBuilder.java:128) > 2018-04-24 12:54:11,687 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > org.apache.kylin.engine.spark.SparkCubingByLayer$EncodeBaseCuboid.call(SparkCubingByLayer.java:309) > 2018-04-24 12:54:11,687 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > org.apache.kylin.engine.spark.SparkCubingByLayer$EncodeBaseCuboid.call(SparkCubingByLayer.java:271) > 2018-04-24 12:54:11,687 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1043) > 2018-04-24 12:54:11,687 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1043) > 2018-04-24 12:54:11,687 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > scala.collection.Iterator$$anon$11.next(Iterator.scala:409) > 2018-04-24 12:54:11,687 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:193) > 2018-04-24 12:54:11,687 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63) > 2018-04-24 12:54:11,687 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96) > 2018-04-24 12:54:11,687 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-26
[jira] [Commented] (KYLIN-3349) Cube Build NumberFormatException when using Spark
[ https://issues.apache.org/jira/browse/KYLIN-3349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16451911#comment-16451911 ] ASF GitHub Bot commented on KYLIN-3349: --- imsong opened a new pull request #136: KYLIN-3349 : Spark Numberformat Exception solved URL: https://github.com/apache/kylin/pull/136 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Cube Build NumberFormatException when using Spark > - > > Key: KYLIN-3349 > URL: https://issues.apache.org/jira/browse/KYLIN-3349 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.2.0, v2.3.0, v2.3.1 >Reporter: Hokyung Song >Priority: Major > > When I use spark engine to build cube, I have this error in spark when > building cube. > In my opinion, data has 0.00 as string, it cannot cast to long or double. > stack trace as follows > {code:java} > 2018-04-24 12:54:11,685 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : 18/04/24 > 12:54:11 WARN TaskSetManager: Lost task 193.0 in stage 0.0 (TID 1, hadoop, > executor 1): java.lang.NumberFormatException: For input string: "0." > 2018-04-24 12:54:11,686 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > 2018-04-24 12:54:11,686 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > java.lang.Long.parseLong(Long.java:589) > 2018-04-24 12:54:11,686 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > java.lang.Long.valueOf(Long.java:803) > 2018-04-24 12:54:11,686 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > org.apache.kylin.measure.basic.LongIngester.valueOf(LongIngester.java:38) > 2018-04-24 12:54:11,686 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > org.apache.kylin.measure.basic.LongIngester.valueOf(LongIngester.java:28) > 2018-04-24 12:54:11,686 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > org.apache.kylin.engine.mr.common.BaseCuboidBuilder.buildValueOf(BaseCuboidBuilder.java:163) > 2018-04-24 12:54:11,686 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > org.apache.kylin.engine.mr.common.BaseCuboidBuilder.buildValueObjects(BaseCuboidBuilder.java:128) > 2018-04-24 12:54:11,687 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > org.apache.kylin.engine.spark.SparkCubingByLayer$EncodeBaseCuboid.call(SparkCubingByLayer.java:309) > 2018-04-24 12:54:11,687 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > org.apache.kylin.engine.spark.SparkCubingByLayer$EncodeBaseCuboid.call(SparkCubingByLayer.java:271) > 2018-04-24 12:54:11,687 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1043) > 2018-04-24 12:54:11,687 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1043) > 2018-04-24 12:54:11,687 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > scala.collection.Iterator$$anon$11.next(Iterator.scala:409) > 2018-04-24 12:54:11,687 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:193) > 2018-04-24 12:54:11,687 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63) > 2018-04-24 12:54:11,687 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96) > 2018-04-24 12:54:11,687 INFO [Scheduler 1401715751 Job > c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at > org.apache.spark.sc
[jira] [Commented] (KYLIN-3347) QueryService Exception when using calcite function ex : {fn CURRENT_TIMESTAMP(0)}
[ https://issues.apache.org/jira/browse/KYLIN-3347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16451908#comment-16451908 ] ASF GitHub Bot commented on KYLIN-3347: --- imsong opened a new pull request #135: KYLIN-3347 : calcite function QueryService Exception URL: https://github.com/apache/kylin/pull/135 sorry. Request again. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > QueryService Exception when using calcite function ex : {fn > CURRENT_TIMESTAMP(0)} > - > > Key: KYLIN-3347 > URL: https://issues.apache.org/jira/browse/KYLIN-3347 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: v2.3.0, v2.3.1 >Reporter: Hokyung Song >Priority: Major > > Hello. :) > I'm trying to upgrade kylin version to v2.3.x > But, when I use cacite function query, it occured error. (v2.2.0 is ok) > In my opinion, the reason is that regular expression matcher of > *DefaultQueryTransformer.java* has changed at v2.3.x > I will contribute a path on this problem soon. > thanks > (ps. I'm using Kylin with Tableau 10) > > stack trace as follows (v2.3.0) > {code:java} > 2018-04-23 14:22:12,475 INFO [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > service.QueryService:429 : The original query: SELECT {fn > CURRENT_TIMESTAMP(0)} AS "COL" > 2018-04-23 14:22:12,526 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > runtime.CalciteException:59 : > org.apache.calcite.sql.validate.SqlValidatorException: Function '{fn > CURRENT_TIMESTAMP}' is not defined > 2018-04-23 14:22:12,535 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > runtime.CalciteException:61 : > org.apache.calcite.runtime.CalciteContextException: From line 1, column 8 to > line 1, column 32: Function '{fn CURRENT_TIMESTAMP}' is not defined > 2018-04-23 14:22:12,535 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > service.QueryService:536 : Exception while executing query > java.sql.SQLException: Error while preparing statement [SELECT {fn > CURRENT_TIMESTAMP(0)} AS "COL"] > at org.apache.calcite.avatica.Helper.createException(Helper.java:56) > at org.apache.calcite.avatica.Helper.createException(Helper.java:41) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement_(CalciteConnectionImpl.java:203) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:185) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:86) > at > org.apache.calcite.avatica.AvaticaConnection.prepareStatement(AvaticaConnection.java:168) > at > org.apache.kylin.rest.service.QueryService.getPrepareOnlySqlResponse(QueryService.java:994) > at org.apache.kylin.rest.service.QueryService.execute(QueryService.java:916) > at > org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:661) > at org.apache.kylin.rest.service.QueryService.query(QueryService.java:188) > at > org.apache.kylin.rest.service.QueryService.queryAndUpdateCache(QueryService.java:496) > at > org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:455) > at > org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:390) > at > org.apache.kylin.rest.controller.QueryController.prepareQuery(QueryController.java:99) > at sun.reflect.GeneratedMethodAccessor138.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) > at > org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) > at > org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) > at > org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) > at > org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967) > at > org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.j
[jira] [Commented] (KYLIN-3347) QueryService Exception when using calcite function ex : {fn CURRENT_TIMESTAMP(0)}
[ https://issues.apache.org/jira/browse/KYLIN-3347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16451909#comment-16451909 ] ASF GitHub Bot commented on KYLIN-3347: --- asfgit commented on issue #135: KYLIN-3347 : calcite function QueryService Exception URL: https://github.com/apache/kylin/pull/135#issuecomment-384211836 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > QueryService Exception when using calcite function ex : {fn > CURRENT_TIMESTAMP(0)} > - > > Key: KYLIN-3347 > URL: https://issues.apache.org/jira/browse/KYLIN-3347 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: v2.3.0, v2.3.1 >Reporter: Hokyung Song >Priority: Major > > Hello. :) > I'm trying to upgrade kylin version to v2.3.x > But, when I use cacite function query, it occured error. (v2.2.0 is ok) > In my opinion, the reason is that regular expression matcher of > *DefaultQueryTransformer.java* has changed at v2.3.x > I will contribute a path on this problem soon. > thanks > (ps. I'm using Kylin with Tableau 10) > > stack trace as follows (v2.3.0) > {code:java} > 2018-04-23 14:22:12,475 INFO [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > service.QueryService:429 : The original query: SELECT {fn > CURRENT_TIMESTAMP(0)} AS "COL" > 2018-04-23 14:22:12,526 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > runtime.CalciteException:59 : > org.apache.calcite.sql.validate.SqlValidatorException: Function '{fn > CURRENT_TIMESTAMP}' is not defined > 2018-04-23 14:22:12,535 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > runtime.CalciteException:61 : > org.apache.calcite.runtime.CalciteContextException: From line 1, column 8 to > line 1, column 32: Function '{fn CURRENT_TIMESTAMP}' is not defined > 2018-04-23 14:22:12,535 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > service.QueryService:536 : Exception while executing query > java.sql.SQLException: Error while preparing statement [SELECT {fn > CURRENT_TIMESTAMP(0)} AS "COL"] > at org.apache.calcite.avatica.Helper.createException(Helper.java:56) > at org.apache.calcite.avatica.Helper.createException(Helper.java:41) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement_(CalciteConnectionImpl.java:203) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:185) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:86) > at > org.apache.calcite.avatica.AvaticaConnection.prepareStatement(AvaticaConnection.java:168) > at > org.apache.kylin.rest.service.QueryService.getPrepareOnlySqlResponse(QueryService.java:994) > at org.apache.kylin.rest.service.QueryService.execute(QueryService.java:916) > at > org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:661) > at org.apache.kylin.rest.service.QueryService.query(QueryService.java:188) > at > org.apache.kylin.rest.service.QueryService.queryAndUpdateCache(QueryService.java:496) > at > org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:455) > at > org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:390) > at > org.apache.kylin.rest.controller.QueryController.prepareQuery(QueryController.java:99) > at sun.reflect.GeneratedMethodAccessor138.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) > at > org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) > at > org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) > at > org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) > at > org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967) > at > org.springframework.web.servlet.DispatcherSe
[jira] [Commented] (KYLIN-3347) QueryService Exception when using calcite function ex : {fn CURRENT_TIMESTAMP(0)}
[ https://issues.apache.org/jira/browse/KYLIN-3347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16451878#comment-16451878 ] ASF GitHub Bot commented on KYLIN-3347: --- imsong closed pull request #134: KYLIN-3347 : calcite function QueryService Exception URL: https://github.com/apache/kylin/pull/134 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/core-metadata/src/main/java/org/apache/kylin/measure/basic/DoubleIngester.java b/core-metadata/src/main/java/org/apache/kylin/measure/basic/DoubleIngester.java index 90ecb0d583..3707fdcc5d 100644 --- a/core-metadata/src/main/java/org/apache/kylin/measure/basic/DoubleIngester.java +++ b/core-metadata/src/main/java/org/apache/kylin/measure/basic/DoubleIngester.java @@ -18,6 +18,7 @@ package org.apache.kylin.measure.basic; +import java.math.BigDecimal; import java.util.Map; import org.apache.kylin.common.util.Dictionary; @@ -35,7 +36,7 @@ public Double valueOf(String[] values, MeasureDesc measureDesc, Map QueryService Exception when using calcite function ex : {fn > CURRENT_TIMESTAMP(0)} > - > > Key: KYLIN-3347 > URL: https://issues.apache.org/jira/browse/KYLIN-3347 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: v2.3.0, v2.3.1 >Reporter: Hokyung Song >Priority: Major > > Hello. :) > I'm trying to upgrade kylin version to v2.3.x > But, when I use cacite function query, it occured error. (v2.2.0 is ok) > In my opinion, the reason is that regular expression matcher of > *DefaultQueryTransformer.java* has changed at v2.3.x > I will contribute a path on this problem soon. > thanks > (ps. I'm using Kylin with Tableau 10) > > stack trace as follows (v2.3.0) > {code:java} > 2018-04-23 14:22:12,475 INFO [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > service.QueryService:429 : The original query: SELECT {fn > CURRENT_TIMESTAMP(0)} AS "COL" > 2018-04-23 14:22:12,526 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > runtime.CalciteException:59 : > org.apache.calcite.sql.validate.SqlValidatorException: Function '{fn > CURRENT_TIMESTAMP}' is not defined > 2018-04-23 14:22:12,535 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > runtime.CalciteException:61 : > org.apache.calcite.runtime.CalciteContextException: From line 1, column 8 to > line 1, column 32: Function '{fn CURRENT_TIMESTAMP}' is not defined > 2018-04-23 14:22:12,535 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > service.QueryService:536 : Exception while executing query > java.sql.SQLException: Error while preparing statement [SELECT {fn > CURRENT_TIMESTAMP(0)} AS "COL"] > at org.apache.calcite.avatica.Helper.createException(Helper.java:56) > at org.apache.calcite.avatica.Helper.createException(Helper.java:41) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement_(CalciteConnectionImpl.java:203) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:185) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:86) > at > org.apache.calcite.avatica.AvaticaConnection.prepareStatement(AvaticaConnection.java:168) > at > org.apache.kylin.rest.service.QueryService.getPrepareOnlySqlResponse(QueryService.java:994) > at org.apache.kylin.rest.service.QueryService.execute(QueryService.java:916) > at > org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:661) > at org.apache.kylin.rest.service.QueryService.query(QueryService.java:188) > at > org.apache.kylin.rest.service.QueryService.queryAndUpdateCache(QueryService.java:496) > at > org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:455) > at > org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:390) > at > org.apache.kylin.rest.controller.QueryController.prepareQuery(QueryController.java:99) > at sun.reflect.GeneratedMethodAccessor138.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) > at > org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) > at > org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) > at > org.springframework.web.servlet.mvc.
[jira] [Updated] (KYLIN-3347) QueryService Exception when using calcite function ex : {fn CURRENT_TIMESTAMP(0)}
[ https://issues.apache.org/jira/browse/KYLIN-3347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hokyung Song updated KYLIN-3347: Summary: QueryService Exception when using calcite function ex : {fn CURRENT_TIMESTAMP(0)} (was: QueryService Exception : {fn CURRENT_TIMESTAMP(0)}) > QueryService Exception when using calcite function ex : {fn > CURRENT_TIMESTAMP(0)} > - > > Key: KYLIN-3347 > URL: https://issues.apache.org/jira/browse/KYLIN-3347 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: v2.3.0, v2.3.1 >Reporter: Hokyung Song >Priority: Major > > Hello. :) > I'm trying to upgrade kylin version to v2.3.x > But, when I use cacite function query, it occured error. (v2.2.0 is ok) > In my opinion, the reason is that regular expression matcher of > *DefaultQueryTransformer.java* has changed at v2.3.x > I will contribute a path on this problem soon. > thanks > (ps. I'm using Kylin with Tableau 10) > > stack trace as follows (v2.3.0) > {code:java} > 2018-04-23 14:22:12,475 INFO [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > service.QueryService:429 : The original query: SELECT {fn > CURRENT_TIMESTAMP(0)} AS "COL" > 2018-04-23 14:22:12,526 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > runtime.CalciteException:59 : > org.apache.calcite.sql.validate.SqlValidatorException: Function '{fn > CURRENT_TIMESTAMP}' is not defined > 2018-04-23 14:22:12,535 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > runtime.CalciteException:61 : > org.apache.calcite.runtime.CalciteContextException: From line 1, column 8 to > line 1, column 32: Function '{fn CURRENT_TIMESTAMP}' is not defined > 2018-04-23 14:22:12,535 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > service.QueryService:536 : Exception while executing query > java.sql.SQLException: Error while preparing statement [SELECT {fn > CURRENT_TIMESTAMP(0)} AS "COL"] > at org.apache.calcite.avatica.Helper.createException(Helper.java:56) > at org.apache.calcite.avatica.Helper.createException(Helper.java:41) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement_(CalciteConnectionImpl.java:203) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:185) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:86) > at > org.apache.calcite.avatica.AvaticaConnection.prepareStatement(AvaticaConnection.java:168) > at > org.apache.kylin.rest.service.QueryService.getPrepareOnlySqlResponse(QueryService.java:994) > at org.apache.kylin.rest.service.QueryService.execute(QueryService.java:916) > at > org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:661) > at org.apache.kylin.rest.service.QueryService.query(QueryService.java:188) > at > org.apache.kylin.rest.service.QueryService.queryAndUpdateCache(QueryService.java:496) > at > org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:455) > at > org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:390) > at > org.apache.kylin.rest.controller.QueryController.prepareQuery(QueryController.java:99) > at sun.reflect.GeneratedMethodAccessor138.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) > at > org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) > at > org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) > at > org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) > at > org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967) > at > org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) > at > org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) > at > org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:650) > at > org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) > at javax.servlet.http.HttpServlet
[jira] [Created] (KYLIN-3349) Cube Build NumberFormatException when using Spark
Hokyung Song created KYLIN-3349: --- Summary: Cube Build NumberFormatException when using Spark Key: KYLIN-3349 URL: https://issues.apache.org/jira/browse/KYLIN-3349 Project: Kylin Issue Type: Bug Components: Job Engine Affects Versions: v2.3.1, v2.3.0, v2.2.0 Reporter: Hokyung Song When I use spark engine to build cube, I have this error in spark when building cube. In my opinion, data has 0.00 as string, it cannot cast to long or double. stack trace as follows {code:java} 2018-04-24 12:54:11,685 INFO [Scheduler 1401715751 Job c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : 18/04/24 12:54:11 WARN TaskSetManager: Lost task 193.0 in stage 0.0 (TID 1, hadoop, executor 1): java.lang.NumberFormatException: For input string: "0." 2018-04-24 12:54:11,686 INFO [Scheduler 1401715751 Job c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) 2018-04-24 12:54:11,686 INFO [Scheduler 1401715751 Job c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at java.lang.Long.parseLong(Long.java:589) 2018-04-24 12:54:11,686 INFO [Scheduler 1401715751 Job c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at java.lang.Long.valueOf(Long.java:803) 2018-04-24 12:54:11,686 INFO [Scheduler 1401715751 Job c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at org.apache.kylin.measure.basic.LongIngester.valueOf(LongIngester.java:38) 2018-04-24 12:54:11,686 INFO [Scheduler 1401715751 Job c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at org.apache.kylin.measure.basic.LongIngester.valueOf(LongIngester.java:28) 2018-04-24 12:54:11,686 INFO [Scheduler 1401715751 Job c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at org.apache.kylin.engine.mr.common.BaseCuboidBuilder.buildValueOf(BaseCuboidBuilder.java:163) 2018-04-24 12:54:11,686 INFO [Scheduler 1401715751 Job c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at org.apache.kylin.engine.mr.common.BaseCuboidBuilder.buildValueObjects(BaseCuboidBuilder.java:128) 2018-04-24 12:54:11,687 INFO [Scheduler 1401715751 Job c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at org.apache.kylin.engine.spark.SparkCubingByLayer$EncodeBaseCuboid.call(SparkCubingByLayer.java:309) 2018-04-24 12:54:11,687 INFO [Scheduler 1401715751 Job c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at org.apache.kylin.engine.spark.SparkCubingByLayer$EncodeBaseCuboid.call(SparkCubingByLayer.java:271) 2018-04-24 12:54:11,687 INFO [Scheduler 1401715751 Job c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1043) 2018-04-24 12:54:11,687 INFO [Scheduler 1401715751 Job c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1043) 2018-04-24 12:54:11,687 INFO [Scheduler 1401715751 Job c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at scala.collection.Iterator$$anon$11.next(Iterator.scala:409) 2018-04-24 12:54:11,687 INFO [Scheduler 1401715751 Job c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:193) 2018-04-24 12:54:11,687 INFO [Scheduler 1401715751 Job c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63) 2018-04-24 12:54:11,687 INFO [Scheduler 1401715751 Job c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96) 2018-04-24 12:54:11,687 INFO [Scheduler 1401715751 Job c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53) 2018-04-24 12:54:11,688 INFO [Scheduler 1401715751 Job c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at org.apache.spark.scheduler.Task.run(Task.scala:99) 2018-04-24 12:54:11,688 INFO [Scheduler 1401715751 Job c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325) 2018-04-24 12:54:11,688 INFO [Scheduler 1401715751 Job c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 2018-04-24 12:54:11,688 INFO [Scheduler 1401715751 Job c1e5e47c-89fc-4ad6-8ae0-629879919aa5-264] spark.SparkExecutable:38 : at java.util.concurrent.ThreadPoolExecutor$Worker.run(Threa
[jira] [Updated] (KYLIN-3347) QueryService Exception : {fn CURRENT_TIMESTAMP(0)}
[ https://issues.apache.org/jira/browse/KYLIN-3347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hokyung Song updated KYLIN-3347: Description: Hello. :) I'm trying to upgrade kylin version to v2.3.x But, when I use cacite function query, it occured error. (v2.2.0 is ok) In my opinion, the reason is that regular expression matcher of *DefaultQueryTransformer.java* has changed at v2.3.x I will contribute a path on this problem soon. thanks (ps. I'm using Kylin with Tableau 10) stack trace as follows (v2.3.0) {code:java} 2018-04-23 14:22:12,475 INFO [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] service.QueryService:429 : The original query: SELECT {fn CURRENT_TIMESTAMP(0)} AS "COL" 2018-04-23 14:22:12,526 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] runtime.CalciteException:59 : org.apache.calcite.sql.validate.SqlValidatorException: Function '{fn CURRENT_TIMESTAMP}' is not defined 2018-04-23 14:22:12,535 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] runtime.CalciteException:61 : org.apache.calcite.runtime.CalciteContextException: From line 1, column 8 to line 1, column 32: Function '{fn CURRENT_TIMESTAMP}' is not defined 2018-04-23 14:22:12,535 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] service.QueryService:536 : Exception while executing query java.sql.SQLException: Error while preparing statement [SELECT {fn CURRENT_TIMESTAMP(0)} AS "COL"] at org.apache.calcite.avatica.Helper.createException(Helper.java:56) at org.apache.calcite.avatica.Helper.createException(Helper.java:41) at org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement_(CalciteConnectionImpl.java:203) at org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:185) at org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:86) at org.apache.calcite.avatica.AvaticaConnection.prepareStatement(AvaticaConnection.java:168) at org.apache.kylin.rest.service.QueryService.getPrepareOnlySqlResponse(QueryService.java:994) at org.apache.kylin.rest.service.QueryService.execute(QueryService.java:916) at org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:661) at org.apache.kylin.rest.service.QueryService.query(QueryService.java:188) at org.apache.kylin.rest.service.QueryService.queryAndUpdateCache(QueryService.java:496) at org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:455) at org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:390) at org.apache.kylin.rest.controller.QueryController.prepareQuery(QueryController.java:99) at sun.reflect.GeneratedMethodAccessor138.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872) at javax.servlet.http.HttpServlet.service(HttpServlet.java:650) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127) at org.springframework.security.
[jira] [Commented] (KYLIN-3347) QueryService Exception : {fn CURRENT_TIMESTAMP(0)}
[ https://issues.apache.org/jira/browse/KYLIN-3347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16451837#comment-16451837 ] ASF GitHub Bot commented on KYLIN-3347: --- imsong opened a new pull request #134: KYLIN-3347 : calcite function QueryService Exception URL: https://github.com/apache/kylin/pull/134 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > QueryService Exception : {fn CURRENT_TIMESTAMP(0)} > -- > > Key: KYLIN-3347 > URL: https://issues.apache.org/jira/browse/KYLIN-3347 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: v2.3.0, v2.3.1 >Reporter: Hokyung Song >Priority: Major > > Hello. :) > I'm trying to upgrade kylin version to v2.3.x > But, when I use cacite function query, it occured error. (v2.2.0 is ok) > In my opinion, the reason is that regular expression matcher of > *DefaultQueryTransformer.java* has changed. > I will contribute a path on this problem soon. > thanks > (ps. I'm using Kylin with Tableau 10) > > stack trace as follows (v2.3.0) > {code:java} > 2018-04-23 14:22:12,475 INFO [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > service.QueryService:429 : The original query: SELECT {fn > CURRENT_TIMESTAMP(0)} AS "COL" > 2018-04-23 14:22:12,526 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > runtime.CalciteException:59 : > org.apache.calcite.sql.validate.SqlValidatorException: Function '{fn > CURRENT_TIMESTAMP}' is not defined > 2018-04-23 14:22:12,535 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > runtime.CalciteException:61 : > org.apache.calcite.runtime.CalciteContextException: From line 1, column 8 to > line 1, column 32: Function '{fn CURRENT_TIMESTAMP}' is not defined > 2018-04-23 14:22:12,535 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > service.QueryService:536 : Exception while executing query > java.sql.SQLException: Error while preparing statement [SELECT {fn > CURRENT_TIMESTAMP(0)} AS "COL"] > at org.apache.calcite.avatica.Helper.createException(Helper.java:56) > at org.apache.calcite.avatica.Helper.createException(Helper.java:41) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement_(CalciteConnectionImpl.java:203) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:185) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:86) > at > org.apache.calcite.avatica.AvaticaConnection.prepareStatement(AvaticaConnection.java:168) > at > org.apache.kylin.rest.service.QueryService.getPrepareOnlySqlResponse(QueryService.java:994) > at org.apache.kylin.rest.service.QueryService.execute(QueryService.java:916) > at > org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:661) > at org.apache.kylin.rest.service.QueryService.query(QueryService.java:188) > at > org.apache.kylin.rest.service.QueryService.queryAndUpdateCache(QueryService.java:496) > at > org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:455) > at > org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:390) > at > org.apache.kylin.rest.controller.QueryController.prepareQuery(QueryController.java:99) > at sun.reflect.GeneratedMethodAccessor138.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) > at > org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) > at > org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) > at > org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) > at > org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967) > at > org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) > at > org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServl
[jira] [Commented] (KYLIN-3347) QueryService Exception : {fn CURRENT_TIMESTAMP(0)}
[ https://issues.apache.org/jira/browse/KYLIN-3347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16451838#comment-16451838 ] ASF GitHub Bot commented on KYLIN-3347: --- asfgit commented on issue #134: KYLIN-3347 : calcite function QueryService Exception URL: https://github.com/apache/kylin/pull/134#issuecomment-384197575 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > QueryService Exception : {fn CURRENT_TIMESTAMP(0)} > -- > > Key: KYLIN-3347 > URL: https://issues.apache.org/jira/browse/KYLIN-3347 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: v2.3.0, v2.3.1 >Reporter: Hokyung Song >Priority: Major > > Hello. :) > I'm trying to upgrade kylin version to v2.3.x > But, when I use cacite function query, it occured error. (v2.2.0 is ok) > In my opinion, the reason is that regular expression matcher of > *DefaultQueryTransformer.java* has changed. > I will contribute a path on this problem soon. > thanks > (ps. I'm using Kylin with Tableau 10) > > stack trace as follows (v2.3.0) > {code:java} > 2018-04-23 14:22:12,475 INFO [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > service.QueryService:429 : The original query: SELECT {fn > CURRENT_TIMESTAMP(0)} AS "COL" > 2018-04-23 14:22:12,526 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > runtime.CalciteException:59 : > org.apache.calcite.sql.validate.SqlValidatorException: Function '{fn > CURRENT_TIMESTAMP}' is not defined > 2018-04-23 14:22:12,535 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > runtime.CalciteException:61 : > org.apache.calcite.runtime.CalciteContextException: From line 1, column 8 to > line 1, column 32: Function '{fn CURRENT_TIMESTAMP}' is not defined > 2018-04-23 14:22:12,535 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > service.QueryService:536 : Exception while executing query > java.sql.SQLException: Error while preparing statement [SELECT {fn > CURRENT_TIMESTAMP(0)} AS "COL"] > at org.apache.calcite.avatica.Helper.createException(Helper.java:56) > at org.apache.calcite.avatica.Helper.createException(Helper.java:41) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement_(CalciteConnectionImpl.java:203) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:185) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:86) > at > org.apache.calcite.avatica.AvaticaConnection.prepareStatement(AvaticaConnection.java:168) > at > org.apache.kylin.rest.service.QueryService.getPrepareOnlySqlResponse(QueryService.java:994) > at org.apache.kylin.rest.service.QueryService.execute(QueryService.java:916) > at > org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:661) > at org.apache.kylin.rest.service.QueryService.query(QueryService.java:188) > at > org.apache.kylin.rest.service.QueryService.queryAndUpdateCache(QueryService.java:496) > at > org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:455) > at > org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:390) > at > org.apache.kylin.rest.controller.QueryController.prepareQuery(QueryController.java:99) > at sun.reflect.GeneratedMethodAccessor138.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) > at > org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) > at > org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) > at > org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) > at > org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967) > at > org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) > at > org.springframework.we
[jira] [Updated] (KYLIN-3347) QueryService Exception : {fn CURRENT_TIMESTAMP(0)}
[ https://issues.apache.org/jira/browse/KYLIN-3347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hokyung Song updated KYLIN-3347: Description: Hello. :) I'm trying to upgrade kylin version to v2.3.x But, when I use cacite function query, it occured error. (v2.2.0 is ok) In my opinion, the reason is that regular expression matcher of *DefaultQueryTransformer.java* has changed. I will contribute a path on this problem soon. thanks (ps. I'm using with Tableau 10) stack trace as follows (v2.3.0) {code:java} 2018-04-23 14:22:12,475 INFO [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] service.QueryService:429 : The original query: SELECT {fn CURRENT_TIMESTAMP(0)} AS "COL" 2018-04-23 14:22:12,526 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] runtime.CalciteException:59 : org.apache.calcite.sql.validate.SqlValidatorException: Function '{fn CURRENT_TIMESTAMP}' is not defined 2018-04-23 14:22:12,535 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] runtime.CalciteException:61 : org.apache.calcite.runtime.CalciteContextException: From line 1, column 8 to line 1, column 32: Function '{fn CURRENT_TIMESTAMP}' is not defined 2018-04-23 14:22:12,535 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] service.QueryService:536 : Exception while executing query java.sql.SQLException: Error while preparing statement [SELECT {fn CURRENT_TIMESTAMP(0)} AS "COL"] at org.apache.calcite.avatica.Helper.createException(Helper.java:56) at org.apache.calcite.avatica.Helper.createException(Helper.java:41) at org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement_(CalciteConnectionImpl.java:203) at org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:185) at org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:86) at org.apache.calcite.avatica.AvaticaConnection.prepareStatement(AvaticaConnection.java:168) at org.apache.kylin.rest.service.QueryService.getPrepareOnlySqlResponse(QueryService.java:994) at org.apache.kylin.rest.service.QueryService.execute(QueryService.java:916) at org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:661) at org.apache.kylin.rest.service.QueryService.query(QueryService.java:188) at org.apache.kylin.rest.service.QueryService.queryAndUpdateCache(QueryService.java:496) at org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:455) at org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:390) at org.apache.kylin.rest.controller.QueryController.prepareQuery(QueryController.java:99) at sun.reflect.GeneratedMethodAccessor138.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872) at javax.servlet.http.HttpServlet.service(HttpServlet.java:650) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127) at org.springframework.security.web.access.inte
[jira] [Updated] (KYLIN-3347) QueryService Exception : {fn CURRENT_TIMESTAMP(0)}
[ https://issues.apache.org/jira/browse/KYLIN-3347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hokyung Song updated KYLIN-3347: Description: Hello. :) I'm trying to upgrade kylin version to v2.3.x But, when I use cacite function query, it occured error. (v2.2.0 is ok) In my opinion, the reason is that regular expression matcher of *DefaultQueryTransformer.java* has changed. I will contribute a path on this problem soon. thanks (ps. I'm using Kylin with Tableau 10) stack trace as follows (v2.3.0) {code:java} 2018-04-23 14:22:12,475 INFO [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] service.QueryService:429 : The original query: SELECT {fn CURRENT_TIMESTAMP(0)} AS "COL" 2018-04-23 14:22:12,526 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] runtime.CalciteException:59 : org.apache.calcite.sql.validate.SqlValidatorException: Function '{fn CURRENT_TIMESTAMP}' is not defined 2018-04-23 14:22:12,535 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] runtime.CalciteException:61 : org.apache.calcite.runtime.CalciteContextException: From line 1, column 8 to line 1, column 32: Function '{fn CURRENT_TIMESTAMP}' is not defined 2018-04-23 14:22:12,535 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] service.QueryService:536 : Exception while executing query java.sql.SQLException: Error while preparing statement [SELECT {fn CURRENT_TIMESTAMP(0)} AS "COL"] at org.apache.calcite.avatica.Helper.createException(Helper.java:56) at org.apache.calcite.avatica.Helper.createException(Helper.java:41) at org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement_(CalciteConnectionImpl.java:203) at org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:185) at org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:86) at org.apache.calcite.avatica.AvaticaConnection.prepareStatement(AvaticaConnection.java:168) at org.apache.kylin.rest.service.QueryService.getPrepareOnlySqlResponse(QueryService.java:994) at org.apache.kylin.rest.service.QueryService.execute(QueryService.java:916) at org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:661) at org.apache.kylin.rest.service.QueryService.query(QueryService.java:188) at org.apache.kylin.rest.service.QueryService.queryAndUpdateCache(QueryService.java:496) at org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:455) at org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:390) at org.apache.kylin.rest.controller.QueryController.prepareQuery(QueryController.java:99) at sun.reflect.GeneratedMethodAccessor138.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872) at javax.servlet.http.HttpServlet.service(HttpServlet.java:650) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127) at org.springframework.security.web.acces
[jira] [Updated] (KYLIN-3347) QueryService Exception : {fn CURRENT_TIMESTAMP(0)}
[ https://issues.apache.org/jira/browse/KYLIN-3347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hokyung Song updated KYLIN-3347: Description: Hello. :) I'm trying to upgrade kylin version to v2.3.x But, when I use cacite function query, it occured error. (v2.2.0 is ok) In my opinion, the reason is that regular expression matcher of *DefaultQueryTransformer.java* has changed. I will contribute a path on this problem soon. thanks. stack trace as follows (v2.3.0) {code:java} 2018-04-23 14:22:12,475 INFO [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] service.QueryService:429 : The original query: SELECT {fn CURRENT_TIMESTAMP(0)} AS "COL" 2018-04-23 14:22:12,526 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] runtime.CalciteException:59 : org.apache.calcite.sql.validate.SqlValidatorException: Function '{fn CURRENT_TIMESTAMP}' is not defined 2018-04-23 14:22:12,535 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] runtime.CalciteException:61 : org.apache.calcite.runtime.CalciteContextException: From line 1, column 8 to line 1, column 32: Function '{fn CURRENT_TIMESTAMP}' is not defined 2018-04-23 14:22:12,535 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] service.QueryService:536 : Exception while executing query java.sql.SQLException: Error while preparing statement [SELECT {fn CURRENT_TIMESTAMP(0)} AS "COL"] at org.apache.calcite.avatica.Helper.createException(Helper.java:56) at org.apache.calcite.avatica.Helper.createException(Helper.java:41) at org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement_(CalciteConnectionImpl.java:203) at org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:185) at org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:86) at org.apache.calcite.avatica.AvaticaConnection.prepareStatement(AvaticaConnection.java:168) at org.apache.kylin.rest.service.QueryService.getPrepareOnlySqlResponse(QueryService.java:994) at org.apache.kylin.rest.service.QueryService.execute(QueryService.java:916) at org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:661) at org.apache.kylin.rest.service.QueryService.query(QueryService.java:188) at org.apache.kylin.rest.service.QueryService.queryAndUpdateCache(QueryService.java:496) at org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:455) at org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:390) at org.apache.kylin.rest.controller.QueryController.prepareQuery(QueryController.java:99) at sun.reflect.GeneratedMethodAccessor138.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872) at javax.servlet.http.HttpServlet.service(HttpServlet.java:650) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.
[jira] [Updated] (KYLIN-3347) QueryService Exception : {fn CURRENT_TIMESTAMP(0)}
[ https://issues.apache.org/jira/browse/KYLIN-3347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hokyung Song updated KYLIN-3347: Summary: QueryService Exception : {fn CURRENT_TIMESTAMP(0)} (was: {fn CURRENT_TIMESTAMP(0)} QueryService Exception) > QueryService Exception : {fn CURRENT_TIMESTAMP(0)} > -- > > Key: KYLIN-3347 > URL: https://issues.apache.org/jira/browse/KYLIN-3347 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: v2.3.0, v2.3.1 >Reporter: Hokyung Song >Priority: Major > > Hello. :) > I'm trying to upgrade kylin version to v2.3.x > But, when I use cacite function query, it occured error. (v2.2.0 is ok) > In my opinion, the reason is that regular expression matcher of > *DefaultQueryTransformer.java* has changed. > I will contribute a path on this problem soon. > thanks. > > stack trace as follows (v2.3.0) > {code:java} > 2018-04-23 14:22:12,475 INFO [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > service.QueryService:429 : The original query: SELECT {fn > CURRENT_TIMESTAMP(0)} AS "COL" > 2018-04-23 14:22:12,526 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > runtime.CalciteException:59 : > org.apache.calcite.sql.validate.SqlValidatorException: Function '{fn > CURRENT_TIMESTAMP}' is not defined > 2018-04-23 14:22:12,535 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > runtime.CalciteException:61 : > org.apache.calcite.runtime.CalciteContextException: From line 1, column 8 to > line 1, column 32: Function '{fn CURRENT_TIMESTAMP}' is not defined > 2018-04-23 14:22:12,535 ERROR [Query f8811967-be03-4fd0-b11f-85c68de0abcb-50] > service.QueryService:536 : Exception while executing query > java.sql.SQLException: Error while preparing statement [SELECT {fn > CURRENT_TIMESTAMP(0)} AS "COL"] > at org.apache.calcite.avatica.Helper.createException(Helper.java:56) > at org.apache.calcite.avatica.Helper.createException(Helper.java:41) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement_(CalciteConnectionImpl.java:203) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:185) > at > org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:86) > at > org.apache.calcite.avatica.AvaticaConnection.prepareStatement(AvaticaConnection.java:168) > at > org.apache.kylin.rest.service.QueryService.getPrepareOnlySqlResponse(QueryService.java:994) > at org.apache.kylin.rest.service.QueryService.execute(QueryService.java:916) > at > org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:661) > at org.apache.kylin.rest.service.QueryService.query(QueryService.java:188) > at > org.apache.kylin.rest.service.QueryService.queryAndUpdateCache(QueryService.java:496) > at > org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:455) > at > org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:390) > at > org.apache.kylin.rest.controller.QueryController.prepareQuery(QueryController.java:99) > at sun.reflect.GeneratedMethodAccessor138.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) > at > org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) > at > org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) > at > org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) > at > org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967) > at > org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) > at > org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) > at > org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:650) > at > org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) > at > o
[jira] [Created] (KYLIN-3348) "missing LastBuildJobID" error when building new cube segment
liyang created KYLIN-3348: - Summary: "missing LastBuildJobID" error when building new cube segment Key: KYLIN-3348 URL: https://issues.apache.org/jira/browse/KYLIN-3348 Project: Kylin Issue Type: Bug Reporter: liyang An unstable exception. Likely to happen when there are multiple concurrent builds. {{2018-04-18 20:11:16,856 ERROR [pool-33-thread-11] threadpool.DefaultScheduler : ExecuteException job:cc08da19-f53e-4344-a6c5-05e764834924}} {{ org.apache.kylin.job.exception.ExecuteException: org.apache.kylin.job.exception.ExecuteException: java.lang.IllegalStateException: For cube CUBE[name=cube2], segment cube2[2018041423000_2018041423001] missing LastBuildJobID}} \{{ at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:140)}} \{{ at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:307)}} \{{ at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)}} \{{ at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)}} \{{ at java.lang.Thread.run(Thread.java:748)}} {{ Caused by: org.apache.kylin.job.exception.ExecuteException: java.lang.IllegalStateException: For cube CUBE[name=cube2], segment cube2[2018041423000_2018041423001] missing LastBuildJobID}} \{{ at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:140)}} \{{ at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:67)}} \{{ at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:129)}} \{{ ... 4 more}} {{ Caused by: java.lang.IllegalStateException: For cube CUBE[name=cube2], segment cube2[2018041423000_2018041423001] missing LastBuildJobID}} \{{ at org.apache.kylin.cube.CubeManager$SegmentAssist.promoteNewlyBuiltSegments(CubeManager.java:810)}} \{{ at org.apache.kylin.cube.CubeManager.promoteNewlyBuiltSegments(CubeManager.java:535)}} \{{ at org.apache.kylin.engine.mr.steps.UpdateCubeInfoAfterBuildStep.doWork(UpdateCubeInfoAfterBuildStep.java:78)}} \{{ at io.kyligence.kap.engine.mr.steps.KapUpdateCubeInfoAfterBuildStep.doWork(SourceFile:47)}} \{{ at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:129)}} \{{ ... 6 more}} -- This message was sent by Atlassian JIRA (v7.6.3#76005)