[jira] [Commented] (KYLIN-3057) In some cases，query failed when sql with comments
[
https://issues.apache.org/jira/browse/KYLIN-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270334#comment-16270334
]
peng.jianhua commented on KYLIN-3057:
-
Hi,[~Shaofengshi],I just wonder that why we need the Calcite parse the comments
in the sql again.
Kylin has checked the sql is select statement or not and remove the comments in
sql.
why we still using the original sql as parameters to query instead of sql with
no comments.
especially the Calcite can't parse the sql very well.
> In some cases，query failed when sql with comments
> -
>
> Key: KYLIN-3057
> URL: https://issues.apache.org/jira/browse/KYLIN-3057
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v2.3.0
>Reporter: peng.jianhua
>Assignee: peng.jianhua
> Attachments:
> 0001-KYLIN-3057-In-some-cases-query-failed-when-sql-with-.patch,
> no_comments_sql.PNG, with_comments_sql.PNG
>
>
> no_comments_sql.PNG: select trans_id from kylin_sales;
> with_comments_sql.PNG : select trans_id from kylin_sales;--comments
> if the sql end with semicolon and there are some comments behind semicolon,we
> will meet the error
> the following is the result of with_comments_sql.PNG:
> Results
> Encountered ";" at line 1, column 33. Was expecting one of: "ORDER" ...
> "LIMIT" ... "OFFSET" ... "FETCH" ... "NATURAL" ... "JOIN" ... "INNER" ...
> "LEFT" ... "RIGHT" ... "FULL" ... "CROSS" ... "," ... "OUTER" ... "EXTEND"
> ... "(" ... "MATCH_RECOGNIZE" ... "." ... "AS" ... ...
> ... ...
> ... ... "TABLESAMPLE"
> ... "WHERE" ... "GROUP" ... "HAVING" ... "WINDOW" ... "UNION" ... "INTERSECT"
> ... "EXCEPT" ... "MINUS" ..
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Closed] (KYLIN-1133) when adding 2 same name columns from different dimension, the aggregation group and row key only use one of them
[ https://issues.apache.org/jira/browse/KYLIN-1133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-1133. --- Resolution: Fixed Fix Version/s: v2.0.0 This is fixed in Kylin 2.0, which allow each table has an alias to differenciate > when adding 2 same name columns from different dimension, the aggregation > group and row key only use one of them > > > Key: KYLIN-1133 > URL: https://issues.apache.org/jira/browse/KYLIN-1133 > Project: Kylin > Issue Type: Bug > Components: Web >Reporter: Haowei Ji >Assignee: Zhong,Jason > Fix For: v2.0.0 > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KYLIN-3067) Provide web interface for dimension capping feature
Roger Shi created KYLIN-3067: Summary: Provide web interface for dimension capping feature Key: KYLIN-3067 URL: https://issues.apache.org/jira/browse/KYLIN-3067 Project: Kylin Issue Type: Improvement Components: Web Reporter: Roger Shi Assignee: Zhong,Jason As [KYLIN-2363] has provided the dimension capping feature in the backend, we need a proper web interface. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-1131) Enabling saving incomplete cube
[ https://issues.apache.org/jira/browse/KYLIN-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-1131: Issue Type: Wish (was: Improvement) > Enabling saving incomplete cube > --- > > Key: KYLIN-1131 > URL: https://issues.apache.org/jira/browse/KYLIN-1131 > Project: Kylin > Issue Type: Wish > Components: Web >Reporter: Haowei Ji >Assignee: Zhong,Jason > > When adding cubes, the cube could only be saved when it comes to the last > step, and if user switch to other functions and back, it will disappear, > seems the web does not caching what user has just done, it's annoying to > start up again, especially when some cubes are complicated. So suggest to > caching the cube and adding save button in the process of creating the cube. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (KYLIN-1123) Make kylin start script vender independent
[ https://issues.apache.org/jira/browse/KYLIN-1123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-1123. --- Resolution: Fixed This has been fixed. > Make kylin start script vender independent > -- > > Key: KYLIN-1123 > URL: https://issues.apache.org/jira/browse/KYLIN-1123 > Project: Kylin > Issue Type: Bug >Reporter: hongbin ma >Assignee: hongbin ma > > currently some start up scripts assume that give hadoop env is hdp. we need > to remove such assumptions and make kylin work in all hadoops envs (at least > both hdp and cdh) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (KYLIN-2363) Prune cuboids by capping number of dimensions
[
https://issues.apache.org/jira/browse/KYLIN-2363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roger Shi resolved KYLIN-2363.
--
Resolution: Fixed
> Prune cuboids by capping number of dimensions
> -
>
> Key: KYLIN-2363
> URL: https://issues.apache.org/jira/browse/KYLIN-2363
> Project: Kylin
> Issue Type: Improvement
>Reporter: fengYu
>Assignee: Roger Shi
> Fix For: v2.3.0
>
> Attachments: Dimension Capping.md
>
>
> the scene like this:
> I have 20+ dimensions, However the query will only use at most 5 dimensions
> in all dimensions, so cuboid that contains 5+ dimensions(except base cuboid)
> is useless.
> I think we can add a configuration in cube, which limit the max dimensions
> that cuboid includes.
> What's more, we can config which level(number of dimension) need to
> calculate. in above scene, we only calculate leve 1,2,3,4,5. and skip level 5+
> =
> The dimension capping is turned on by adding dim_cap property in
> aggregation_groups definition.
> For example, the following aggregation group sets the dimension cap to 3. All
> cuboids containing more than 3 dimensions are skipped in this aggregation
> group.
> {code:none}
> "aggregation_groups" : [ {
> "includes" : [ "PART_DT", "META_CATEG_NAME", "CATEG_LVL2_NAME",
> "CATEG_LVL3_NAME", "LEAF_CATEG_ID", "LSTG_FORMAT_NAME", "LSTG_SITE_ID",
> "OPS_USER_ID", "OPS_REGION",
>"BUYER_ACCOUNT.ACCOUNT_BUYER_LEVEL",
> "SELLER_ACCOUNT.ACCOUNT_SELLER_LEVEL", "BUYER_ACCOUNT.ACCOUNT_COUNTRY",
> "SELLER_ACCOUNT.ACCOUNT_COUNTRY", "BUYER_COUNTRY.NAME", "SELLER_COUNTRY.NAME"
> ],
> "select_rule" : {
> "hierarchy_dims" : [ [ "META_CATEG_NAME", "CATEG_LVL2_NAME",
> "CATEG_LVL3_NAME", "LEAF_CATEG_ID" ] ],
> "mandatory_dims" : [ "PART_DT" ],
> "joint_dims" : [ [ "BUYER_ACCOUNT.ACCOUNT_COUNTRY",
> "BUYER_COUNTRY.NAME" ], [ "SELLER_ACCOUNT.ACCOUNT_COUNTRY",
> "SELLER_COUNTRY.NAME" ],
>[ "BUYER_ACCOUNT.ACCOUNT_BUYER_LEVEL",
> "SELLER_ACCOUNT.ACCOUNT_SELLER_LEVEL" ], [ "LSTG_FORMAT_NAME", "LSTG_SITE_ID"
> ], [ "OPS_USER_ID", "OPS_REGION" ] ],
> "dim_cap" : 3
> }
> } ]
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Closed] (KYLIN-1464) org.apache.kylin.cube.CubeInstance.getDataModelDesc throw java.lang.NullPointerException
[
https://issues.apache.org/jira/browse/KYLIN-1464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI closed KYLIN-1464.
---
Resolution: Fixed
This should have been fixed, please upgrade to a newer version.
> org.apache.kylin.cube.CubeInstance.getDataModelDesc throw
> java.lang.NullPointerException
>
>
> Key: KYLIN-1464
> URL: https://issues.apache.org/jira/browse/KYLIN-1464
> Project: Kylin
> Issue Type: Bug
>Reporter: yunjiong zhao
>Assignee: yunjiong zhao
> Attachments: KYLIN-1464.patch
>
>
> User report can't see models from Web UI.
> After I check log, I found java.lang.NullPointerException
> {quote}
> 2016-03-02 08:06:59,645 ERROR [http-bio-7070-exec-22]
> controller.BasicController:44 :
> java.lang.NullPointerException
> at
> org.apache.kylin.cube.CubeInstance.getDataModelDesc(CubeInstance.java:128)
> at
> org.apache.kylin.metadata.MetadataManager.getModels(MetadataManager.java:326)
> at
> org.apache.kylin.rest.service.ModelService.listAllModels(ModelService.java:55)
> at
> org.apache.kylin.rest.service.ModelService.getModels(ModelService.java:74)
> at
> org.apache.kylin.rest.service.ModelService$$FastClassByCGLIB$$eb0205be.invoke()
> at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
> at
> org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:618)
> at
> org.apache.kylin.rest.service.ModelService$$EnhancerByCGLIB$$2e750d6d.getModels()
> at
> org.apache.kylin.rest.controller.ModelController.getModels(ModelController.java:71)
> at sun.reflect.GeneratedMethodAccessor164.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:213)
> at
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:126)
> at
> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:96)
> at
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:617)
> at
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:578)
> at
> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80)
> at
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:923)
> at
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852)
> at
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)
> {quote}
> And before that we can see from log why it happens.
> {quote}
> 2016-03-02 08:06:59,238 INFO [http-bio-7070-exec-22]
> cube.CubeDescManager:197 : Reloading Cube Metadata from folder
> _kylin_metadata_qa2(key='/cube_desc')@_kylin_metadata_qa2@hbase
> 2016-03-02 08:06:59,632 ERROR [http-bio-7070-exec-22]
> cube.CubeDescManager:207 : Error loading cube desc /cube_desc/test4.json
> java.lang.IllegalStateException: Derived can only be defined on lookup table,
> cube CubeDesc [name=test4], DimensionDesc
> [name=DEFAULT.SLNG_CATEG_META_LVL2_DERIVED, join=null, hierarchy=null,
> table=DEFAULT.SLNG_CATEG_META_LVL2, column=null, derived=[CATEG_LVL2_NAME,
> META_CATEG_NAME]]
> at
> org.apache.kylin.cube.model.DimensionDesc.init(DimensionDesc.java:111)
> at org.apache.kylin.cube.model.CubeDesc.init(CubeDesc.java:461)
> at
> org.apache.kylin.cube.CubeDescManager.loadCubeDesc(CubeDescManager.java:134)
> at
> org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc(CubeDescManager.java:205)
> at
> org.apache.kylin.cube.CubeDescManager.(CubeDescManager.java:94)
> at
> org.apache.kylin.cube.CubeDescManager.getInstance(CubeDescManager.java:74)
> at
> org.apache.kylin.cube.CubeInstance.getDescriptor(CubeInstance.java:123)
> at
> org.apache.kylin.cube.CubeInstance.getDataModelDesc(CubeInstance.java:128)
> at
> org.apache.kylin.metadata.MetadataManager.getModels(MetadataManager.java:326)
> at
> org.apache.kylin.rest.service.ModelService.listAllModels(ModelService.java:55)
> at
> org.apache.kylin.rest.service.ModelService.getModels(ModelService.java:74)
> at
>
[jira] [Closed] (KYLIN-1003) add make patch tool
[ https://issues.apache.org/jira/browse/KYLIN-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-1003. --- Resolution: Fixed The how to contribute doc has added this: https://kylin.apache.org/development/howto_contribute.html > add make patch tool > > > Key: KYLIN-1003 > URL: https://issues.apache.org/jira/browse/KYLIN-1003 > Project: Kylin > Issue Type: Wish >Affects Versions: v1.4.0, v1.1 >Reporter: hongbin ma >Assignee: hongbin ma > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2363) Prune cuboids by capping number of dimensions
[
https://issues.apache.org/jira/browse/KYLIN-2363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roger Shi updated KYLIN-2363:
-
Description:
the scene like this:
I have 20+ dimensions, However the query will only use at most 5 dimensions in
all dimensions, so cuboid that contains 5+ dimensions(except base cuboid) is
useless.
I think we can add a configuration in cube, which limit the max dimensions that
cuboid includes.
What's more, we can config which level(number of dimension) need to calculate.
in above scene, we only calculate leve 1,2,3,4,5. and skip level 5+
=
The dimension capping is turned on by adding dim_cap property in
aggregation_groups definition.
For example, the following aggregation group sets the dimension cap to 3. All
cuboids containing more than 3 dimensions are skipped in this aggregation
group.
{code:none}
"aggregation_groups" : [ {
"includes" : [ "PART_DT", "META_CATEG_NAME", "CATEG_LVL2_NAME",
"CATEG_LVL3_NAME", "LEAF_CATEG_ID", "LSTG_FORMAT_NAME", "LSTG_SITE_ID",
"OPS_USER_ID", "OPS_REGION",
"BUYER_ACCOUNT.ACCOUNT_BUYER_LEVEL",
"SELLER_ACCOUNT.ACCOUNT_SELLER_LEVEL", "BUYER_ACCOUNT.ACCOUNT_COUNTRY",
"SELLER_ACCOUNT.ACCOUNT_COUNTRY", "BUYER_COUNTRY.NAME", "SELLER_COUNTRY.NAME" ],
"select_rule" : {
"hierarchy_dims" : [ [ "META_CATEG_NAME", "CATEG_LVL2_NAME",
"CATEG_LVL3_NAME", "LEAF_CATEG_ID" ] ],
"mandatory_dims" : [ "PART_DT" ],
"joint_dims" : [ [ "BUYER_ACCOUNT.ACCOUNT_COUNTRY", "BUYER_COUNTRY.NAME"
], [ "SELLER_ACCOUNT.ACCOUNT_COUNTRY", "SELLER_COUNTRY.NAME" ],
[ "BUYER_ACCOUNT.ACCOUNT_BUYER_LEVEL",
"SELLER_ACCOUNT.ACCOUNT_SELLER_LEVEL" ], [ "LSTG_FORMAT_NAME", "LSTG_SITE_ID"
], [ "OPS_USER_ID", "OPS_REGION" ] ],
"dim_cap" : 3
}
} ]
{code}
was:
the scene like this:
I have 20+ dimensions, However the query will only use at most 5 dimensions in
all dimensions, so cuboid that contains 5+ dimensions(except base cuboid) is
useless.
I think we can add a configuration in cube, which limit the max dimensions that
cuboid includes.
What's more, we can config which level(number of dimension) need to calculate.
in above scene, we only calculate leve 1,2,3,4,5. and skip level 5+
> Prune cuboids by capping number of dimensions
> -
>
> Key: KYLIN-2363
> URL: https://issues.apache.org/jira/browse/KYLIN-2363
> Project: Kylin
> Issue Type: Improvement
>Reporter: fengYu
>Assignee: Roger Shi
> Fix For: v2.3.0
>
> Attachments: Dimension Capping.md
>
>
> the scene like this:
> I have 20+ dimensions, However the query will only use at most 5 dimensions
> in all dimensions, so cuboid that contains 5+ dimensions(except base cuboid)
> is useless.
> I think we can add a configuration in cube, which limit the max dimensions
> that cuboid includes.
> What's more, we can config which level(number of dimension) need to
> calculate. in above scene, we only calculate leve 1,2,3,4,5. and skip level 5+
> =
> The dimension capping is turned on by adding dim_cap property in
> aggregation_groups definition.
> For example, the following aggregation group sets the dimension cap to 3. All
> cuboids containing more than 3 dimensions are skipped in this aggregation
> group.
> {code:none}
> "aggregation_groups" : [ {
> "includes" : [ "PART_DT", "META_CATEG_NAME", "CATEG_LVL2_NAME",
> "CATEG_LVL3_NAME", "LEAF_CATEG_ID", "LSTG_FORMAT_NAME", "LSTG_SITE_ID",
> "OPS_USER_ID", "OPS_REGION",
>"BUYER_ACCOUNT.ACCOUNT_BUYER_LEVEL",
> "SELLER_ACCOUNT.ACCOUNT_SELLER_LEVEL", "BUYER_ACCOUNT.ACCOUNT_COUNTRY",
> "SELLER_ACCOUNT.ACCOUNT_COUNTRY", "BUYER_COUNTRY.NAME", "SELLER_COUNTRY.NAME"
> ],
> "select_rule" : {
> "hierarchy_dims" : [ [ "META_CATEG_NAME", "CATEG_LVL2_NAME",
> "CATEG_LVL3_NAME", "LEAF_CATEG_ID" ] ],
> "mandatory_dims" : [ "PART_DT" ],
> "joint_dims" : [ [ "BUYER_ACCOUNT.ACCOUNT_COUNTRY",
> "BUYER_COUNTRY.NAME" ], [ "SELLER_ACCOUNT.ACCOUNT_COUNTRY",
> "SELLER_COUNTRY.NAME" ],
>[ "BUYER_ACCOUNT.ACCOUNT_BUYER_LEVEL",
> "SELLER_ACCOUNT.ACCOUNT_SELLER_LEVEL" ], [ "LSTG_FORMAT_NAME", "LSTG_SITE_ID"
> ], [ "OPS_USER_ID", "OPS_REGION" ] ],
> "dim_cap" : 3
> }
> } ]
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KYLIN-1111) Ignore unsupported hive column types when sync hive table
[ https://issues.apache.org/jira/browse/KYLIN-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-: Labels: newbie (was: ) Issue Type: Improvement (was: Bug) > Ignore unsupported hive column types when sync hive table > - > > Key: KYLIN- > URL: https://issues.apache.org/jira/browse/KYLIN- > Project: Kylin > Issue Type: Improvement >Reporter: liyang > Labels: newbie > > Currently unsupported hive column type like map, array will cause failing to > load hive table. Instead, we could just ignore the unsupported column, and > still load the table with remaining columns. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (KYLIN-1094) improve performance of spark cubing
[ https://issues.apache.org/jira/browse/KYLIN-1094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-1094. --- Resolution: Won't Fix Kylin 2.0 has the newly implemented by-layer spark cubing, so this task is not needed. > improve performance of spark cubing > --- > > Key: KYLIN-1094 > URL: https://issues.apache.org/jira/browse/KYLIN-1094 > Project: Kylin > Issue Type: Sub-task > Components: Spark Engine >Affects Versions: v1.4.0 >Reporter: ZhouQianhao >Assignee: Dong Li > Fix For: Backlog > > > POC result of spark cubing shows that, on a dataset of 150 million records, > MR is about 100% faster than Spark, however we believe that Spark could be at > least at same speed as MR, so optimization is needed here. > We are asking Spark community for help now. > the cluster info: > vm: 8 nodes * (128G mem + 64 core) > hadoop cluster: hdp 2.2.6 > spark running mode: yarn-client > spark version: 1.5.1 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (KYLIN-1028) Keep Migration tool working for v1 cubes and v2 cubes
[ https://issues.apache.org/jira/browse/KYLIN-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-1028. --- Resolution: Fixed > Keep Migration tool working for v1 cubes and v2 cubes > - > > Key: KYLIN-1028 > URL: https://issues.apache.org/jira/browse/KYLIN-1028 > Project: Kylin > Issue Type: Improvement >Reporter: hongbin ma >Assignee: hongbin ma > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-1463) When edit a model, if the old fact table be changed, it will still exist for dimensions selection
[ https://issues.apache.org/jira/browse/KYLIN-1463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270306#comment-16270306 ] Shaofeng SHI commented on KYLIN-1463: - Yanghong, Dipesh, what's the status of this issue? > When edit a model, if the old fact table be changed, it will still exist for > dimensions selection > - > > Key: KYLIN-1463 > URL: https://issues.apache.org/jira/browse/KYLIN-1463 > Project: Kylin > Issue Type: Bug >Reporter: Zhong Yanghong >Assignee: Dipesh > > Firstly, I cloned a model. During editing the model, after selecting another > fact table, during the dimension select step, the old fact table still exists > in UI. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (KYLIN-1397) Query result cached even when underly HTABLE scan failed
[ https://issues.apache.org/jira/browse/KYLIN-1397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-1397. --- Resolution: Fixed This issue has been fixed I believe. > Query result cached even when underly HTABLE scan failed > > > Key: KYLIN-1397 > URL: https://issues.apache.org/jira/browse/KYLIN-1397 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: v1.4.0 >Reporter: Su Ralph >Assignee: Edward Zhang > > Query result cached even when underly HTABLE scan failed. When the HTABLE > fixed, the query would still get the cached result. > In my case, there is an Hbase table(one hung htable in all of 14 htables > involved) issue happened when user issue an query. The query hungs for long > time. Later, when the the htable hung issue fixed. User still get the empty > result. > But if user just simple change a query condition, the result would return > correct result. The empty result fixed when kylin admin "Reload Metadata". > This indicates that kylin query cache are *incorrectly build* and util admin > "Reload Metadata" we don't have an way invalidate the cache. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (KYLIN-1452) document improvement: remote debug kylin map/reduce job
[ https://issues.apache.org/jira/browse/KYLIN-1452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-1452. --- Resolution: Won't Fix > document improvement: remote debug kylin map/reduce job > --- > > Key: KYLIN-1452 > URL: https://issues.apache.org/jira/browse/KYLIN-1452 > Project: Kylin > Issue Type: Improvement >Affects Versions: Future >Reporter: Edward Zhang >Assignee: Edward Zhang >Priority: Trivial > Fix For: Future > > > in sandbox/mapred-site.xml > > mapreduce.map.java.opts > -Xdebug -Xrunjdwp:transport=dt_socket,server=y,address=51234 > -Xmx512m > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (KYLIN-1364) Job type always displays as BUILD in RestAPI results
[
https://issues.apache.org/jira/browse/KYLIN-1364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI closed KYLIN-1364.
---
Resolution: Not A Problem
> Job type always displays as BUILD in RestAPI results
>
>
> Key: KYLIN-1364
> URL: https://issues.apache.org/jira/browse/KYLIN-1364
> Project: Kylin
> Issue Type: Improvement
> Components: Job Engine
>Affects Versions: v1.4.0, v1.2
>Reporter: Dong Li
>Assignee: Dipesh
>Priority: Minor
> Labels: newbie
>
> In restAPI result of /jobs, type is always BUILD, no matter it's
> BUILD/MERGE/REFRESH
> eg. {...type: "BUILD"...}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Closed] (KYLIN-1369) Display Build Time on Cube segment tabpage
[ https://issues.apache.org/jira/browse/KYLIN-1369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-1369. --- Resolution: Won't Fix > Display Build Time on Cube segment tabpage > -- > > Key: KYLIN-1369 > URL: https://issues.apache.org/jira/browse/KYLIN-1369 > Project: Kylin > Issue Type: Improvement > Components: Web >Reporter: Dong Li >Assignee: Dipesh >Priority: Minor > Labels: newbie > Attachments: Screen Shot 2016-02-22 at 4.18.34 PM.png > > > User may want to know when did the cube segment built, so he can do some > maintenance job. > Now we have the field in RestAPI JSON Result, one thing to do is to display > it on WebUI. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (KYLIN-1359) developer guide should mention how to manage hive job dependencies
[ https://issues.apache.org/jira/browse/KYLIN-1359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-1359. --- Resolution: Cannot Reproduce > developer guide should mention how to manage hive job dependencies > -- > > Key: KYLIN-1359 > URL: https://issues.apache.org/jira/browse/KYLIN-1359 > Project: Kylin > Issue Type: Improvement > Components: Documentation >Affects Versions: v1.2 > Environment: kylin 1.2, mac osx, hdp sandbox 2.2.4.2-2 >Reporter: Edward Zhang >Assignee: Edward Zhang >Priority: Minor > Fix For: Future > > Original Estimate: 12h > Remaining Estimate: 12h > > When start cubing jobs in Kylin dev environment for example from Intellij, we > should explicitly set hive dependent jars which include > FactDistinctColumnsMapper etc. > This is missing in http://kylin.apache.org/development/dev_env.html. We can > tell developer to do the following: > 1) In IDE, > set -Dkylin.hive.dependency=/usr/hdp/current/kylinjob/kylin-job-1.2-job.jar > 2) copy the corresponding jar file > under kylin/job/target/kylin-job-1.2-job.jar to sandbox if you use sandbox > as development env. > BuildCubeWithEngineTest may directly run some scrip to get > hive dependent jars, but in IDE, looks we have to do manual work above. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (KYLIN-1353) Validate Cuboid Before Create/Update CubeDesc
[ https://issues.apache.org/jira/browse/KYLIN-1353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-1353. --- Resolution: Won't Fix > Validate Cuboid Before Create/Update CubeDesc > - > > Key: KYLIN-1353 > URL: https://issues.apache.org/jira/browse/KYLIN-1353 > Project: Kylin > Issue Type: Bug >Affects Versions: v1.4.0, v1.2 >Reporter: Zhong,Jason >Assignee: Shaofeng SHI > > When user create or Update CubeDesc, we should validate cuboid first and then > save CubeDesc, but now we save first and then validate, it's not right way, > need to fix this. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (KYLIN-1339) N/A Job Tracking URL in Build WebUI
[ https://issues.apache.org/jira/browse/KYLIN-1339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-1339. --- Resolution: Won't Fix > N/A Job Tracking URL in Build WebUI > --- > > Key: KYLIN-1339 > URL: https://issues.apache.org/jira/browse/KYLIN-1339 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Reporter: Yerui Sun >Assignee: Su Ralph > Labels: newbie > > By designing, the cuboid building steps on web ui has the url directed to the > job page on yarn. > In fact, the url can't be always retrieved successfully. The url is 'N/A' > sometimes, or added a prefix 'unsafe:' sometimes, or the right link > sometimes. With some debug log, founded that the url was fetched from status > info, which filled by job.getTrackingURL() in HadoopCmdOutput.getInfo() when > check job status in MapReduceExecutable. The job tracking url was 'N/A' at > the first 20 seconds since job started, and became the right link after that. > One possible reason is that the tracking url is valid until AppMaster > started, which started slowly. > There's still one question, the job status checking was executed > periodically, and the status info should filled with job tracking url after > some seconds later, but the web ui link was not updated. > Some more digging is needed for this issue. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (KYLIN-1278) Query engine optimization: skip the condition checking on partition column if a segment's time is totaly included in the query range
[ https://issues.apache.org/jira/browse/KYLIN-1278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-1278. --- Resolution: Fixed This has already been implemented. > Query engine optimization: skip the condition checking on partition column if > a segment's time is totaly included in the query range > > > Key: KYLIN-1278 > URL: https://issues.apache.org/jira/browse/KYLIN-1278 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Reporter: Shaofeng SHI >Assignee: liyang > Fix For: Backlog > > > See an example: a cube (which is partitioned by date column "CAL_DT") has > three segments: > S1, S2, S3; > Now a query like "select from fact_table where CAL_DT > '-mm-dd' > group by dimA, dimB..." comes, and the "CAL_DT" is only used for filtering, > not in group by; > By checking the startTime/endTime, Kylin found S2 and S3 need be scanned, and > S3 is totally match the condition "CAL_DT > -mm-dd". Then when scan S3, > Kylin can ignore the condition on "CAL_DT", selecting a smaller cuboid, this > should gain better performance. > Further more, this can be generalized to all filtering condition columns: if > all values of a dimension matches with user's filtering condition, and it > isn't in "group by", Kylin can ignore that column when selecting the cuboid. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (KYLIN-3058) We should limit the integer type ID and Port for "Kafka Setting" in "Streaming Cluster" page
[
https://issues.apache.org/jira/browse/KYLIN-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16268110#comment-16268110
]
peng.jianhua edited comment on KYLIN-3058 at 11/29/17 7:13 AM:
---
Hi [~Zhixiong Chen], I have checked the all web pages, and find the other input
limition issues, refer to
[^advanced_setting(supplement).png][^kafka_setting(supplement).png][^measures(supplement).png].
Then I repaired them, and add a new patch refer to
[^0001-KYLIN-3058-the-other-number-input-limition-issues.patch], please check
it, thanks!
was (Author: peng.jianhua):
Hi [~Zhixiong Chen], I have checked the all web pages, and find the other input
limition issues, refer to
[^advanced_setting(supplement).png][^kafka_setting(supplement).png][^measures(supplement).png].
Then I repaired them, and update the patch, please check it, thanks!
> We should limit the integer type ID and Port for "Kafka Setting" in
> "Streaming Cluster" page
>
>
> Key: KYLIN-3058
> URL: https://issues.apache.org/jira/browse/KYLIN-3058
> Project: Kylin
> Issue Type: Bug
> Components: Web
>Reporter: peng.jianhua
>Assignee: peng.jianhua
> Labels: patch
> Fix For: v2.3.0
>
> Attachments:
> 0001-KYLIN-3058-We-should-limit-the-integer-type-ID-and-P.patch,
> 0001-KYLIN-3058-the-other-number-input-limition-issues.patch,
> advanced_setting(supplement).png, create_non_interger_ID.png,
> edit_non_interger_ID.png, kafka_setting(supplement).png,
> measures(supplement).png, streaming_table_dashboard.png
>
>
> When you create or edit one streaming table, then set a non-integer type ID
> for "Kafka Setting" in "Streaming Cluster" page( refer to
> [^streaming_table_dashboard.png]), at last save it, you will find an error
> message page will pop up.
> 1. Create a streaming table, then set a non-integer type ID to save, such as
> "23a", you will get the error message page, refer to
> [^create_non_interger_ID.png];
> 2. Edit a streaming table, then modify a non-integer type ID to save, such as
> "23a", you will also get another error message page, refer to
> [^edit_non_interger_ID.png];
> In order to find out the cause of the problem, I analyze the Kylin server
> log, then I find the error, as follow:
> {panel:title=Kylin server log}
> 2017-11-24 18:46:07,413 ERROR [http-bio-7070-exec-14]
> controller.StreamingController:280 : The data KafkaConfig definition is
> invalid.
> {color:red}com.fasterxml.jackson.databind.exc.InvalidFormatException: Can not
> construct instance of int from String value '23a': not a valid Integer
> value{color}
> at [Source:
> {"uuid":"261e36cd-599e-4306-93b2-add0291de588","last_modified":1511520352482,"version":"2.3.0.20500","name":"DEFAULT.STREAMING_TABLE","clusters":[{"uuid":null,"last_modified":0,"version":"2.3.0.20500","brokers":[{"id":"23a","host":"1","port":1}],"newBroker":{"id":"23a","host":"1","port":1}}],"topic":"kafka
>
> topic","timeout":6,"parserName":"org.apache.kylin.source.kafka.TimedJsonStreamParser","timestampField":null,"margin":0,"parserProperties":"tsColName=datetime"};
> {panel}
> Then I find the server code, it also show the ID and Port must be integer, as
> follow:
> {code:java}
> public class BrokerConfig implements Serializable{
> @JsonProperty("id")
> private int id;
> ..
> @JsonProperty("port")
> private int port;
> {code}
> It means the parameter ID and Port are both not allowed for non-integer
> types, so we should solve them at the web page.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KYLIN-3058) We should limit the integer type ID and Port for "Kafka Setting" in "Streaming Cluster" page
[
https://issues.apache.org/jira/browse/KYLIN-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270294#comment-16270294
]
Zhixiong Chen commented on KYLIN-3058:
--
OK.
Thanks!
> We should limit the integer type ID and Port for "Kafka Setting" in
> "Streaming Cluster" page
>
>
> Key: KYLIN-3058
> URL: https://issues.apache.org/jira/browse/KYLIN-3058
> Project: Kylin
> Issue Type: Bug
> Components: Web
>Reporter: peng.jianhua
>Assignee: peng.jianhua
> Labels: patch
> Fix For: v2.3.0
>
> Attachments:
> 0001-KYLIN-3058-We-should-limit-the-integer-type-ID-and-P.patch,
> 0001-KYLIN-3058-the-other-number-input-limition-issues.patch,
> advanced_setting(supplement).png, create_non_interger_ID.png,
> edit_non_interger_ID.png, kafka_setting(supplement).png,
> measures(supplement).png, streaming_table_dashboard.png
>
>
> When you create or edit one streaming table, then set a non-integer type ID
> for "Kafka Setting" in "Streaming Cluster" page( refer to
> [^streaming_table_dashboard.png]), at last save it, you will find an error
> message page will pop up.
> 1. Create a streaming table, then set a non-integer type ID to save, such as
> "23a", you will get the error message page, refer to
> [^create_non_interger_ID.png];
> 2. Edit a streaming table, then modify a non-integer type ID to save, such as
> "23a", you will also get another error message page, refer to
> [^edit_non_interger_ID.png];
> In order to find out the cause of the problem, I analyze the Kylin server
> log, then I find the error, as follow:
> {panel:title=Kylin server log}
> 2017-11-24 18:46:07,413 ERROR [http-bio-7070-exec-14]
> controller.StreamingController:280 : The data KafkaConfig definition is
> invalid.
> {color:red}com.fasterxml.jackson.databind.exc.InvalidFormatException: Can not
> construct instance of int from String value '23a': not a valid Integer
> value{color}
> at [Source:
> {"uuid":"261e36cd-599e-4306-93b2-add0291de588","last_modified":1511520352482,"version":"2.3.0.20500","name":"DEFAULT.STREAMING_TABLE","clusters":[{"uuid":null,"last_modified":0,"version":"2.3.0.20500","brokers":[{"id":"23a","host":"1","port":1}],"newBroker":{"id":"23a","host":"1","port":1}}],"topic":"kafka
>
> topic","timeout":6,"parserName":"org.apache.kylin.source.kafka.TimedJsonStreamParser","timestampField":null,"margin":0,"parserProperties":"tsColName=datetime"};
> {panel}
> Then I find the server code, it also show the ID and Port must be integer, as
> follow:
> {code:java}
> public class BrokerConfig implements Serializable{
> @JsonProperty("id")
> private int id;
> ..
> @JsonProperty("port")
> private int port;
> {code}
> It means the parameter ID and Port are both not allowed for non-integer
> types, so we should solve them at the web page.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (KYLIN-3029) The warning window of existing cube name does not work
[ https://issues.apache.org/jira/browse/KYLIN-3029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270280#comment-16270280 ] Zhixiong Chen edited comment on KYLIN-3029 at 11/29/17 7:12 AM: Hi,[~peng.jianhua] I can't merge it into KYLIN master. Please check this patch has not conflicts with master branch. Thanks! was (Author: zhixiong chen): Hi,[~peng.jianhua] This patch is fine to me. I will merge it into KYLIN master. Thanks! > The warning window of existing cube name does not work > -- > > Key: KYLIN-3029 > URL: https://issues.apache.org/jira/browse/KYLIN-3029 > Project: Kylin > Issue Type: Bug > Components: Web >Reporter: peng.jianhua >Assignee: peng.jianhua > Labels: scope > Fix For: v2.3.0 > > Attachments: > 0001-KYLIN-3029-The-warning-window-of-exist-cube-name-is-.patch, 01.PNG, > 02.PNG, 03.PNG, 04.png > > > The dealing logical of the warning window of exist cube name is error. The > issue can be reproduced using the following steps: > Add a same name cube such as 'kylin_sales_cube', then click the 'next' button > to enter a new page, we can't get the warning window. > Click 'Prev' button in the page to return back to the first step,then click > 'next' button again,we can get the warning window. > Further if you keep editing the cube,you will get the error message like > 01.png. > It should like 02.png anyway. > The 03.PNG shows that the code in 'cubeSchema.js' can check whether the cube > name exists, but it's logical is error. > The right dealing logical should be as following: > The program should first check whether the cube exists when we clicked the > "next" button. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-3058) We should limit the integer type ID and Port for "Kafka Setting" in "Streaming Cluster" page
[
https://issues.apache.org/jira/browse/KYLIN-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
peng.jianhua updated KYLIN-3058:
Attachment: 0001-KYLIN-3058-the-other-number-input-limition-issues.patch
> We should limit the integer type ID and Port for "Kafka Setting" in
> "Streaming Cluster" page
>
>
> Key: KYLIN-3058
> URL: https://issues.apache.org/jira/browse/KYLIN-3058
> Project: Kylin
> Issue Type: Bug
> Components: Web
>Reporter: peng.jianhua
>Assignee: peng.jianhua
> Labels: patch
> Fix For: v2.3.0
>
> Attachments:
> 0001-KYLIN-3058-We-should-limit-the-integer-type-ID-and-P.patch,
> 0001-KYLIN-3058-the-other-number-input-limition-issues.patch,
> advanced_setting(supplement).png, create_non_interger_ID.png,
> edit_non_interger_ID.png, kafka_setting(supplement).png,
> measures(supplement).png, streaming_table_dashboard.png
>
>
> When you create or edit one streaming table, then set a non-integer type ID
> for "Kafka Setting" in "Streaming Cluster" page( refer to
> [^streaming_table_dashboard.png]), at last save it, you will find an error
> message page will pop up.
> 1. Create a streaming table, then set a non-integer type ID to save, such as
> "23a", you will get the error message page, refer to
> [^create_non_interger_ID.png];
> 2. Edit a streaming table, then modify a non-integer type ID to save, such as
> "23a", you will also get another error message page, refer to
> [^edit_non_interger_ID.png];
> In order to find out the cause of the problem, I analyze the Kylin server
> log, then I find the error, as follow:
> {panel:title=Kylin server log}
> 2017-11-24 18:46:07,413 ERROR [http-bio-7070-exec-14]
> controller.StreamingController:280 : The data KafkaConfig definition is
> invalid.
> {color:red}com.fasterxml.jackson.databind.exc.InvalidFormatException: Can not
> construct instance of int from String value '23a': not a valid Integer
> value{color}
> at [Source:
> {"uuid":"261e36cd-599e-4306-93b2-add0291de588","last_modified":1511520352482,"version":"2.3.0.20500","name":"DEFAULT.STREAMING_TABLE","clusters":[{"uuid":null,"last_modified":0,"version":"2.3.0.20500","brokers":[{"id":"23a","host":"1","port":1}],"newBroker":{"id":"23a","host":"1","port":1}}],"topic":"kafka
>
> topic","timeout":6,"parserName":"org.apache.kylin.source.kafka.TimedJsonStreamParser","timestampField":null,"margin":0,"parserProperties":"tsColName=datetime"};
> {panel}
> Then I find the server code, it also show the ID and Port must be integer, as
> follow:
> {code:java}
> public class BrokerConfig implements Serializable{
> @JsonProperty("id")
> private int id;
> ..
> @JsonProperty("port")
> private int port;
> {code}
> It means the parameter ID and Port are both not allowed for non-integer
> types, so we should solve them at the web page.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Reopened] (KYLIN-3029) The warning window of existing cube name does not work
[ https://issues.apache.org/jira/browse/KYLIN-3029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhixiong Chen reopened KYLIN-3029: -- > The warning window of existing cube name does not work > -- > > Key: KYLIN-3029 > URL: https://issues.apache.org/jira/browse/KYLIN-3029 > Project: Kylin > Issue Type: Bug > Components: Web >Reporter: peng.jianhua >Assignee: peng.jianhua > Labels: scope > Fix For: v2.3.0 > > Attachments: > 0001-KYLIN-3029-The-warning-window-of-exist-cube-name-is-.patch, 01.PNG, > 02.PNG, 03.PNG, 04.png > > > The dealing logical of the warning window of exist cube name is error. The > issue can be reproduced using the following steps: > Add a same name cube such as 'kylin_sales_cube', then click the 'next' button > to enter a new page, we can't get the warning window. > Click 'Prev' button in the page to return back to the first step,then click > 'next' button again,we can get the warning window. > Further if you keep editing the cube,you will get the error message like > 01.png. > It should like 02.png anyway. > The 03.PNG shows that the code in 'cubeSchema.js' can check whether the cube > name exists, but it's logical is error. > The right dealing logical should be as following: > The program should first check whether the cube exists when we clicked the > "next" button. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (KYLIN-1163) Add more job information in Kylin WebUI -> Job Tab
[ https://issues.apache.org/jira/browse/KYLIN-1163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-1163. --- Resolution: Won't Fix > Add more job information in Kylin WebUI -> Job Tab > -- > > Key: KYLIN-1163 > URL: https://issues.apache.org/jira/browse/KYLIN-1163 > Project: Kylin > Issue Type: Improvement > Components: Web >Affects Versions: v1.1.1 >Reporter: Yi Song >Priority: Critical > Attachments: Screen Shot 2015-11-24 at 14.48.46.png, Screen Shot > 2015-11-24 at 14.49.02.png > > > In most cases, we need to know who have ever submitted a job. > We already have such information in JobInstance.java -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (KYLIN-1156) Tutorial web application with sample data
[ https://issues.apache.org/jira/browse/KYLIN-1156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-1156. --- Resolution: Won't Fix > Tutorial web application with sample data > - > > Key: KYLIN-1156 > URL: https://issues.apache.org/jira/browse/KYLIN-1156 > Project: Kylin > Issue Type: New Feature > Components: Documentation, Tools, Build and Test >Reporter: Luke Han >Assignee: Chen Luwei > > To build a tutorial web application with sample data for people > to learn and try Kylin with more easy way, more description informaion > for Kylin's concept, term and all others. > Also will be a good reference for build custom web application > with Kylin's API. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (KYLIN-1221) Allow user to kill Hadoop job in the job list page
[ https://issues.apache.org/jira/browse/KYLIN-1221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-1221. --- Resolution: Fixed This has already been implemented, but I'm not sure the detail version. > Allow user to kill Hadoop job in the job list page > -- > > Key: KYLIN-1221 > URL: https://issues.apache.org/jira/browse/KYLIN-1221 > Project: Kylin > Issue Type: Wish > Components: Job Engine >Affects Versions: v1.1, v1.0, v0.7.2, v0.7.1 >Reporter: Shaofeng SHI >Assignee: Zhong Yanghong > Labels: newbie > Fix For: Backlog > > > Sometimes user expects to abondon the current running hadoop job, he can > choose to kill the related job directly from the Kylin job page, instead of > manually run yarn or mapred CLI to kill it. > Besides, when user discards a Kylin job, the underlying running hadoop job > should also be killed automatically, to release the computing resources in > time. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (KYLIN-1189) Automatically resume running jobs when job engine failover
[ https://issues.apache.org/jira/browse/KYLIN-1189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-1189. --- Resolution: Fixed This also has been implemented I believe. > Automatically resume running jobs when job engine failover > -- > > Key: KYLIN-1189 > URL: https://issues.apache.org/jira/browse/KYLIN-1189 > Project: Kylin > Issue Type: Sub-task > Components: Job Engine >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI > Fix For: Backlog > > > When Kylin's job engine failover, it will put all running job to "ERROR". > User need manually resume such jobs to move ahead. This should be automated > when we implement the auto-balance and HA. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (KYLIN-1263) remove FuzzyRowFilterV2 as it is not being used
[ https://issues.apache.org/jira/browse/KYLIN-1263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-1263. --- Resolution: Fixed > remove FuzzyRowFilterV2 as it is not being used > --- > > Key: KYLIN-1263 > URL: https://issues.apache.org/jira/browse/KYLIN-1263 > Project: Kylin > Issue Type: Improvement >Affects Versions: v1.5.0 >Reporter: hongbin ma >Assignee: hongbin ma > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (KYLIN-1229) CubeMigrationCLI should not need set KYLIN_HOME
[
https://issues.apache.org/jira/browse/KYLIN-1229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI closed KYLIN-1229.
---
Resolution: Won't Fix
> CubeMigrationCLI should not need set KYLIN_HOME
> ---
>
> Key: KYLIN-1229
> URL: https://issues.apache.org/jira/browse/KYLIN-1229
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v1.2, v1.1, v1.0
>Reporter: Shaofeng SHI
>Assignee: Shaofeng SHI
>Priority: Minor
> Fix For: Future
>
>
> When release a cube from one Kylin server to another Kylin server, if didn't
> set KYLIN_HOME environment variable, it will throw exception:
> {code}
> hbase -Dlog4j.configuration=/conf/log4j.properties
> org.apache.hadoop.util.RunJar lib/kylin-job-1.2-SNAPSHOT.jar
> org.apache.kylin.job.tools.CubeMigrationCLI kylin-qa:80 kylin-prod:80
> CUBE_NAME PROJ_NAME true true false
> L4J [2015-12-14
> 00:44:53,578][ERROR][org.apache.kylin.common.persistence.ResourceStore] -
> Create new store instance failed
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:76)
> at
> org.apache.kylin.job.tools.CubeMigrationCLI.moveCube(CubeMigrationCLI.java:94)
> at
> org.apache.kylin.job.tools.CubeMigrationCLI.moveCube(CubeMigrationCLI.java:138)
> at
> org.apache.kylin.job.tools.CubeMigrationCLI.main(CubeMigrationCLI.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> Caused by: java.lang.RuntimeException: Didn't find KYLIN_CONF or KYLIN_HOME,
> please set one of them
> at
> org.apache.kylin.common.KylinConfig.getKylinProperties(KylinConfig.java:600)
> at
> org.apache.kylin.common.KylinConfig.getKylinPropertiesAsInputSteam(KylinConfig.java:608)
> at
> org.apache.kylin.common.KylinConfig.loadKylinConfig(KylinConfig.java:249)
> at
> org.apache.kylin.common.KylinConfig.getInstanceFromEnv(KylinConfig.java:145)
> at
> org.apache.kylin.common.util.HadoopUtil.getCurrentHBaseConfiguration(HadoopUtil.java:56)
> at
> org.apache.kylin.common.persistence.HBaseConnection.get(HBaseConnection.java:68)
> at
> org.apache.kylin.common.persistence.HBaseResourceStore.getConnection(HBaseResourceStore.java:81)
> at
> org.apache.kylin.common.persistence.HBaseResourceStore.createHTableIfNeeded(HBaseResourceStore.java:100)
> at
> org.apache.kylin.common.persistence.HBaseResourceStore.(HBaseResourceStore.java:95)
> ... 13 more
> Exception in thread "main" java.lang.IllegalArgumentException: Failed to find
> metadata store by url: kylin_metadata_qa@hbase
> at
> org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:91)
> at
> org.apache.kylin.job.tools.CubeMigrationCLI.moveCube(CubeMigrationCLI.java:94)
> at
> org.apache.kylin.job.tools.CubeMigrationCLI.moveCube(CubeMigrationCLI.java:138)
> at
> org.apache.kylin.job.tools.CubeMigrationCLI.main(CubeMigrationCLI.java:82)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> {code}
> The problem is between HBaseResourceStore.getConnection() and
> HBaseConnection.get(url), which depends on an implicit KylinConfig,
> insteading using the given KylinConfig.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Closed] (KYLIN-1520) Port Kylin-1249 to 1.5.x branch
[ https://issues.apache.org/jira/browse/KYLIN-1520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-1520. --- Resolution: Won't Fix > Port Kylin-1249 to 1.5.x branch > --- > > Key: KYLIN-1520 > URL: https://issues.apache.org/jira/browse/KYLIN-1520 > Project: Kylin > Issue Type: Bug >Reporter: hongbin ma >Assignee: nichunen > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (KYLIN-1523) Kylin server hang when many bad query running
[ https://issues.apache.org/jira/browse/KYLIN-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-1523. --- Resolution: Won't Fix > Kylin server hang when many bad query running > - > > Key: KYLIN-1523 > URL: https://issues.apache.org/jira/browse/KYLIN-1523 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: v1.4.0 >Reporter: qianqiaoneng >Assignee: liyang >Priority: Critical > Attachments: kylin-425.jstack > > > When there are some bad query running, kylin server will be exhausted then > crashed. Better to have some control on the resource that bad query consumer, > or kill the bad query with some timeout. Should not let them bring the server > down. > The root reason is there are too many slow queries like below exhaust kylin: > 2016-03-21 14:28:33,487 INFO [BadQueryDetector] service.BadQueryDetector:57 : > Slow query has been running 3557 seconds (thread id 0x1841) – > 2016-03-21 14:28:33,489 INFO [BadQueryDetector] service.BadQueryDetector:57 : > Slow query has been running 3557 seconds (thread id 0x1840) – > 2016-03-21 14:28:33,490 INFO [BadQueryDetector] service.BadQueryDetector:57 : > Slow query has been running 3556 seconds (thread id 0x1842) – > 2016-03-21 14:28:33,491 INFO [BadQueryDetector] service.BadQueryDetector:57 : > Slow query has been running 3556 seconds (thread id 0x1843) – > 2016-03-21 14:28:33,493 INFO [BadQueryDetector] service.BadQueryDetector:57 : > Slow query has been running 3556 seconds (thread id 0x1844) – > 2016-03-21 14:28:33,494 INFO [BadQueryDetector] service.BadQueryDetector:57 : > Slow query has been running 3553 seconds (thread id 0x1845) – > 2016-03-21 14:28:33,495 INFO [BadQueryDetector] service.BadQueryDetector:57 : > Slow query has been running 3553 seconds (thread id 0x1847) – > 2016-03-21 14:28:33,509 INFO [BadQueryDetector] service.BadQueryDetector:57 : > Slow query has been running 3552 seconds (thread id 0x1848) – > 2016-03-21 14:28:33,512 INFO [BadQueryDetector] service.BadQueryDetector:57 : > Slow query has been running 3551 seconds (thread id 0x184a) – > 2016-03-21 14:28:33,525 INFO [BadQueryDetector] service.BadQueryDetector:57 : > Slow query has been running 3510 seconds (thread id 0x184c) – > 2016-03-21 14:28:33,554 INFO [BadQueryDetector] service.BadQueryDetector:57 : > Slow query has been running 3510 seconds (thread id 0x184d) – -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2363) Prune cuboids by capping number of dimensions
[ https://issues.apache.org/jira/browse/KYLIN-2363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270288#comment-16270288 ] Roger Shi commented on KYLIN-2363: -- [~Shaofengshi] Thanks for your kind remind. I'll update it in the description. > Prune cuboids by capping number of dimensions > - > > Key: KYLIN-2363 > URL: https://issues.apache.org/jira/browse/KYLIN-2363 > Project: Kylin > Issue Type: Improvement >Reporter: fengYu >Assignee: Roger Shi > Fix For: v2.3.0 > > Attachments: Dimension Capping.md > > > the scene like this: > I have 20+ dimensions, However the query will only use at most 5 dimensions > in all dimensions, so cuboid that contains 5+ dimensions(except base cuboid) > is useless. > I think we can add a configuration in cube, which limit the max dimensions > that cuboid includes. > What's more, we can config which level(number of dimension) need to > calculate. in above scene, we only calculate leve 1,2,3,4,5. and skip level 5+ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (KYLIN-1450) kylin metadata inconsistent
[ https://issues.apache.org/jira/browse/KYLIN-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-1450. --- Resolution: Cannot Reproduce > kylin metadata inconsistent > --- > > Key: KYLIN-1450 > URL: https://issues.apache.org/jira/browse/KYLIN-1450 > Project: Kylin > Issue Type: Bug >Affects Versions: Future >Reporter: Edward Zhang >Assignee: Edward Zhang >Priority: Minor > Fix For: Future > > > In production environment, we saw the following problem of metadata > inconsistency > 1. user could not see table/column metadata in insight page > 2. after we reload metadata, it works -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (KYLIN-1451) no valid state transfer from:DISCARDED to:READY
[ https://issues.apache.org/jira/browse/KYLIN-1451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-1451. --- Resolution: Cannot Reproduce > no valid state transfer from:DISCARDED to:READY > --- > > Key: KYLIN-1451 > URL: https://issues.apache.org/jira/browse/KYLIN-1451 > Project: Kylin > Issue Type: Bug >Reporter: Edward Zhang >Assignee: Edward Zhang >Priority: Minor > > Somehow the state is changed to N/A > 2016-02-25 18:06:33,162 INFO [http-bio-7070-exec-1 ExecutableManager:273]: > job id:a3cadaa3-0167-4d33-8f85-1294078c67d1 from ERROR to DISCARDED > 2016-02-25 18:06:37,884 ERROR [http-bio-7070-exec-4 JobController:183]: there > is no valid state transfer from:DISCARDED to:READY > org.apache.kylin.job.exception.IllegalStateTranferException: there is no > valid state transfer from:DISCARDED to:READY > at > org.apache.kylin.job.manager.ExecutableManager.updateJobOutput(ExecutableManager.java:262) > at > org.apache.kylin.job.manager.ExecutableManager.resumeJob(ExecutableManager.java:230) > at > org.apache.kylin.rest.service.JobService.resumeJob(JobService.java:372) > at > org.apache.kylin.rest.service.JobService$$FastClassByCGLIB$$83a44b2a.invoke() > at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) > at > org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:689) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) > at > org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:64) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) > at > org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:622) > at > org.apache.kylin.rest.service.JobService$$EnhancerByCGLIB$$70963f18.resumeJob() > at > org.apache.kylin.rest.controller.JobController.resume(JobController.java:180) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:213) > at > org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:126) > at > org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:96) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:617) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:578) > at > org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80) > at > org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:923) > at > org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852) > at > org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882) > at > org.springframework.web.servlet.FrameworkServlet.doPut(FrameworkServlet.java:800) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:649) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:727) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) > at > org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118) > at > org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84) > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) > at > org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) > at >
[jira] [Closed] (KYLIN-2164) Cube metadata broken error message missing
[ https://issues.apache.org/jira/browse/KYLIN-2164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-2164. --- Resolution: Won't Fix > Cube metadata broken error message missing > -- > > Key: KYLIN-2164 > URL: https://issues.apache.org/jira/browse/KYLIN-2164 > Project: Kylin > Issue Type: Improvement >Reporter: Roger Shi >Assignee: Roger Shi >Priority: Minor > > Once the cube's metadata gets broken, the Web UI shows its state as BROKEN, > while there's no detailed error even in log. We should print the detailed log > in kylin.log -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-3058) We should limit the integer type ID and Port for "Kafka Setting" in "Streaming Cluster" page
[
https://issues.apache.org/jira/browse/KYLIN-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
peng.jianhua updated KYLIN-3058:
Attachment: 0001-KYLIN-3058-We-should-limit-the-integer-type-ID-and-P.patch
> We should limit the integer type ID and Port for "Kafka Setting" in
> "Streaming Cluster" page
>
>
> Key: KYLIN-3058
> URL: https://issues.apache.org/jira/browse/KYLIN-3058
> Project: Kylin
> Issue Type: Bug
> Components: Web
>Reporter: peng.jianhua
>Assignee: peng.jianhua
> Labels: patch
> Fix For: v2.3.0
>
> Attachments:
> 0001-KYLIN-3058-We-should-limit-the-integer-type-ID-and-P.patch,
> advanced_setting(supplement).png, create_non_interger_ID.png,
> edit_non_interger_ID.png, kafka_setting(supplement).png,
> measures(supplement).png, streaming_table_dashboard.png
>
>
> When you create or edit one streaming table, then set a non-integer type ID
> for "Kafka Setting" in "Streaming Cluster" page( refer to
> [^streaming_table_dashboard.png]), at last save it, you will find an error
> message page will pop up.
> 1. Create a streaming table, then set a non-integer type ID to save, such as
> "23a", you will get the error message page, refer to
> [^create_non_interger_ID.png];
> 2. Edit a streaming table, then modify a non-integer type ID to save, such as
> "23a", you will also get another error message page, refer to
> [^edit_non_interger_ID.png];
> In order to find out the cause of the problem, I analyze the Kylin server
> log, then I find the error, as follow:
> {panel:title=Kylin server log}
> 2017-11-24 18:46:07,413 ERROR [http-bio-7070-exec-14]
> controller.StreamingController:280 : The data KafkaConfig definition is
> invalid.
> {color:red}com.fasterxml.jackson.databind.exc.InvalidFormatException: Can not
> construct instance of int from String value '23a': not a valid Integer
> value{color}
> at [Source:
> {"uuid":"261e36cd-599e-4306-93b2-add0291de588","last_modified":1511520352482,"version":"2.3.0.20500","name":"DEFAULT.STREAMING_TABLE","clusters":[{"uuid":null,"last_modified":0,"version":"2.3.0.20500","brokers":[{"id":"23a","host":"1","port":1}],"newBroker":{"id":"23a","host":"1","port":1}}],"topic":"kafka
>
> topic","timeout":6,"parserName":"org.apache.kylin.source.kafka.TimedJsonStreamParser","timestampField":null,"margin":0,"parserProperties":"tsColName=datetime"};
> {panel}
> Then I find the server code, it also show the ID and Port must be integer, as
> follow:
> {code:java}
> public class BrokerConfig implements Serializable{
> @JsonProperty("id")
> private int id;
> ..
> @JsonProperty("port")
> private int port;
> {code}
> It means the parameter ID and Port are both not allowed for non-integer
> types, so we should solve them at the web page.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KYLIN-3058) We should limit the integer type ID and Port for "Kafka Setting" in "Streaming Cluster" page
[
https://issues.apache.org/jira/browse/KYLIN-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
peng.jianhua updated KYLIN-3058:
Attachment: (was:
0001-KYLIN-3058-We-should-limit-the-integer-type-ID-and-P.patch)
> We should limit the integer type ID and Port for "Kafka Setting" in
> "Streaming Cluster" page
>
>
> Key: KYLIN-3058
> URL: https://issues.apache.org/jira/browse/KYLIN-3058
> Project: Kylin
> Issue Type: Bug
> Components: Web
>Reporter: peng.jianhua
>Assignee: peng.jianhua
> Labels: patch
> Fix For: v2.3.0
>
> Attachments: advanced_setting(supplement).png,
> create_non_interger_ID.png, edit_non_interger_ID.png,
> kafka_setting(supplement).png, measures(supplement).png,
> streaming_table_dashboard.png
>
>
> When you create or edit one streaming table, then set a non-integer type ID
> for "Kafka Setting" in "Streaming Cluster" page( refer to
> [^streaming_table_dashboard.png]), at last save it, you will find an error
> message page will pop up.
> 1. Create a streaming table, then set a non-integer type ID to save, such as
> "23a", you will get the error message page, refer to
> [^create_non_interger_ID.png];
> 2. Edit a streaming table, then modify a non-integer type ID to save, such as
> "23a", you will also get another error message page, refer to
> [^edit_non_interger_ID.png];
> In order to find out the cause of the problem, I analyze the Kylin server
> log, then I find the error, as follow:
> {panel:title=Kylin server log}
> 2017-11-24 18:46:07,413 ERROR [http-bio-7070-exec-14]
> controller.StreamingController:280 : The data KafkaConfig definition is
> invalid.
> {color:red}com.fasterxml.jackson.databind.exc.InvalidFormatException: Can not
> construct instance of int from String value '23a': not a valid Integer
> value{color}
> at [Source:
> {"uuid":"261e36cd-599e-4306-93b2-add0291de588","last_modified":1511520352482,"version":"2.3.0.20500","name":"DEFAULT.STREAMING_TABLE","clusters":[{"uuid":null,"last_modified":0,"version":"2.3.0.20500","brokers":[{"id":"23a","host":"1","port":1}],"newBroker":{"id":"23a","host":"1","port":1}}],"topic":"kafka
>
> topic","timeout":6,"parserName":"org.apache.kylin.source.kafka.TimedJsonStreamParser","timestampField":null,"margin":0,"parserProperties":"tsColName=datetime"};
> {panel}
> Then I find the server code, it also show the ID and Port must be integer, as
> follow:
> {code:java}
> public class BrokerConfig implements Serializable{
> @JsonProperty("id")
> private int id;
> ..
> @JsonProperty("port")
> private int port;
> {code}
> It means the parameter ID and Port are both not allowed for non-integer
> types, so we should solve them at the web page.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Closed] (KYLIN-2188) Kylin does not compile with HBase 1.2
[ https://issues.apache.org/jira/browse/KYLIN-2188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-2188. --- Resolution: Fixed Please upgrade Kylin to a newer version like 2.1, which supports HBase 1.1+ > Kylin does not compile with HBase 1.2 > - > > Key: KYLIN-2188 > URL: https://issues.apache.org/jira/browse/KYLIN-2188 > Project: Kylin > Issue Type: Bug > Components: Storage - HBase >Affects Versions: v1.5.3, v1.5.4, v1.5.4.1 > Environment: CentOS Linux release 7.2.1511、maven 3.2.5、jdk1.7.0_80 >Reporter: Qiang Zhang >Assignee: liyang > Labels: patch > > The RegionScanner and InternalScanner interface was different between > hbase-server-1.2.x.jar and hbase-server-0.9.8.jar. The > AggregationRegionScanner class inherited the RegionScanner interface. But it > couldn't be compatible with hbase-server-1.2.x.jar and hbase-server-0.9.8.jar. > The reappear steps is as following: > 1. Get source code using git from git://git.apache.org/kylin.git; > 2. Run the mvn command to compile the kylin project; > 3. Compile successfully; > 4. Modify the value of the to 1.2.0 in the pom.xml of > the kylin project; > 5. Run the mvn command to compile the kylin project; > 3. Compile fail; -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (KYLIN-1959) Streaming cube error when Kafka don't send message
[ https://issues.apache.org/jira/browse/KYLIN-1959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-1959. --- Resolution: Fixed Fix Version/s: v1.6.0 The v1.6.0 fixed this issue: when no new message comes, the build job will not be submitted. > Streaming cube error when Kafka don't send message > -- > > Key: KYLIN-1959 > URL: https://issues.apache.org/jira/browse/KYLIN-1959 > Project: Kylin > Issue Type: Bug >Affects Versions: v1.5.2 >Reporter: Le Van Ha > Fix For: v1.6.0 > > Attachments: kylin.log, > streaming_Streaming_Cube_147125430_147125460.log, > streaming_Streaming_Cube_147125700_147125730.log > > > Hi all, > > I'm trying Kylin Cube from Streaming (Kafka). > I using Linux crontab to schedule the build every 5 minute. > But, when Kafka don't send message to streaming cube. The streaming cube’s > build don't auto stop, it keeps running. And then, new jobs will error: > OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00070ccc, > 724828160, 0) failed; error='Cannot allocate memory' (errno=12) > # > # There is insufficient memory for the Java Runtime Environment to continue. > # Native memory allocation (malloc) failed to allocate 724828160 bytes for > committing reserved memory. > # An error report file with more information is saved as: > # /var/log/hbase/hs_err_pid104196.log -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (KYLIN-3029) The warning window of existing cube name does not work
[ https://issues.apache.org/jira/browse/KYLIN-3029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhixiong Chen resolved KYLIN-3029. -- Resolution: Fixed > The warning window of existing cube name does not work > -- > > Key: KYLIN-3029 > URL: https://issues.apache.org/jira/browse/KYLIN-3029 > Project: Kylin > Issue Type: Bug > Components: Web >Reporter: peng.jianhua >Assignee: peng.jianhua > Labels: scope > Fix For: v2.3.0 > > Attachments: > 0001-KYLIN-3029-The-warning-window-of-exist-cube-name-is-.patch, 01.PNG, > 02.PNG, 03.PNG, 04.png > > > The dealing logical of the warning window of exist cube name is error. The > issue can be reproduced using the following steps: > Add a same name cube such as 'kylin_sales_cube', then click the 'next' button > to enter a new page, we can't get the warning window. > Click 'Prev' button in the page to return back to the first step,then click > 'next' button again,we can get the warning window. > Further if you keep editing the cube,you will get the error message like > 01.png. > It should like 02.png anyway. > The 03.PNG shows that the code in 'cubeSchema.js' can check whether the cube > name exists, but it's logical is error. > The right dealing logical should be as following: > The program should first check whether the cube exists when we clicked the > "next" button. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-3029) The warning window of existing cube name does not work
[ https://issues.apache.org/jira/browse/KYLIN-3029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270280#comment-16270280 ] Zhixiong Chen commented on KYLIN-3029: -- Hi,[~peng.jianhua] This patch is fine to me. I will merge it into KYLIN master. Thanks! > The warning window of existing cube name does not work > -- > > Key: KYLIN-3029 > URL: https://issues.apache.org/jira/browse/KYLIN-3029 > Project: Kylin > Issue Type: Bug > Components: Web >Reporter: peng.jianhua >Assignee: peng.jianhua > Labels: scope > Fix For: v2.3.0 > > Attachments: > 0001-KYLIN-3029-The-warning-window-of-exist-cube-name-is-.patch, 01.PNG, > 02.PNG, 03.PNG, 04.png > > > The dealing logical of the warning window of exist cube name is error. The > issue can be reproduced using the following steps: > Add a same name cube such as 'kylin_sales_cube', then click the 'next' button > to enter a new page, we can't get the warning window. > Click 'Prev' button in the page to return back to the first step,then click > 'next' button again,we can get the warning window. > Further if you keep editing the cube,you will get the error message like > 01.png. > It should like 02.png anyway. > The 03.PNG shows that the code in 'cubeSchema.js' can check whether the cube > name exists, but it's logical is error. > The right dealing logical should be as following: > The program should first check whether the cube exists when we clicked the > "next" button. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-1865) segment auto merge based on calendar week, month and year
[ https://issues.apache.org/jira/browse/KYLIN-1865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-1865: Issue Type: Wish (was: Improvement) > segment auto merge based on calendar week, month and year > - > > Key: KYLIN-1865 > URL: https://issues.apache.org/jira/browse/KYLIN-1865 > Project: Kylin > Issue Type: Wish >Reporter: hongbin ma >Assignee: hongbin ma > > Currently kylin allows users to specify some time ranges like (7days, 30days, > 365days) to instruct auto merging daily segments. However, the design fails > to respect the boundary of calendar week/month/year, which is often useful > for real applications. For example, if a query only wants a month's data by > providing a filter like `date > 2010-01-01 and data < 2010-02-01`, with > current auto merging design we may end up with visiting two adjacent 30 days > long segments. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-1869) When building snapshot for lookup tables, should we build those dimensions used by model or the whole
[ https://issues.apache.org/jira/browse/KYLIN-1869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270272#comment-16270272 ] Shaofeng SHI commented on KYLIN-1869: - [~yaho]yanghong, what's the status of it? please update status in time. > When building snapshot for lookup tables, should we build those dimensions > used by model or the whole > - > > Key: KYLIN-1869 > URL: https://issues.apache.org/jira/browse/KYLIN-1869 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > > Currently when building a snapshot for a lookup table, the input is the whole > value set of the lookup table, which may be not so reasonable. In some cases, > a lookup table owns tens columns. However, the columns used by a model or a > cube is only a few, 1 to 5. Those unused columns will make the snapshot too > large, which will bring burdens for both storing and loading. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (KYLIN-1864) keep volatile time range out of auto merging
[ https://issues.apache.org/jira/browse/KYLIN-1864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-1864. --- Resolution: Duplicate > keep volatile time range out of auto merging > > > Key: KYLIN-1864 > URL: https://issues.apache.org/jira/browse/KYLIN-1864 > Project: Kylin > Issue Type: Improvement >Reporter: hongbin ma >Assignee: hongbin ma > > Current segment auto merging logic will try to merge all segments, including > all the recent segments. In some scenarios user don't want recent segments > being merged because they may still undergo data updates. A "volatile time > range" may be introduced to ease such cases. A "volatile time range" specify > a recent time range, segments in this range will be immune from being auto > merged. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (KYLIN-3049) Filter the invalid zero value of "Auto Merge Thresholds" parameter when you create or upate a cube.
[ https://issues.apache.org/jira/browse/KYLIN-3049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhixiong Chen resolved KYLIN-3049. -- Resolution: Fixed > Filter the invalid zero value of "Auto Merge Thresholds" parameter when you > create or upate a cube. > --- > > Key: KYLIN-3049 > URL: https://issues.apache.org/jira/browse/KYLIN-3049 > Project: Kylin > Issue Type: Bug > Components: Web >Reporter: peng.jianhua >Assignee: peng.jianhua > Labels: patch, scope > Fix For: v2.3.0 > > Attachments: > 0001-KYLIN-3049-Filter-the-invalid-zero-value-of-Auto-Mer.patch, > add_or_update_cube.png, show_cube.png > > > 1. Open the Model page; > 2. Create a new cube, click the "New Thresholds" button in the "Refresh > Setting" tab, and don't input any value( refer to [^add_or_update_cube.png]), > then complete the follow steps, at last save it; > 3. Choose the cube you just created, go to the "Refresh Setting" tab, you > will find the zero hours in "Auto Merge Thresholds" list, which is invalid > but saved in the server( refer to [^show_cube.png]); > 4. Similarly, update a existent cube, do the same steps, you will also find > the invalid zero value saved in the server. > As we know, the zero value of "Auto Merge Thresholds" parameter is invalid, > so we should filter it before sending it to server. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (KYLIN-2482) build a streaming cube, I meet a 404 error
[ https://issues.apache.org/jira/browse/KYLIN-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI resolved KYLIN-2482. - Resolution: Fixed Assignee: Shaofeng SHI Fix Version/s: v2.2.0 > build a streaming cube, I meet a 404 error > -- > > Key: KYLIN-2482 > URL: https://issues.apache.org/jira/browse/KYLIN-2482 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.0.0 >Reporter: liyang >Assignee: Shaofeng SHI > Fix For: v2.2.0 > > > Reported by mqb <1511013...@qq.com>: > When I using kylin-2.0.0-beta web interface to build a streaming cube, I > meet a 404 error. Does anyone meet the same problem? The messages of chorme > browser's console is as follows: > http://*.*.*.*:7070/kylin/streamingBuild.html 404 (OK) > 7da2a5dd-81c9-4395-8972-61cb2e91010a200xhr > scripts.min.20170224095843.js:8660245 B > SSCube1200xhrscripts.min.20170224095843.js:86601.0 KB > DEFAULT.TEST200xhrscripts.min.20170224095843.js:86601.2 KB > streamingBuild.html404xhr scripts.min.20170224095843.js:8660 > 1.4 KB -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (KYLIN-2482) build a streaming cube, I meet a 404 error
[ https://issues.apache.org/jira/browse/KYLIN-2482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-2482. --- > build a streaming cube, I meet a 404 error > -- > > Key: KYLIN-2482 > URL: https://issues.apache.org/jira/browse/KYLIN-2482 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.0.0 >Reporter: liyang >Assignee: Shaofeng SHI > Fix For: v2.2.0 > > > Reported by mqb <1511013...@qq.com>: > When I using kylin-2.0.0-beta web interface to build a streaming cube, I > meet a 404 error. Does anyone meet the same problem? The messages of chorme > browser's console is as follows: > http://*.*.*.*:7070/kylin/streamingBuild.html 404 (OK) > 7da2a5dd-81c9-4395-8972-61cb2e91010a200xhr > scripts.min.20170224095843.js:8660245 B > SSCube1200xhrscripts.min.20170224095843.js:86601.0 KB > DEFAULT.TEST200xhrscripts.min.20170224095843.js:86601.2 KB > streamingBuild.html404xhr scripts.min.20170224095843.js:8660 > 1.4 KB -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (KYLIN-2467) TPC-H queries run on Kylin
[ https://issues.apache.org/jira/browse/KYLIN-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-2467. --- Resolution: Fixed Fix Version/s: v2.1.0 > TPC-H queries run on Kylin > -- > > Key: KYLIN-2467 > URL: https://issues.apache.org/jira/browse/KYLIN-2467 > Project: Kylin > Issue Type: New Feature >Reporter: liyang >Assignee: liyang > Fix For: v2.1.0 > > > This is an umbrella JIRA to link all tasks about TPC-H queries. > The goal at this stage is functionality rather than performance. Enabling all > TPC-H queries is a new milestone of Kylin's SQL capability. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (KYLIN-2427) Auto adjust join order to make query executable
[
https://issues.apache.org/jira/browse/KYLIN-2427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI closed KYLIN-2427.
---
Resolution: Fixed
Fix Version/s: v2.1.0
> Auto adjust join order to make query executable
> ---
>
> Key: KYLIN-2427
> URL: https://issues.apache.org/jira/browse/KYLIN-2427
> Project: Kylin
> Issue Type: Bug
>Reporter: Kaige Liu
>Assignee: Kaige Liu
> Fix For: v2.1.0
>
>
> KYLIN-2406 reports an issue: The order of joins will affect the result of
> query. For example, below query leads to "No model found"
> Below query triggers NPE
> {code}
> with tmp3 as (
> select l_partkey, 0.5 * sum(l_quantity) as sum_quantity, l_suppkey
> from v_lineitem
> inner join supplier on l_suppkey = s_suppkey
> inner join nation on s_nationkey = n_nationkey
> inner join part on l_partkey = p_partkey
> where l_shipdate >= '1992-01-01' and l_shipdate <= '1995-01-01'
> and n_name = 'CANADA'
> and p_name like 'forest%'
> group by l_partkey, l_suppkey
> )
> select
> s_name,
> s_address
> from
> v_partsupp
> inner join tmp3 on ps_partkey = l_partkey and ps_suppkey = l_suppkey
> inner join supplier on ps_suppkey = s_suppkey
> where
> ps_availqty > sum_quantity
> group by
> s_name, s_address
> order by
> s_name
> {code}
> While below query is OK. Only difference being the order of "inner join tmp3"
> and "inner join supplier"
> {code}
> with tmp3 as (
> select l_partkey, 0.5 * sum(l_quantity) as sum_quantity, l_suppkey
> from v_lineitem
> inner join supplier on l_suppkey = s_suppkey
> inner join nation on s_nationkey = n_nationkey
> inner join part on l_partkey = p_partkey
> where l_shipdate >= '1992-01-01' and l_shipdate <= '1995-01-01'
> and n_name = 'CANADA'
> and p_name like 'forest%'
> group by l_partkey, l_suppkey
> )
> select
> s_name,
> s_address
> from
> v_partsupp
> inner join supplier on ps_suppkey = s_suppkey
> inner join tmp3 on ps_partkey = l_partkey and ps_suppkey = l_suppkey
> where
> ps_availqty > sum_quantity
> group by
> s_name, s_address
> order by
> s_name
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Closed] (KYLIN-2466) Dynamic resource allocation in spark cubing
[ https://issues.apache.org/jira/browse/KYLIN-2466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-2466. --- Resolution: Not A Problem Just follow Spark configuration document to enable the dynamic resource allocation. Kylin doesn't need extra change. > Dynamic resource allocation in spark cubing > --- > > Key: KYLIN-2466 > URL: https://issues.apache.org/jira/browse/KYLIN-2466 > Project: Kylin > Issue Type: Improvement > Components: Spark Engine >Affects Versions: v2.0.0 >Reporter: XIE FAN >Assignee: XIE FAN > > Spark provides a mechanism to dynamically adjust the resources your > application needed. This means that in spark cubing procedure, kylin can > apply for resources dynamically, which is very helpful in many cases such as > saving resources and avoid out of memory error. So we need to provide users a > easy way to enable this feature in spark cubing. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2431) StorageCleanupJob will remove intermediate tables created by other kylin instances
[ https://issues.apache.org/jira/browse/KYLIN-2431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270261#comment-16270261 ] Shaofeng SHI commented on KYLIN-2431: - [~lidong_sjtu] what's the status of this task? > StorageCleanupJob will remove intermediate tables created by other kylin > instances > -- > > Key: KYLIN-2431 > URL: https://issues.apache.org/jira/browse/KYLIN-2431 > Project: Kylin > Issue Type: Improvement >Reporter: Dong Li >Assignee: Dong Li >Priority: Minor > > If QA and PROD instances are in same hive environment, running > StorageCleanupJob on QA will remove intermediate tables created by PROD, > which might fail Kylin jobs of PROD. > A solution is to add metastore name to hive table prefix, then filter table > names with metastore name during cleanup job. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-3049) Filter the invalid zero value of "Auto Merge Thresholds" parameter when you create or upate a cube.
[ https://issues.apache.org/jira/browse/KYLIN-3049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270262#comment-16270262 ] Zhixiong Chen commented on KYLIN-3049: -- Hi,peng.jianhua.I have checked this patch.I will merge it into KYLIN master. Thanks! > Filter the invalid zero value of "Auto Merge Thresholds" parameter when you > create or upate a cube. > --- > > Key: KYLIN-3049 > URL: https://issues.apache.org/jira/browse/KYLIN-3049 > Project: Kylin > Issue Type: Bug > Components: Web >Reporter: peng.jianhua >Assignee: peng.jianhua > Labels: patch, scope > Fix For: v2.3.0 > > Attachments: > 0001-KYLIN-3049-Filter-the-invalid-zero-value-of-Auto-Mer.patch, > add_or_update_cube.png, show_cube.png > > > 1. Open the Model page; > 2. Create a new cube, click the "New Thresholds" button in the "Refresh > Setting" tab, and don't input any value( refer to [^add_or_update_cube.png]), > then complete the follow steps, at last save it; > 3. Choose the cube you just created, go to the "Refresh Setting" tab, you > will find the zero hours in "Auto Merge Thresholds" list, which is invalid > but saved in the server( refer to [^show_cube.png]); > 4. Similarly, update a existent cube, do the same steps, you will also find > the invalid zero value saved in the server. > As we know, the zero value of "Auto Merge Thresholds" parameter is invalid, > so we should filter it before sending it to server. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (KYLIN-2402) execute query hits No registered coprocessor service
[ https://issues.apache.org/jira/browse/KYLIN-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-2402. --- Resolution: Cannot Reproduce > execute query hits No registered coprocessor service > > > Key: KYLIN-2402 > URL: https://issues.apache.org/jira/browse/KYLIN-2402 > Project: Kylin > Issue Type: Improvement > Environment: hadoop2.6.4 /hbase1.1.5/hive1.2.1/zk3.4.6/kylin1.6.0 >Reporter: answer > > today I add kylin.coprocessor.local.jar in kylin.property and when build cube > done . I execute query "select * from xxx" hits error below: > org.apache.hadoop.hbase.exceptions.UnknownProtocolException: No registered > coprocessor service found for name CubeVisitService in region > KYLIN_GA464RXYQX,,1484635706236.9aac092cc3bd8855999c7a8734f09842. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (KYLIN-2373) kyin1.5.3, the exposed tables often disappear from the kylin Insight page.
[ https://issues.apache.org/jira/browse/KYLIN-2373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-2373. --- Resolution: Fixed > kyin1.5.3, the exposed tables often disappear from the kylin Insight page. > -- > > Key: KYLIN-2373 > URL: https://issues.apache.org/jira/browse/KYLIN-2373 > Project: Kylin > Issue Type: Bug > Components: REST Service >Affects Versions: v1.5.3 >Reporter: chelubaiq >Assignee: Zhong,Jason > Attachments: KYLIN-2373_v1.5.4.1-beeline.patch > > > 1 environment: > kylin1.5.3 > tow nodes: "query" server a and "all" server b, > with config: kylin.rest.servers=a_ip,b_ip > 2 problem: > the exposed tables often disappear from the kylin Insight page. > the log says: > ERROR [http-bio-7070-exec-9] project.ProjectL2Cache:240 : Realization > 'CUBE[name=custom_out_sales4]' reports column 'PROJECT1.TABLE1.COLUMN1', but > it is not equal to 'ColumnDesc [name=COLUMN1,table=PROJECT1.TABLE1]' > according to MetadataManager > 3 one way to reproduce: > in project1, cube1 is ready; > in server a, choose project1, and reload one table from the DataSource of > Model page, succeed; the table exist in Insight page. > in server b, choose project1, refresh Insight page, no tables found: "No > Result." > if click "reload metadata" in the system page, tables will show up. > 4 > reason of table not found in server b: > in ProjectL2Cache, ColumnDesc from project realization is not equal to > ColumnDesc from MetadataManager, > because table.equals(other.table) is false in the ColumnDesc.equals() > method, > because table's lastModified not equal. > and table's lastModified from MetadataManager > from project realization > the bug may exist in CacheService.rebuildCache(): > case TABLE: > getMetadataManager().reloadTableCache(cacheKey); > CubeDescManager.clearCache(); > break; > MetadataManager.reloadAllDataModel() may be also needed here. > or MetadataManager.reloadDataModelDesc(forEachModelNameOfTheProject) > or MetadataManager.clearCache() > 5 > why table exist in server a's Insight page? > according to 4, server a should also have same equality problem and should > also disappear? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2363) Prune cuboids by capping number of dimensions
[ https://issues.apache.org/jira/browse/KYLIN-2363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-2363: Fix Version/s: v2.3.0 > Prune cuboids by capping number of dimensions > - > > Key: KYLIN-2363 > URL: https://issues.apache.org/jira/browse/KYLIN-2363 > Project: Kylin > Issue Type: Improvement >Reporter: fengYu >Assignee: Roger Shi > Fix For: v2.3.0 > > Attachments: Dimension Capping.md > > > the scene like this: > I have 20+ dimensions, However the query will only use at most 5 dimensions > in all dimensions, so cuboid that contains 5+ dimensions(except base cuboid) > is useless. > I think we can add a configuration in cube, which limit the max dimensions > that cuboid includes. > What's more, we can config which level(number of dimension) need to > calculate. in above scene, we only calculate leve 1,2,3,4,5. and skip level 5+ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2363) Prune cuboids by capping number of dimensions
[ https://issues.apache.org/jira/browse/KYLIN-2363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270249#comment-16270249 ] Shaofeng SHI commented on KYLIN-2363: - [~R0ger] Roger, could you please provide a document and connect with Guosheng to add the GUI? > Prune cuboids by capping number of dimensions > - > > Key: KYLIN-2363 > URL: https://issues.apache.org/jira/browse/KYLIN-2363 > Project: Kylin > Issue Type: Improvement >Reporter: fengYu >Assignee: Roger Shi > Attachments: Dimension Capping.md > > > the scene like this: > I have 20+ dimensions, However the query will only use at most 5 dimensions > in all dimensions, so cuboid that contains 5+ dimensions(except base cuboid) > is useless. > I think we can add a configuration in cube, which limit the max dimensions > that cuboid includes. > What's more, we can config which level(number of dimension) need to > calculate. in above scene, we only calculate leve 1,2,3,4,5. and skip level 5+ -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (KYLIN-2339) Get error when refresh segment of a streaming cube
[
https://issues.apache.org/jira/browse/KYLIN-2339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI closed KYLIN-2339.
---
Resolution: Won't Fix
> Get error when refresh segment of a streaming cube
> --
>
> Key: KYLIN-2339
> URL: https://issues.apache.org/jira/browse/KYLIN-2339
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v1.6.0
>Reporter: Shaofeng SHI
>Assignee: Dong Li
>
> Try to refresh a segment of a "streaming" cube, get error as below; Although
> for a streaming case, the refresh does change any data, it shouldn't report
> error.
> {code}
> 2016-12-30 02:22:45,609 ERROR [http-bio-7070-exec-41]
> controller.BasicController:44 :
> org.apache.kylin.rest.exception.InternalErrorException: The new refreshing
> segment twitter_tag_cube5[20161230002000_20161230012000] does not match any
> existing segment in cube CUBE[name=twitter_tag_cube5]
> at
> org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:299)
> at
> org.apache.kylin.rest.controller.CubeController.rebuild(CubeController.java:273)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)
> at
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)
> at
> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104)
> at
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:743)
> at
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:672)
> at
> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:82)
> at
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:933)
> at
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:867)
> at
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:951)
> at
> org.springframework.web.servlet.FrameworkServlet.doPut(FrameworkServlet.java:864)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:653)
> at
> org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:827)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Closed] (KYLIN-2336) Add benchmark test tools for Kylin
[ https://issues.apache.org/jira/browse/KYLIN-2336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-2336. --- Resolution: Workaround > Add benchmark test tools for Kylin > -- > > Key: KYLIN-2336 > URL: https://issues.apache.org/jira/browse/KYLIN-2336 > Project: Kylin > Issue Type: Improvement > Components: Client - CLI >Reporter: XIE FAN >Assignee: XIE FAN > > Add performance testing tools that allows developers to evaluate the > comprehensive performance of Kylin. By using those tools, developers can > easily compare the performance differences of the various versions of Kylin, > to ensure that their optimization is effective. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (KYLIN-2229) "/build" folder was excluded in assembly
[ https://issues.apache.org/jira/browse/KYLIN-2229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-2229. --- Resolution: Won't Fix > "/build" folder was excluded in assembly > > > Key: KYLIN-2229 > URL: https://issues.apache.org/jira/browse/KYLIN-2229 > Project: Kylin > Issue Type: Bug > Components: Tools, Build and Test >Reporter: Shaofeng SHI >Assignee: hongbin ma > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2336) Add benchmark test tools for Kylin
[ https://issues.apache.org/jira/browse/KYLIN-2336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270243#comment-16270243 ] Shaofeng SHI commented on KYLIN-2336: - Apache JMeter with Kylin JDBC driver can do benchmark test. > Add benchmark test tools for Kylin > -- > > Key: KYLIN-2336 > URL: https://issues.apache.org/jira/browse/KYLIN-2336 > Project: Kylin > Issue Type: Improvement > Components: Client - CLI >Reporter: XIE FAN >Assignee: XIE FAN > > Add performance testing tools that allows developers to evaluate the > comprehensive performance of Kylin. By using those tools, developers can > easily compare the performance differences of the various versions of Kylin, > to ensure that their optimization is effective. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2325) kylin sql optimization is very slow about 100ms or so, and consumes a lot of cpu
[ https://issues.apache.org/jira/browse/KYLIN-2325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270240#comment-16270240 ] Shaofeng SHI commented on KYLIN-2325: - Please provide more information like sample sql, profiling statistics for the developer to understand the scenario. Not everyone can understand the problem quickly. > kylin sql optimization is very slow about 100ms or so, and consumes a lot of > cpu > > > Key: KYLIN-2325 > URL: https://issues.apache.org/jira/browse/KYLIN-2325 > Project: Kylin > Issue Type: Improvement >Reporter: WangTaiZe > Labels: performance > Original Estimate: 12h > Remaining Estimate: 12h > > kylin sql optimization is too slow, and consumes too much cpu. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2001) PreparedStatement Caching Support
[ https://issues.apache.org/jira/browse/KYLIN-2001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270233#comment-16270233 ] Shaofeng SHI commented on KYLIN-2001: - [~siddiqut] Would you like to contribute a patch to Kylin? Thanks! > PreparedStatement Caching Support > - > > Key: KYLIN-2001 > URL: https://issues.apache.org/jira/browse/KYLIN-2001 > Project: Kylin > Issue Type: Improvement >Affects Versions: v1.5.2 >Reporter: Tajdar Siddiqui >Assignee: Billy Liu > > Please refer to : > Discussion on "Bindable cache" on calcite-dev mailing list (Aug 2016) > and > http://apache-kylin.74782.x6.nabble.com/PreparedStatement-caching-in-kylin-td5618.html > Ability to reuse prepared statements gives a very significant performance > boost. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (KYLIN-2654) Kylin failed to build cube at 3rd step on Hadoop 2.6.0-cdh5.4.4 + jdk 1.8
[
https://issues.apache.org/jira/browse/KYLIN-2654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI closed KYLIN-2654.
---
Resolution: Cannot Reproduce
> Kylin failed to build cube at 3rd step on Hadoop 2.6.0-cdh5.4.4 + jdk 1.8
> -
>
> Key: KYLIN-2654
> URL: https://issues.apache.org/jira/browse/KYLIN-2654
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v2.0.0
>Reporter: Shaofeng SHI
>Assignee: Dong Li
>
> From community user qiumingming:
> When I use kylin 2.0 to build cube, I got a error at Extract Fact Table
> Distinct Columns step, mr job failed. My Hadoop version is 2.6.0-cdh5.4.4.
> Below is my error info:
> {code}
> Error: java.io.IOException: Initialization of all the collectors failed.
> Error in last collector was :Cannot find serializer for
> org.apache.kylin.engine.mr.steps.SelfDefineSortableKey at
> org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:423) at
> org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:82) at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:718)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:790) at
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:350) at
> org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at
> java.security.AccessController.doPrivileged(Native Method) at
> javax.security.auth.Subject.doAs(Subject.java:422) at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by:
> java.io.IOException: Cannot find serializer for
> org.apache.kylin.engine.mr.steps.SelfDefineSortableKey at
> org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator.init(NativeMapOutputCollectorDelegator.java:147)
> at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:411)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Closed] (KYLIN-2570) source code is that TopNCounter merge method problem
[ https://issues.apache.org/jira/browse/KYLIN-2570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-2570. --- Resolution: Not A Problem > source code is that TopNCounter merge method problem > > > Key: KYLIN-2570 > URL: https://issues.apache.org/jira/browse/KYLIN-2570 > Project: Kylin > Issue Type: Test >Reporter: aijuanbo > > In org.apache.kylin.measure.topn.TopNCounter have a merge method, > I have a question that why notDuplicateItems offer(item, m2); > I think is this.offer(item,entry.getValue().getValue().count); > because item is not found in TopNCounter,it is first offer -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2570) source code is that TopNCounter merge method problem
[ https://issues.apache.org/jira/browse/KYLIN-2570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270222#comment-16270222 ] Shaofeng SHI commented on KYLIN-2570: - If you have further question, please subscribe the mailing list for discussion: https://kylin.apache.org/community/ JIRA is for bug and task tracking, we don't do QA usually. Thanks for the understanding. > source code is that TopNCounter merge method problem > > > Key: KYLIN-2570 > URL: https://issues.apache.org/jira/browse/KYLIN-2570 > Project: Kylin > Issue Type: Test >Reporter: aijuanbo > > In org.apache.kylin.measure.topn.TopNCounter have a merge method, > I have a question that why notDuplicateItems offer(item, m2); > I think is this.offer(item,entry.getValue().getValue().count); > because item is not found in TopNCounter,it is first offer -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2570) source code is that TopNCounter merge method problem
[ https://issues.apache.org/jira/browse/KYLIN-2570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270218#comment-16270218 ] Shaofeng SHI commented on KYLIN-2570: - Hi, sorry for the late response; This is a good question; for the answer you can refer to this blog: https://kylin.apache.org/blog/2016/03/19/approximate-topn-measure/ And the algorithm that it uses: https://arxiv.org/pdf/1401.0702.pdf Btw, the code has been improved for performance since what you saw, please check the latest version. > source code is that TopNCounter merge method problem > > > Key: KYLIN-2570 > URL: https://issues.apache.org/jira/browse/KYLIN-2570 > Project: Kylin > Issue Type: Test >Reporter: aijuanbo > > In org.apache.kylin.measure.topn.TopNCounter have a merge method, > I have a question that why notDuplicateItems offer(item, m2); > I think is this.offer(item,entry.getValue().getValue().count); > because item is not found in TopNCounter,it is first offer -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (KYLIN-2847) Avoid doing useless work by checking query deadline
[ https://issues.apache.org/jira/browse/KYLIN-2847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI resolved KYLIN-2847. - Resolution: Fixed > Avoid doing useless work by checking query deadline > --- > > Key: KYLIN-2847 > URL: https://issues.apache.org/jira/browse/KYLIN-2847 > Project: Kylin > Issue Type: Improvement > Components: Query Engine, Storage - HBase >Reporter: Dayue Gao >Assignee: Dayue Gao > Labels: scope > Fix For: v2.3.0 > > Attachments: KYLIN-2847-v2.patch, KYLIN-2847.patch > > > Under high load, requests can spend a long time waiting in RPC queue, > probably longer than query timeout. However, current coprocessor timeout > mechanism doesn't take RPC queue time into account. As a result, handling > these requests not only waste server resources while doing nothing useful, > but can also cause cascading failure when server crashes and client retries. > To recover from server overload ASAP, we should check query deadline at each > stages of query processing, and avoid spending resources on query that will > exceed their deadline. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2847) Avoid doing useless work by checking query deadline
[ https://issues.apache.org/jira/browse/KYLIN-2847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-2847: Affects Version/s: (was: v2.1.0) Fix Version/s: v2.3.0 Summary: Avoid doing useless work by checking query deadline (was: avoid doing useless work by checking query deadline) Merged in master, thanks Dayue! > Avoid doing useless work by checking query deadline > --- > > Key: KYLIN-2847 > URL: https://issues.apache.org/jira/browse/KYLIN-2847 > Project: Kylin > Issue Type: Improvement > Components: Query Engine, Storage - HBase >Reporter: Dayue Gao >Assignee: Dayue Gao > Labels: scope > Fix For: v2.3.0 > > Attachments: KYLIN-2847-v2.patch, KYLIN-2847.patch > > > Under high load, requests can spend a long time waiting in RPC queue, > probably longer than query timeout. However, current coprocessor timeout > mechanism doesn't take RPC queue time into account. As a result, handling > these requests not only waste server resources while doing nothing useful, > but can also cause cascading failure when server crashes and client retries. > To recover from server overload ASAP, we should check query deadline at each > stages of query processing, and avoid spending resources on query that will > exceed their deadline. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2847) avoid doing useless work by checking query deadline
[ https://issues.apache.org/jira/browse/KYLIN-2847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-2847: Attachment: (was: KYLIN-2847.patch.1) > avoid doing useless work by checking query deadline > --- > > Key: KYLIN-2847 > URL: https://issues.apache.org/jira/browse/KYLIN-2847 > Project: Kylin > Issue Type: Improvement > Components: Query Engine, Storage - HBase >Affects Versions: v2.1.0 >Reporter: Dayue Gao >Assignee: Dayue Gao > Labels: scope > Attachments: KYLIN-2847-v2.patch, KYLIN-2847.patch > > > Under high load, requests can spend a long time waiting in RPC queue, > probably longer than query timeout. However, current coprocessor timeout > mechanism doesn't take RPC queue time into account. As a result, handling > these requests not only waste server resources while doing nothing useful, > but can also cause cascading failure when server crashes and client retries. > To recover from server overload ASAP, we should check query deadline at each > stages of query processing, and avoid spending resources on query that will > exceed their deadline. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2847) avoid doing useless work by checking query deadline
[ https://issues.apache.org/jira/browse/KYLIN-2847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-2847: Attachment: KYLIN-2847-v2.patch > avoid doing useless work by checking query deadline > --- > > Key: KYLIN-2847 > URL: https://issues.apache.org/jira/browse/KYLIN-2847 > Project: Kylin > Issue Type: Improvement > Components: Query Engine, Storage - HBase >Affects Versions: v2.1.0 >Reporter: Dayue Gao >Assignee: Dayue Gao > Labels: scope > Attachments: KYLIN-2847-v2.patch, KYLIN-2847.patch > > > Under high load, requests can spend a long time waiting in RPC queue, > probably longer than query timeout. However, current coprocessor timeout > mechanism doesn't take RPC queue time into account. As a result, handling > these requests not only waste server resources while doing nothing useful, > but can also cause cascading failure when server crashes and client retries. > To recover from server overload ASAP, we should check query deadline at each > stages of query processing, and avoid spending resources on query that will > exceed their deadline. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Closed] (KYLIN-2952) dynamic cube build for time(statTime and endTime)
[
https://issues.apache.org/jira/browse/KYLIN-2952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI closed KYLIN-2952.
---
Resolution: Not A Problem
> dynamic cube build for time(statTime and endTime)
> -
>
> Key: KYLIN-2952
> URL: https://issues.apache.org/jira/browse/KYLIN-2952
> Project: Kylin
> Issue Type: Improvement
> Components: REST Service
>Affects Versions: v1.6.0
> Environment: linux
>Reporter: wenxue lin
>Assignee: Zhong,Jason
>Priority: Minor
>
> ex => curl -X PUT -u "ADMIN:KYLIN" -H
> "Content-Type:application/json;charset=utf-8" -d
> '{"startTime":150860160,"endTime":150868800,"buildType":"BUILD"}'
> http://host:port/kylin/api/cubes/bi_dispatch_waiting_service_cube/rebuild
> desc :
> rest api param is startTime:150860160(2017-10-22) and
> endTime:150868800(2017-10-23)， but the actual time of building the cube
> is 1 day ahead of schedule (actually 8 hours ahead of schedule)
> =》【2017-10-21~2017-10-22】，But using kylinUI build without question, the
> actual view kylin code, found that is because the kylin on the server side
> source code for configuration of GMT rather than use the timezone of fixed
> GMT + 8, and front-end UI will according to the configuration of the timezone
> is transformed to the time of the page to add GMT + 8 time, then the back-end
> to GMT + 8 time in into GMT time, so the kylinUI cube build time without
> error, and using restAPI build cube time not making timezone 8 hours is not
> accurate time difference problem。
> *for code:*
> kylinProperties.js
> this.getTimeZone = function () {
> if (!this.timezone) {
> this.timezone = this.getProperty("kylin.rest.timezone").trim();
> }
> return this.timezone;
> }
> org.apache.kylin.cube.CubeSegment
> public static String makeSegmentName(long startDate, long endDate, long
> startOffset, long endOffset) {
> if (startOffset != 0 || endOffset != 0) {
> if (startOffset == 0 && (endOffset == 0 || endOffset ==
> Long.MAX_VALUE)) {
> return "FULL_BUILD";
> }
> return startOffset + "_" + endOffset;
> }
> // using time
> SimpleDateFormat dateFormat = new SimpleDateFormat("MMddHHmmss");
> dateFormat.setTimeZone(TimeZone.getTimeZone("GMT"));
> return dateFormat.format(startDate) + "_" + dateFormat.format(endDate);
> }
> org.apache.kylin.common.util.DateFormat:
> public static FastDateFormat getDateFormat(String datePattern) {
> FastDateFormat r = formatMap.get(datePattern);
> if (r == null) {
> r = FastDateFormat.getInstance(datePattern,
> TimeZone.getTimeZone("GMT")); // NOTE: this must be GMT to calculate epoch
> date correctly
> formatMap.put(datePattern, r);
> }
> return r;
> }
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Closed] (KYLIN-2827) cube已经建好了，再insight 查询回报错
[ https://issues.apache.org/jira/browse/KYLIN-2827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-2827. --- Resolution: Cannot Reproduce > cube已经建好了，再insight 查询回报错 > - > > Key: KYLIN-2827 > URL: https://issues.apache.org/jira/browse/KYLIN-2827 > Project: Kylin > Issue Type: Bug >Reporter: liuxd > > Error while executing SQL "select * from GDL_LOANRPY_DET LIMIT 5": > org.apache.hadoop.hbase.exceptions.UnknownProtocolException: > org.apache.hadoop.hbase.exceptions.UnknownProtocolException: No registered > coprocessor service found for name CubeVisitService in region > KYLIN_ZW5LTEKVAL,,1504166387142.432256ac271b98fc6afd29bc7e054502. at > org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:7582) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:1890) > at > org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:1872) > at > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32389) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2117) at > org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:104) at > org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133) at > org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108) at > java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2953) List readable project not correct if add limit and offset
[ https://issues.apache.org/jira/browse/KYLIN-2953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269958#comment-16269958 ] Shaofeng SHI commented on KYLIN-2953: - Hi Julian, it makes sense, would you like to contribute a patch? > List readable project not correct if add limit and offset > - > > Key: KYLIN-2953 > URL: https://issues.apache.org/jira/browse/KYLIN-2953 > Project: Kylin > Issue Type: Bug > Components: REST Service >Affects Versions: v2.1.0 >Reporter: Pan, Julian >Assignee: Pan, Julian > > There is one line getReadableProjects in ProjectController: > List projectInstances = > projectService.listAllProjects(limit, offset); > That means just the part of projects, then validate the ACL. > If user's project beyond the limit, then he will never get the projects if he > put limit and offset. > It should be similar with cube list, get all the projects then validate the > ACL, at last get the sub list by limit and offset. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2959) SAML logout issue
[ https://issues.apache.org/jira/browse/KYLIN-2959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269962#comment-16269962 ] Shaofeng SHI commented on KYLIN-2959: - Hi Julian, please provide detailed description about the issue. > SAML logout issue > - > > Key: KYLIN-2959 > URL: https://issues.apache.org/jira/browse/KYLIN-2959 > Project: Kylin > Issue Type: Bug > Components: REST Service, Web >Reporter: Pan, Julian >Assignee: Pan, Julian >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (KYLIN-2945) Global dict specific info should not use absolute path when saved it
[
https://issues.apache.org/jira/browse/KYLIN-2945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI reassigned KYLIN-2945:
---
Assignee: Yifei Wu
Priority: Minor (was: Major)
Summary: Global dict specific info should not use absolute path when saved
it (was: global dict specific info should not use absolute path when saved it)
> Global dict specific info should not use absolute path when saved it
>
>
> Key: KYLIN-2945
> URL: https://issues.apache.org/jira/browse/KYLIN-2945
> Project: Kylin
> Issue Type: Improvement
>Reporter: Yifei Wu
>Assignee: Yifei Wu
>Priority: Minor
>
> Currently, absolute path is used in the global dictionary for saving the
> location of specific info like the next json.
> {code}
> {
> "uuid" : "b82505d8-5b40-4009-8839-8456500ea6a8",
> "last_modified" : 0,
> "version" : "2.3.0.20500",
> "source_table" : "KYLIN_YIFEI_1011_TESTGLOBALDICTMIGRATION.KYLIN_SALES",
> "source_column" : "PRICE",
> "source_column_index" : 6,
> "data_type" : "decimal(19,4)",
> "input" : {
> "path" :
> "hdfs://sandbox.hortonworks.com:8020/kylin/kylin_default/kylin-d213a963-b5e3-4e75-a1a2-8af93ca83a80/global_dict_cube/fact_distinct_columns/KYLIN_SALES.PRICE",
> "size" : 81150,
> "last_modified_time" : 1508133611074
> },
> "dictionary_class" : "org.apache.kylin.dict.AppendTrieDictionary",
> "cardinality" : 8878
> }^@hdfs://master:8020/kylin/kylin_yifei_1011_testGlobalDictMigration/resources/GlobalDict/dict/KYLIN_YIFEI_1011_TESTGLOBALDICTMIGRATION.KYLIN_SALES/PRICE/
> {code}
> But if you change the path configuration or migrate the data to another
> cluster which node name is different or original host cannot be reached, you
> will meet problems in reusing it, like rebuilding and merge dictionary or
> transfer column from dictionary in building cube, for the dictionary info
> cannot be found.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KYLIN-2935) Improve the way to deploy coprocessor
[
https://issues.apache.org/jira/browse/KYLIN-2935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269954#comment-16269954
]
Zhong Yanghong commented on KYLIN-2935:
---
[~Shaofengshi], generally we don't need to deploy coprocessors for the tables
whose git commit is the same as the one from {{commit_SHA1}}
> Improve the way to deploy coprocessor
> -
>
> Key: KYLIN-2935
> URL: https://issues.apache.org/jira/browse/KYLIN-2935
> Project: Kylin
> Issue Type: Improvement
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
>
> There are two improvements for deploying coprocessor:
> # Make thread number configurable
> # Filter tables by git commit
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KYLIN-2951) Fix couple of NullPointerException in Kylin code base
[ https://issues.apache.org/jira/browse/KYLIN-2951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269953#comment-16269953 ] Shaofeng SHI commented on KYLIN-2951: - Thanks Ken, welcome to contribute on Kylin! > Fix couple of NullPointerException in Kylin code base > - > > Key: KYLIN-2951 > URL: https://issues.apache.org/jira/browse/KYLIN-2951 > Project: Kylin > Issue Type: Bug >Reporter: Wang Ken >Assignee: Wang Ken >Priority: Minor > > I create this ticket as the umbrella to cover some NPE bugs in Kylin 2.1 & > 2.2 code base. > We see some NPE errors from our server log, some of them are tiny, it is > trivial to create individual ticket to cover them one by one. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2932) Simplify the thread model for in-memory cubing
[ https://issues.apache.org/jira/browse/KYLIN-2932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269949#comment-16269949 ] Shaofeng SHI commented on KYLIN-2932: - Hi Ken, could you please share the patch? Thanks! > Simplify the thread model for in-memory cubing > -- > > Key: KYLIN-2932 > URL: https://issues.apache.org/jira/browse/KYLIN-2932 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: Wang Ken >Assignee: Wang Ken > > The current implementation uses split threads, task threads and main thread > to do the cube building, there is complex join and error handling logic. > The new implement leverages the ForkJoinPool from JDK, the event split logic > is handled in > main thread. Cuboid task and sub-tasks are handled in fork join pool, cube > results are collected > async and can be write to output earlier. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2933) Fix compilation against the Kafka 1.0.0 release
[
https://issues.apache.org/jira/browse/KYLIN-2933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269948#comment-16269948
]
Shaofeng SHI commented on KYLIN-2933:
-
Ted, you are correct; Since it is in test module, I will fix it with a lower
priority.
> Fix compilation against the Kafka 1.0.0 release
> ---
>
> Key: KYLIN-2933
> URL: https://issues.apache.org/jira/browse/KYLIN-2933
> Project: Kylin
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Shaofeng SHI
>
> Due to change of class location of SecurityProtocol, KafkaClusterConfig.java
> needs to be modified so that it compiles against the Kafka 1.0.0 release:
> {code}
> -import org.apache.kafka.common.protocol.SecurityProtocol;
> +import org.apache.kafka.common.security.auth.SecurityProtocol;
> {code}
> A compatibility layer may be needed if we support multiple Kafka releases
> simultaneously.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KYLIN-2935) Improve the way to deploy coprocessor
[ https://issues.apache.org/jira/browse/KYLIN-2935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269944#comment-16269944 ] Shaofeng SHI commented on KYLIN-2935: - I couldn't understand the #2, could you elaborate it? > Improve the way to deploy coprocessor > - > > Key: KYLIN-2935 > URL: https://issues.apache.org/jira/browse/KYLIN-2935 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > > There are two improvements for deploying coprocessor: > # Make thread number configurable > # Filter tables by git commit -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2929) speed up Dump file performance
[ https://issues.apache.org/jira/browse/KYLIN-2929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269928#comment-16269928 ] Shaofeng SHI commented on KYLIN-2929: - In the spill() method, I didn't see the close for InputStream and OutputStream; > speed up Dump file performance > -- > > Key: KYLIN-2929 > URL: https://issues.apache.org/jira/browse/KYLIN-2929 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: v2.0.0 >Reporter: fengYu >Assignee: fengYu > Labels: Performance > Attachments: > 0002-KYLIN-2929-speed-up-dump-performance-write-dump-file.patch > > > when I work on KYLIN-2926, I find coprocessor will dump to disk once > estimatedMemSize is bigger than spillThreshold, and found that spill data > size is extraordinary smaller that estimatedMemSize, in my case dump file > size is about 8MB and spillThreshold is setting to 3GB. > So, I try to keep the spill data in memory rather than write the file to disk > immediately, and when those in-memory spill data reach the threshold, write > all spill files together. > In my case, the coprocessor process cost time drop from 22s to 16s, it is > about 30% upgrade。 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-3043) Don't need create materialized view for lookup tables without snapshot
[ https://issues.apache.org/jira/browse/KYLIN-3043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong updated KYLIN-3043: -- Attachment: APACHE-KYLIN-3043.patch > Don't need create materialized view for lookup tables without snapshot > -- > > Key: KYLIN-3043 > URL: https://issues.apache.org/jira/browse/KYLIN-3043 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > Attachments: APACHE-KYLIN-3043.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2923) FILTER clause support
[ https://issues.apache.org/jira/browse/KYLIN-2923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269903#comment-16269903 ] Shaofeng SHI commented on KYLIN-2923: - Hi Alexander, could you please provide detail description here? For example, a sample query. > FILTER clause support > - > > Key: KYLIN-2923 > URL: https://issues.apache.org/jira/browse/KYLIN-2923 > Project: Kylin > Issue Type: New Feature >Reporter: Alexander Sterligov > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2923) FILTER clause support
[ https://issues.apache.org/jira/browse/KYLIN-2923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-2923: Issue Type: Wish (was: New Feature) > FILTER clause support > - > > Key: KYLIN-2923 > URL: https://issues.apache.org/jira/browse/KYLIN-2923 > Project: Kylin > Issue Type: Wish >Reporter: Alexander Sterligov > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2912) Remove "hfile" folder after bulk load to HBase
[ https://issues.apache.org/jira/browse/KYLIN-2912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-2912: Priority: Minor (was: Major) > Remove "hfile" folder after bulk load to HBase > -- > > Key: KYLIN-2912 > URL: https://issues.apache.org/jira/browse/KYLIN-2912 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI >Priority: Minor > Fix For: v2.3.0 > > > Kylin generates HFile with MR, and then bulk load to HBase, which is a mv > operation. But if blob storage like S3 is used, the old HFile won't be > deleted. > Suggestion: automatically removes the hfile sub folder of that job in the > garbage collection step. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (KYLIN-2918) Table ACL needs GUI
[ https://issues.apache.org/jira/browse/KYLIN-2918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI reassigned KYLIN-2918: --- Assignee: luguosheng (was: Zhong,Jason) Fix Version/s: v2.3.0 > Table ACL needs GUI > --- > > Key: KYLIN-2918 > URL: https://issues.apache.org/jira/browse/KYLIN-2918 > Project: Kylin > Issue Type: Bug > Components: Web >Affects Versions: v2.2.0 >Reporter: liyang >Assignee: luguosheng > Labels: scope > Fix For: v2.3.0 > > > The backend of Table ACL is done, but the GUI part still requires work. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (KYLIN-2912) Remove "hfile" folder after bulk load to HBase
[ https://issues.apache.org/jira/browse/KYLIN-2912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI resolved KYLIN-2912. - Resolution: Fixed > Remove "hfile" folder after bulk load to HBase > -- > > Key: KYLIN-2912 > URL: https://issues.apache.org/jira/browse/KYLIN-2912 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI > Fix For: v2.3.0 > > > Kylin generates HFile with MR, and then bulk load to HBase, which is a mv > operation. But if blob storage like S3 is used, the old HFile won't be > deleted. > Suggestion: automatically removes the hfile sub folder of that job in the > garbage collection step. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (KYLIN-2912) Remove "hfile" folder after bulk load to HBase
[ https://issues.apache.org/jira/browse/KYLIN-2912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI reassigned KYLIN-2912: --- Assignee: Shaofeng SHI (was: Dong Li) Fix Version/s: v2.3.0 > Remove "hfile" folder after bulk load to HBase > -- > > Key: KYLIN-2912 > URL: https://issues.apache.org/jira/browse/KYLIN-2912 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI > Fix For: v2.3.0 > > > Kylin generates HFile with MR, and then bulk load to HBase, which is a mv > operation. But if blob storage like S3 is used, the old HFile won't be > deleted. > Suggestion: automatically removes the hfile sub folder of that job in the > garbage collection step. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2900) Web Interface MR Link BUG
[ https://issues.apache.org/jira/browse/KYLIN-2900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269893#comment-16269893 ] Shaofeng SHI commented on KYLIN-2900: - The URL is from YARN; I occasionally see this, usually manually remove the prefix will work. > Web Interface MR Link BUG > - > > Key: KYLIN-2900 > URL: https://issues.apache.org/jira/browse/KYLIN-2900 > Project: Kylin > Issue Type: Bug > Components: REST Service >Reporter: lynn >Assignee: Zhong,Jason > Attachments: 1.png > > > When I notice the detail info of the MR Job, I find the Link starts with > “unsafe” occasionally , as the picture "1.png". > !1.png|thumbnail! -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2903) support cardinality calculation for Hive view
[ https://issues.apache.org/jira/browse/KYLIN-2903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269891#comment-16269891 ] Shaofeng SHI commented on KYLIN-2903: - Hi Wang Gang, yes this is a known issue in Kylin. Would you like to contribute a patch for this? Thanks for making Kylin better! > support cardinality calculation for Hive view > - > > Key: KYLIN-2903 > URL: https://issues.apache.org/jira/browse/KYLIN-2903 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: Wang, Gang >Assignee: Wang, Gang >Priority: Minor > > Currently, Kylin leverage HCatlog to calculate column cardinality for Hive > tables. While, HCatlog does not support Hive view actually. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2886) export csv file error
[
https://issues.apache.org/jira/browse/KYLIN-2886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269889#comment-16269889
]
Shaofeng SHI commented on KYLIN-2886:
-
Hi Lynn, thanks for the reporting, it is a bug in front-end web, which seems
always sends the latest sql to backend. Would you like contribute a patch?
> export csv file error
> -
>
> Key: KYLIN-2886
> URL: https://issues.apache.org/jira/browse/KYLIN-2886
> Project: Kylin
> Issue Type: Bug
> Components: REST Service
>Reporter: lynn
>Assignee: Zhong,Jason
> Labels: scope
> Attachments: 1.png, 21.png
>
>
> SQL:
> select price, sell_id from kylin_sales order by sell_id compute max(price) by
> sell_id;
> when i run a "compute by sub statement"，obviously, it does not support the
> statement.
> !1.png|thumbnail!
> It doesn't matter! But when i want to export the results which previously
> executed successfully to the csv file, it still throws exception!
> !21.png|thumbnail!
> REST Service return message:
> {
> "code": "999",
> "data": null,
> "msg": "Encountered \"compute\" at line 1, column 57. Was expecting one
> of: \"LIMIT\" ... \"OFFSET\" ... \"FETCH\" ...
> \",\" ... \"ASC\" ... \"DESC\" ... \"NULLS\" ... \"NOT\" ...
>\"IN\" ... \"BETWEEN\" ... \"LIKE\" ... \"SIMILAR\" ...
> \"=\" ... \">\" ... \"<\" ... \"<=\" ... \">=\" ...
> \"<>\" ... \"!=\" ... \"+\" ... \"-\" ... \"*\" ... \"/\"
> ... \"||\" ... \"AND\" ... \"OR\" ... \"IS\" ...
> \"MEMBER\" ... \"SUBMULTISET\" ... \"MULTISET\" ... \"[\" ...
> \".\" ... \"(\" ... ",
> "stacktrace": "org.apache.kylin.rest.exception.InternalErrorException:
> Encountered \"compute\" at line 1, column 57. Was expecting one of:
> \"LIMIT\" ... \"OFFSET\" ... \"FETCH\" ... \",\" ...
> \"ASC\" ... \"DESC\" ... \"NULLS\" ... \"NOT\" ... \"IN\" ...
> \"BETWEEN\" ... \"LIKE\" ... \"SIMILAR\" ... \"=\" ...
> \">\" ... \"<\" ... \"<=\" ... \">=\" ... \"<>\" ...
> \"!=\" ... \"+\" ... \"-\" ... \"*\" ... \"/\" ... \"||\"
> ... \"AND\" ... \"OR\" ... \"IS\" ... \"MEMBER\" ...
> \"SUBMULTISET\" ... \"MULTISET\" ... \"[\" ... \".\" ...
> \"(\" ... \n\tat
> org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:440)\n\tat
>
> org.apache.kylin.rest.controller.QueryController.downloadQueryResult(QueryController.java:121)\n\tat
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat
> java.lang.reflect.Method.invoke(Method.java:498)\n\tat
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)\n\tat
>
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)\n\tat
>
> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110)\n\tat
>
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:832)\n\tat
>
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:743)\n\tat
>
> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)\n\tat
>
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:961)\n\tat
>
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:895)\n\tat
>
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:967)\n\tat
>
> org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:858)\n\tat
> javax.servlet.http.HttpServlet.service(HttpServlet.java:624)\n\tat
> org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:843)\n\tat
> javax.servlet.http.HttpServlet.service(HttpServlet.java:731)\n\tat
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)\n\tat
>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)\n\tat
> org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)\n\tat
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)\n\tat
>
>
[jira] [Assigned] (KYLIN-3062) Hide RAW measure
[ https://issues.apache.org/jira/browse/KYLIN-3062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI reassigned KYLIN-3062: --- Assignee: Zhixiong Chen Component/s: Web > Hide RAW measure > > > Key: KYLIN-3062 > URL: https://issues.apache.org/jira/browse/KYLIN-3062 > Project: Kylin > Issue Type: Task > Components: Web >Reporter: Shaofeng SHI >Assignee: Zhixiong Chen > > See the discussion here: > http://apache-kylin.74782.x6.nabble.com/Discuss-Disable-hide-quot-RAW-quot-measure-in-Kylin-web-GUI-td6636.html -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-3057) In some cases，query failed when sql with comments
[
https://issues.apache.org/jira/browse/KYLIN-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269868#comment-16269868
]
Shaofeng SHI commented on KYLIN-3057:
-
Hi Jianhua, Calcite can parse the comments in a SQL, we don't need additional
handing for that. You can try the SQL "select count(*) from KYLIN_SALES
--comment" or "select count(*) from KYLIN_SALES /* comment */"
In the sample case you provided, please remove ";" or move it after the
comment, that will work.
> In some cases，query failed when sql with comments
> -
>
> Key: KYLIN-3057
> URL: https://issues.apache.org/jira/browse/KYLIN-3057
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v2.3.0
>Reporter: peng.jianhua
>Assignee: peng.jianhua
> Attachments:
> 0001-KYLIN-3057-In-some-cases-query-failed-when-sql-with-.patch,
> no_comments_sql.PNG, with_comments_sql.PNG
>
>
> no_comments_sql.PNG: select trans_id from kylin_sales;
> with_comments_sql.PNG : select trans_id from kylin_sales;--comments
> if the sql end with semicolon and there are some comments behind semicolon,we
> will meet the error
> the following is the result of with_comments_sql.PNG:
> Results
> Encountered ";" at line 1, column 33. Was expecting one of: "ORDER" ...
> "LIMIT" ... "OFFSET" ... "FETCH" ... "NATURAL" ... "JOIN" ... "INNER" ...
> "LEFT" ... "RIGHT" ... "FULL" ... "CROSS" ... "," ... "OUTER" ... "EXTEND"
> ... "(" ... "MATCH_RECOGNIZE" ... "." ... "AS" ... ...
> ... ...
> ... ... "TABLESAMPLE"
> ... "WHERE" ... "GROUP" ... "HAVING" ... "WINDOW" ... "UNION" ... "INTERSECT"
> ... "EXCEPT" ... "MINUS" ..
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KYLIN-3043) Don't need create materialized view for lookup tables without snapshot
[ https://issues.apache.org/jira/browse/KYLIN-3043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269864#comment-16269864 ] Zhong Yanghong commented on KYLIN-3043: --- [~Shaofengshi], yes. Today or tomorrow.:) > Don't need create materialized view for lookup tables without snapshot > -- > > Key: KYLIN-3043 > URL: https://issues.apache.org/jira/browse/KYLIN-3043 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (KYLIN-3055) NullPointerException in MutableRoaringBitmap.or
[
https://issues.apache.org/jira/browse/KYLIN-3055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI reassigned KYLIN-3055:
---
Assignee: yuchuqian
Fix Version/s: v2.3.0
> NullPointerException in MutableRoaringBitmap.or
> ---
>
> Key: KYLIN-3055
> URL: https://issues.apache.org/jira/browse/KYLIN-3055
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.2.0
>Reporter: yuchuqian
>Assignee: yuchuqian
> Fix For: v2.3.0
>
> Attachments: KYLIN-3055.patch
>
>
> 2017-11-21 19:55:17,363 ERROR [Query
> b1fbcd45-6524-4b1e-8844-1d6d6277a1bf-120] service.QueryService:459 :
> Exception while executing query
> java.sql.SQLException: Error while executing SQL "select part_dt,
> intersect_count(item_count, part_dt, array[date'2012-01-01']) as first_day,
> intersect_count(item_count, part_dt, array[date'2012-01-02']) as second_day,
> intersect_count(item_count, part_dt, array[date'2012-01-03']) as third_day,
> intersect_count(item_count, part_dt,
> array[date'2012-01-01',date'2012-01-02']) as retention_oneday,
> intersect_count(item_count, part_dt,
> array[date'2012-01-01',date'2012-01-02',date'2012-01-03']) as retention_twoday
> from kylin_sales
> where part_dt in (date'2012-01-01',date'2012-01-02',date'2012-01-03')
> group by PART_DT
> LIMIT 5": null
> at org.apache.calcite.avatica.Helper.createException(Helper.java:56)
> at org.apache.calcite.avatica.Helper.createException(Helper.java:41)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:218)
> at org.apache.kylin.rest.service.QueryService.execute(QueryService.java:834)
> at
> org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:561)
> at org.apache.kylin.rest.service.QueryService.query(QueryService.java:181)
> at
> org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:415)
> at
> org.apache.kylin.rest.controller.QueryController.query(QueryController.java:78)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
> ..
> Caused by: java.lang.NullPointerException
> at
> org.roaringbitmap.buffer.MutableRoaringBitmap.or(MutableRoaringBitmap.java:1041)
> at
> org.apache.kylin.measure.bitmap.RoaringBitmapCounter.orWith(RoaringBitmapCounter.java:72)
> at
> org.apache.kylin.measure.bitmap.BitmapIntersectDistinctCountAggFunc$RetentionPartialResult.add(BitmapIntersectDistinctCountAggFunc.java:57)
> at
> org.apache.kylin.measure.bitmap.BitmapIntersectDistinctCountAggFunc.add(BitmapIntersectDistinctCountAggFunc.java:90)
> at Baz$4.apply(ANONYMOUS.java:136)
> at Baz$4.apply(ANONYMOUS.java:158)
> at Baz$4.apply(ANONYMOUS.java)
> at
> org.apache.calcite.linq4j.EnumerableDefaults.groupBy_(EnumerableDefaults.java:832)
> at
> org.apache.calcite.linq4j.EnumerableDefaults.groupBy(EnumerableDefaults.java:761)
> at
> org.apache.calcite.linq4j.DefaultEnumerable.groupBy(DefaultEnumerable.java:302)
> at Baz.bind(Baz.java:99)
> How to re-produce:
> 1. run $KYLIN_HOME/bin/sample.sh
> 2. then create a cube like
> {
> "uuid": "9554f6f6-74dc-489e-b780-2f48f281576c",
> "last_modified": 1511247707372,
> "version": "2.2.0.0",
> "name": "test",
> "is_draft": false,
> "model_name": "kylin_sales_model",
> "description": "",
> "null_string": null,
> "dimensions": [
> {
> "name": "PART_DT",
> "table": "KYLIN_SALES",
> "column": "PART_DT",
> "derived": null
> },
> {
> "name": "LEAF_CATEG_ID",
> "table": "KYLIN_SALES",
> "column": "LEAF_CATEG_ID",
> "derived": null
> },
> {
> "name": "LSTG_SITE_ID",
> "table": "KYLIN_SALES",
> "column": "LSTG_SITE_ID",
> "derived": null
> },
> {
> "name": "CAL_DT",
> "table": "KYLIN_CAL_DT",
> "column": null,
> "derived": [
> "CAL_DT"
> ]
> },
> {
> "name": "LEAF_CATEG_ID",
> "table": "KYLIN_CATEGORY_GROUPINGS",
> "column": null,
> "derived": [
> "LEAF_CATEG_ID"
> ]
> },
> {
> "name": "USER_DEFINED_FIELD1",
> "table": "KYLIN_CATEGORY_GROUPINGS",
> "column": null,
> "derived": [
> "USER_DEFINED_FIELD1"
> ]
> },
> {
> "name": "USER_DEFINED_FIELD3",
> "table": "KYLIN_CATEGORY_GROUPINGS",
> "column": null,
> "derived": [
[jira] [Commented] (KYLIN-3055) NullPointerException in MutableRoaringBitmap.or
[
https://issues.apache.org/jira/browse/KYLIN-3055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16269844#comment-16269844
]
Shaofeng SHI commented on KYLIN-3055:
-
Hi Chuqian, the patch is invalid, please re-submit a new patch by following the
guide here: https://kylin.apache.org/development/howto_contribute.html
Thanks for reporting the issue.
> NullPointerException in MutableRoaringBitmap.or
> ---
>
> Key: KYLIN-3055
> URL: https://issues.apache.org/jira/browse/KYLIN-3055
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.2.0
>Reporter: yuchuqian
> Attachments: KYLIN-3055.patch
>
>
> 2017-11-21 19:55:17,363 ERROR [Query
> b1fbcd45-6524-4b1e-8844-1d6d6277a1bf-120] service.QueryService:459 :
> Exception while executing query
> java.sql.SQLException: Error while executing SQL "select part_dt,
> intersect_count(item_count, part_dt, array[date'2012-01-01']) as first_day,
> intersect_count(item_count, part_dt, array[date'2012-01-02']) as second_day,
> intersect_count(item_count, part_dt, array[date'2012-01-03']) as third_day,
> intersect_count(item_count, part_dt,
> array[date'2012-01-01',date'2012-01-02']) as retention_oneday,
> intersect_count(item_count, part_dt,
> array[date'2012-01-01',date'2012-01-02',date'2012-01-03']) as retention_twoday
> from kylin_sales
> where part_dt in (date'2012-01-01',date'2012-01-02',date'2012-01-03')
> group by PART_DT
> LIMIT 5": null
> at org.apache.calcite.avatica.Helper.createException(Helper.java:56)
> at org.apache.calcite.avatica.Helper.createException(Helper.java:41)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:218)
> at org.apache.kylin.rest.service.QueryService.execute(QueryService.java:834)
> at
> org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:561)
> at org.apache.kylin.rest.service.QueryService.query(QueryService.java:181)
> at
> org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:415)
> at
> org.apache.kylin.rest.controller.QueryController.query(QueryController.java:78)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
> ..
> Caused by: java.lang.NullPointerException
> at
> org.roaringbitmap.buffer.MutableRoaringBitmap.or(MutableRoaringBitmap.java:1041)
> at
> org.apache.kylin.measure.bitmap.RoaringBitmapCounter.orWith(RoaringBitmapCounter.java:72)
> at
> org.apache.kylin.measure.bitmap.BitmapIntersectDistinctCountAggFunc$RetentionPartialResult.add(BitmapIntersectDistinctCountAggFunc.java:57)
> at
> org.apache.kylin.measure.bitmap.BitmapIntersectDistinctCountAggFunc.add(BitmapIntersectDistinctCountAggFunc.java:90)
> at Baz$4.apply(ANONYMOUS.java:136)
> at Baz$4.apply(ANONYMOUS.java:158)
> at Baz$4.apply(ANONYMOUS.java)
> at
> org.apache.calcite.linq4j.EnumerableDefaults.groupBy_(EnumerableDefaults.java:832)
> at
> org.apache.calcite.linq4j.EnumerableDefaults.groupBy(EnumerableDefaults.java:761)
> at
> org.apache.calcite.linq4j.DefaultEnumerable.groupBy(DefaultEnumerable.java:302)
> at Baz.bind(Baz.java:99)
> How to re-produce:
> 1. run $KYLIN_HOME/bin/sample.sh
> 2. then create a cube like
> {
> "uuid": "9554f6f6-74dc-489e-b780-2f48f281576c",
> "last_modified": 1511247707372,
> "version": "2.2.0.0",
> "name": "test",
> "is_draft": false,
> "model_name": "kylin_sales_model",
> "description": "",
> "null_string": null,
> "dimensions": [
> {
> "name": "PART_DT",
> "table": "KYLIN_SALES",
> "column": "PART_DT",
> "derived": null
> },
> {
> "name": "LEAF_CATEG_ID",
> "table": "KYLIN_SALES",
> "column": "LEAF_CATEG_ID",
> "derived": null
> },
> {
> "name": "LSTG_SITE_ID",
> "table": "KYLIN_SALES",
> "column": "LSTG_SITE_ID",
> "derived": null
> },
> {
> "name": "CAL_DT",
> "table": "KYLIN_CAL_DT",
> "column": null,
> "derived": [
> "CAL_DT"
> ]
> },
> {
> "name": "LEAF_CATEG_ID",
> "table": "KYLIN_CATEGORY_GROUPINGS",
> "column": null,
> "derived": [
> "LEAF_CATEG_ID"
> ]
> },
> {
> "name": "USER_DEFINED_FIELD1",
> "table": "KYLIN_CATEGORY_GROUPINGS",
> "column": null,
> "derived": [
> "USER_DEFINED_FIELD1"
> ]
> },
> {
> "name":
[jira] [Updated] (KYLIN-3057) In some cases，query failed when sql with comments
[
https://issues.apache.org/jira/browse/KYLIN-3057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
peng.jianhua updated KYLIN-3057:
Description:
no_comments_sql.PNG: select trans_id from kylin_sales;
with_comments_sql.PNG : select trans_id from kylin_sales;--comments
if the sql end with semicolon and there are some comments behind semicolon,we
will meet the error
the following is the result of with_comments_sql.PNG:
Results
Encountered ";" at line 1, column 33. Was expecting one of: "ORDER" ...
"LIMIT" ... "OFFSET" ... "FETCH" ... "NATURAL" ... "JOIN" ... "INNER" ...
"LEFT" ... "RIGHT" ... "FULL" ... "CROSS" ... "," ... "OUTER" ... "EXTEND" ...
"(" ... "MATCH_RECOGNIZE" ... "." ... "AS" ... ...
... ...
... ... "TABLESAMPLE"
... "WHERE" ... "GROUP" ... "HAVING" ... "WINDOW" ... "UNION" ... "INTERSECT"
... "EXCEPT" ... "MINUS" ..
was:
select trans_id from kylin_sales; refer to no_comments_sql.PNG
select trans_id from kylin_sales;--comments refer to with_comments_sql.PNG
Results
Encountered ";" at line 1, column 33. Was expecting one of: "ORDER" ...
"LIMIT" ... "OFFSET" ... "FETCH" ... "NATURAL" ... "JOIN" ... "INNER" ...
"LEFT" ... "RIGHT" ... "FULL" ... "CROSS" ... "," ... "OUTER" ... "EXTEND" ...
"(" ... "MATCH_RECOGNIZE" ... "." ... "AS" ... ...
... ...
... ... "TABLESAMPLE"
... "WHERE" ... "GROUP" ... "HAVING" ... "WINDOW" ... "UNION" ... "INTERSECT"
... "EXCEPT" ... "MINUS" ..
> In some cases，query failed when sql with comments
> -
>
> Key: KYLIN-3057
> URL: https://issues.apache.org/jira/browse/KYLIN-3057
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v2.3.0
>Reporter: peng.jianhua
>Assignee: peng.jianhua
> Attachments:
> 0001-KYLIN-3057-In-some-cases-query-failed-when-sql-with-.patch,
> no_comments_sql.PNG, with_comments_sql.PNG
>
>
> no_comments_sql.PNG: select trans_id from kylin_sales;
> with_comments_sql.PNG : select trans_id from kylin_sales;--comments
> if the sql end with semicolon and there are some comments behind semicolon,we
> will meet the error
> the following is the result of with_comments_sql.PNG:
> Results
> Encountered ";" at line 1, column 33. Was expecting one of: "ORDER" ...
> "LIMIT" ... "OFFSET" ... "FETCH" ... "NATURAL" ... "JOIN" ... "INNER" ...
> "LEFT" ... "RIGHT" ... "FULL" ... "CROSS" ... "," ... "OUTER" ... "EXTEND"
> ... "(" ... "MATCH_RECOGNIZE" ... "." ... "AS" ... ...
> ... ...
> ... ... "TABLESAMPLE"
> ... "WHERE" ... "GROUP" ... "HAVING" ... "WINDOW" ... "UNION" ... "INTERSECT"
> ... "EXCEPT" ... "MINUS" ..
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
