[jira] [Resolved] (KYLIN-2377) Add kylin client query timeout
[ https://issues.apache.org/jira/browse/KYLIN-2377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kangkaisen resolved KYLIN-2377. --- Resolution: Fixed Fix Version/s: v2.0.0 > Add kylin client query timeout > -- > > Key: KYLIN-2377 > URL: https://issues.apache.org/jira/browse/KYLIN-2377 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Affects Versions: v1.6.0 >Reporter: kangkaisen >Assignee: kangkaisen > Fix For: v2.0.0 > > Attachments: KYLIN-2377.patch > > > Add kylin client query timeout to make query server more robust -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KYLIN-2377) Add kylin client query timeout
[ https://issues.apache.org/jira/browse/KYLIN-2377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15857594#comment-15857594 ] kangkaisen commented on KYLIN-2377: --- commit: https://github.com/apache/kylin/commit/8263752a499158342e0588fda851a4006e8b1669 > Add kylin client query timeout > -- > > Key: KYLIN-2377 > URL: https://issues.apache.org/jira/browse/KYLIN-2377 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Affects Versions: v1.6.0 >Reporter: kangkaisen >Assignee: kangkaisen > Fix For: v2.0.0 > > Attachments: KYLIN-2377.patch > > > Add kylin client query timeout to make query server more robust -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (KYLIN-2436) add a configuration knob to disable spilling of aggregation cache
Dayue Gao created KYLIN-2436: Summary: add a configuration knob to disable spilling of aggregation cache Key: KYLIN-2436 URL: https://issues.apache.org/jira/browse/KYLIN-2436 Project: Kylin Issue Type: Improvement Components: Storage - HBase Affects Versions: v1.6.0 Reporter: Dayue Gao Assignee: Dayue Gao Kylin's aggregation operator can spill intermediate results to disk when its estimated memory usage exceeds some threshold (kylin.query.coprocessor.mem.gb to be specific). While it's a useful feature in general to prevent RegionServer from OOM, there are times when aborting this kind of memory-hungry query immediately is a more suitable choice to users. To accommodate this requirement, I suggest adding a new configuration named "kylin.storage.hbase.coprocessor-spill-enabled". The default value would be true, which will keep the same behavior as before. If changed to false, query that uses more aggregation memory than threshold will fail immediately. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (KYLIN-2316) Build Base Cuboid Data ERROR
[ https://issues.apache.org/jira/browse/KYLIN-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15857574#comment-15857574 ] kangkaisen edited comment on KYLIN-2316 at 2/8/17 7:31 AM: --- Hi, konglei, It looks like you use Global Dictionary for a dimension column? The Global Dictionary could only used for Measure column currently. I will add this check later. was (Author: kangkaisen): Hi, konglei, It looks like you use Global Dictionary for a dimension column? > Build Base Cuboid Data ERROR > - > > Key: KYLIN-2316 > URL: https://issues.apache.org/jira/browse/KYLIN-2316 > Project: Kylin > Issue Type: Test > Components: General >Affects Versions: v1.6.0 > Environment: hadoop:cdh 5.7.0 > kylin:apache-kylin-1.6.0-cdh.5.7.0 > system:centos 6.8 >Reporter: konglei >Assignee: kangkaisen > Fix For: v2.0.0 > > > Dec 23, 2016 3:24:40 PM com.google.common.cache.LocalCache > processPendingNotifications > WARNING: Exception thrown by removal listener > java.lang.RuntimeException: unexpected evict reason COLLECTED > at > org.apache.kylin.dict.CachedTreeMap$1.onRemoval(CachedTreeMap.java:155) > at > com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:2004) > at > com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3490) > at > com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3466) > at > com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2304) > at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228) > at com.google.common.cache.LocalCache.get(LocalCache.java:3965) > at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3969) > at > com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4829) > at org.apache.kylin.dict.CachedTreeMap.get(CachedTreeMap.java:292) > at org.apache.kylin.dict.CachedTreeMap.get(CachedTreeMap.java:52) > at > org.apache.kylin.dict.AppendTrieDictionary.getIdFromValueBytesImpl(AppendTrieDictionary.java:971) > at > org.apache.kylin.common.util.Dictionary.getIdFromValueBytes(Dictionary.java:160) > at > org.apache.kylin.dimension.DictionaryDimEnc.encode(DictionaryDimEnc.java:91) > at > org.apache.kylin.cube.kv.RowKeyColumnIO.writeColumn(RowKeyColumnIO.java:55) > at > org.apache.kylin.cube.kv.RowKeyEncoder.fillColumnValue(RowKeyEncoder.java:195) > at org.apache.kylin.cube.kv.RowKeyEncoder.encode(RowKeyEncoder.java:163) > at > org.apache.kylin.engine.mr.steps.BaseCuboidMapperBase.buildKey(BaseCuboidMapperBase.java:151) > at > org.apache.kylin.engine.mr.steps.BaseCuboidMapperBase.outputKV(BaseCuboidMapperBase.java:198) > at > org.apache.kylin.engine.mr.steps.HiveToBaseCuboidMapper.map(HiveToBaseCuboidMapper.java:52) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KYLIN-2316) Build Base Cuboid Data ERROR
[ https://issues.apache.org/jira/browse/KYLIN-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15857574#comment-15857574 ] kangkaisen commented on KYLIN-2316: --- Hi, konglei, It looks like you use Global Dictionary for a dimension column? > Build Base Cuboid Data ERROR > - > > Key: KYLIN-2316 > URL: https://issues.apache.org/jira/browse/KYLIN-2316 > Project: Kylin > Issue Type: Test > Components: General >Affects Versions: v1.6.0 > Environment: hadoop:cdh 5.7.0 > kylin:apache-kylin-1.6.0-cdh.5.7.0 > system:centos 6.8 >Reporter: konglei >Assignee: kangkaisen > Fix For: v2.0.0 > > > Dec 23, 2016 3:24:40 PM com.google.common.cache.LocalCache > processPendingNotifications > WARNING: Exception thrown by removal listener > java.lang.RuntimeException: unexpected evict reason COLLECTED > at > org.apache.kylin.dict.CachedTreeMap$1.onRemoval(CachedTreeMap.java:155) > at > com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:2004) > at > com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3490) > at > com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3466) > at > com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2304) > at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228) > at com.google.common.cache.LocalCache.get(LocalCache.java:3965) > at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3969) > at > com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4829) > at org.apache.kylin.dict.CachedTreeMap.get(CachedTreeMap.java:292) > at org.apache.kylin.dict.CachedTreeMap.get(CachedTreeMap.java:52) > at > org.apache.kylin.dict.AppendTrieDictionary.getIdFromValueBytesImpl(AppendTrieDictionary.java:971) > at > org.apache.kylin.common.util.Dictionary.getIdFromValueBytes(Dictionary.java:160) > at > org.apache.kylin.dimension.DictionaryDimEnc.encode(DictionaryDimEnc.java:91) > at > org.apache.kylin.cube.kv.RowKeyColumnIO.writeColumn(RowKeyColumnIO.java:55) > at > org.apache.kylin.cube.kv.RowKeyEncoder.fillColumnValue(RowKeyEncoder.java:195) > at org.apache.kylin.cube.kv.RowKeyEncoder.encode(RowKeyEncoder.java:163) > at > org.apache.kylin.engine.mr.steps.BaseCuboidMapperBase.buildKey(BaseCuboidMapperBase.java:151) > at > org.apache.kylin.engine.mr.steps.BaseCuboidMapperBase.outputKV(BaseCuboidMapperBase.java:198) > at > org.apache.kylin.engine.mr.steps.HiveToBaseCuboidMapper.map(HiveToBaseCuboidMapper.java:52) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KYLIN-2321) Message: Error while executing SQL "select * from root.id_card LIMIT 50000": AppendTrieDictionary can't retrive value from id
[ https://issues.apache.org/jira/browse/KYLIN-2321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15857569#comment-15857569 ] YongBo Wu commented on KYLIN-2321: -- I just encounter the same problem, could there be a temporary solution before v2.0.0 is released? > Message: Error while executing SQL "select * from root.id_card LIMIT 5": > AppendTrieDictionary can't retrive value from id > - > > Key: KYLIN-2321 > URL: https://issues.apache.org/jira/browse/KYLIN-2321 > Project: Kylin > Issue Type: Bug > Components: 3rd Party, Job Engine >Affects Versions: v1.6.0 > Environment: Message: Error while executing SQL "select * from > root.id_card LIMIT 5": AppendTrieDictionary can't retrive value from id >Reporter: konglei >Assignee: Dong Li > Fix For: v2.0.0 > > > Message: Error while executing SQL "select * from root.id_card LIMIT 5": > AppendTrieDictionary can't retrive value from id > ==[QUERY]=== > 2016-12-26 16:59:16,553 ERROR [http-bio-7070-exec-10] > controller.BasicController:44 : > org.apache.kylin.rest.exception.InternalErrorException: Error while executing > SQL "select * from root.id_card LIMIT 5": AppendTrieDictionary can't > retrive value from id > at > org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:389) > at > org.apache.kylin.rest.controller.QueryController.query(QueryController.java:69) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221) > at > org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136) > at > org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:743) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:672) > at > org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:82) > at > org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:933) > at > org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:867) > at > org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:951) > at > org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:853) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:650) > at > org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:827) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) > at > org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330) > at > org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118) > at > org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84) > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) > at > org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) > at > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342) > at > org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103) >
[jira] [Updated] (KYLIN-2268) Potential NPE in ModelDimensionDesc#init()
[
https://issues.apache.org/jira/browse/KYLIN-2268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liyang updated KYLIN-2268:
--
Labels: newbie (was: )
> Potential NPE in ModelDimensionDesc#init()
> --
>
> Key: KYLIN-2268
> URL: https://issues.apache.org/jira/browse/KYLIN-2268
> Project: Kylin
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Minor
> Labels: newbie
>
> {code}
> if (columns != null) {
> StringUtil.toUpperCaseArray(columns, columns);
> }
> if (model != null) {
> table = model.findTable(table).getAlias();
> for (int i = 0; i < columns.length; i++) {
> {code}
> In the second if block, columns is accessed without null check.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (KYLIN-2316) Build Base Cuboid Data ERROR
[ https://issues.apache.org/jira/browse/KYLIN-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15857550#comment-15857550 ] liyang commented on KYLIN-2316: --- Looks like about Global Dictionary. > Build Base Cuboid Data ERROR > - > > Key: KYLIN-2316 > URL: https://issues.apache.org/jira/browse/KYLIN-2316 > Project: Kylin > Issue Type: Test > Components: General >Affects Versions: v1.6.0 > Environment: hadoop:cdh 5.7.0 > kylin:apache-kylin-1.6.0-cdh.5.7.0 > system:centos 6.8 >Reporter: konglei >Assignee: kangkaisen > Fix For: v2.0.0 > > > Dec 23, 2016 3:24:40 PM com.google.common.cache.LocalCache > processPendingNotifications > WARNING: Exception thrown by removal listener > java.lang.RuntimeException: unexpected evict reason COLLECTED > at > org.apache.kylin.dict.CachedTreeMap$1.onRemoval(CachedTreeMap.java:155) > at > com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:2004) > at > com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3490) > at > com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3466) > at > com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2304) > at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228) > at com.google.common.cache.LocalCache.get(LocalCache.java:3965) > at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3969) > at > com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4829) > at org.apache.kylin.dict.CachedTreeMap.get(CachedTreeMap.java:292) > at org.apache.kylin.dict.CachedTreeMap.get(CachedTreeMap.java:52) > at > org.apache.kylin.dict.AppendTrieDictionary.getIdFromValueBytesImpl(AppendTrieDictionary.java:971) > at > org.apache.kylin.common.util.Dictionary.getIdFromValueBytes(Dictionary.java:160) > at > org.apache.kylin.dimension.DictionaryDimEnc.encode(DictionaryDimEnc.java:91) > at > org.apache.kylin.cube.kv.RowKeyColumnIO.writeColumn(RowKeyColumnIO.java:55) > at > org.apache.kylin.cube.kv.RowKeyEncoder.fillColumnValue(RowKeyEncoder.java:195) > at org.apache.kylin.cube.kv.RowKeyEncoder.encode(RowKeyEncoder.java:163) > at > org.apache.kylin.engine.mr.steps.BaseCuboidMapperBase.buildKey(BaseCuboidMapperBase.java:151) > at > org.apache.kylin.engine.mr.steps.BaseCuboidMapperBase.outputKV(BaseCuboidMapperBase.java:198) > at > org.apache.kylin.engine.mr.steps.HiveToBaseCuboidMapper.map(HiveToBaseCuboidMapper.java:52) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (KYLIN-2316) Build Base Cuboid Data ERROR
[ https://issues.apache.org/jira/browse/KYLIN-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyang reassigned KYLIN-2316: - Assignee: kangkaisen (was: Zhong,Jason) > Build Base Cuboid Data ERROR > - > > Key: KYLIN-2316 > URL: https://issues.apache.org/jira/browse/KYLIN-2316 > Project: Kylin > Issue Type: Test > Components: General >Affects Versions: v1.6.0 > Environment: hadoop:cdh 5.7.0 > kylin:apache-kylin-1.6.0-cdh.5.7.0 > system:centos 6.8 >Reporter: konglei >Assignee: kangkaisen > Fix For: v2.0.0 > > > Dec 23, 2016 3:24:40 PM com.google.common.cache.LocalCache > processPendingNotifications > WARNING: Exception thrown by removal listener > java.lang.RuntimeException: unexpected evict reason COLLECTED > at > org.apache.kylin.dict.CachedTreeMap$1.onRemoval(CachedTreeMap.java:155) > at > com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:2004) > at > com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3490) > at > com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3466) > at > com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2304) > at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228) > at com.google.common.cache.LocalCache.get(LocalCache.java:3965) > at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3969) > at > com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4829) > at org.apache.kylin.dict.CachedTreeMap.get(CachedTreeMap.java:292) > at org.apache.kylin.dict.CachedTreeMap.get(CachedTreeMap.java:52) > at > org.apache.kylin.dict.AppendTrieDictionary.getIdFromValueBytesImpl(AppendTrieDictionary.java:971) > at > org.apache.kylin.common.util.Dictionary.getIdFromValueBytes(Dictionary.java:160) > at > org.apache.kylin.dimension.DictionaryDimEnc.encode(DictionaryDimEnc.java:91) > at > org.apache.kylin.cube.kv.RowKeyColumnIO.writeColumn(RowKeyColumnIO.java:55) > at > org.apache.kylin.cube.kv.RowKeyEncoder.fillColumnValue(RowKeyEncoder.java:195) > at org.apache.kylin.cube.kv.RowKeyEncoder.encode(RowKeyEncoder.java:163) > at > org.apache.kylin.engine.mr.steps.BaseCuboidMapperBase.buildKey(BaseCuboidMapperBase.java:151) > at > org.apache.kylin.engine.mr.steps.BaseCuboidMapperBase.outputKV(BaseCuboidMapperBase.java:198) > at > org.apache.kylin.engine.mr.steps.HiveToBaseCuboidMapper.map(HiveToBaseCuboidMapper.java:52) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (KYLIN-2316) Build Base Cuboid Data ERROR
[ https://issues.apache.org/jira/browse/KYLIN-2316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyang updated KYLIN-2316: -- Component/s: (was: Web ) General > Build Base Cuboid Data ERROR > - > > Key: KYLIN-2316 > URL: https://issues.apache.org/jira/browse/KYLIN-2316 > Project: Kylin > Issue Type: Test > Components: General >Affects Versions: v1.6.0 > Environment: hadoop:cdh 5.7.0 > kylin:apache-kylin-1.6.0-cdh.5.7.0 > system:centos 6.8 >Reporter: konglei >Assignee: kangkaisen > Fix For: v2.0.0 > > > Dec 23, 2016 3:24:40 PM com.google.common.cache.LocalCache > processPendingNotifications > WARNING: Exception thrown by removal listener > java.lang.RuntimeException: unexpected evict reason COLLECTED > at > org.apache.kylin.dict.CachedTreeMap$1.onRemoval(CachedTreeMap.java:155) > at > com.google.common.cache.LocalCache.processPendingNotifications(LocalCache.java:2004) > at > com.google.common.cache.LocalCache$Segment.runUnlockedCleanup(LocalCache.java:3490) > at > com.google.common.cache.LocalCache$Segment.postWriteCleanup(LocalCache.java:3466) > at > com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2304) > at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228) > at com.google.common.cache.LocalCache.get(LocalCache.java:3965) > at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3969) > at > com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4829) > at org.apache.kylin.dict.CachedTreeMap.get(CachedTreeMap.java:292) > at org.apache.kylin.dict.CachedTreeMap.get(CachedTreeMap.java:52) > at > org.apache.kylin.dict.AppendTrieDictionary.getIdFromValueBytesImpl(AppendTrieDictionary.java:971) > at > org.apache.kylin.common.util.Dictionary.getIdFromValueBytes(Dictionary.java:160) > at > org.apache.kylin.dimension.DictionaryDimEnc.encode(DictionaryDimEnc.java:91) > at > org.apache.kylin.cube.kv.RowKeyColumnIO.writeColumn(RowKeyColumnIO.java:55) > at > org.apache.kylin.cube.kv.RowKeyEncoder.fillColumnValue(RowKeyEncoder.java:195) > at org.apache.kylin.cube.kv.RowKeyEncoder.encode(RowKeyEncoder.java:163) > at > org.apache.kylin.engine.mr.steps.BaseCuboidMapperBase.buildKey(BaseCuboidMapperBase.java:151) > at > org.apache.kylin.engine.mr.steps.BaseCuboidMapperBase.outputKV(BaseCuboidMapperBase.java:198) > at > org.apache.kylin.engine.mr.steps.HiveToBaseCuboidMapper.map(HiveToBaseCuboidMapper.java:52) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KYLIN-2325) kylin sql optimization is very slow about 100ms or so, and consumes a lot of cpu
[ https://issues.apache.org/jira/browse/KYLIN-2325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15857546#comment-15857546 ] liyang commented on KYLIN-2325: --- Hi [~elasticlog], thanks for bring up this topic. 100ms is not much in big data scenario and it's meant to use CPU. That said, what is your requirement regarding the SQL parsing performance? Why is 100ms a concern? Is it a bottleneck in your env? Any performance goal you want to achieve? Please share more background. > kylin sql optimization is very slow about 100ms or so, and consumes a lot of > cpu > > > Key: KYLIN-2325 > URL: https://issues.apache.org/jira/browse/KYLIN-2325 > Project: Kylin > Issue Type: Improvement >Reporter: WangTaiZe > Labels: performance > Fix For: v2.0.0 > > Original Estimate: 12h > Remaining Estimate: 12h > > kylin sql optimization is too slow, and consumes too much cpu. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (KYLIN-2435) two EXTRACT on a column will fail if there exists NULL values for the column
hongbin ma created KYLIN-2435:
-
Summary: two EXTRACT on a column will fail if there exists NULL
values for the column
Key: KYLIN-2435
URL: https://issues.apache.org/jira/browse/KYLIN-2435
Project: Kylin
Issue Type: Bug
Reporter: hongbin ma
Assignee: hongbin ma
2000-01-01 19:12:33,US,android,10.22
2001-01-01 9:12:33,US,windows,9.12
2002-05-02 20:12:03,CN,windows,3.33
\N,CN,windows,3.32
create table testtable (starttime TIMESTAMP,country STRING, client STRING,
price DECIMAL(18,4)) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',';
the following query will succeed:
{code}
select sum(price),extract (year from starttime) from testtable group by extract
(year from starttime)
{code}
but the following will fail:
{code}
select sum(price) from testtable group by extract (year from starttime),
extract (month from starttime)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Closed] (KYLIN-1455) HBase ScanMetrics are not properly logged in query log
[
https://issues.apache.org/jira/browse/KYLIN-1455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dayue Gao closed KYLIN-1455.
Resolution: Won't Fix
> HBase ScanMetrics are not properly logged in query log
> --
>
> Key: KYLIN-1455
> URL: https://issues.apache.org/jira/browse/KYLIN-1455
> Project: Kylin
> Issue Type: Improvement
> Components: Storage - HBase
>Affects Versions: v1.2
>Reporter: Dayue Gao
>Assignee: Dayue Gao
>Priority: Minor
> Attachments: KYLIN-1455-1.x-staging.patch
>
>
> HBase's ScanMetrics provide users valuable information when troubleshooting
> query performance issues. But I found it was not properly logged, sometimes
> missing from the log, sometimes duplicated.
> Below is an example of duplicated scan log, this is due to
> {{CubeSegmentTupleIterator#closeScanner()}} method is invoked two times,
> first in hasNext(), second in close().
> {noformat}
> [http-bio-8080-exec-8]:[2016-02-26
> 17:31:50,227][DEBUG][org.apache.kylin.storage.hbase.CubeSegmentTupleIterator.closeScanner(CubeSegmentTupleIterator.java:146)]
> - Scan
> {"loadColumnFamiliesOnDemand":null,"filter":"FuzzyRowFilter{fuzzyKeysData={\\x00\\x00\\x00\\x00\\x00\\x00\\x09\\xC7\\x00\\x00\\x00\\x17\\x00\\x00\\x17\\x00:\\xFF\\xFF\\xFF\\xFF\\xFF\\xFF\\xFF\\xFF\\x00\\x00\\x00\\xFF\\x00\\x00\\xFF\\x00}},
>
> ","startRow":"\\x00\\x00\\x00\\x00\\x00\\x00\\x09\\xC7\\x00\\x00\\x00\\x17\\x00\\x00\\x17\\x00","stopRow":"\\x00\\x00\\x00\\x00\\x00\\x00\\x09\\xC7\\x08\\x8F\\xFF\\x17\\xFF\\xFF\\x17\\xFF\\x00","batch":-1,"cacheBlocks":true,"totalColumns":1,"maxResultSize":5242880,"families":{"F1":["M"]},"caching":1024,"maxVersions":1,"timeRange":[0,9223372036854775807]}
> [http-bio-8080-exec-8]:[2016-02-26
> 17:31:50,229][DEBUG][org.apache.kylin.storage.hbase.CubeSegmentTupleIterator.closeScanner(CubeSegmentTupleIterator.java:150)]
> - HBase Metrics: count=17357, ms=3194, bytes=905594, remote_bytes=905594,
> regions=1, not_serving_region=0, rpc=19, rpc_retries=0, remote_rpc=19,
> remote_rpc_retries=0
> [http-bio-8080-exec-8]:[2016-02-26
> 17:32:58,016][DEBUG][org.apache.kylin.storage.hbase.CubeSegmentTupleIterator.closeScanner(CubeSegmentTupleIterator.java:146)]
> - Scan
> {"loadColumnFamiliesOnDemand":null,"filter":"FuzzyRowFilter{fuzzyKeysData={\\x00\\x00\\x00\\x00\\x00\\x00\\x09\\xC7\\x00\\x00\\x00\\x17\\x00\\x00\\x17\\x00:\\xFF\\xFF\\xFF\\xFF\\xFF\\xFF\\xFF\\xFF\\x00\\x00\\x00\\xFF\\x00\\x00\\xFF\\x00}},
>
> ","startRow":"\\x00\\x00\\x00\\x00\\x00\\x00\\x09\\xC7\\x00\\x00\\x00\\x17\\x00\\x00\\x17\\x00","stopRow":"\\x00\\x00\\x00\\x00\\x00\\x00\\x09\\xC7\\x08\\x8F\\xFF\\x17\\xFF\\xFF\\x17\\xFF\\x00","batch":-1,"cacheBlocks":true,"totalColumns":1,"maxResultSize":5242880,"families":{"F1":["M"]},"caching":1024,"maxVersions":1,"timeRange":[0,9223372036854775807]}
> [http-bio-8080-exec-8]:[2016-02-26
> 17:33:04,443][DEBUG][org.apache.kylin.storage.hbase.CubeSegmentTupleIterator.closeScanner(CubeSegmentTupleIterator.java:150)]
> - HBase Metrics: count=17357, ms=3194, bytes=905594, remote_bytes=905594,
> regions=1, not_serving_region=0, rpc=19, rpc_retries=0, remote_rpc=19,
> remote_rpc_retries=0
> {noformat}
> And sometimes ScanMetrics is missing from the log, showed below. I think this
> is due to {{CubeSegmentTupleIterator#closeScanner()}} trying to get
> ScanMetrics before close the current ResultScanner. After looking into HBase
> client source, I found that ScanMetrics will not be written out until the
> scanner is closed or exhausted (no cache entries). So it'd be better to get
> ScanMetrics after closing the scanner.
> {noformat}
> [http-bio-8080-exec-2]:[2016-02-26
> 17:18:43,928][DEBUG][org.apache.kylin.storage.hbase.CubeSegmentTupleIterator.closeScanner(CubeSegmentTupleIterator.java:146)]
> - Scan
> {"loadColumnFamiliesOnDemand":null,"filter":"FuzzyRowFilter{fuzzyKeysData={\\x00\\x00\\x00\\x00\\x00\\x00\\x09\\xC7\\x00\\x00\\x00\\x17\\x00\\x00\\x17\\x00:\\xFF\\xFF\\xFF\\xFF\\xFF\\xFF\\xFF\\xFF\\x00\\x00\\x00\\xFF\\x00\\x00\\xFF\\x00}},
>
> ","startRow":"\\x00\\x00\\x00\\x00\\x00\\x00\\x09\\xC7\\x01\\x1C\\x04\\x0Cx\\x03\\x08Y","stopRow":"\\x00\\x00\\x00\\x00\\x00\\x00\\x09\\xC7\\x08\\x8F\\xFF\\x17\\xFF\\xFF\\x17\\xFF\\x00","batch":-1,"cacheBlocks":true,"totalColumns":1,"maxResultSize":5242880,"families":{"F1":["M"]},"caching":1024,"maxVersions":1,"timeRange":[0,9223372036854775807]}
> [http-bio-8080-exec-2]:[2016-02-26
> 17:19:38,228][INFO][org.apache.kylin.rest.service.QueryService.logQuery(QueryService.java:242)]
> -
> ==[QUERY]===
> {noformat}
> This should be easy to fix, I will submit a patch for this.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (KYLIN-2058) Make Kylin more resilient to bad queries
[ https://issues.apache.org/jira/browse/KYLIN-2058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dayue Gao resolved KYLIN-2058. -- Resolution: Fixed Fix Version/s: v1.6.0 There are still works to be done to defend Kylin against bad query. However since 1.6.0 has been released, I'll fire new JIRAs to continue. > Make Kylin more resilient to bad queries > > > Key: KYLIN-2058 > URL: https://issues.apache.org/jira/browse/KYLIN-2058 > Project: Kylin > Issue Type: Improvement > Components: Query Engine, Storage - HBase >Affects Versions: v1.6.0 >Reporter: Dayue Gao >Assignee: Dayue Gao > Fix For: v1.6.0 > > > Bad/Big queries are a huge threat to the overall performance and stability of > Kylin. We occasionally saw some of these queries either causing heavy GC > activities or crashing regionservers. I'd like to start a series of work to > make Kylin more resilient to bad queries. > This is an umbrella jira to relating works. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (KYLIN-2341) sum(case .. when ..) is not supported
[
https://issues.apache.org/jira/browse/KYLIN-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liyang updated KYLIN-2341:
--
Summary: sum(case .. when ..) is not supported (was: sum(case .. when ..)
problem)
> sum(case .. when ..) is not supported
> -
>
> Key: KYLIN-2341
> URL: https://issues.apache.org/jira/browse/KYLIN-2341
> Project: Kylin
> Issue Type: Bug
>Reporter: liyang
>Assignee: liyang
>
> Query like below should either fail, or return correct result. Currently it
> returns incorrect result.
> {code}
> SELECT
> sum(case
> when lstg_format_name like 'Other%'
> then price
> else 0
> end) as gmv
>
> FROM test_kylin_fact
> inner JOIN edw.test_cal_dt as test_cal_dt
> ON test_kylin_fact.cal_dt = test_cal_dt.cal_dt
> inner JOIN test_category_groupings
> ON test_kylin_fact.leaf_categ_id = test_category_groupings.leaf_categ_id
> AND test_kylin_fact.lstg_site_id = test_category_groupings.site_id
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (KYLIN-2428) Cleanup unnecessary shaded libraries for job/coprocessor/jdbc/server
[ https://issues.apache.org/jira/browse/KYLIN-2428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu updated KYLIN-2428: - Attachment: KYLIN-2428.patch > Cleanup unnecessary shaded libraries for job/coprocessor/jdbc/server > > > Key: KYLIN-2428 > URL: https://issues.apache.org/jira/browse/KYLIN-2428 > Project: Kylin > Issue Type: Improvement > Components: General >Affects Versions: v1.6.0 >Reporter: Billy Liu >Assignee: Billy Liu > Attachments: KYLIN-2428.patch > > > Kylin releases three libraries: kylin-coprocessor, kylin-jdbc, kylin-job and > one web application: server. > Currently, all libraries have shaded some used third party libraries into the > package. For example, guava, curator, commons, kyro in kylin-job. The > duplicate libraries in runtime classpath may have potential class loading > conflicts and waste computing resource. We should leverage the hadoop > provided libraries at runtime instead of the shaded one. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (KYLIN-2428) Cleanup unnecessary shaded libraries for job/coprocessor/jdbc/server
[ https://issues.apache.org/jira/browse/KYLIN-2428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu updated KYLIN-2428: - Attachment: (was: KYLIN-2428.patch) > Cleanup unnecessary shaded libraries for job/coprocessor/jdbc/server > > > Key: KYLIN-2428 > URL: https://issues.apache.org/jira/browse/KYLIN-2428 > Project: Kylin > Issue Type: Improvement > Components: General >Affects Versions: v1.6.0 >Reporter: Billy Liu >Assignee: Billy Liu > Attachments: KYLIN-2428.patch > > > Kylin releases three libraries: kylin-coprocessor, kylin-jdbc, kylin-job and > one web application: server. > Currently, all libraries have shaded some used third party libraries into the > package. For example, guava, curator, commons, kyro in kylin-job. The > duplicate libraries in runtime classpath may have potential class loading > conflicts and waste computing resource. We should leverage the hadoop > provided libraries at runtime instead of the shaded one. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (KYLIN-2388) Hot load kylin config from web
[ https://issues.apache.org/jira/browse/KYLIN-2388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kangkaisen resolved KYLIN-2388. --- Resolution: Fixed > Hot load kylin config from web > -- > > Key: KYLIN-2388 > URL: https://issues.apache.org/jira/browse/KYLIN-2388 > Project: Kylin > Issue Type: New Feature > Components: Web >Affects Versions: v1.6.0 >Reporter: kangkaisen >Assignee: kangkaisen > Fix For: v2.0.0 > > Attachments: KYLIN-2388.patch > > > Allow admin user reload kylin config from web, which could improve > operational efficiency and service stability. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Comment Edited] (KYLIN-2388) Hot load kylin config from web
[ https://issues.apache.org/jira/browse/KYLIN-2388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15855824#comment-15855824 ] kangkaisen edited comment on KYLIN-2388 at 2/8/17 3:18 AM: --- After The configs as follows update, The Kylin Server still need to restart. kylin.job.max-concurrent-jobs kylin.job.lock kylin.job.scheduler.default kylin.query.badquery-persistent-enabled kylin.query.badquery-detect-interval kylin.query.badquery-alerting-seconds kylin.server.mode kylin.server.init-tasks kylin.server.query-metrics-enabled kylin.web.timezone was (Author: kangkaisen): After The configs as follows update, The Kylin Server still nedd to restart. kylin.job.max-concurrent-jobs kylin.job.lock kylin.job.scheduler.default kylin.query.badquery-persistent-enabled kylin.server.mode kylin.server.init-tasks kylin.server.query-metrics-enabled kylin.web.timezone I will change code to make These configs could hot load. kylin.query.badquery-detect-interval kylin.query.badquery-alerting-seconds > Hot load kylin config from web > -- > > Key: KYLIN-2388 > URL: https://issues.apache.org/jira/browse/KYLIN-2388 > Project: Kylin > Issue Type: New Feature > Components: Web >Affects Versions: v1.6.0 >Reporter: kangkaisen >Assignee: kangkaisen > Fix For: v2.0.0 > > Attachments: KYLIN-2388.patch > > > Allow admin user reload kylin config from web, which could improve > operational efficiency and service stability. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KYLIN-2424) Optimize the integration test's performance
[ https://issues.apache.org/jira/browse/KYLIN-2424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15857252#comment-15857252 ] Billy Liu commented on KYLIN-2424: -- Thanks [~Shaofengshi], it reduces my IT from 3 hours to 1 hour. > Optimize the integration test's performance > --- > > Key: KYLIN-2424 > URL: https://issues.apache.org/jira/browse/KYLIN-2424 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI > Fix For: v2.0.0 > > > Kylin's integration test is slow, especially the ITCombinationTest. Most of > time are spent on H2 to execute the test queries. In a latest integration > test, this test case take 90 minutes to finish. > By checking H2's document, I think the main problem is the absence of index > on the tables, while index is very important for a relational database's > query performance. So when Kylin create the tables in H2, shoud create index > on the columns that will be used in the queries, like the pk/fk, the > filtering columns etc. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KYLIN-2431) StorageCleanupJob will remove intermediate tables created by other kylin instances
[ https://issues.apache.org/jira/browse/KYLIN-2431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15857231#comment-15857231 ] Billy Liu commented on KYLIN-2431: -- +1 > StorageCleanupJob will remove intermediate tables created by other kylin > instances > -- > > Key: KYLIN-2431 > URL: https://issues.apache.org/jira/browse/KYLIN-2431 > Project: Kylin > Issue Type: Improvement >Reporter: Dong Li >Assignee: Dong Li >Priority: Minor > > If QA and PROD instances are in same hive environment, running > StorageCleanupJob on QA will remove intermediate tables created by PROD, > which might fail Kylin jobs of PROD. > A solution is to add metastore name to hive table prefix, then filter table > names with metastore name during cleanup job. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (KYLIN-2431) StorageCleanupJob will remove intermediate tables created by other kylin instances
[ https://issues.apache.org/jira/browse/KYLIN-2431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li updated KYLIN-2431: --- Description: If QA and PROD instances are in same hive environment, running StorageCleanupJob on QA will remove immediate tables created by PROD, which might fail Kylin jobs of PROD. A solution is to add metastore name to hive table prefix, then filter table names with metastore name during cleanup job. was: If QA and PROD instances are in same hive database, running StorageCleanupJob on QA will remove immediate tables created by PROD, which might fail Kylin jobs of PROD. A solution is to add metastore name to hive table prefix, then filter table names with metastore name during cleanup job. > StorageCleanupJob will remove intermediate tables created by other kylin > instances > -- > > Key: KYLIN-2431 > URL: https://issues.apache.org/jira/browse/KYLIN-2431 > Project: Kylin > Issue Type: Improvement >Reporter: Dong Li >Assignee: Dong Li >Priority: Minor > > If QA and PROD instances are in same hive environment, running > StorageCleanupJob on QA will remove immediate tables created by PROD, which > might fail Kylin jobs of PROD. > A solution is to add metastore name to hive table prefix, then filter table > names with metastore name during cleanup job. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (KYLIN-2431) StorageCleanupJob will remove intermediate tables created by other kylin instances
[ https://issues.apache.org/jira/browse/KYLIN-2431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li updated KYLIN-2431: --- Description: If QA and PROD instances are in same hive environment, running StorageCleanupJob on QA will remove intermediate tables created by PROD, which might fail Kylin jobs of PROD. A solution is to add metastore name to hive table prefix, then filter table names with metastore name during cleanup job. was: If QA and PROD instances are in same hive environment, running StorageCleanupJob on QA will remove immediate tables created by PROD, which might fail Kylin jobs of PROD. A solution is to add metastore name to hive table prefix, then filter table names with metastore name during cleanup job. > StorageCleanupJob will remove intermediate tables created by other kylin > instances > -- > > Key: KYLIN-2431 > URL: https://issues.apache.org/jira/browse/KYLIN-2431 > Project: Kylin > Issue Type: Improvement >Reporter: Dong Li >Assignee: Dong Li >Priority: Minor > > If QA and PROD instances are in same hive environment, running > StorageCleanupJob on QA will remove intermediate tables created by PROD, > which might fail Kylin jobs of PROD. > A solution is to add metastore name to hive table prefix, then filter table > names with metastore name during cleanup job. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (KYLIN-2431) StorageCleanupJob will remove intermediate tables created by other kylin instances
[ https://issues.apache.org/jira/browse/KYLIN-2431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Li updated KYLIN-2431: --- Request participants: (was: ) Summary: StorageCleanupJob will remove intermediate tables created by other kylin instances (was: StorageCleanupJob will remove immediate tables created by other kylin instances) > StorageCleanupJob will remove intermediate tables created by other kylin > instances > -- > > Key: KYLIN-2431 > URL: https://issues.apache.org/jira/browse/KYLIN-2431 > Project: Kylin > Issue Type: Improvement >Reporter: Dong Li >Assignee: Dong Li >Priority: Minor > > If QA and PROD instances are in same hive database, running StorageCleanupJob > on QA will remove immediate tables created by PROD, which might fail Kylin > jobs of PROD. > A solution is to add metastore name to hive table prefix, then filter table > names with metastore name during cleanup job. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KYLIN-2388) Hot load kylin config from web
[ https://issues.apache.org/jira/browse/KYLIN-2388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15855846#comment-15855846 ] kangkaisen commented on KYLIN-2388: --- commit: https://github.com/apache/kylin/commit/39afa5197e197d5143843a3bb62c528749b03418 > Hot load kylin config from web > -- > > Key: KYLIN-2388 > URL: https://issues.apache.org/jira/browse/KYLIN-2388 > Project: Kylin > Issue Type: New Feature > Components: Web >Affects Versions: v1.6.0 >Reporter: kangkaisen >Assignee: kangkaisen > Fix For: v2.0.0 > > Attachments: KYLIN-2388.patch > > > Allow admin user reload kylin config from web, which could improve > operational efficiency and service stability. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KYLIN-2388) Hot load kylin config from web
[ https://issues.apache.org/jira/browse/KYLIN-2388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15855824#comment-15855824 ] kangkaisen commented on KYLIN-2388: --- After The configs as follows update, The Kylin Server still nedd to restart. kylin.job.max-concurrent-jobs kylin.job.lock kylin.job.scheduler.default kylin.query.badquery-persistent-enabled kylin.server.mode kylin.server.init-tasks kylin.server.query-metrics-enabled kylin.web.timezone I will change code to make These configs could hot load. kylin.query.badquery-detect-interval kylin.query.badquery-alerting-seconds > Hot load kylin config from web > -- > > Key: KYLIN-2388 > URL: https://issues.apache.org/jira/browse/KYLIN-2388 > Project: Kylin > Issue Type: New Feature > Components: Web >Affects Versions: v1.6.0 >Reporter: kangkaisen >Assignee: kangkaisen > Fix For: v2.0.0 > > Attachments: KYLIN-2388.patch > > > Allow admin user reload kylin config from web, which could improve > operational efficiency and service stability. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KYLIN-2430) Unnecessary exception catching in BulkLoadJob
[
https://issues.apache.org/jira/browse/KYLIN-2430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15855648#comment-15855648
]
kangkaisen commented on KYLIN-2430:
---
commit:
https://github.com/apache/kylin/commit/7b860adb0a23b53d6e2e40337f4317a3a2b067ba
> Unnecessary exception catching in BulkLoadJob
> -
>
> Key: KYLIN-2430
> URL: https://issues.apache.org/jira/browse/KYLIN-2430
> Project: Kylin
> Issue Type: Bug
> Components: Storage - HBase
>Affects Versions: v1.6.0
>Reporter: kangkaisen
>Assignee: kangkaisen
> Fix For: v2.0.0
>
>
> FsShell.run has caught all exceptions, So we should get exitCode instead of
> catching exception.
> Currently code potentially result in infinite loop in
> {{LoadIncrementalHFiles}} if user use HBase 0.98.13 and don't set
> {{hbase.bulkload.retries.number}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (KYLIN-2430) Unnecessary exception catching in BulkLoadJob
[
https://issues.apache.org/jira/browse/KYLIN-2430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kangkaisen resolved KYLIN-2430.
---
Resolution: Fixed
Fix Version/s: v2.0.0
> Unnecessary exception catching in BulkLoadJob
> -
>
> Key: KYLIN-2430
> URL: https://issues.apache.org/jira/browse/KYLIN-2430
> Project: Kylin
> Issue Type: Bug
> Components: Storage - HBase
>Affects Versions: v1.6.0
>Reporter: kangkaisen
>Assignee: kangkaisen
> Fix For: v2.0.0
>
>
> FsShell.run has caught all exceptions, So we should get exitCode instead of
> catching exception.
> Currently code potentially result in infinite loop in
> {{LoadIncrementalHFiles}} if user use HBase 0.98.13 and don't set
> {{hbase.bulkload.retries.number}}.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (KYLIN-2434) Spark cubing does not respect config kylin.source.hive.database-for-flat-table
Dong Li created KYLIN-2434: -- Summary: Spark cubing does not respect config kylin.source.hive.database-for-flat-table Key: KYLIN-2434 URL: https://issues.apache.org/jira/browse/KYLIN-2434 Project: Kylin Issue Type: Bug Reporter: Dong Li Assignee: Dong Li Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (KYLIN-2433) NPE in MergeCuboidMapper
[
https://issues.apache.org/jira/browse/KYLIN-2433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kangkaisen resolved KYLIN-2433.
---
Resolution: Fixed
Fix Version/s: v2.0.0
> NPE in MergeCuboidMapper
>
>
> Key: KYLIN-2433
> URL: https://issues.apache.org/jira/browse/KYLIN-2433
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v1.6.0
>Reporter: kangkaisen
>Assignee: kangkaisen
> Fix For: v2.0.0
>
>
> If all records of one column is null in a segment, there will be a NPE in
> {{sourceCubeSegment.getDictionary}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (KYLIN-2433) NPE in MergeCuboidMapper
[
https://issues.apache.org/jira/browse/KYLIN-2433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15855628#comment-15855628
]
kangkaisen commented on KYLIN-2433:
---
commit: b32cc9545a7665757
> NPE in MergeCuboidMapper
>
>
> Key: KYLIN-2433
> URL: https://issues.apache.org/jira/browse/KYLIN-2433
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v1.6.0
>Reporter: kangkaisen
>Assignee: kangkaisen
> Fix For: v2.0.0
>
>
> If all records of one column is null in a segment, there will be a NPE in
> {{sourceCubeSegment.getDictionary}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (KYLIN-2433) NPE in MergeCuboidMapper
kangkaisen created KYLIN-2433:
-
Summary: NPE in MergeCuboidMapper
Key: KYLIN-2433
URL: https://issues.apache.org/jira/browse/KYLIN-2433
Project: Kylin
Issue Type: Bug
Components: Job Engine
Affects Versions: v1.6.0
Reporter: kangkaisen
Assignee: kangkaisen
If all records of one column is null in a segment, there will be a NPE in
{{sourceCubeSegment.getDictionary}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (KYLIN-2432) Couldn't select partition column in some old browser (such as Google Chrome 18.0.1025.162)
luguosheng created KYLIN-2432: - Summary: Couldn't select partition column in some old browser (such as Google Chrome 18.0.1025.162) Key: KYLIN-2432 URL: https://issues.apache.org/jira/browse/KYLIN-2432 Project: Kylin Issue Type: Bug Components: Web Affects Versions: v2.0.0 Reporter: luguosheng Assignee: luguosheng Fix For: v2.0.0 Couldn't select partition column in some old browser (such as Google Chrome 18.0.1025.162) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
