Hi All, I tried using Kylin on Hadoop environment, but overhead of hadoop is too much for our medium scale need. Is there any way we can use kylin Olap engine with minimal requirements of underlaying storage layer. What are the best practices and architecture to support this, where we possibly can use it without hadoop components. Thanks
Asim Ali *Software Developer* Email: [email protected] <[email protected]> Phone: 1300 855 642 <1300855642> Website: www.easyemployer.com [image: easyemployer] <http://www.easyemployer.com/> On Fri, 21 Jun 2019 at 15:22, 敏丞 <[email protected]> wrote: > Hi, > After check cube.json provided by you, I can reproduce this error in > my development env. Looks like this kind exception occurs when you have a > cube which *have both Raw measure and CountDistinct(Bitmap) on the same > column*. I find the reason should be the Raw Measure choose the wrong > dictionary (AppendTrieDictionary cannot used to decode). Maybe you should > try use two cubes in this situation. > And if you don't mind, I have a question, have you ever use this type > of query "select * from FACT_TABLE" in old version of kylin in such kind of > cube(raw measure and count_distinct both on the same column) and get > correct result successfully? > > > If you have find anything wrong or other information, please let me > know. Thank you. > > > > > *-----------------* > *-----------------* > *Best wishes to you ! * > *From ：**Xiaoxiang Yu* > > > At 2019-06-20 16:00:10, "[email protected]" <[email protected]> > wrote: > >hi,all: > > > > I got an error like "AppendTrieDictionary can't retrieve value from id" > > while query the cube data, the following is the cube info and exception > > info. > > > >I found the same error in this > >thread(https://lists.apache.org/thread.html/63981bc08ef7b97c41921ed692de79ef9a744f6329192e5199074ba3@%3Cdev.kylin.apache.org%3E), > > but I just use the bitmaps (count distinct) as measure, and never use it > >as dimension. > > > >So please help me to resolve it. > > > >cube data: > >{ > > "uuid": "eb0b4a32-fbc0-b197-b3f0-4c9cd5fb3916", > > "last_modified": 1561014544528, > > "version": "2.6.2.0", > > "name": "dev_cube_user_currency", > > "is_draft": false, > > "model_name": "user_currency", > > "description": "", > > "null_string": null, > > "dimensions": [ > > { > > "name": "TYPE", > > "table": "DEV_DWD_USER_CURRENCY", > > "column": "TYPE", > > "derived": null > > }, > > { > > "name": "SUB_TYPE", > > "table": "DEV_DWD_USER_CURRENCY", > > "column": "SUB_TYPE", > > "derived": null > > }, > > { > > "name": "SOURCE_TYPE", > > "table": "DEV_DWD_USER_CURRENCY", > > "column": "SOURCE_TYPE", > > "derived": null > > }, > > { > > "name": "SOURCE", > > "table": "DEV_DWD_USER_CURRENCY", > > "column": "SOURCE", > > "derived": null > > }, > > { > > "name": "STATUS", > > "table": "DEV_DWD_USER_CURRENCY", > > "column": "STATUS", > > "derived": null > > }, > > { > > "name": "CREATE_TIME", > > "table": "DEV_DWD_USER_CURRENCY", > > "column": "CREATE_TIME", > > "derived": null > > }, > > { > > "name": "UPDATE_TIME", > > "table": "DEV_DWD_USER_CURRENCY", > > "column": "UPDATE_TIME", > > "derived": null > > }, > > { > > "name": "END_TIME", > > "table": "DEV_DWD_USER_CURRENCY", > > "column": "END_TIME", > > "derived": null > > }, > > { > > "name": "ACTIVATION_TIME", > > "table": "DEV_DWD_USER_CURRENCY", > > "column": "ACTIVATION_TIME", > > "derived": null > > }, > > { > > "name": "CODE", > > "table": "DEV_DIM_CURRENCY_TYPE", > > "column": null, > > "derived": [ > > "CODE" > > ] > > }, > > { > > "name": "SHORT_NAME", > > "table": "DEV_DIM_CURRENCY_TYPE", > > "column": null, > > "derived": [ > > "SHORT_NAME" > > ] > > } > > ], > > "measures": [ > > { > > "name": "_COUNT_", > > "function": { > > "expression": "COUNT", > > "parameter": { > > "type": "constant", > > "value": "1" > > }, > > "returntype": "bigint" > > } > > }, > > { > > "name": "SUM_TOTAL", > > "function": { > > "expression": "SUM", > > "parameter": { > > "type": "column", > > "value": "DEV_DWD_USER_CURRENCY.TOTAL" > > }, > > "returntype": "double" > > } > > }, > > { > > "name": "SUM_BALANCE", > > "function": { > > "expression": "SUM", > > "parameter": { > > "type": "column", > > "value": "DEV_DWD_USER_CURRENCY.BALANCE" > > }, > > "returntype": "double" > > } > > }, > > { > > "name": "SUM_VALID_PERIOD", > > "function": { > > "expression": "SUM", > > "parameter": { > > "type": "column", > > "value": "DEV_DWD_USER_CURRENCY.VALID_PERIOD" > > }, > > "returntype": "bigint" > > } > > }, > > { > > "name": "MAX_TOTAL", > > "function": { > > "expression": "MAX", > > "parameter": { > > "type": "column", > > "value": "DEV_DWD_USER_CURRENCY.TOTAL" > > }, > > "returntype": "double" > > } > > }, > > { > > "name": "MAX_BALANCE", > > "function": { > > "expression": "MAX", > > "parameter": { > > "type": "column", > > "value": "DEV_DWD_USER_CURRENCY.BALANCE" > > }, > > "returntype": "double" > > } > > }, > > { > > "name": "MAX_VALID_PERIOD", > > "function": { > > "expression": "MAX", > > "parameter": { > > "type": "column", > > "value": "DEV_DWD_USER_CURRENCY.VALID_PERIOD" > > }, > > "returntype": "integer" > > } > > }, > > { > > "name": "MIN_TOTAL", > > "function": { > > "expression": "MIN", > > "parameter": { > > "type": "column", > > "value": "DEV_DWD_USER_CURRENCY.TOTAL" > > }, > > "returntype": "double" > > } > > }, > > { > > "name": "MIN_BALANCE", > > "function": { > > "expression": "MIN", > > "parameter": { > > "type": "column", > > "value": "DEV_DWD_USER_CURRENCY.BALANCE" > > }, > > "returntype": "double" > > } > > }, > > { > > "name": "MIN_VALID_PERIOD", > > "function": { > > "expression": "MIN", > > "parameter": { > > "type": "column", > > "value": "DEV_DWD_USER_CURRENCY.VALID_PERIOD" > > }, > > "returntype": "integer" > > } > > }, > > { > > "name": "RAW_PARENT_ID", > > "function": { > > "expression": "RAW", > > "parameter": { > > "type": "column", > > "value": "DEV_DWD_USER_CURRENCY.PARENT_ID" > > }, > > "returntype": "raw" > > } > > }, > > { > > "name": "RAW_ORDER_NO", > > "function": { > > "expression": "RAW", > > "parameter": { > > "type": "column", > > "value": "DEV_DWD_USER_CURRENCY.ORDER_NO" > > }, > > "returntype": "raw" > > } > > }, > > { > > "name": "COUNT_PARENTID", > > "function": { > > "expression": "COUNT_DISTINCT", > > "parameter": { > > "type": "column", > > "value": "DEV_DWD_USER_CURRENCY.PARENT_ID" > > }, > > "returntype": "bitmap" > > } > > }, > > { > > "name": "COUNT_ORDERNO", > > "function": { > > "expression": "COUNT_DISTINCT", > > "parameter": { > > "type": "column", > > "value": "DEV_DWD_USER_CURRENCY.ORDER_NO" > > }, > > "returntype": "bitmap" > > } > > } > > ], > > "dictionaries": [ > > { > > "column": "DEV_DWD_USER_CURRENCY.PARENT_ID", > > "builder": "org.apache.kylin.dict.GlobalDictionaryBuilder" > > }, > > { > > "column": "DEV_DWD_USER_CURRENCY.ORDER_NO", > > "builder": "org.apache.kylin.dict.GlobalDictionaryBuilder" > > } > > ], > > "rowkey": { > > "rowkey_columns": [ > > { > > "column": "DEV_DWD_USER_CURRENCY.TYPE", > > "encoding": "integer:2", > > "encoding_version": 1, > > "isShardBy": false > > }, > > { > > "column": "DEV_DWD_USER_CURRENCY.SUB_TYPE", > > "encoding": "integer:2", > > "encoding_version": 1, > > "isShardBy": false > > }, > > { > > "column": "DEV_DWD_USER_CURRENCY.SOURCE_TYPE", > > "encoding": "integer:2", > > "encoding_version": 1, > > "isShardBy": false > > }, > > { > > "column": "DEV_DWD_USER_CURRENCY.STATUS", > > "encoding": "integer:2", > > "encoding_version": 1, > > "isShardBy": false > > }, > > { > > "column": "DEV_DWD_USER_CURRENCY.CREATE_TIME", > > "encoding": "time", > > "encoding_version": 1, > > "isShardBy": false > > }, > > { > > "column": "DEV_DWD_USER_CURRENCY.UPDATE_TIME", > > "encoding": "time", > > "encoding_version": 1, > > "isShardBy": false > > }, > > { > > "column": "DEV_DWD_USER_CURRENCY.END_TIME", > > "encoding": "time", > > "encoding_version": 1, > > "isShardBy": false > > }, > > { > > "column": "DEV_DWD_USER_CURRENCY.ACTIVATION_TIME", > > "encoding": "time", > > "encoding_version": 1, > > "isShardBy": false > > }, > > { > > "column": "DEV_DWD_USER_CURRENCY.SOURCE", > > "encoding": "fixed_length:10", > > "encoding_version": 1, > > "isShardBy": false > > } > > ] > > }, > > "hbase_mapping": { > > "column_family": [ > > { > > "name": "F1", > > "columns": [ > > { > > "qualifier": "M", > > "measure_refs": [ > > "_COUNT_", > > "SUM_TOTAL", > > "SUM_BALANCE", > > "SUM_VALID_PERIOD", > > "MAX_TOTAL", > > "MAX_BALANCE", > > "MAX_VALID_PERIOD", > > "MIN_TOTAL", > > "MIN_BALANCE", > > "MIN_VALID_PERIOD", > > "RAW_PARENT_ID", > > "RAW_ORDER_NO" > > ] > > } > > ] > > }, > > { > > "name": "F2", > > "columns": [ > > { > > "qualifier": "M", > > "measure_refs": [ > > "COUNT_PARENTID", > > "COUNT_ORDERNO" > > ] > > } > > ] > > } > > ] > > }, > > "aggregation_groups": [ > > { > > "includes": [ > > "DEV_DWD_USER_CURRENCY.TYPE", > > "DEV_DWD_USER_CURRENCY.SUB_TYPE", > > "DEV_DWD_USER_CURRENCY.SOURCE_TYPE", > > "DEV_DWD_USER_CURRENCY.SOURCE", > > "DEV_DWD_USER_CURRENCY.STATUS", > > "DEV_DWD_USER_CURRENCY.CREATE_TIME", > > "DEV_DWD_USER_CURRENCY.UPDATE_TIME", > > "DEV_DWD_USER_CURRENCY.END_TIME", > > "DEV_DWD_USER_CURRENCY.ACTIVATION_TIME" > > ], > > "select_rule": { > > "hierarchy_dims": [], > > "mandatory_dims": [], > > "joint_dims": [] > > } > > } > > ], > > "signature": "eiGIwgaD57yCz/jfbRCscg==", > > "notify_list": [], > > "status_need_notify": [ > > "ERROR", > > "DISCARDED", > > "SUCCEED" > > ], > > "partition_date_start": 0, > > "partition_date_end": 3153600000000, > > "auto_merge_time_ranges": [], > > "volatile_range": 0, > > "retention_range": 0, > > "engine_type": 2, > > "storage_type": 2, > > "override_kylin_properties": {}, > > "cuboid_black_list": [], > > "parent_forward": 3, > > "mandatory_dimension_set_list": [], > > "snapshot_table_desc_list": [] > >} > > > >=====exception info==================== > >org.apache.kylin.rest.exception.InternalErrorException: AppendTrieDictionary > >can't retrieve value from id > >while executing SQL: "select * from mmears_dev.DEV_DWD_USER_CURRENCY limit > >10" > > at > > org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:441) > > at > > org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:367) > > at > > org.apache.kylin.rest.controller.QueryController.query(QueryController.java:93) > > at sun.reflect.GeneratedMethodAccessor209.invoke(Unknown Source) > > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:498) > > at > > org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) > > at > > org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) > > at > > org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) > > at > > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) > > at > > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) > > at > > org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) > > at > > org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967) > > at > > org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) > > at > > org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) > > at > > org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872) > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:661) > > at > > org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) > > at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) > > at > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) > > at > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) > > at > > org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) > > at > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) > > at > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) > > at > > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317) > > at > > org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127) > > at > > org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91) > > at > > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) > > at > > org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114) > > at > > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) > > at > > org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137) > > at > > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) > > at > > org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111) > > at > > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) > > at > > org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170) > > at > > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) > > at > > org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) > > at > > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) > > at > > org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:158) > > at > > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) > > at > > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) > > at > > org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200) > > at > > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) > > at > > org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116) > > at > > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) > > at > > org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:64) > > at > > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) > > at > > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) > > at > > org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56) > > at > > org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) > > at > > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) > > at > > org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105) > > at > > org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331) > > at > > org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214) > > at > > org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177) > > at > > org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346) > > at > > org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262) > > at > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) > > at > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) > > at > > com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:209) > > at > > com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:244) > > at > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) > > at > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) > > at > > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198) > > at > > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) > > at > > org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493) > > at > > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) > > at > > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) > > at > > org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650) > > at > > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) > > at > > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) > > at > > org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:800) > > at > > org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) > > at > > org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:806) > > at > > org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498) > > at > > org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) > > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > at > > org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) > > at java.lang.Thread.run(Thread.java:748) > > > >
