[
https://issues.apache.org/jira/browse/KYLIN-958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14710970#comment-14710970
]
Shaofeng SHI commented on KYLIN-958:
------------------------------------
Summary for today's discussion on call:
Disable the edit action for data model in "Edit Cube" UI, this includes: a)
change lookup table; b) add/update/remove any lookup tables and their join
condition with fact table; c) partition column info; d) filter conditions; If
user does need change the data model, he/she should create a new cube; In the
backend java, only update CubeDesc on saving; If need support on UI part,
please contact Jason;
As this will disallow user to change the data model once the cube is created,
it may add user's labor; Another possible way is, when user saves the cube, in
backend Kylin takes the following steps: 1) drop cube and cube desc; 2) drop
data model; 3) create data model; 4) create cube desc and cube; Each step will
trigger a BroadCast event, but it will not cause metadata error; In the end all
things get re-created in meta store; The problem is, if encounter error during
these steps, the cube will be inconsistent state; But the possibility is low
and user can re-create at least;
Just evaluate and pick a solution you like when you implement the patch; thanks!
> update cube data model may fail and leave metadata in inconsistent state
> ------------------------------------------------------------------------
>
> Key: KYLIN-958
> URL: https://issues.apache.org/jira/browse/KYLIN-958
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
> Affects Versions: v0.7.2
> Reporter: Dayue Gao
> Assignee: Shaofeng SHI
> Priority: Critical
>
> When user updates cube data model, say drop one lookup table and related
> dimensions, the operation will fail with message "Failed to deal with the
> request: null". What's worse is that *metadata is corrupted*, data model is
> in post-update state where cube desc is in pre-update state.
> Reproduction Steps:
> 1. Define a cube "test_cube" with two lookup tables, EDW.TEST_SITES and
> EDW.TEST_SELLER_TYPE_DIM. Define a derived dimension SITE_NAME from
> EDW.TEST_SITES.
> 2. Edit the cube, drop one lookup table EDW.TEST_SITES. The system will
> automatically drop dimensions from EDW.TEST_SITES.
> 3. Save the new cube. The error message may occur. (race condition)
> From logs
> -------------
> {noformat}
> java.lang.IllegalStateException: Derived can only be defined on lookup table,
> cube CubeDesc [name=test_cube], DimensionDesc [name=SITE_NAME, join=null,
> hierarchy=null, table=EDW.TEST_SITES, column=null, derived=[SITE_NAME]]
> at org.apache.kylin.cube.model.DimensionDesc.init(DimensionDesc.java:113)
> at org.apache.kylin.cube.model.CubeDesc.init(CubeDesc.java:446)
> at
> org.apache.kylin.cube.CubeDescManager.loadCubeDesc(CubeDescManager.java:129)
> at
> org.apache.kylin.cube.CubeDescManager.reloadAllCubeDesc(CubeDescManager.java:200)
> at org.apache.kylin.cube.CubeDescManager.<init>(CubeDescManager.java:93)
> at
> org.apache.kylin.cube.CubeDescManager.getInstance(CubeDescManager.java:73)
> at org.apache.kylin.cube.CubeInstance.getDescriptor(CubeInstance.java:121)
> at
> org.apache.kylin.rest.service.CubeService.updateCubeAndDesc(CubeService.java:237)
> at
> org.apache.kylin.rest.service.CubeService$$FastClassByCGLIB$$17a07c0e.invoke(<generated>)
> at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
> at
> org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:689)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at
> org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:64)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:622)
> at
> org.apache.kylin.rest.service.CubeService$$EnhancerByCGLIB$$57a9f12a.updateCubeAndDesc(<generated>)
> at
> org.apache.kylin.rest.controller.CubeController.updateCubeDesc(CubeController.java:385)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:213)
> at
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:126)
> at
> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:96)
> at
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:617)
> at
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:578)
> at
> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80)
> at
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:923)
> at
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852)
> at
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)
> at
> org.springframework.web.servlet.FrameworkServlet.doPut(FrameworkServlet.java:800)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:649)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
> at
> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
> at
> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
> at
> org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
> .... // ommited
> {noformat}
> The cause of the problem is in {{CubeController.updateCubeDesc}}, data model
> is first updated {{metaManager.updateDataModelDesc(modelDesc);}}, which will
> cause a {{BroadcastEvent type=data_model, name=test_cube, action=update}} to
> be sent. As a result CubeDesc cache will be cleared.
> {code:title=CacheService.java}
> rebuildCache
> case DATA_MODEL:
> getMetadataManager().reloadDataModelDesc(cacheKey);
> IIDescManager.clearCache();
> CubeDescManager.clearCache();
> break;
> {code}
> Then when the cube is being updated through
> {{cubeService.updateCubeAndDesc(cube, desc, projectName);}}, all cube desc is
> reloaded in CubeDescManager and the old cube desc can't match the new data
> model.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
