Shaofeng SHI commented on KYLIN-2012:

Dayue, in our Kylin env there is already some cubes marked as "DESCBROKEN" 
(released in, when deploy the new code, I guess there will be error 
unless doing a metadata migration. Can we keep the old name? or if it can be 
compitable with previous name, that would avoid the migration...

> more robust approach to hive schema changes
> -------------------------------------------
>                 Key: KYLIN-2012
>                 URL: https://issues.apache.org/jira/browse/KYLIN-2012
>             Project: Kylin
>          Issue Type: Bug
>          Components: Metadata, REST Service, Web 
>    Affects Versions: v1.5.3
>            Reporter: Dayue Gao
>            Assignee: Dayue Gao
>             Fix For: v1.6.0
> Our users occasionally want to change their existing cube, such as 
> adding/renaming/removing a dimension. Some of these changes require 
> modifications to its source hive table. So our user changed the table schema 
> and reloaded its metadata in Kylin, then several issues can happen depends on 
> what he changed.
> I did some schema changing tests based on 1.5.3, the results after reloading 
> table are listed below
> || type of changes || fact table || lookup table ||
> | *minor* | both query and build still works | query can fail or return wrong 
> answer |
> | *major* | fail to load related cube | fail to load related cube |
> {{minor}} changes refer to those doesn't change columns used in cubes, such 
> as insert/append new column, remove/change unused column.
> {{major}} changes are the opposite, like remove/rename/change type of used 
> column.
> Clearly from the table, reload a changed table is problematic in certain 
> cases. KYLIN-1536 reports a similar problem.
> So what can we do to support this kind of iterative development process (load 
> -> define cube -> build -> reload -> change cube -> rebuild)?
> My first thought is simply detect-and-prohibit reloading used table. User 
> should be able to know which cube is preventing him from reloading, and then 
> he could drop and recreate cube after reloading. However, defining a cube is 
> not an easy task (consider editing 100 measures). Force users to recreate 
> their cube over and over again will certainly not make them happy.
> A better idea is to allow cube to be editable even if it's broken due to some 
> columns changed after reloading. Broken cube can't be built or queried, it 
> can only be edit or dropped. In fact, there is a cube status called 
> {{RealizationStatusEnum.DESCBROKEN}} in code, but was never used. We should 
> take advantage of it.
> An enabled cube shouldn't allow schema changes, otherwise an unintentional 
> reload could make it unavailable. Similarly, a disabled but unpurged cube 
> shouldn't allow schema changes since it still has data in it.

This message was sent by Atlassian JIRA

Reply via email to