[jira] [Commented] (KYLIN-2672) Only clean necessary cache for CubeMigrationCLI
[ https://issues.apache.org/jira/browse/KYLIN-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358058#comment-16358058 ] kangkaisen commented on KYLIN-2672: --- This is the test commit in KYLIN-2672 branch: [https://github.com/apache/kylin/commit/0b393851718a17d3cae470e67e628d69cabf4ad8] please review. thanks. > Only clean necessary cache for CubeMigrationCLI > --- > > Key: KYLIN-2672 > URL: https://issues.apache.org/jira/browse/KYLIN-2672 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Affects Versions: v2.0.0 >Reporter: kangkaisen >Assignee: kangkaisen >Priority: Major > Fix For: v2.3.0 > > Attachments: KYLIN-2672.patch > > > Currently, we simply clear ALL cache in CubeMigrationCLI. which will make a > few of queries slower in prod env when we have many tables, models, cubes and > migrate cube often. > So, we could only clean necessary cache for CubeMigrationCLI. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-2672) Only clean necessary cache for CubeMigrationCLI
[ https://issues.apache.org/jira/browse/KYLIN-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16352231#comment-16352231 ] kangkaisen commented on KYLIN-2672: --- Thanks liyang and Billy. OK, I will add a test for the new API. > Only clean necessary cache for CubeMigrationCLI > --- > > Key: KYLIN-2672 > URL: https://issues.apache.org/jira/browse/KYLIN-2672 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Affects Versions: v2.0.0 >Reporter: kangkaisen >Assignee: kangkaisen >Priority: Major > Fix For: v2.3.0 > > Attachments: KYLIN-2672.patch > > > Currently, we simply clear ALL cache in CubeMigrationCLI. which will make a > few of queries slower in prod env when we have many tables, models, cubes and > migrate cube often. > So, we could only clean necessary cache for CubeMigrationCLI. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-2672) Only clean necessary cache for CubeMigrationCLI
[ https://issues.apache.org/jira/browse/KYLIN-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16352228#comment-16352228 ] liyang commented on KYLIN-2672: --- I rebased and merged the patch into branch KYLIN-2672. The code looks good. The only concern is the lack of test cases. [~kangkaisen], could you add test cases to cover the newly added feature? > Only clean necessary cache for CubeMigrationCLI > --- > > Key: KYLIN-2672 > URL: https://issues.apache.org/jira/browse/KYLIN-2672 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Affects Versions: v2.0.0 >Reporter: kangkaisen >Assignee: kangkaisen >Priority: Major > Fix For: v2.3.0 > > Attachments: KYLIN-2672.patch > > > Currently, we simply clear ALL cache in CubeMigrationCLI. which will make a > few of queries slower in prod env when we have many tables, models, cubes and > migrate cube often. > So, we could only clean necessary cache for CubeMigrationCLI. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-2672) Only clean necessary cache for CubeMigrationCLI
[ https://issues.apache.org/jira/browse/KYLIN-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351728#comment-16351728 ] Billy Liu commented on KYLIN-2672: -- [~liyang.g...@gmail.com], could you review this patch? > Only clean necessary cache for CubeMigrationCLI > --- > > Key: KYLIN-2672 > URL: https://issues.apache.org/jira/browse/KYLIN-2672 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Affects Versions: v2.0.0 >Reporter: kangkaisen >Assignee: kangkaisen >Priority: Major > Fix For: v2.3.0 > > Attachments: KYLIN-2672.patch > > > Currently, we simply clear ALL cache in CubeMigrationCLI. which will make a > few of queries slower in prod env when we have many tables, models, cubes and > migrate cube often. > So, we could only clean necessary cache for CubeMigrationCLI. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-2672) Only clean necessary cache for CubeMigrationCLI
[ https://issues.apache.org/jira/browse/KYLIN-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16342470#comment-16342470 ] kangkaisen commented on KYLIN-2672: --- BTW，this patch has nothing to do with KYLIN-2717. > Only clean necessary cache for CubeMigrationCLI > --- > > Key: KYLIN-2672 > URL: https://issues.apache.org/jira/browse/KYLIN-2672 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Affects Versions: v2.0.0 >Reporter: kangkaisen >Assignee: kangkaisen >Priority: Major > Fix For: v2.3.0 > > Attachments: KYLIN-2672.patch > > > Currently, we simply clear ALL cache in CubeMigrationCLI. which will make a > few of queries slower in prod env when we have many tables, models, cubes and > migrate cube often. > So, we could only clean necessary cache for CubeMigrationCLI. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-2672) Only clean necessary cache for CubeMigrationCLI
[ https://issues.apache.org/jira/browse/KYLIN-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16342469#comment-16342469 ] kangkaisen commented on KYLIN-2672: --- During user rolling upgrade to Kylin 2.3.0, user could use org.apache.kylin.storage.hbase.util.CubeMigrationCLI instead of org.apache.kylin.tool.CubeMigrationCLI, or user could still use org.apache.kylin.tool.CubeMigrationCLI and manually clear all cache in web. > Only clean necessary cache for CubeMigrationCLI > --- > > Key: KYLIN-2672 > URL: https://issues.apache.org/jira/browse/KYLIN-2672 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Affects Versions: v2.0.0 >Reporter: kangkaisen >Assignee: kangkaisen >Priority: Major > Fix For: v2.3.0 > > Attachments: KYLIN-2672.patch > > > Currently, we simply clear ALL cache in CubeMigrationCLI. which will make a > few of queries slower in prod env when we have many tables, models, cubes and > migrate cube often. > So, we could only clean necessary cache for CubeMigrationCLI. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-2672) Only clean necessary cache for CubeMigrationCLI
[ https://issues.apache.org/jira/browse/KYLIN-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16342468#comment-16342468 ] kangkaisen commented on KYLIN-2672: --- The basic idea for this patch is adding a new api for cube migration and only reloading necessary cache， not any clear operation. The metadata reloading must be by the follwing order: - table - project - model - cube desc - cube instance - project l2 cache The reason please refer the code comment. The meaning for this patch is improving the Kylin QueryServer availability and Fixing the "Table not found" bug when migrate cube. > Only clean necessary cache for CubeMigrationCLI > --- > > Key: KYLIN-2672 > URL: https://issues.apache.org/jira/browse/KYLIN-2672 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Affects Versions: v2.0.0 >Reporter: kangkaisen >Assignee: kangkaisen >Priority: Major > Fix For: v2.3.0 > > Attachments: KYLIN-2672.patch > > > Currently, we simply clear ALL cache in CubeMigrationCLI. which will make a > few of queries slower in prod env when we have many tables, models, cubes and > migrate cube often. > So, we could only clean necessary cache for CubeMigrationCLI. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-2672) Only clean necessary cache for CubeMigrationCLI
[ https://issues.apache.org/jira/browse/KYLIN-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16339146#comment-16339146 ] kangkaisen commented on KYLIN-2672: --- I will add a patch tomorrow. > Only clean necessary cache for CubeMigrationCLI > --- > > Key: KYLIN-2672 > URL: https://issues.apache.org/jira/browse/KYLIN-2672 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Affects Versions: v2.0.0 >Reporter: kangkaisen >Assignee: kangkaisen >Priority: Major > Fix For: v2.3.0 > > Attachments: KYLIN-2672.patch > > > Currently, we simply clear ALL cache in CubeMigrationCLI. which will make a > few of queries slower in prod env when we have many tables, models, cubes and > migrate cube often. > So, we could only clean necessary cache for CubeMigrationCLI. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-2672) Only clean necessary cache for CubeMigrationCLI
[ https://issues.apache.org/jira/browse/KYLIN-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16097175#comment-16097175 ] kangkaisen commented on KYLIN-2672: --- KYLIN-2717 is great. After KYLIN-2717, we could only reload the related tables. I will update this patch when you finish KYLIN-2717. Thanks you. > Only clean necessary cache for CubeMigrationCLI > --- > > Key: KYLIN-2672 > URL: https://issues.apache.org/jira/browse/KYLIN-2672 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Affects Versions: v2.0.0 >Reporter: kangkaisen >Assignee: kangkaisen > Attachments: KYLIN-2672.patch > > > Currently, we simply clear ALL cache in CubeMigrationCLI. which will make a > few of queries slower in prod env when we have many tables, models, cubes and > migrate cube often. > So, we could only clean necessary cache for CubeMigrationCLI. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2672) Only clean necessary cache for CubeMigrationCLI
[ https://issues.apache.org/jira/browse/KYLIN-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095331#comment-16095331 ] liyang commented on KYLIN-2672: --- True, tables requires reload too. I'm working on KYLIN-2717 that may help this JIRA too. Once table is project local, not shared by multiple projects globally, then reload a TableDesc won't impact multiple projects. Will this behavior solve the problem addressed by this JIRA? > Only clean necessary cache for CubeMigrationCLI > --- > > Key: KYLIN-2672 > URL: https://issues.apache.org/jira/browse/KYLIN-2672 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Affects Versions: v2.0.0 >Reporter: kangkaisen >Assignee: kangkaisen > Attachments: KYLIN-2672.patch > > > Currently, we simply clear ALL cache in CubeMigrationCLI. which will make a > few of queries slower in prod env when we have many tables, models, cubes and > migrate cube often. > So, we could only clean necessary cache for CubeMigrationCLI. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2672) Only clean necessary cache for CubeMigrationCLI
[ https://issues.apache.org/jira/browse/KYLIN-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16088910#comment-16088910 ] kangkaisen commented on KYLIN-2672: --- The SrcTable and SrcTableExt only listen on ClearAll and itself events, So I think we need to reload tables explicitly. > Only clean necessary cache for CubeMigrationCLI > --- > > Key: KYLIN-2672 > URL: https://issues.apache.org/jira/browse/KYLIN-2672 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Affects Versions: v2.0.0 >Reporter: kangkaisen >Assignee: kangkaisen > Attachments: KYLIN-2672.patch > > > Currently, we simply clear ALL cache in CubeMigrationCLI. which will make a > few of queries slower in prod env when we have many tables, models, cubes and > migrate cube often. > So, we could only clean necessary cache for CubeMigrationCLI. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2672) Only clean necessary cache for CubeMigrationCLI
[ https://issues.apache.org/jira/browse/KYLIN-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084004#comment-16084004 ] liyang commented on KYLIN-2672: --- Would broadcast a Model Update and a Cube Update event be sufficient? They should the model and cube metadata loaded, and purge project level caches as well. > Only clean necessary cache for CubeMigrationCLI > --- > > Key: KYLIN-2672 > URL: https://issues.apache.org/jira/browse/KYLIN-2672 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Affects Versions: v2.0.0 >Reporter: kangkaisen >Assignee: kangkaisen > Attachments: KYLIN-2672.patch > > > Currently, we simply clear ALL cache in CubeMigrationCLI. which will make a > few of queries slower in prod env when we have many tables, models, cubes and > migrate cube often. > So, we could only clean necessary cache for CubeMigrationCLI. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2672) Only clean necessary cache for CubeMigrationCLI
[ https://issues.apache.org/jira/browse/KYLIN-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16072345#comment-16072345 ] kangkaisen commented on KYLIN-2672: --- I agree with you. Do you have better idea or advice? Thanks. > Only clean necessary cache for CubeMigrationCLI > --- > > Key: KYLIN-2672 > URL: https://issues.apache.org/jira/browse/KYLIN-2672 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Affects Versions: v2.0.0 >Reporter: kangkaisen >Assignee: kangkaisen > Attachments: KYLIN-2672.patch > > > Currently, we simply clear ALL cache in CubeMigrationCLI. which will make a > few of queries slower in prod env when we have many tables, models, cubes and > migrate cube often. > So, we could only clean necessary cache for CubeMigrationCLI. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2672) Only clean necessary cache for CubeMigrationCLI
[ https://issues.apache.org/jira/browse/KYLIN-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068016#comment-16068016 ] liyang commented on KYLIN-2672: --- I'm OK with the patch in general. However this complicates the Broadcast system, which is not a happy thing. > Only clean necessary cache for CubeMigrationCLI > --- > > Key: KYLIN-2672 > URL: https://issues.apache.org/jira/browse/KYLIN-2672 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Affects Versions: v2.0.0 >Reporter: kangkaisen >Assignee: kangkaisen > Attachments: KYLIN-2672.patch > > > Currently, we simply clear ALL cache in CubeMigrationCLI. which will make a > few of queries slower in prod env when we have many tables, models, cubes and > migrate cube often. > So, we could only clean necessary cache for CubeMigrationCLI. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2672) Only clean necessary cache for CubeMigrationCLI
[ https://issues.apache.org/jira/browse/KYLIN-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16051687#comment-16051687 ] kangkaisen commented on KYLIN-2672: --- The core idea is adding a the special entity for cube migration, add only clean necessary cache for project , cube and model. > Only clean necessary cache for CubeMigrationCLI > --- > > Key: KYLIN-2672 > URL: https://issues.apache.org/jira/browse/KYLIN-2672 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Affects Versions: v2.0.0 >Reporter: kangkaisen >Assignee: kangkaisen > Attachments: KYLIN-2672.patch > > > Currently, we simply clear ALL cache in CubeMigrationCLI. which will make a > few of queries slower in prod env when we have many tables, models, cubes and > migrate cube often. > So, we could only clean necessary cache for CubeMigrationCLI. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
