[ 
https://issues.apache.org/jira/browse/KYLIN-1079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15050141#comment-15050141
 ] 

hongbin ma commented on KYLIN-1079:
-----------------------------------

I was just afraid this will complex the server code too much. Currently a lot 
of admin functions are provided as command tool(for example cube migration tool 
CubeMigrationCLI, metadata store clean tool MetadataCleanupJob, storage cleanup 
tool StorageCleanupJob), it might not be the optimal solution, but at least 
they're independent and more stable. Is it possible to borrow the design? 
Parameters in command tool is much easier to implement & use.

BTW, here is an example of how to run the tools:"hbase  
org.apache.hadoop.util.RunJar lib/kylin-job-2.0-incubating-SNAPSHOT.jar 
org.apache.kylin.job.tools.CubeMigrationCLI ..."

> Manager large number of entries in metadata store
> -------------------------------------------------
>
>                 Key: KYLIN-1079
>                 URL: https://issues.apache.org/jira/browse/KYLIN-1079
>             Project: Kylin
>          Issue Type: Improvement
>            Reporter: hongbin ma
>            Assignee: hongbin ma
>              Labels: newbie
>
> Kylin saves cube metadata, table metadata as well as job history/output in a 
> metadata store. The HBaseMetadataStore is a fault tolerant implementation 
> which brings no extra dependencies to the system. We use it in real world 
> deployments.
> When cube or hive table is updated, the correspond entries in metadata store 
> simply updated.(so there's no way to trace history cube definitions, anyway 
> this is not very expected function).However Job histories and outputs are a 
> little special, each cubing job's definition and output are saved as new 
> entries in the metadata store. As more and more jobs accumulate, a lot of job 
> histories will reside in the metadata store. This might harm frontend 
> performance when user wants to query job histories.
> We should tackle the problem from two perspectives:
> 1.Backend tool to delete/archive job history based on given conditions,e.g. 
> "all jobs that is older than one month and not referenced by any cube 
> segment(each cube segment keeps track of which job created it)"
> 2.Frontend display enforce timestamp filter to retrieve from metadata store 
> for efficiency. When showing job lists, for example, a "Show last N days" 
> filter is enforced, where N is configurable by the user. For 
> HBaseMetadataStore, we saved timestamp for each entry in a separate column, 
> this is where HBase SingleColumnValueFilter can help.
> We can start working this on 2.x-staging branch(as it is the latest dev 
> branch, and is more friendly to developers), and backport it to 1.x-staging 
> branch if necessary.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to