[ 
https://issues.apache.org/jira/browse/KYLIN-1079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15049935#comment-15049935
 ] 

wangxianbin commented on KYLIN-1079:
------------------------------------

about delete/archive job history based on given conditions, I think we can have 
a configurable option in serve-side config files which can trigger a periodic 
job history clean up in hbase, it disable by default, by the way, do you think 
archive is necessary, if you do, any suggestion for that?

> Manager large number of entries in metadata store
> -------------------------------------------------
>
>                 Key: KYLIN-1079
>                 URL: https://issues.apache.org/jira/browse/KYLIN-1079
>             Project: Kylin
>          Issue Type: Improvement
>            Reporter: hongbin ma
>            Assignee: hongbin ma
>              Labels: newbie
>
> Kylin saves cube metadata, table metadata as well as job history/output in a 
> metadata store. The HBaseMetadataStore is a fault tolerant implementation 
> which brings no extra dependencies to the system. We use it in real world 
> deployments.
> When cube or hive table is updated, the correspond entries in metadata store 
> simply updated.(so there's no way to trace history cube definitions, anyway 
> this is not very expected function).However Job histories and outputs are a 
> little special, each cubing job's definition and output are saved as new 
> entries in the metadata store. As more and more jobs accumulate, a lot of job 
> histories will reside in the metadata store. This might harm frontend 
> performance when user wants to query job histories.
> We should tackle the problem from two perspectives:
> 1.Backend tool to delete/archive job history based on given conditions,e.g. 
> "all jobs that is older than one month and not referenced by any cube 
> segment(each cube segment keeps track of which job created it)"
> 2.Frontend display enforce timestamp filter to retrieve from metadata store 
> for efficiency. When showing job lists, for example, a "Show last N days" 
> filter is enforced, where N is configurable by the user. For 
> HBaseMetadataStore, we saved timestamp for each entry in a separate column, 
> this is where HBase SingleColumnValueFilter can help.
> We can start working this on 2.x-staging branch(as it is the latest dev 
> branch, and is more friendly to developers), and backport it to 1.x-staging 
> branch if necessary.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to