Zhong Yanghong created KYLIN-3335:

             Summary: Add project & cube related info to the job id for better 
                 Key: KYLIN-3335
                 URL: https://issues.apache.org/jira/browse/KYLIN-3335
             Project: Kylin
          Issue Type: Improvement
            Reporter: Zhong Yanghong

Currently it's painful to search cube or project related jobs, since those 
infos are hidden in values. Especially, when users want to list job in a period 
under one project, by current design, all of the job output info have to be 
read into memory. If this kind of operation is very often with a short period, 
it's easy to get OOM.

If the job id is prefixed with project and cube names, then we can push down 
prefix filters, which is efficient and safe.

This kind of change will cause backward compatibility issue. There're two ways 
to deal with this:
* Set a milestone with time tag, if a search relates to data earlier than this 
time, then just need to do one scan with prefix filter. Otherwise, two scans 
are needed. One with prefix filter and the other use current strategy. As time 
goes on, old job infos will be deleted. Once there's no data older than the 
time, only one scan is needed.
* Do migration for the old data once.

This message was sent by Atlassian JIRA

Reply via email to