[
https://issues.apache.org/jira/browse/KYLIN-2328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dayue Gao updated KYLIN-2328:
-----------------------------
Description:
Currently, each MR job uploads all the metadata belonging to a cube to
distributed cache. When the total size of metadata increases, the submission
time ("MapReduce Waiting" at Monitor UI) also increases and could become a
significant problem.
We could actually optimize the amount of metadata uploaded according to the
type of job, for example
* CuboidJob only needs dictionary of the building segment
* CubeHFileJob doesn't need any dictionary
was:
Currently, each MR job uploads all the metadata belonging to a cube to
distributed cache. When the total size of metadata increases, the submission
time ("MapReduce Waiting" at Monitor UI) also increases and could become
notable.
We could actually optimize the amount of metadata uploaded according to the
type of job, for example
* CuboidJob only needs dictionary of the building segment
* CubeHFileJob doesn't need any dictionary
> Reduce the size of metadata uploaded to distributed cache
> ---------------------------------------------------------
>
> Key: KYLIN-2328
> URL: https://issues.apache.org/jira/browse/KYLIN-2328
> Project: Kylin
> Issue Type: Improvement
> Components: Job Engine
> Affects Versions: all
> Reporter: Dayue Gao
> Assignee: Dayue Gao
> Fix For: v2.0.0
>
>
> Currently, each MR job uploads all the metadata belonging to a cube to
> distributed cache. When the total size of metadata increases, the submission
> time ("MapReduce Waiting" at Monitor UI) also increases and could become a
> significant problem.
> We could actually optimize the amount of metadata uploaded according to the
> type of job, for example
> * CuboidJob only needs dictionary of the building segment
> * CubeHFileJob doesn't need any dictionary
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
