[
https://issues.apache.org/jira/browse/KYLIN-1323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15149722#comment-15149722
]
Shaofeng SHI commented on KYLIN-1323:
-------------------------------------
Hi Yerui, just two comments: 1) parameter "kylin.hbase.hfile.per.region" seems
be a little difficult to understand for user, and may have confusion as he
already specified how large a region; 2) this should a dynamic value based on
the calculated region size, instead of a static global value, because sometimes
the region is small which doesn't need multiple reducers, sometimes the region
is very large which need a couple of reducers.
My proposal would be, don't expose this detail or configuration to user;
Leverage the "kylin.job.mapreduce.default.reduce.input.mb" parameter to split
one region into multiple reducers based on the estimated size. Please correct
me if I understand the patch wrongly. Thanks!
> Improve performance of converting data to hfile
> -----------------------------------------------
>
> Key: KYLIN-1323
> URL: https://issues.apache.org/jira/browse/KYLIN-1323
> Project: Kylin
> Issue Type: Improvement
> Components: Job Engine
> Affects Versions: v1.2
> Reporter: Yerui Sun
> Assignee: Yerui Sun
> Fix For: v2.0, v1.3
>
> Attachments: KYLIN-1323-1.x-staging.patch
>
>
> Supposed that we got 100GB data after cuboid building, and with setting that
> 10GB per region. For now, 10 split keys was calculated, and 10 region
> created, 10 reducer used in ‘convert to hfile’ step.
> With optimization, we could calculate 100 (or more) split keys, and use all
> them in ‘covert to file’ step, but sampled 10 keys in them to create regions.
> The result is still 10 region created, but 100 reducer used in ‘convert to
> file’ step. Of course, the hfile created is also 100, and load 10 files per
> region. That’s should be fine, doesn’t affect the query performance
> dramatically.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
