[
https://issues.apache.org/jira/browse/KYLIN-4833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17257891#comment-17257891
]
ASF subversion and git services commented on KYLIN-4833:
--------------------------------------------------------
Commit 804334f79f2eee2351ff060b20e9057a875f1f58 in kylin's branch
refs/heads/master from fengpod
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=804334f ]
KYLIN-4833 create hfile on hadoop hdfs and distcp to hbase hdfs (#1494)
* create hfile on hadoop hdfs and distcp to hbase hdfs
* code style
Co-authored-by: zhangrusong <[email protected]>
> use distcp to control the speed of writting hfile data to hbase cluster
> -----------------------------------------------------------------------
>
> Key: KYLIN-4833
> URL: https://issues.apache.org/jira/browse/KYLIN-4833
> Project: Kylin
> Issue Type: Improvement
> Components: Storage - HBase
> Affects Versions: v3.1.1
> Reporter: fengpod
> Assignee: fengpod
> Priority: Minor
> Fix For: v3.1.2
>
>
> When a large data is written to hbase cluster at the same time，the cluster
> load will become very high，which will affect the query performance. This pr
> allows data to be written data to hadoop hdfs when doing step “Convert Cuboid
> Data to HFile”，and then hfile will be transferred to the hbase cluster by
> DistCp。DistCp controls the speed of write data so as to reduce the pressure
> of cluster。 This pr adds a new step " HFile Distcp To HBase" between “Convert
> Cuboid Data to HFile” and "Load HFile to HBase Table" 。As look like this：
> !https://user-images.githubusercontent.com/4843586/100835711-013fae00-34a9-11eb-8de8-e69228ba0991.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
