[jira] [Commented] (KYLIN-4833) use distcp to control the speed of writting hfile data to hbase cluster

Tue, 01 Dec 2020 22:39:05 -0800 


    [ 
https://issues.apache.org/jira/browse/KYLIN-4833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17242080#comment-17242080
 ] 

ASF GitHub Bot commented on KYLIN-4833:
---------------------------------------

coveralls commented on pull request #1494:
URL: https://github.com/apache/kylin/pull/1494#issuecomment-737026735


   ## Pull Request Test Coverage Report for [Build 
6598](https://coveralls.io/builds/35381222)
   
   * **0** of **94**   **(0.0%)**  changed or added relevant lines in **9** 
files are covered.
   * **11** unchanged lines in **5** files lost coverage.
   * Overall coverage decreased (**-0.02%**) to **28.015%**
   
   ---
   
   |  Changes Missing Coverage | Covered Lines | Changed/Added Lines | % |
   | :-----|--------------|--------|---: |
   | 
[storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HDFSPathGarbageCollectionStep.java](https://coveralls.io/builds/35381222/source?filename=storage-hbase%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fstorage%2Fhbase%2Fsteps%2FHDFSPathGarbageCollectionStep.java#L94)
 | 0 | 2 | 0.0%
   | 
[core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java](https://coveralls.io/builds/35381222/source?filename=core-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcommon%2FKylinConfigBase.java#L2692)
 | 0 | 3 | 0.0%
   | 
[storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseFlinkOutputTransition.java](https://coveralls.io/builds/35381222/source?filename=storage-hbase%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fstorage%2Fhbase%2Fsteps%2FHBaseFlinkOutputTransition.java#L58)
 | 0 | 4 | 0.0%
   | 
[storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseSparkOutputTransition.java](https://coveralls.io/builds/35381222/source?filename=storage-hbase%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fstorage%2Fhbase%2Fsteps%2FHBaseSparkOutputTransition.java#L58)
 | 0 | 4 | 0.0%
   | 
[storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseMROutput2Transition.java](https://coveralls.io/builds/35381222/source?filename=storage-hbase%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fstorage%2Fhbase%2Fsteps%2FHBaseMROutput2Transition.java#L81)
 | 0 | 6 | 0.0%
   | 
[storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseFlinkSteps.java](https://coveralls.io/builds/35381222/source?filename=storage-hbase%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fstorage%2Fhbase%2Fsteps%2FHBaseFlinkSteps.java#L50)
 | 0 | 8 | 0.0%
   | 
[storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseMRSteps.java](https://coveralls.io/builds/35381222/source?filename=storage-hbase%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fstorage%2Fhbase%2Fsteps%2FHBaseMRSteps.java#L43)
 | 0 | 8 | 0.0%
   | 
[storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseJobSteps.java](https://coveralls.io/builds/35381222/source?filename=storage-hbase%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fstorage%2Fhbase%2Fsteps%2FHBaseJobSteps.java#L67)
 | 0 | 20 | 0.0%
   | 
[storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HFileDistcpJob.java](https://coveralls.io/builds/35381222/source?filename=storage-hbase%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fstorage%2Fhbase%2Fsteps%2FHFileDistcpJob.java#L45)
 | 0 | 39 | 0.0%
   <!-- | **Total:** | **0** | **94** | **0.0%** | -->
   
   |  Files with Coverage Reduction | New Missed Lines | % |
   | :-----|--------------|--: |
   | 
[core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java](https://coveralls.io/builds/35381222/source?filename=core-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcommon%2FKylinConfigBase.java#L2704)
 | 1 | 13.57% |
   | 
[tool/src/main/java/org/apache/kylin/tool/query/ProbabilityGenerator.java](https://coveralls.io/builds/35381222/source?filename=tool%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Ftool%2Fquery%2FProbabilityGenerator.java#L44)
 | 1 | 81.58% |
   | 
[core-cube/src/main/java/org/apache/kylin/cube/cuboid/TreeCuboidScheduler.java](https://coveralls.io/builds/35381222/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Fcuboid%2FTreeCuboidScheduler.java#L124)
 | 2 | 68.46% |
   | 
[stream-core/src/main/java/org/apache/kylin/stream/core/storage/CheckPointStore.java](https://coveralls.io/builds/35381222/source?filename=stream-core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fstream%2Fcore%2Fstorage%2FCheckPointStore.java#L98)
 | 2 | 73.74% |
   | 
[core-cube/src/main/java/org/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://coveralls.io/builds/35381222/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Finmemcubing%2FMemDiskStore.java#L439)
 | 5 | 77.81% |
   <!-- | **Total:** | **11** |  | -->
   
   |  Totals | [![Coverage 
Status](https://coveralls.io/builds/35381222/badge)](https://coveralls.io/builds/35381222)
 |
   | :-- | --: |
   | Change from base [Build 6595](https://coveralls.io/builds/35328877): |  
-0.02% |
   | Covered Lines: | 26253 |
   | Relevant Lines: | 93709 |
   
   ---
   ##### đź’›  - [Coveralls](https://coveralls.io)
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
[email protected]


> use distcp to control the speed of writting hfile data to hbase cluster
> -----------------------------------------------------------------------
>
>                 Key: KYLIN-4833
>                 URL: https://issues.apache.org/jira/browse/KYLIN-4833
>             Project: Kylin
>          Issue Type: Improvement
>          Components: Storage - HBase
>    Affects Versions: v3.1.1
>            Reporter: fengpod
>            Priority: Minor
>
> When a large data is written to hbase cluster at the same time，the cluster 
> load will become very high，which will affect the query performance. This pr 
> allows data to be written data to hadoop hdfs when doing step “Convert Cuboid 
> Data to HFile”，and then hfile will be transferred to the hbase cluster by 
> DistCp。DistCp controls the speed of write data so as to reduce the pressure 
> of cluster。 This pr adds a new step " HFile Distcp To HBase" between “Convert 
> Cuboid Data to HFile” and "Load HFile to HBase Table" 。As look like this：
> !https://user-images.githubusercontent.com/4843586/100835711-013fae00-34a9-11eb-8de8-e69228ba0991.png!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Reply via email to