[
https://issues.apache.org/jira/browse/KYLIN-702?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14502255#comment-14502255
]
Shaofeng SHI commented on KYLIN-702:
------------------------------------
Hive can enable file merge by setting "hive.merge.mapfiles" and
"hive.merge.mapredfiles" configurations to true; By defaul the merged size is
256M per file, which might be too large but you can customize by setting
"hive.merge.size.per.task", such like 16M;
<property>
<name>hive.merge.mapfiles</name>
<value>true</value>
<description>Enable hive file merge on mapper only job</description>
</property>
<property>
<name>hive.merge.mapredfiles</name>
<value>true</value>
<description>Enable hive file merge on map-reduce job</description>
</property>
<property>
<name>hive.merge.size.per.task</name>
<value>16000000</value>
<description>Size for the merged file: 16M</description>
</property>
> When Kylin create the flat hive table, it generates large number of small
> files in HDFS
> ----------------------------------------------------------------------------------------
>
> Key: KYLIN-702
> URL: https://issues.apache.org/jira/browse/KYLIN-702
> Project: Kylin
> Issue Type: Improvement
> Components: General
> Affects Versions: v0.7.1
> Reporter: Shaofeng SHI
>
> When I build a cube, I noticed that when build the dictionary and calculate
> the cube, there are a large number of mappers be started (more than 10,000);
> With the log I noticed many mappers has 0 or much less records to process,
> this confused me;
> Then I checked the storage location of the flat table, found there are many
> files; I did a count and found it is the same number as the mappers;
> Too many mappers will cause much overhead, and downgrade the cluster's
> performance; Kylin should ask Hive to merge those small files during creating
> the flat table step.
> In my hadoop cluster, the hive.merge.mapredfiles was set to false (default
> value); After changing it to true for Kylin's job, the intermediate table's
> file number was reduced to 4, each be up to 256M, looks good; Check hive
> configuration at:
> https://cwiki.apache.org/confluence/display/Hive/AdminManual+Configuration
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
