[
https://issues.apache.org/jira/browse/HIVE-16483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HIVE-16483:
----------------------------
Description:
There are several split related configurations, such as {{MAPREDMINSPLITSIZE}},
{{MAPREDMINSPLITSIZEPERNODE}}, {{MAPREDMINSPLITSIZEPERRACK}}, etc., that should
be populated to HiveConf. Currently we only do this for {{MAPREDMINSPLITSIZE}}.
All the others, if not set, will be using the default value, which is 1.
Without these, Spark sometimes will not merge small files for file formats such
as text.
was:
There are several split related configurations, such as {{MAPREDMINSPLITSIZE}},
{{MAPREDMINSPLITSIZEPERNODE}}, {{MAPREDMINSPLITSIZEPERRACK}}, etc., that should
be populated to HiveConf. Currently we only do this for {{MAPREDMINSPLITSIZE}}.
Without these, Spark sometimes will not merge small files for file formats such
as text.
> HoS should populate split related configurations to HiveConf
> ------------------------------------------------------------
>
> Key: HIVE-16483
> URL: https://issues.apache.org/jira/browse/HIVE-16483
> Project: Hive
> Issue Type: Bug
> Components: Spark
> Reporter: Chao Sun
> Assignee: Chao Sun
> Attachments: HIVE-16483.1.patch
>
>
> There are several split related configurations, such as
> {{MAPREDMINSPLITSIZE}}, {{MAPREDMINSPLITSIZEPERNODE}},
> {{MAPREDMINSPLITSIZEPERRACK}}, etc., that should be populated to HiveConf.
> Currently we only do this for {{MAPREDMINSPLITSIZE}}.
> All the others, if not set, will be using the default value, which is 1.
> Without these, Spark sometimes will not merge small files for file formats
> such as text.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
