[
https://issues.apache.org/jira/browse/HIVE-7567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chengxiang Li updated HIVE-7567:
--------------------------------
Attachment: HIVE-7567.1-spark.patch
support automatic adjusting reducer number same as MR, configure through 3
following parameters:
# In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
# In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
# In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Currently Tez specified optimization(committed with
[HIVE-7158|https://issues.apache.org/jira/browse/HIVE-7158]) is not supported
as it's based on Tez feature, refer to [dynamic graph
reconfiguration|http://hortonworks.com/blog/apache-tez-dynamic-graph-reconfiguration/]
for details.
> support automatic calculating reduce task number
> ------------------------------------------------
>
> Key: HIVE-7567
> URL: https://issues.apache.org/jira/browse/HIVE-7567
> Project: Hive
> Issue Type: Task
> Components: Spark
> Reporter: Chengxiang Li
> Assignee: Chengxiang Li
> Attachments: HIVE-7567.1-spark.patch
>
>
> Hive have its own machenism to calculate reduce task number, we need to
> implement it on spark job.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
