[
https://issues.apache.org/jira/browse/HIVE-9153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14255650#comment-14255650
]
Rui Li commented on HIVE-9153:
------------------------------
[~xuefuz] - I was wrong about turning off delay schedule. Actually you can set
{{spark.locality.wait}} to 0 to turn it off. I tried doing that and parallelism
won't drop during execution now.
Besides, I find tez uses its own property to control the size of combined
spits: {{tez.grouping.max-size}} which defaults to 1G, while
{{mapreduce.input.fileinputformat.split.maxsize}} defaults to less than 256M
(these two properties are a little different in that
{{mapreduce.input.fileinputformat.split.maxsize}} is more like a target size
and {{tez.grouping.max-size}} is an upper bound, but they have similar effect
when data size is big). So I changed
{{mapreduce.input.fileinputformat.split.maxsize}} to 1G as well and spark now
spawns 317 mappers for the previous test (332 for tez). Spark finishes the
query in 155s with these new settings.
> Evaluate CombineHiveInputFormat versus HiveInputFormat [Spark Branch]
> ---------------------------------------------------------------------
>
> Key: HIVE-9153
> URL: https://issues.apache.org/jira/browse/HIVE-9153
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
> Affects Versions: spark-branch
> Reporter: Brock Noland
> Assignee: Rui Li
> Attachments: screenshot.PNG
>
>
> The default InputFormat is {{CombineHiveInputFormat}} and thus HOS uses this.
> However, Tez uses {{HiveInputFormat}}. Since tasks are relatively cheap in
> Spark, it might make sense for us to use {{HiveInputFormat}} as well. We
> should evaluate this on a query which has many input splits such as {{select
> count(\*) from store_sales where something is not null}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
