[
https://issues.apache.org/jira/browse/PIG-2397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171464#comment-13171464
]
Jie Li commented on PIG-2397:
-----------------------------
Thanks Dmitriy for your analysis.
bq. Hive ranged between 160 and 240 seconds, while Pig ranged between 290 and
350 (ish) on several runs of Q1.
Seems your results were not stable. Most of our benchmark results have a
variance within 5%.
bq. 1) The hive TPC-H scripts set mapred.min.split.size=536870912 while Pig
ones do not.
Yeah we should have mentioned this. Actually we removed this parameter in
Hive's query file, and we found it didn't make much difference to Hive. I agree
that on this case it might be better to have less number of map tasks, as we
don't need to overlap them with the reduce tasks. For our project we didn't
optimize the configuration.
bq. 2) We generate a sampling job for an ORDER-BY even when the parallelism of
that operator is set to 1
Yeah it would be nice to dynamically decide the sampling job.
> Running TPC-H on Pig
> --------------------
>
> Key: PIG-2397
> URL: https://issues.apache.org/jira/browse/PIG-2397
> Project: Pig
> Issue Type: Task
> Reporter: Jie Li
> Attachments: TPC-H_on_Pig.tgz, pig_tpch.ppt
>
>
> For a class project we developed a whole set of Pig scripts for TPC-H. Our
> goals are:
> 1) identifying the bottlenecks of Pig's performance especially of its
> relational operators,
> 2) studying how to write efficient scripts by making full use of Pig Latin's
> features,
> 3) comparing with Hive's TPC-H results for verifying both 1) and 2).
> We will update the JIRA with our scripts, results and analysis soon.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
