[
https://issues.apache.org/jira/browse/PIG-2397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13171991#comment-13171991
]
Dmitriy V. Ryaboy commented on PIG-2397:
----------------------------------------
The variability in my numbers is pretty much completely due to delays in task
scheduling (busy cluster). The amount of time the tasks themselves took stayed
quite consistent. That's part of my point about the fallacy of just measuring
elapsed wall-clock time as a single metric. You have to account for degree of
parallelism and separate out framework overhead from time spent on work.
I find it hard to imagine that changing the split size by 8x didn't affect Hive
performance on an under-provisioned cluster (meaning, in this case, a cluster
that can't run all the tasks in parallel simultaneously). Perhaps Hive is doing
some task aggregation for you under the covers. What was the number of map
tasks it spun up? What was the number of tasks Pig spun up?
> Running TPC-H on Pig
> --------------------
>
> Key: PIG-2397
> URL: https://issues.apache.org/jira/browse/PIG-2397
> Project: Pig
> Issue Type: Task
> Reporter: Jie Li
> Attachments: TPC-H_on_Pig.tgz, pig_tpch.ppt
>
>
> For a class project we developed a whole set of Pig scripts for TPC-H. Our
> goals are:
> 1) identifying the bottlenecks of Pig's performance especially of its
> relational operators,
> 2) studying how to write efficient scripts by making full use of Pig Latin's
> features,
> 3) comparing with Hive's TPC-H results for verifying both 1) and 2).
> We will update the JIRA with our scripts, results and analysis soon.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
