[
https://issues.apache.org/jira/browse/PIG-162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12600225#action_12600225
]
Shravan Matthur Narayanamurthy commented on PIG-162:
----------------------------------------------------
Here are the performance numbers. These are only tests that find the compare
method's performance of WritableComparable basic types against the same
embedded in a Tuple.
To do a million compares for the WritableComparable(WC) types (Basic vs
Embedded in Tuple)
BOOLEAN : Tuple :: 14.8 : 66.26
BYTEARRAY : Tuple :: 54.46 : 109.66
CHARARRAY : Tuple :: 53.06 : 109.72
FLOAT : Tuple :: 14.92 : 79.36
INTEGER : Tuple :: 15.4 : 79.3
LONG : Tuple :: 16.62 : 95.28
These are not too bad considering the fact that on every tuple that is
processed, we need to convert a pig type into WC and back into pig type in the
earlier method. Now we just do a lookup into the tuple and collect it instead.
We should probably check the performance again during the perf tests of the
total thing and come to a conclusion. Another thing is the overheads might not
be applicable if we use hadoop 17 which doesn't mandate us to use WCs. So if we
figure a way out to using pig types without the overhead, maybe we should
reconsider.
Alan, I now leave it to you. We can either leave it as is and take this up
again during perf test or we can apply the patch and think about perf impact
later.
> Rework mapreduce submission and monitoring
> ------------------------------------------
>
> Key: PIG-162
> URL: https://issues.apache.org/jira/browse/PIG-162
> Project: Pig
> Issue Type: Sub-task
> Environment: This bug tracks works to rework the submission and
> monitoring interface to map reduce as described in
> http://wiki.apache.org/pig/PigTypesFunctionalSpec
> Reporter: Alan Gates
> Assignee: Alan Gates
> Attachments: changes1.patch, incr7.patch, incr8.patch,
> mapreduceJumbo.patch, mapreduceJumboWithComInc.patch, modJumbo.patch,
> split.png, TEST-org.apache.pig.test.TestMRCompiler.txt,
> TEST-org.apache.pig.test.TestMRCompiler.txt,
> TEST-org.apache.pig.test.TestUnion.txt
>
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
