[
https://issues.apache.org/jira/browse/PIG-688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12783720#action_12783720
]
Dmitriy V. Ryaboy commented on PIG-688:
---------------------------------------
I am not sure what you mean by processing remaining tuples in the close()
method.
My understanding is more like this: in the first next() call on the leaf
operator, a next() call cascades down the tree, and a bag of tuples gets
processed as it comes back up the tree. The next b calls get a pre-computed
result from the buffer. Once the buffer is empty, the operation is repeated,
until all tuples at the bottom of the tree have been fetched. The close method
is unaffected.
> PERFORMANCE: Vectorize operators
> --------------------------------
>
> Key: PIG-688
> URL: https://issues.apache.org/jira/browse/PIG-688
> Project: Pig
> Issue Type: Improvement
> Reporter: Thejas M Nair
>
> By Vectorization, I mean passing multiple (/vector of) records at a time
> between operators (and potentially other functions like udfs)
> Vectorization of pig operators can improve performance by
> 1. improving locality and cache utilization
> 2. Reducing number of function calls. Many functions calls are likely to be
> dynamically resolved. There may be some checks in each function that we might
> be able to do once for several recs.
> 3. Potentially benefit from cpu pipeline architecture. ( But I don't know how
> good java VM is at that ..)
> To do vectorization in map stage, we need to use MapRunner - see PIG-687.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
