[ https://issues.apache.org/jira/browse/PIG-688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12783734#action_12783734 ]
Jeff Zhang commented on PIG-688: -------------------------------- Dmitriy, I think I get your meaning. You mean create a buffer in operator which store the output of this operator, and each time when you call next(), first check this buffer, if buffer is not empty, poll tuple from this buffer, or if buffer is empty, call cascading next() of its parents to fill this buffer. But here's a problem that you can not poll data from ReaderReader, actually hadoop push the record to map class, it is conflicted with operator's poll model. And my idea is to create a tuple buffer in PigMapBase, and store the tuple from RecordReader in this buffer. The buffer is the input of the root of mapper plan. And do not run the runPipeLine until the buffer is full. When the buffer is full, transfer this bag (buffer) through the tree. And in the close() method, I will check the buffer, if it's not empty, run the runPipeLine() to process the remaining tuples in buffer. > PERFORMANCE: Vectorize operators > -------------------------------- > > Key: PIG-688 > URL: https://issues.apache.org/jira/browse/PIG-688 > Project: Pig > Issue Type: Improvement > Reporter: Thejas M Nair > > By Vectorization, I mean passing multiple (/vector of) records at a time > between operators (and potentially other functions like udfs) > Vectorization of pig operators can improve performance by > 1. improving locality and cache utilization > 2. Reducing number of function calls. Many functions calls are likely to be > dynamically resolved. There may be some checks in each function that we might > be able to do once for several recs. > 3. Potentially benefit from cpu pipeline architecture. ( But I don't know how > good java VM is at that ..) > To do vectorization in map stage, we need to use MapRunner - see PIG-687. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.