Dmitriy V. Ryaboy commented on PIG-688:

Jeff --
Iterator means one by one, so there is no conflict between what Olga is saying 
and the code you are reading.

The idea is, instead of getting results by pulling a single tuple through a 
chain of operators one at a time, pull batches through. At the root of the 
operator tree, the batches would have to be built using several successive 
next() calls. The number of next() calls on the reader would stay the same, of 
course, but the number of next() calls on the operators would be reduced by a 
factor of b, where b is the size of a batch (or vector, or bag).

> PERFORMANCE: Vectorize operators
> --------------------------------
>                 Key: PIG-688
>                 URL: https://issues.apache.org/jira/browse/PIG-688
>             Project: Pig
>          Issue Type: Improvement
>            Reporter: Thejas M Nair
> By Vectorization, I mean passing multiple (/vector of) records at a time 
> between operators (and potentially other functions like udfs)
> Vectorization of pig operators can improve performance by 
> 1. improving locality and cache utilization
> 2. Reducing number of function calls. Many functions calls are likely to be 
> dynamically resolved. There may be some checks in each function that we might 
> be able to do once for several recs.
> 3. Potentially benefit from cpu pipeline architecture. ( But I don't know how 
> good java VM is at that ..)
> To do vectorization in map stage, we need to use MapRunner - see PIG-687.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

Reply via email to