[
https://issues.apache.org/jira/browse/PIG-161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12588644#action_12588644
]
Alan Gates commented on PIG-161:
--------------------------------
Questions on incr4 patch
1) According to the PigExecutionModel document:
The Local Rearrange takes the input tuple and outputs a key, value pair with
the group field as the key and the tuple as the value. For eg., (1,R) will be
converted to {1,(1,R)}. Also the tuple is tagged with the input index it
originated from. In our case, if (1,R) came from A it would be tagged 1 and if
it was from B it would be tagged 2.
Yet in LocalRearrange.constructLROutput() you are creating [key, {[index,
tuple]}] where [] indicates tuple and {} bag. So you are creating a tuple with
the key and a bag, and in that bag you are putting the indexed tuple. Why the
bag? You always have only one element, the bag seems like it's just baggage.
2) In POPackage, we want eventually to be able to push eval funcs into the
getNext loop of this. As currently implemented, all of the bags are
materialized (that is, copied off the disk). This is fine for now. But
eventually we want to be able to push eval functions into that loop so that the
loop
{code}
//For each bag in the inp, read
//the indexexd tuples and sort them
//into their corresponding bags based
//on the index
while (inpBagIter.hasNext()) {
DataBag db = inpBagIter.next();
for (Tuple tuple : db) {
IndexedTuple it = (IndexedTuple)tuple;
dbs[it.index].add(it.toTuple());
}
}
{code}
can become instead:
{code}
//For each bag in the inp, read
//the indexexd tuples and sort them
//into their corresponding bags based
//on the index
while (inpBagIter.hasNext()) {
DataBag db = inpBagIter.next();
for (Tuple tuple : db) {
IndexedTuple it = (IndexedTuple)tuple;
evalfunc[it.index].push(it.toTuple());
}
}
{code}
Note, I'm not saying we want to add this now. I'm just noting this here to get
your feedback. I want to make sure nothing in the current implementation will
prevent this future enhancement.
> Rework physical plan
> --------------------
>
> Key: PIG-161
> URL: https://issues.apache.org/jira/browse/PIG-161
> Project: Pig
> Issue Type: Sub-task
> Reporter: Alan Gates
> Assignee: Alan Gates
> Attachments: arithmeticOperators.patch, incr2.patch, incr3.patch,
> incr4.patch, Phy_AbsClass.patch, pogenerate.patch, pogenerate.patch,
> pogenerate.patch
>
>
> This bug tracks work to rework all of the physical operators as described in
> http://wiki.apache.org/pig/PigTypesFunctionalSpec
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
