[
https://issues.apache.org/jira/browse/PIG-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12832072#action_12832072
]
Alan Gates commented on PIG-1178:
---------------------------------
Responses to Ashutosh's questions and comments:
bq. b) At various places Runtime Exceptions are thrown. Do we need a new
Exception class something like ExperimentalOptimizerException so as to easily
spot those exceptions? That will aid in debugging. Also error messages are
pretty terse. More details can be put in messages, again to aid in debugging
later on.
The error messages are sparse. Part of moving this from prototype to
production will be fleshing out those error messages, adding error codes, etc.
As for adding a new exception, I'm not sure I see the value. Hopefully
RuntimeException is only used in appropriate places. If it's being used where
a different exception should be used, then we can change it.
bq. c) At couple of places, changes are made outside of experimental package.
Will be useful to put comments there for why are those needed.
Changes were made to FuncSpec and FileSpec to give them equals functions so
that comparisons can be done between two nodes that both contain a FuncSpec.
For example, we may want to determine if two load operators are the same. Part
of that will be determining if they use the same load function and load the
same file.
bq. Was wondering about different optimizations that we do on a complied MR
plan. ... [E]ssentially those optimizations are also done through visitors and
would benefit greatly if there is a framework for them just as there is one for
front-end. Is there any plan to also subsume those visitors (possibly by
rewriting them as rule-transform pairs) in this new optimizer or will they be
dealt with separately later on?
Plan is too strong a word; there is a hope. I agree that the MR optimizer is a
mess, and it needs a framework. The question is whether the same framework
will suffice for both. I hope that it will. But to avoid taking on too much
we have defined the scope of this current work to just be the logical
optimizer. After we have the logical optimizer in good shape we will need to
address issues in the MR optimizer. Hopefully this framework being developed
will work with minimal refactoring and changes. If not, a different framework
will need to be designed for the MR case.
> LogicalPlan and Optimizer are too complex and hard to work with
> ---------------------------------------------------------------
>
> Key: PIG-1178
> URL: https://issues.apache.org/jira/browse/PIG-1178
> Project: Pig
> Issue Type: Improvement
> Reporter: Alan Gates
> Assignee: Ying He
> Attachments: expressions-2.patch, expressions.patch, lp.patch,
> lp.patch, pig_1178.patch, pig_1178.patch, PIG_1178.patch
>
>
> The current implementation of the logical plan and the logical optimizer in
> Pig has proven to not be easily extensible. Developer feedback has indicated
> that adding new rules to the optimizer is quite burdensome. In addition, the
> logical plan has been an area of numerous bugs, many of which have been
> difficult to fix. Developers also feel that the logical plan is difficult to
> understand and maintain. The root cause for these issues is that a number of
> design decisions that were made as part of the 0.2 rewrite of the front end
> have now proven to be sub-optimal. The heart of this proposal is to revisit a
> number of those proposals and rebuild the logical plan with a simpler design
> that will make it much easier to maintain the logical plan as well as extend
> the logical optimizer.
> See http://wiki.apache.org/pig/PigLogicalPlanOptimizerRewrite for full
> details.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
