Yes, physical properties is important for an optimizer. To optimize Pig
well, we need to know the underlying hadoop execution environment, such
as # of map-reduce jobs, how many maps/reducers, how the job is
configured, etc. This is true even for a rule based optimizer.
Unfortunately, physical layer does not provide much physical information
as the name suggests. Basically physical layer is a rephrase of logical
layer using physical operators. Compare to logical operators, physical
operators include implementation of pipeline processing but strip away
many logical details such as "schema". Also, in logical layer, we have
infrastructure to restructure logical operator such as move nodes
around, swap nodes, etc, which does not exist in physical layer. From
optimizer's point of view, physical layer does not give necessary
information but more harder to deal with. If you would like to work with
physical details, I think map-reduce layer is the right place to look
at. However, restructure map-reduce layer is hard cuz we do not have all
the infrastructure to move things around. Another approach is to use a
combined logical layer and map-reduce layer for the optimization. In
this, you restructure the logical layer by observing the physical
details from map-reduce layer. The down side is that we have to tightly
couple Pig to hadoop. But now Pig is a subproject of hadoop and almost
all Pig users are using hadoop, I think it is fine to optimize thing
towards hadoop.
Dmitriy Ryaboy wrote:
Our initial survey of related literature showed that the usual place
for a CBO tends to be between the physical and logical layer (in fact,
the famous Cascades paper advocates removing the distinction between
physical and logical operators altogether, and using an "is_logical"
and "is_physical" flag instead -- meaning an operator can be one,
both, or neither).
The reasoning is that you cannot properly determine a cost of a plan
if you don't know the physical "properties" of the operators that
implement it. An optimizer that works at a logical layer would by
definition create the same plan whether in local or mapreduce mode
(since such differences are abstracted from it). This is clearly
incorrect, as the properties of the environment in which these plans
are executed are drastically different. Working at the physical layer
lets us stay close to the iron and adjust based on the specifics of
the execution environment.
Certainly one can posit a framework for a CBO that would set up the
necessary interfaces and plumbing for optimizing in any execution
mode, and invoke the proper implementations at run time; we are not
discounting that possibility (haven't gotten quite that far in the
design, to be honest). But we feel that the implementations have to
be execution mode specific.
-Dmitriy
On Tue, Sep 1, 2009 at 6:26 PM, Jianyong Dai<[email protected]> wrote:
I am still reading but one interesting question is why you decide to put CBO
in physical layer?
Dmitriy Ryaboy wrote:
Whoops :-)
Here's the Google doc:
http://docs.google.com/Doc?docid=0Adqb7pZsloe6ZGM4Z3o1OG1fMjFrZjViZ21jdA&hl=en
-Dmitriy
On Tue, Sep 1, 2009 at 12:51 PM, Santhosh Srinivasan<[email protected]>
wrote:
Dmitriy and Gang,
The mailing list does not allow attachments. Can you post it on a
website and just send the URL ?
Thanks,
Santhosh
-----Original Message-----
From: Dmitriy Ryaboy [mailto:[email protected]]
Sent: Tuesday, September 01, 2009 9:48 AM
To: [email protected]
Subject: Request for feedback: cost-based optimizer
Hi everyone,
Attached is a (very) preliminary document outlining a rough design we
are proposing for a cost-based optimizer for Pig.
This is being done as a capstone project by three CMU Master's students
(myself, Ashutosh Chauhan, and Tejal Desai). As such, it is not
necessarily meant for immediate incorporation into the Pig codebase,
although it would be nice if it, or parts of it, are found to be useful
in the mainline.
We would love to get some feedback from the developer community
regarding the ideas expressed in the document, any concerns about the
design, suggestions for improvement, etc.
Thanks,
Dmitriy, Ashutosh, Tejal
