Dear Wiki user,
You have subscribed to a wiki page or wiki category on "Pig Wiki" for change
notification.
The following page has been changed by PiSong:
http://wiki.apache.org/pig/PlanTestingHelper
New page:
= Plan Testing Helper =
This is a small utility that I developed for testing my type checking logic. I
think it might be useful for other people as well so I have refactored a bit to
make it more generic.
== Use cases ==
Here are steps that I do for type checking:-
* Construct a plan
* Run type-checking logic against the plan
* Construct the expected plan
* Compare structures of the actual plan and the expected plan.
Here are steps that one might do for query parser:-
* Given a query string, construct the plan.
* Construct the expected plan
* Compare two plans
Here for testing plan optimizer:-
* Construct a plan
* Run optimizer
* Construct the expected plan
* Compare structures of the actual plan and the expected plan.
== What can be facilitated? ==
So there are two common bits from above use cases:-
1. Construct the expected plan
1. Compare two plans
== Construct a plan ==
==== What is Dot Language? ====
Dot language is a text graph description language. There are three main object
types: node, edge, and graph. All of them can have custom attributes.
==== Sample Dot graph ====
{{{
digraph plan1 {
load [color="black"]
load -> distinct -> split -> splitOut1 [style=dotted] ;
split -> splitOut2 ;
splitOut1 -> cross ;
splitOut2 -> cross ;
}
}}}
'''Note''': "digraph" dictates that this is a description of directed graph
which is the domain we're interested in.
'''Note''': "load [color="black"]" is attaching an attribute to the node. This
is optional.
By extending Dot a bit, we can encode our logical plan in the following format:-
{{{
digraph graph1 {
load [key="114", type="LOLoad", schema="field1: int, field2: float"]
;
distinct [key="115", type="LODistinct", schema="field1: int, field2:
float"] ;
split [key="116", type="LOSplit", schema="field1: int, field2:
float"] ;
splitout1 [key="117", type="LOForEach", schema="field1: int, field2:
float"] ;
splitout2 [key="117", type="LOForEach", schema="field1: int, field2: float"]
cross [key="119", type="LOCross", schema="field1: int, field2: float, field3: chararray"] ;
load -> distinct -> split -> splitOut1 ;
split -> splitOut2 ;
splitOut1 -> cross ;
splitOut2 -> cross ;
}
}}}
And this can be translated to a plan using a loader class (API will be provided)
== Compare two plans ==
I will provide API like this:-
{{{
/***
* This abstract class is a base for plan comparer
*/
public abstract class PlanStructuralComparer<E extends Operator,
P extends OperatorPlan<E>> {
/***
* This method does structural comparison of two plans based on:-
* - Graph connectivity
*
* The current implementation is based on simple key-based
* vertex matching.
*
* @param plan1 the first plan
* @param plan2 the second plan
* @param messages where the error messages go
* @return
*/
public boolean structurallyEquals(P plan1, P plan2, StringBuilder messages)
;
/***
* Same as above in case just want to compare but
* don't want to know the error messages
* @param plan1
* @param plan2
* @return
*/
public boolean structurallyEquals(P plan1, P plan2) ;
}
}}}
A subtype which is interested in type information would look like this:-
{{{
/***
* This class is used for LogicalPlan comparison
*/
public class LogicalPlanComparer
extends PlanStructuralComparer<LogicalOperator, LogicalPlan> {
/***
* This method does naive structural comparison of two plans.
*
* Things we compare :-
* - Things compared in the super class
* - Types of matching nodes
* - Schema associated with each operator
*
* @param plan1
* @param plan2
* @param messages
* @return
*/
@Override
public boolean structurallyEquals(LogicalPlan plan1,
LogicalPlan plan2,
StringBuilder messages) {
// Stage 1: Compare connectivity
if (!super.structurallyEquals(plan1, plan2, messages)) return false ;
// Stage 2: Compare node types
if (isMismatchNodeType(plan1, plan2, messages)) return false ;
// Stage 3: Compare schemas
if (isMismatchSchemas(plan1, plan2, messages)) return false ;
// else
return true ;
}
}}}
== Dot Trick ==
One can plot a graph written in Dot language by just doing like:-
{{{
dot -Tpng dot1.dot > dot1.png
}}}
Or alternatively,
{{{
dotty dot1.dot
}}}
NOTE: You need graphviz installed on your machine to do these things.
Here is a sample graph generated from the given sample.
http://people.apache.org/~pisong/dot1.png
= Current Status & Issues =
* Working code will be available in 1-2 days (Today = 26th May)
* Doesn't work with inner plans yet. Inner plans may have to be constructed
and compare separately.
== Appendix ==
The API:-
OperatorPlanLoader - This class is an abstract base class for loading a plan from Dot
{{{
public abstract class OperatorPlanLoader<E extends Operator,
P extends OperatorPlan<E>> {
/***
* This method is used for loading an operator plan encoded in Dot format
* @param dotContent the dot content
* @param clazz the plan type to be created
* @return
*/
public P load(String dotContent) {
/***
* This method has be overridden to instantiate the correct node type
*
* @param node
* @param plan
* @return
*/
protected abstract E createOperator(Node node, P plan) ;
}
}}}
Structures captured from Dot (Before being converted to plan):-
{{{
/***
* This represents graph structure in DOT format
*/
public class DotGraph {
public String name;
public List<Edge> edges = new ArrayList<Edge>() ;
public List<Node> nodes = new ArrayList<Node>() ;
public Map<String, String> attributes = new HashMap<String,String>() ;
public DotGraph(String name) {
this.name = name ;
}
}
}}}
{{{
/***
* This represents a node in DOT format
*/
public class Node {
public String name ;
public Map<String, String> attributes = new HashMap<String,String>() ;
}
}}}
{{{
/**
* This represents an edge in DOT format.
* An edge in DOT can have attributes but we're not interested
*/
public class Edge {
public String fromNode ;
public String toNode ;
}
}}}
