pig-commits  

[Pig Wiki] Update of "ExampleGenerator" by ShubhamChopra

Apache Wiki
Mon, 14 Apr 2008 00:02:44 -0700

Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Pig Wiki" for change 
notification.

The following page has been changed by ShubhamChopra:
http://wiki.apache.org/pig/ExampleGenerator

------------------------------------------------------------------------------
+ == Illustrate ==
- ILLUSTRATE Command : 
- 
  Illustrate is a new addition to pig that helps users debug their pig scripts. 
  
  The idea is to select a few example data items, and illustrate how they are 
transformed by the sequence of Pig commands in the user's program. The 
ExampleGenerator algorithm can select an appropriate and concise set of example 
data items automatically. It does a better job than random sampling would do; 
for example, random sampling suffers from the drawback that selective 
operations such as filters or joins can eliminate all the sampled data items, 
giving you empty results which is of no help in debugging.
  
  This "ILLUSTRATE" functionality will avoid people having to test their Pig 
programs on large data sets, which has a long turnaround time and wastes system 
resources. The algorithm uses the "Local" execution operators (it does not run 
on hadoop), so as to generate illustrative example data in near-real-time for 
the user.
  
- Usage :
+ === Usage ===
  Illustrate command can be used in the following way:
  
  Say the input file is 'visits.txt' containing the following data :