Dear Wiki user, You have subscribed to a wiki page or wiki category on "Pig Wiki" for change notification.
The following page has been changed by ChrisOlston: http://wiki.apache.org/pig/PigOptimizationWishList ------------------------------------------------------------------------------ * push cheap filters (move filters known to be cheap, e.g. ones with simple logic predicates, earlier in the plan) * eliminate cartesian products when possible, e.g. convert CROSS followed by FILTER into JOIN - === Medium hanging fruit === + === Fruit that can be reached with a short ladder === * look for ways to do multiple group/cogroup/join operations in a single map-reduce job --- this would occur if the keys share a common prefix. Example: group by userid+hour, then count, then group by userid, then take max --- can be done in one map-reduce job with userid as the reduce key. * choose a join strategy (symmetric hashing, fragment-and-replicate, ...); can probably make a reasonable choice based on file sizes [but first, we have to implement various join strategies in the execution layer -- currently pig only supports symmetric hashing] @@ -25, +25 @@ === Probably won't get there, and may not even want to go there === * query optimization techniques found in any database textbook - * reordering filters (need to estimate selectivity based on histograms, or maybe adaptively reorder) + * reordering filters (need to estimate selectivity based on (a-priori or on-the-fly) histograms, or maybe adaptively reorder) * ...
