Dear Wiki user, You have subscribed to a wiki page or wiki category on "Pig Wiki" for change notification.
The following page has been changed by AlanGates: http://wiki.apache.org/pig/PigDeveloperCookbook ------------------------------------------------------------------------------ == Pig and Eclipse == To use Pig with the Eclipse IDE, see ["Eclipse Environment"]. - + == Performance Enhancers == + + The following are a list of tips that people have discovered for making their pig queries run faster. Please feel free to add any tips you have. + + '''Project Early and Often''' + + Pig does not (yet) determine when a field is no longer needed and drop the field from the row. For example, say you have a query like: + + {{{ + A = load 'myfile' as (t, u, v); + B = load 'myotherfile' as (x, y, z); + C = join A by t, B by x; + D = group C by u; + E = foreach D generate group, COUNT($1); + }}} + + There is no need for v, y, or z to participate in this query. And there is no need to carry both t and x past the join, just one will suffice. Changing + the above query to + + {{{ + A = load 'myfile' as (t, u, v); + A1 = foreach A generate t, u; + B = load 'myotherfile' as (x, y, z); + B1 = foreach B generate x; + C = join A1 by t, B1 by x; + C1 = foreach C generate t, u; + D = group C1 by u; + E = foreach D generate group, COUNT($1); + }}} + + will greatly reduce the amount of data being carried through the map and reduce phases by pig. Depending on your data, this can produce significant time savings. In + queries similar to the example given we have seen total time drop by 50%. + + '''Drop Nulls Before a Join''' + + This comment only applies to pig on the types branch, as pig 0.1.0 does not have nulls. + + With the introduction of nulls, join and cogroup semantics were altered to work with nulls. The semantic for cogrouping with nulls is that nulls from a given input are + grouped together, but nulls across inputs are not grouped together. This preserves the semantics of grouping (nulls are collected together from a single input to be + passed to aggregate functions like COUNT) and the semantics of join (nulls are not joined across inputs). Since flattening an empty bag results in an empty row, in a + standard join the rows with a null key will always be dropped. The join: + + {{{ + A = load 'myfile' as (t, u, v); + B = load 'myotherfile' as (x, y, z); + C = join A by t, B by x; + }}} + + is rewritten by pig to + + {{{ + A = load 'myfile' as (t, u, v); + B = load 'myotherfile' as (x, y, z); + C1 = cogroup A by t, B by x; + C = foreach C1 generate flatten(A), flatten(B); + }}} + + Since the nulls from A and B won't be collected together, when the nulls are flattened we're guaranteed to have an empty bag, which will result in no output. So the null + keys will be dropped. But they will not be dropped until the last possible moment. If the query is rewritten to + + {{{ + A = load 'myfile' as (t, u, v); + B = load 'myotherfile' as (x, y, z); + A1 = filter A by t is not null; + B1 = filter B by x is not null; + C = join A1 by t, B1 by x; + }}} + + then the nulls will be dropped before the join. Since all null keys go to a single reducer, if your key is null even a small percentage of the time the gain can be + significant. In one test where the key was null 7% of the time and the data was spread across 200 reducers, we saw a 6x speed up in the query by adding the early + filters. +
