Dear Wiki user, You have subscribed to a wiki page or wiki category on "Pig Wiki" for change notification.
The following page has been changed by CorinneC: http://wiki.apache.org/pig/PigTutorial ------------------------------------------------------------------------------ * Calls the !NonPornDetector UDF to remove records if the query field contains porn terms. * Calls the !ExtractHour UDF to extract the hour from the time field. * Calls the N!GramGenerator UDF to compose the n-grams of the query. - * Uses the DISTINCT command to get the unique n-grams for all records. - * Uses the GROUP command to group records by n-gram and hour. + * Uses the [http://wiki.apache.org/pig/PigLatin#DISTINCT:_Eliminating_duplicates_in_data DISTINCT] command to get the unique n-grams for all records. + * Uses the [http://wiki.apache.org/pig/PigLatin#COGROUP:_Getting_the_relevant_data_together GROUP] command to group records by n-gram and hour. * Uses the COUNT function to get the count (occurrences) of each n-gram. - * Uses the GROUP command to group records by n-gram only. + * Uses the [http://wiki.apache.org/pig/PigLatin#COGROUP:_Getting_the_relevant_data_together GROUP] command to group records by n-gram only. * Calls the !ScoreGenerator UDF to calculate a "popularity" score for the n-gram. * Uses the [http://wiki.apache.org/pig/PigLatin#FOREACH_..._GENERATE:_Applying_transformations_to_the_data FOREACH-GENERATE] command to assign names to the fields. * Uses the [http://wiki.apache.org/pig/PigLatin#FILTER:_Getting_rid_of_data_you_are_not_interested_in_ FILTER] command to move all records with a score less than or equal to 2.0. - * Uses the ORDER command to sort the remaining records by hour and score. + * Uses the [http://wiki.apache.org/pig/PigLatin#ORDER:_Sorting_data_according_to_some_field ORDER] command to sort the remaining records by hour and score. * Uses the !PigStorage function to store the results. The output file contains a list of n-grams with the following fields: '''hour''', '''ngram''', '''score''', '''count''', '''mean''' @@ -120, +120 @@ * Calls the Non!PornDetector UDF to remove records if the query field contains porn terms. * Calls the !ExtractHour UDF to extract the hour from the time field. * Calls the N!GramGenerator UDF to compose the n-grams of the query. - * Uses the DISTINCT command to get the unique n-grams for all records. - * Uses the GROUP command to group the records by n-gram and hour. + * Uses the [http://wiki.apache.org/pig/PigLatin#DISTINCT:_Eliminating_duplicates_in_data DISTINCT] command to get the unique n-grams for all records. + * Uses the [http://wiki.apache.org/pig/PigLatin#COGROUP:_Getting_the_relevant_data_together GROUP] command to group the records by n-gram and hour. * Uses the COUNT function to get the count (occurrences) of each n-gram. * Uses the [http://wiki.apache.org/pig/PigLatin#FOREACH_..._GENERATE:_Applying_transformations_to_the_data FOREACH-GENERATE] command to assign names to the fields. * Uses the [http://wiki.apache.org/pig/PigLatin#FILTER:_Getting_rid_of_data_you_are_not_interested_in_ FILTER] command to get the n-grams for hour â00â