Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Pig Wiki" for change 
notification.

The following page has been changed by CorinneC:
http://wiki.apache.org/pig/PigTutorial

------------------------------------------------------------------------------
  The tutorial pig script (tutorial.pig or tutorial-local.pig) does the 
following:
  
   * Registers the tutorial JAR file so that the user-defined functions (UDFs) 
can be called in the script. 
-  * Uses the !PigStorage function to load the excite log file (excite.log or 
excite-small.log) into the “raw” bag as an array of records with the fields 
'''user''', '''time''', and '''query'''. 
+  * Uses the [http://wiki.apache.org/pig/PigBuiltins PigStorage] function to 
load the excite log file (excite.log or excite-small.log) into the “raw” 
bag as an array of records with the fields '''user''', '''time''', and 
'''query'''. 
   * Calls the NonURLDetector UDF to remove records if the query field is empty 
or a URL. 
   * Calls the !ToLower UDF to change the query field to lowercase.
   * Calls the !NonPornDetector UDF to remove records if the query field 
contains porn terms. 
@@ -100, +100 @@

   * Calls the N!GramGenerator UDF to compose the n-grams of the query. 
   * Uses the 
[http://wiki.apache.org/pig/PigLatin#DISTINCT:_Eliminating_duplicates_in_data 
DISTINCT] command to get the unique n-grams for all records. 
   * Uses the 
[http://wiki.apache.org/pig/PigLatin#COGROUP:_Getting_the_relevant_data_together
 GROUP] command to group records by n-gram and hour.
-  * Uses the COUNT function to get the count (occurrences) of each n-gram.
+  * Uses the [http://wiki.apache.org/pig/PigBuiltins COUNT] function to get 
the count (occurrences) of each n-gram.
   * Uses the 
[http://wiki.apache.org/pig/PigLatin#COGROUP:_Getting_the_relevant_data_together
 GROUP] command to group records by n-gram only.
   * Calls the !ScoreGenerator UDF to calculate a "popularity" score for the 
n-gram.
   * Uses the 
[http://wiki.apache.org/pig/PigLatin#FOREACH_..._GENERATE:_Applying_transformations_to_the_data
 FOREACH-GENERATE] command to assign names to the fields.
   * Uses the 
[http://wiki.apache.org/pig/PigLatin#FILTER:_Getting_rid_of_data_you_are_not_interested_in_
 FILTER] command to move all records with a score less than or equal to 2.0.
   * Uses the 
[http://wiki.apache.org/pig/PigLatin#ORDER:_Sorting_data_according_to_some_field
 ORDER] command to sort the remaining records by hour and score.
-  * Uses the !PigStorage function to store the results. The output file 
contains a list of n-grams with the following fields: '''hour''', '''ngram''', 
'''score''', '''count''', '''mean''' 
+  * Uses the [http://wiki.apache.org/pig/PigBuiltins PigStorage] function to 
store the results. The output file contains a list of n-grams with the 
following fields: '''hour''', '''ngram''', '''score''', '''count''', '''mean''' 
  
  
  == Tutorial-Join Pig Script ==
@@ -114, +114 @@

  The tutorial-join pig script (tutorial-join.pig or tutorial-join-local.pig) 
does the following:
  
   * Registers the tutorial JAR file so that the user-defined functions (UDFs) 
can be called in the script. 
-  * Uses the !PigStorage function to load the excite log file (excite.log or 
excite-small.log) into the “raw” bag as an array of records with the fields 
'''user''', '''time''', and '''query'''. 
+  * Uses the [http://wiki.apache.org/pig/PigBuiltins PigStorage] function to 
load the excite log file (excite.log or excite-small.log) into the “raw” 
bag as an array of records with the fields '''user''', '''time''', and 
'''query'''. 
   * Calls the NonURLDetector UDF to remove records if the query field is empty 
or a URL. 
   * Calls the !ToLower UDF to change the query field to lowercase.
   * Calls the Non!PornDetector UDF to remove records if the query field 
contains porn terms. 
@@ -122, +122 @@

   * Calls the N!GramGenerator UDF to compose the n-grams of the query. 
   * Uses the 
[http://wiki.apache.org/pig/PigLatin#DISTINCT:_Eliminating_duplicates_in_data 
DISTINCT] command to get the unique n-grams for all records. 
   * Uses the 
[http://wiki.apache.org/pig/PigLatin#COGROUP:_Getting_the_relevant_data_together
 GROUP] command to group the records by n-gram and hour. 
-  * Uses the COUNT function to get the count (occurrences) of each n-gram. 
+  * Uses the [http://wiki.apache.org/pig/PigBuiltins COUNT] function to get 
the count (occurrences) of each n-gram. 
   * Uses the 
[http://wiki.apache.org/pig/PigLatin#FOREACH_..._GENERATE:_Applying_transformations_to_the_data
 FOREACH-GENERATE] command to assign names to the fields.
   * Uses the 
[http://wiki.apache.org/pig/PigLatin#FILTER:_Getting_rid_of_data_you_are_not_interested_in_
 FILTER] command to get the n-grams for hour ‘00’ 
   * Uses the 
[http://wiki.apache.org/pig/PigLatin#FILTER:_Getting_rid_of_data_you_are_not_interested_in_
 FILTER] command to get the n-grams for hour ‘12’ 
   * Uses the [http://wiki.apache.org/pig/PigLatin#Joining JOIN] command to 
join the n-grams in hour “00” and  hour “12” by field $0
-  * Uses the COUNT function to get the count (occurrences) of the n-grams in 
both “00” and “12” 
+  * Uses the [http://wiki.apache.org/pig/PigBuiltins COUNT] function to get 
the count (occurrences) of the n-grams in both “00” and “12” 
-  * Uses the !PigStorage function to store the results. The output file 
contains a list of n-grams with the following fields: '''hour''', 
'''count00''', '''count12'''
+  * Uses the [http://wiki.apache.org/pig/PigBuiltins PigStorage] function to 
store the results. The output file contains a list of n-grams with the 
following fields: '''hour''', '''count00''', '''count12'''
  

Reply via email to