[Pig Wiki] Update of PigErrorHandling by SanthoshSrinivasan
Dear Wiki user, You have subscribed to a wiki page or wiki category on Pig Wiki for change notification. The following page has been changed by SanthoshSrinivasan: http://wiki.apache.org/pig/PigErrorHandling -- Errors that occur in each system should be caught as early as possible. Pig relies on Hadoop for run time execution. Detection and reporting errors early will improve turnaround time by avoiding invoking Hadoop till most errors are fixed. A few examples that demonstrate this behavior are: 1. Syntax errors. E.g.: Missing ';' - 2. Semantic errors. E.g: Mismatch in cogroup arity - 3. Validation errors. E.g: Type mismatch when trying to add a string to an integer + 2. Semantic errors. E.g: Mismatch in cogroup arity, Type mismatch when trying to add a string to an integer === Error reporting === - Provide users with readable error messages. Stack traces provide a good mechanism to help debugging but do not mean much to the user. Readable and simple error messages will be presented on STDERR. Error codes will be devised for all error messages. Detailed information like stack trace will be logged into client side logs. The logs will be stored in the current working directory or the user's home directory. Users can send logs that contain the details of the error like stack trace to assist developers in resolving issues. + Provide users with readable error messages and not just for developers. Stack traces provide a good mechanism to help debugging but do not mean much to the user. Readable and simple error messages will be presented on STDERR. Error codes will be devised for all error messages. The error codes will be classified into categories to enable further action based on the type of error. Exceptions will not be swallowed enabling detection of the root cause of the issue. Detailed information like stack trace will be logged into client side logs. The logs will be stored in the current working directory or the user's home directory. The location of the log file will be configurable. Users can send logs that contain the details of the error like stack trace to assist developers in resolving issues. E.g.: For a semantic error like mismatch in cogroup arity, the STDERR messages and the log messages will look like. Note that this example is used to illustrate a possible use case. @@ -86, +85 @@ With the introduction of types and NULLs into Pig, there are several scenarios where Pig warns the user about introduction of casts, divide by zero uses, etc. The warning messages are issued on each occurrence of the warning. While this message is useful, the increased frequency of the messages is annoying and distracts the user from possible error messages. In order to alleviate this problem, warning message aggregation will be supported to report the warning message and the number of occurrences of the warning message. The warning message and the frequency of each warning message will be presented on STDERR. The logs will also contain the same information. The location of the log file will be the same as that of the error log file. - E.g.: + According to the [#ref3 Pig types functional specification], divide by zero will result in NULL. Pig warns the user about these conversions. - Warning: Divide by zero: 30 times + E.g.: Warning: Divide by zero: 30 times - Implicit cast to integer: 12 times + Similarly, Pig emits warning messages when implicit casts are introduced for type promotion (adding an integer and a double) + + E.g.: Warning: Implicit cast to double: 12 times '''Note:''' @@ -101, +102 @@ 1. [[Anchor(ref1)]] Mika Raento, What should Exceptions look like? July 30, 2006, http://www.errorhandling.org/wordpress/ 2. [[Anchor(ref2)]] Bruce Eckel, Thinking in Java, 3rd Edition Revision 4.0, November 20, 2002, http://www.faqs.org/docs/think_java/TIJ3_c.htm + 3. [[Anchor(ref3)]] Alan Gates, Pig Types Functional Specification, May 19, 2008, http://wiki.apache.org/pig/PigTypesFunctionalSpec + == User feedback == + + The following comments have not been incorporated in the requirements but are listed for reference. + + 1. Pig should indicate the expression that caused the warning and/or error message. + + E.g.: B = FILTER A BY (f1 10) and (f2 != 'asdf') and (f3 2L). If f1 requires a cast for the comparison, Pig should indicate that (f1 10) resulted in the warning. + + 2. Pig should indicate the line number that resulted in the error ala general purpose programming languages like C++, Java + + 3. Pig should be more transparent. Retries in Hadoop should be communicated to the user in a programmatic fashion, to allow layers above Pig to take appropriate action. Pig should also log/share information about the processes, machines and logs that are spawned by Hadoop to enable users to take quick action. +
[Pig Wiki] Update of PigErrorHandling by SanthoshSrinivasan
Dear Wiki user, You have subscribed to a wiki page or wiki category on Pig Wiki for change notification. The following page has been changed by SanthoshSrinivasan: http://wiki.apache.org/pig/PigErrorHandling -- The following comments have not been incorporated in the requirements but are listed for reference. - 1. Pig should indicate the expression that caused the warning and/or error message. - - E.g.: B = FILTER A BY (f1 10) and (f2 != 'asdf') and (f3 2L). If f1 requires a cast for the comparison, Pig should indicate that (f1 10) resulted in the warning. + 1. Pig should indicate the expression that caused the warning and/or error message. E.g.: B = FILTER A BY (f1 10) and (f2 != 'asdf') and (f3 2L). If f1 requires a cast for the comparison, Pig should indicate that (f1 10) resulted in the warning. 2. Pig should indicate the line number that resulted in the error ala general purpose programming languages like C++, Java
[Pig Wiki] Update of PigErrorHandling by SanthoshSrinivasan
Dear Wiki user, You have subscribed to a wiki page or wiki category on Pig Wiki for change notification. The following page has been changed by SanthoshSrinivasan: http://wiki.apache.org/pig/PigErrorHandling -- === Error reporting === - Provide users with readable error messages. Stack traces provide a good mechanism to help debugging but do not mean much to the user. Readable and simple error messages will be presented on STDERR. Error codes will be devised for common error messages. Detailed information like stack trace will be logged into client side logs. Users can send logs that contain the details of the error like stack trace to assist developers in resolving issues. + Provide users with readable error messages. Stack traces provide a good mechanism to help debugging but do not mean much to the user. Readable and simple error messages will be presented on STDERR. Error codes will be devised for all error messages. Detailed information like stack trace will be logged into client side logs. The logs will be stored in the current working directory or the user's home directory. Users can send logs that contain the details of the error like stack trace to assist developers in resolving issues. '''Note:''' 1. Users are responsible for purging error logs 2. Users will be able switch on/off the detailed error messages on STDERR. - 3. Since Pig depends on Hadoop for execution, Hadoop error messages will be reported by Pig. An error during execution due to a bug in Pig will be shown differently from that of an error in Hadoop itself. + 3. Since Pig depends on Hadoop for execution, Hadoop error messages will be reported by Pig. An error during execution due to a bug in Pig will be shown differently from that of an error in Hadoop itself. The error code and the error message will indicate that the error was due to Hadoop. === Warning message aggregation === - With the introduction of types and NULLs into Pig, there are several scenarios where Pig warns the user about introduction of casts, divide by zero uses, etc. The warning messages are issued on each occurrence of the warning. While this message is useful, the increased frequency of the messages is annoying and distracts the user from possible error messages. In order to alleviate this problem, warning message aggregation will be supported to report the warning message and the number of occurrences of the warning message. + With the introduction of types and NULLs into Pig, there are several scenarios where Pig warns the user about introduction of casts, divide by zero uses, etc. The warning messages are issued on each occurrence of the warning. While this message is useful, the increased frequency of the messages is annoying and distracts the user from possible error messages. In order to alleviate this problem, warning message aggregation will be supported to report the warning message and the number of occurrences of the warning message. The warning message and the frequency of each warning message will be presented on STDERR. The logs will also contain the same information. The location of the log file will be the same as that of the error log file. + + E.g.: + + Warning: Divide by zero: 30 times + + Implicit cast to integer: 12 times '''Note:''' - 1. Users will be able switch to turn on/off warning message aggregation + 1. Users will be able switch to turn on/off warning message aggregation. Turning warning message aggregation will result in one warning message per warning. == References ==
[Pig Wiki] Update of PigErrorHandling by SanthoshSrinivasan
Dear Wiki user,
You have subscribed to a wiki page or wiki category on Pig Wiki for change
notification.
The following page has been changed by SanthoshSrinivasan:
http://wiki.apache.org/pig/PigErrorHandling
--
Provide users with readable error messages. Stack traces provide a good
mechanism to help debugging but do not mean much to the user. Readable and
simple error messages will be presented on STDERR. Error codes will be devised
for all error messages. Detailed information like stack trace will be logged
into client side logs. The logs will be stored in the current working directory
or the user's home directory. Users can send logs that contain the details of
the error like stack trace to assist developers in resolving issues.
+ E.g.: For a semantic error like mismatch in cogroup arity, the STDERR
messages and the log messages will look like. Note that this example is used to
illustrate a possible use case.
+
+ {{{#!java
+ //STDERR
+ grunt a = load 'input1' as (x, y, z);
+ grunt b = load 'input2' as (m, n, o);
+ grunt c = cogroup a by (x, y), b by m;
+ 2008-10-22 13:54:28,821 [main] ERROR org.apache.pig.tools.grunt.GruntParser -
The arity of the group by columns do not match.
+
+ //LOG
+ 2008-10-22 13:54:28,821 [main] ERROR org.apache.pig.tools.grunt.GruntParser -
java.io.IOException: The arity of the group by columns do not match.
+ at org.apache.pig.PigServer.parseQuery(PigServer.java:298)
+ at org.apache.pig.PigServer.registerQuery(PigServer.java:263)
+ at
org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:439)
+ at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:249)
+ at
org.apache.pig.tools.grunt.GruntParser.parseContOnError(GruntParser.java:94)
+ at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:58)
+ at org.apache.pig.Main.main(Main.java:282)
+ Caused by: org.apache.pig.impl.logicalLayer.parser.ParseException: The arity
of the group by columns do not match.
+ at
org.apache.pig.impl.logicalLayer.parser.QueryParser.parseCogroup(QueryParser.java:169)
+ at
org.apache.pig.impl.logicalLayer.parser.QueryParser.CogroupClause(QueryParser.java:1739)
+ at
org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:941)
+ at
org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:755)
+ at
org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:550)
+ at
org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:60)
+ at org.apache.pig.PigServer.parseQuery(PigServer.java:295)
+ ... 6 more
+
+ 2008-10-22 13:54:28,821 [main] ERROR org.apache.pig.tools.grunt.GruntParser -
The arity of the group by columns do not match.
+ 2008-10-22 13:54:28,821 [main] ERROR org.apache.pig.tools.grunt.GruntParser -
java.io.IOException: The arity of the group by columns do not match.
+ }}}
+
'''Note:'''
1. Users are responsible for purging error logs
[Pig Wiki] Update of PigErrorHandling by SanthoshSrinivasan
Dear Wiki user, You have subscribed to a wiki page or wiki category on Pig Wiki for change notification. The following page has been changed by SanthoshSrinivasan: http://wiki.apache.org/pig/PigErrorHandling -- grunt a = load 'input1' as (x, y, z); grunt b = load 'input2' as (m, n, o); grunt c = cogroup a by (x, y), b by m; - 2008-10-22 13:54:28,821 [main] ERROR org.apache.pig.tools.grunt.GruntParser - The arity of the group by columns do not match. + ERROR org.apache.pig.tools.grunt.GruntParser - The arity of the group by columns do not match. //LOG - 2008-10-22 13:54:28,821 [main] ERROR org.apache.pig.tools.grunt.GruntParser - java.io.IOException: The arity of the group by columns do not match. + ERROR org.apache.pig.tools.grunt.GruntParser - java.io.IOException: The arity of the group by columns do not match. + Stack Trace: at org.apache.pig.PigServer.parseQuery(PigServer.java:298) at org.apache.pig.PigServer.registerQuery(PigServer.java:263) at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:439) @@ -72, +73 @@ at org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:60) at org.apache.pig.PigServer.parseQuery(PigServer.java:295) ... 6 more - - 2008-10-22 13:54:28,821 [main] ERROR org.apache.pig.tools.grunt.GruntParser - The arity of the group by columns do not match. - 2008-10-22 13:54:28,821 [main] ERROR org.apache.pig.tools.grunt.GruntParser - java.io.IOException: The arity of the group by columns do not match. }}} '''Note:'''
[Pig Wiki] Update of PigErrorHandling by SanthoshSrinivasan
Dear Wiki user, You have subscribed to a wiki page or wiki category on Pig Wiki for change notification. The following page has been changed by SanthoshSrinivasan: http://wiki.apache.org/pig/PigErrorHandling -- grunt a = load 'input1' as (x, y, z); grunt b = load 'input2' as (m, n, o); grunt c = cogroup a by (x, y), b by m; - ERROR org.apache.pig.tools.grunt.GruntParser - The arity of the group by columns do not match. + ERROR org.apache.pig.tools.grunt.GruntParser - [Error 42] The arity of the group by columns do not match. //LOG - ERROR org.apache.pig.tools.grunt.GruntParser - java.io.IOException: The arity of the group by columns do not match. + ERROR org.apache.pig.tools.grunt.GruntParser - [Error 42] The arity of the group by columns do not match. Stack Trace: at org.apache.pig.PigServer.parseQuery(PigServer.java:298) at org.apache.pig.PigServer.registerQuery(PigServer.java:263)
