[Pig Wiki] Update of "PigTypesBranchIntegration" by OlgaN
Dear Wiki user, You have subscribed to a wiki page or wiki category on "Pig Wiki" for change notification. The following page has been changed by OlgaN: http://wiki.apache.org/pig/PigTypesBranchIntegration -- || [http://issues.apache.org/jira/browse/PIG-207 PIG-207] || New illustrate command does not work in mapreduce mode. || No || || [http://issues.apache.org/jira/browse/PIG-59 PIG-59] || Illustrate command; I think should be easy to integrate || No || || [http://issues.apache.org/jira/browse/PIG-114 PIG-114] || reversable functions; only added reversable interface; the rest is no longer applicable as the new code does not try to reuse intermediate results || Yes || - || [http://issues.apache.org/jira/browse/PIG-167 PIG-167] || memory management || No || || [http://issues.apache.org/jira/browse/PIG-170 PIG-170] || memory management || No || || [http://issues.apache.org/jira/browse/PIG-172 PIG-172] || NullPointerException thrown if we catch exception with null message || Yes || || [http://issues.apache.org/jira/browse/PIG-164 PIG-164] || purgin weak references || Yes ||
[Pig Wiki] Update of "PigTypesBranchIntegration" by OlgaN
Dear Wiki user, You have subscribed to a wiki page or wiki category on "Pig Wiki" for change notification. The following page has been changed by OlgaN: http://wiki.apache.org/pig/PigTypesBranchIntegration -- || [http://issues.apache.org/jira/browse/PIG-114 PIG-114] || reversable functions; only added reversable interface; the rest is no longer applicable as the new code does not try to reuse intermediate results || Yes || || [http://issues.apache.org/jira/browse/PIG-167 PIG-167] || memory management || No || || [http://issues.apache.org/jira/browse/PIG-170 PIG-170] || memory management || No || - || [http://issues.apache.org/jira/browse/PIG-172 PIG-172] || NullPointerException thrown if we catch exception with null message || No || + || [http://issues.apache.org/jira/browse/PIG-172 PIG-172] || NullPointerException thrown if we catch exception with null message || Yes || || [http://issues.apache.org/jira/browse/PIG-164 PIG-164] || purgin weak references || Yes || || [http://issues.apache.org/jira/browse/PIG-129 PIG-129] || need to create temp files in the task's working directory; should be easy to integrate || Yes || || [http://issues.apache.org/jira/browse/PIG-118 PIG-118] || UNION/CROSS/JOIN operations should not allow 1 operand; trivial change; || Yes ||
svn commit: r677771 - in /incubator/pig/branches/types/src/org/apache/pig/tools/grunt: Grunt.java GruntParser.java Utils.java
Author: olga
Date: Thu Jul 17 16:07:24 2008
New Revision: 61
URL: http://svn.apache.org/viewvc?rev=61&view=rev
Log:
PIG-172: permission error handling
Added:
incubator/pig/branches/types/src/org/apache/pig/tools/grunt/Utils.java
Modified:
incubator/pig/branches/types/src/org/apache/pig/tools/grunt/Grunt.java
incubator/pig/branches/types/src/org/apache/pig/tools/grunt/GruntParser.java
Modified: incubator/pig/branches/types/src/org/apache/pig/tools/grunt/Grunt.java
URL:
http://svn.apache.org/viewvc/incubator/pig/branches/types/src/org/apache/pig/tools/grunt/Grunt.java?rev=61&r1=60&r2=61&view=diff
==
--- incubator/pig/branches/types/src/org/apache/pig/tools/grunt/Grunt.java
(original)
+++ incubator/pig/branches/types/src/org/apache/pig/tools/grunt/Grunt.java Thu
Jul 17 16:07:24 2008
@@ -56,10 +56,12 @@
try {
parser.setInteractive(false);
parser.parseStopOnError();
-} catch (Throwable e) {
-log.error(e);
-throw e;
-}
+} catch (Exception e) {
+Exception pe = Utils.getPermissionException(e);
+if (pe != null)
+log.error("You don't have permission to perform the operation.
Error from the server: " + pe.getMessage());
+throw (e);
+}
}
}
Modified:
incubator/pig/branches/types/src/org/apache/pig/tools/grunt/GruntParser.java
URL:
http://svn.apache.org/viewvc/incubator/pig/branches/types/src/org/apache/pig/tools/grunt/GruntParser.java?rev=61&r1=60&r2=61&view=diff
==
---
incubator/pig/branches/types/src/org/apache/pig/tools/grunt/GruntParser.java
(original)
+++
incubator/pig/branches/types/src/org/apache/pig/tools/grunt/GruntParser.java
Thu Jul 17 16:07:24 2008
@@ -23,6 +23,8 @@
import java.util.Iterator;
import java.util.Map;
import java.util.Properties;
+import java.io.ByteArrayOutputStream;
+import java.io.PrintStream;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
@@ -88,7 +90,17 @@
}
catch(Exception e)
{
-log.error(e.getMessage());
+Exception pe = Utils.getPermissionException(e);
+if (pe != null)
+log.error("You don't have permission to perform the
operation. Error from the server: " + pe.getMessage());
+else {
+ByteArrayOutputStream bs = new ByteArrayOutputStream();
+e.printStackTrace(new PrintStream(bs));
+log.error(bs.toString());
+log.error(e.getMessage());
+log.error(e);
+ }
+
}
}
Added: incubator/pig/branches/types/src/org/apache/pig/tools/grunt/Utils.java
URL:
http://svn.apache.org/viewvc/incubator/pig/branches/types/src/org/apache/pig/tools/grunt/Utils.java?rev=61&view=auto
==
--- incubator/pig/branches/types/src/org/apache/pig/tools/grunt/Utils.java
(added)
+++ incubator/pig/branches/types/src/org/apache/pig/tools/grunt/Utils.java Thu
Jul 17 16:07:24 2008
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.pig.tools.grunt;
+
+class Utils {
+static Exception getPermissionException(Exception top){
+Throwable current = top;
+
+while (current != null && (current.getMessage() == null ||
current.getMessage().indexOf("Permission denied") == -1)){
+current = current.getCause();
+}
+return (Exception)current;
+}
+}
+
[Pig Wiki] Update of "PigTypesBranchIntegration" by OlgaN
Dear Wiki user, You have subscribed to a wiki page or wiki category on "Pig Wiki" for change notification. The following page has been changed by OlgaN: http://wiki.apache.org/pig/PigTypesBranchIntegration -- || [http://issues.apache.org/jira/browse/PIG-235 PIG-235] || Performance issues with memory spills || No || || [http://issues.apache.org/jira/browse/PIG-245 PIG-245] || Need wrapper UDFs for all java.lang.Math functions || No || || [https://issues.apache.org/jira/browse/PIG-271 PIG-271] || Add tutorial files and builds to Pig SVN || No || + || [https://issues.apache.org/jira/browse/PIG-123 PIG-123] || escaping|| yes || + == Step 2 ==
svn commit: r677768 - in /incubator/pig/branches/types: src/org/apache/pig/impl/logicalLayer/parser/QueryParser.jjt src/org/apache/pig/tools/pigscript/parser/PigScriptParser.jj test/org/apache/pig/tes
Author: olga
Date: Thu Jul 17 15:52:31 2008
New Revision: 677768
URL: http://svn.apache.org/viewvc?rev=677768&view=rev
Log:
merge of PIG-123: escaping
Added:
incubator/pig/branches/types/test/org/apache/pig/test/TestPigScriptParser.java
Modified:
incubator/pig/branches/types/src/org/apache/pig/impl/logicalLayer/parser/QueryParser.jjt
incubator/pig/branches/types/src/org/apache/pig/tools/pigscript/parser/PigScriptParser.jj
Modified:
incubator/pig/branches/types/src/org/apache/pig/impl/logicalLayer/parser/QueryParser.jjt
URL:
http://svn.apache.org/viewvc/incubator/pig/branches/types/src/org/apache/pig/impl/logicalLayer/parser/QueryParser.jjt?rev=677768&r1=677767&r2=677768&view=diff
==
---
incubator/pig/branches/types/src/org/apache/pig/impl/logicalLayer/parser/QueryParser.jjt
(original)
+++
incubator/pig/branches/types/src/org/apache/pig/impl/logicalLayer/parser/QueryParser.jjt
Thu Jul 17 15:52:31 2008
@@ -141,7 +141,7 @@
}
static String unquote(String s) {
- return s.substring(1, s.length()-1);
+ return StringUtils.unescapeInputString(s.substring(1,
s.length()-1)) ;
}
static int undollar(String s) {
@@ -397,6 +397,91 @@
}
+
+class StringUtils {
+
+ public static String unescapeInputString(String input) {
+
+if (input == null) {
+return new String() ;
+}
+
+// Needed variables
+// preset the size so our StringBuilders don't have to grow
+int inputlength = input.length();
+StringBuilder unicode = new StringBuilder(4);
+StringBuilder output = new StringBuilder(inputlength) ;
+boolean hadSlash = false;
+boolean inUnicode = false;
+
+// The main loop
+for (int i = 0; i < inputlength; i++) {
+char ch = input.charAt(i);
+// currently doing unicode mode
+if (inUnicode) {
+unicode.append(ch);
+if (unicode.length() == 4) {
+// unicode now contains the four hex digits
+try {
+int value = Integer.parseInt(unicode.toString(),
0x10);
+output.append((char) value) ;
+// reuse the StringBuilder
+unicode.setLength(0);
+inUnicode = false;
+hadSlash = false;
+} catch (NumberFormatException nfe) {
+throw new RuntimeException("Unable to parse
unicode value: " + unicode, nfe);
+}
+}
+continue;
+}
+if (hadSlash) {
+// handle an escaped value
+hadSlash = false;
+switch (ch) {
+case '\\':
+output.append('\\');
+break;
+case '\'':
+output.append('\'');
+break;
+case 'r':
+output.append('\r');
+break;
+case 'f':
+output.append('\f');
+break;
+case 't':
+output.append('\t');
+break;
+case 'n':
+output.append('\n');
+break;
+case 'b':
+output.append('\b');
+break;
+case 'u':
+{
+// switch to unicode mode
+inUnicode = true;
+break;
+}
+default :
+output.append(ch);
+break;
+}
+continue;
+} else if (ch == '\\') {
+hadSlash = true;
+continue;
+}
+output.append(ch);
+}
+
+return output.toString() ;
+}
+}
+
PARSER_END(QueryParser)
@@ -478,7 +563,20 @@
| < FLOATNUMBER: (["f","F"])? >
}
-TOKEN : { }
+TOKEN : { }
+
// Pig has special variables starting with $
TOKEN : { > }
Modified:
incubator/pig/branches/types/src/org/apache/pig/tools/pigscript/parser/PigScriptParser.jj
URL:
http://svn.apache.org/viewvc/incubator/pig/branches/types/src/org/apache/pig/tools/pigscript/parser/PigScriptParser
[Pig Wiki] Trivial Update of "EmbeddedPig" by CorinneC
Dear Wiki user,
You have subscribed to a wiki page or wiki category on "Pig Wiki" for change
notification.
The following page has been changed by CorinneC:
http://wiki.apache.org/pig/EmbeddedPig
--
[[Anchor(Example)]]
=== Example ===
- Lets assume that I need to count the number of occurrences of each word in a
document. Lets also assume that you have EvalFunction `Tokenize` that parses a
line of text and returns all the words for that line. The function is located
in `/mylocation/tokenize.jar`.
+ Let's assume you need to count the number of occurrences of each word in a
document. Let's also assume that you have EvalFunction `Tokenize` that parses a
line of text and returns all the words for that line. The function is located
in `/mylocation/tokenize.jar`.
- PigLatin script for this computation will look as follows:
+ The PigLatin script for the computation will look like this:
{{{
register /mylocation/tokenize.jar
@@ -29, +29 @@
store D into 'myoutput';
}}}
- The same can be accomplished with the following Java program
+ The same computation can be performed with this Java program:
{{{
[Pig Wiki] Trivial Update of "EmbeddedPig" by CorinneC
Dear Wiki user, You have subscribed to a wiki page or wiki category on "Pig Wiki" for change notification. The following page has been changed by CorinneC: http://wiki.apache.org/pig/EmbeddedPig -- * The jar which contains your functions must be registered. * The four calls to `pigServer.registerQuery()` simply cause the query to be parsed and enquired. The query is not actually executed until `pigServer.store()` is called. - * The input data referred to on the load statement, must be on DFS in the specified location. + * The input data referred to on the load statement, must be on HDFS in the specified location. - * The final result is placed into `myoutput` file in the your current working directory on DFS. (By default this is your home directory on DFS.) + * The final result is placed into `myoutput` file in the your current working directory on HDFS. (By default this is your home directory on HDFS.) To run your program, you need to first compile it by using the following command:
[Pig Wiki] Trivial Update of "EmbeddedPig" by CorinneC
Dear Wiki user,
You have subscribed to a wiki page or wiki category on "Pig Wiki" for change
notification.
The following page has been changed by CorinneC:
http://wiki.apache.org/pig/EmbeddedPig
--
try {
pigServer.registerJar("/mylocation/tokenize.jar");
runMyQuery(pigServer, "myinput.txt";
+ }
- } catch (IOException e) {
+ catch (IOException e) {
e.printStackTrace();
- }
+ }
}
public static void runMyQuery(PigServer pigServer, String inputFile)
throws IOException {
[Pig Wiki] Trivial Update of "PigTutorial" by CorinneC
Dear Wiki user, You have subscribed to a wiki page or wiki category on "Pig Wiki" for change notification. The following page has been changed by CorinneC: http://wiki.apache.org/pig/PigTutorial -- - The Pig tutorial shows you how to run Pig scripts in local mode or on a Hadoop cluster. + The Pig tutorial shows you how to run two Pig scripts in local mode or on a Hadoop cluster. * To run the scripts in local mode, no Hadoop or DFS installation is required. All files are installed and run from your local host and file system. * To run the scripts on a Hadoop cluster, you need access to a Hadoop cluster and DFS installation. - The Pig tutorial file (pigtutorial.tar.gz) includes the Pig JAR file (pig.jar) and the tutorial files (tutorial.jar, etc). These files work with Hadoop 0.17 and provide everything you need to run the Pig scripts in local mode or on a Hadoop cluster. To get started, follow these basic steps: + The Pig tutorial file (pigtutorial.tar.gz) includes the Pig JAR file (pig.jar) and the tutorial files (tutorial.jar, Pigs scripts, log files). These files work with Hadoop 0.17 and provide everything you need to run the Pig scripts. To get started, follow these basic steps: 1. Install Java. 1. Download the Pig tutorial file and install Pig.
[Pig Wiki] Trivial Update of "FrontPage" by CorinneC
Dear Wiki user, You have subscribed to a wiki page or wiki category on "Pig Wiki" for change notification. The following page has been changed by CorinneC: http://wiki.apache.org/pig/FrontPage -- * ParameterSubstitution * PigOptimizationWishList * NestedLogicalPlan (still draft version) + * Performance - * PigPerformance - current performance numbers + * PigPerformance (current performance numbers) == Related Resources ==
svn commit: r677662 - in /incubator/pig/trunk: build.xml contrib/piggybank/java/build.xml lib-src/bzip2/bzip2-LICENSE.txt lib/hadoop-LICENSE.txt lib/javacc-LICENSE.txt lib/jsch-LICENSE.txt lib/junit-L
Author: olga Date: Thu Jul 17 11:13:00 2008 New Revision: 677662 URL: http://svn.apache.org/viewvc?rev=677662&view=rev Log: release related change from branch-01 Added: incubator/pig/trunk/lib-src/bzip2/bzip2-LICENSE.txt incubator/pig/trunk/lib/hadoop-LICENSE.txt incubator/pig/trunk/lib/javacc-LICENSE.txt incubator/pig/trunk/lib/jsch-LICENSE.txt incubator/pig/trunk/lib/junit-LICENSE.txt Modified: incubator/pig/trunk/build.xml incubator/pig/trunk/contrib/piggybank/java/build.xml incubator/pig/trunk/tutorial/build.xml Modified: incubator/pig/trunk/build.xml URL: http://svn.apache.org/viewvc/incubator/pig/trunk/build.xml?rev=677662&r1=677661&r2=677662&view=diff == --- incubator/pig/trunk/build.xml (original) +++ incubator/pig/trunk/build.xml Thu Jul 17 11:13:00 2008 @@ -360,7 +360,18 @@ - + + + + + + + + + + + + Modified: incubator/pig/trunk/contrib/piggybank/java/build.xml URL: http://svn.apache.org/viewvc/incubator/pig/trunk/contrib/piggybank/java/build.xml?rev=677662&r1=677661&r2=677662&view=diff == --- incubator/pig/trunk/contrib/piggybank/java/build.xml (original) +++ incubator/pig/trunk/contrib/piggybank/java/build.xml Thu Jul 17 11:13:00 2008 @@ -15,7 +15,6 @@ limitations under the License. --> - Added: incubator/pig/trunk/lib-src/bzip2/bzip2-LICENSE.txt URL: http://svn.apache.org/viewvc/incubator/pig/trunk/lib-src/bzip2/bzip2-LICENSE.txt?rev=677662&view=auto == --- incubator/pig/trunk/lib-src/bzip2/bzip2-LICENSE.txt (added) +++ incubator/pig/trunk/lib-src/bzip2/bzip2-LICENSE.txt Thu Jul 17 11:13:00 2008 @@ -0,0 +1,18 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + Added: incubator/pig/trunk/lib/hadoop-LICENSE.txt URL: http://svn.apache.org/viewvc/incubator/pig/trunk/lib/hadoop-LICENSE.txt?rev=677662&view=auto == --- incubator/pig/trunk/lib/hadoop-LICENSE.txt (added) +++ incubator/pig/trunk/lib/hadoop-LICENSE.txt Thu Jul 17 11:13:00 2008 @@ -0,0 +1,18 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + Added: incubator/pig/trunk/lib/javacc-LICENSE.txt URL: http://svn.apache.org/viewvc/incubator/pig/trunk/lib/javacc-LICENSE.txt?rev=677662&view=auto == --- incubator/pig/trunk/lib/javacc-LICENSE.txt (added) +++ incubator/pig/trunk/lib/javacc-LICENSE.txt Thu Jul 17 11:13:00 2008 @@ -0,0 +1,24 @@ +* Copyright (c) , +* All rights reserved. +* +* Redistribution and use in source and binary forms, with or without +* modification, are permitted provided that the following conditions are met: +* * Redistributions of source code must retain the above copyright +* notice, this list of conditions and the following disclaimer. +* * Redistributions in binary form must reproduce the above copyright +* notice, this list of conditions and the following disclaimer in the +* documentation and/or other materials provided with the distribution. +* * Neither the name of the nor the +*
svn commit: r677661 - in /incubator/pig/branches/branch-0.1: build.xml contrib/piggybank/java/build.xml lib-src/bzip2/bzip2-LICENSE.txt lib/hadoop-LICENSE.txt lib/javacc-LICENSE.txt lib/jsch-LICENSE.t
Author: olga Date: Thu Jul 17 11:11:10 2008 New Revision: 677661 URL: http://svn.apache.org/viewvc?rev=677661&view=rev Log: build/package process fixes + library licenses Added: incubator/pig/branches/branch-0.1/lib-src/bzip2/bzip2-LICENSE.txt incubator/pig/branches/branch-0.1/lib/hadoop-LICENSE.txt incubator/pig/branches/branch-0.1/lib/javacc-LICENSE.txt incubator/pig/branches/branch-0.1/lib/jsch-LICENSE.txt incubator/pig/branches/branch-0.1/lib/junit-LICENSE.txt Modified: incubator/pig/branches/branch-0.1/build.xml incubator/pig/branches/branch-0.1/contrib/piggybank/java/build.xml incubator/pig/branches/branch-0.1/tutorial/build.xml Modified: incubator/pig/branches/branch-0.1/build.xml URL: http://svn.apache.org/viewvc/incubator/pig/branches/branch-0.1/build.xml?rev=677661&r1=677660&r2=677661&view=diff == --- incubator/pig/branches/branch-0.1/build.xml (original) +++ incubator/pig/branches/branch-0.1/build.xml Thu Jul 17 11:11:10 2008 @@ -360,7 +360,18 @@ - + + + + + + + + + + + + Modified: incubator/pig/branches/branch-0.1/contrib/piggybank/java/build.xml URL: http://svn.apache.org/viewvc/incubator/pig/branches/branch-0.1/contrib/piggybank/java/build.xml?rev=677661&r1=677660&r2=677661&view=diff == --- incubator/pig/branches/branch-0.1/contrib/piggybank/java/build.xml (original) +++ incubator/pig/branches/branch-0.1/contrib/piggybank/java/build.xml Thu Jul 17 11:11:10 2008 @@ -15,7 +15,6 @@ limitations under the License. --> - Added: incubator/pig/branches/branch-0.1/lib-src/bzip2/bzip2-LICENSE.txt URL: http://svn.apache.org/viewvc/incubator/pig/branches/branch-0.1/lib-src/bzip2/bzip2-LICENSE.txt?rev=677661&view=auto == --- incubator/pig/branches/branch-0.1/lib-src/bzip2/bzip2-LICENSE.txt (added) +++ incubator/pig/branches/branch-0.1/lib-src/bzip2/bzip2-LICENSE.txt Thu Jul 17 11:11:10 2008 @@ -0,0 +1,18 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + Added: incubator/pig/branches/branch-0.1/lib/hadoop-LICENSE.txt URL: http://svn.apache.org/viewvc/incubator/pig/branches/branch-0.1/lib/hadoop-LICENSE.txt?rev=677661&view=auto == --- incubator/pig/branches/branch-0.1/lib/hadoop-LICENSE.txt (added) +++ incubator/pig/branches/branch-0.1/lib/hadoop-LICENSE.txt Thu Jul 17 11:11:10 2008 @@ -0,0 +1,18 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + Added: incubator/pig/branches/branch-0.1/lib/javacc-LICENSE.txt URL: http://svn.apache.org/viewvc/incubator/pig/branches/branch-0.1/lib/javacc-LICENSE.txt?rev=677661&view=auto == --- incubator/pig/branches/branch-0.1/lib/javacc-LICENSE.txt (added) +++ incubator/pig/branches/branch-0.1/lib/javacc-LICENSE.txt Thu Jul 17 11:11:10 2008 @@ -0,0 +1,24 @@ +* Copyright (c) , +* All rights reserved. +* +* Redistribution and use in source and binary forms, with or without +* modification, are permitted provided that the following conditions are met: +* * Redistri
svn commit: r677655 - /incubator/pig/branches/types/lib/hadoop17.jar
Author: olga Date: Thu Jul 17 10:56:59 2008 New Revision: 677655 URL: http://svn.apache.org/viewvc?rev=677655&view=rev Log: added hadoop17.jar Added: incubator/pig/branches/types/lib/hadoop17.jar (with props) Added: incubator/pig/branches/types/lib/hadoop17.jar URL: http://svn.apache.org/viewvc/incubator/pig/branches/types/lib/hadoop17.jar?rev=677655&view=auto == Binary file - no diff available. Propchange: incubator/pig/branches/types/lib/hadoop17.jar -- svn:mime-type = application/octet-stream
[Pig Wiki] Update of "PigTypesBranchIntegration" by OlgaN
Dear Wiki user, You have subscribed to a wiki page or wiki category on "Pig Wiki" for change notification. The following page has been changed by OlgaN: http://wiki.apache.org/pig/PigTypesBranchIntegration -- || [http://issues.apache.org/jira/browse/PIG-106 PIG-106] || Optimize Pig by replacing String '+' and StringBuffer with StringBuilder; - there are changes in backend/Operators which have changed in structure now || Yes || || [http://issues.apache.org/jira/browse/PIG-156 PIG-156] || unit tests broken under windows can be merged now - depends on streaming and other test cases being merged first|| No || || [http://issues.apache.org/jira/browse/PIG-85 PIG-85] || making PigStorage work with control characters || No || - || [http://issues.apache.org/jira/browse/PIG-250 PIG-250] || speculative execution is broken; || Patch supplied - needs to be committed || + || [http://issues.apache.org/jira/browse/PIG-250 PIG-250] || speculative execution is broken; || Yes || || [http://issues.apache.org/jira/browse/PIG-198 PIG-198] || integration with hadoop 0.17; Files not patched: * src/org/apache/pig/backend/executionengine/PigSlicer.java - not present yet in types * src/org/apache/pig/backend/hadoop/executionengine/HExecutionEngine.java - changes depend on Config properties change * src/org/apache/pig/backend/hadoop/executionengine/mapreduceExec/PigMapReduce.java - this is present in ../impl/mapReduceLayer now and is different in content - depends on some other change going in first which will cause PigMapReduce to "implement" MapRunnable and Reducer (in the types branch it currently does not implement any interface) * src/org/apache/pig/backend/hadoop/executionengine/mapreduceExec/PigCombine.java - not present in types || Yes || || [http://issues.apache.org/jira/browse/PIG-266 PIG-266] || Warnings with hadoop 17 || No || || [http://issues.apache.org/jira/browse/PIG-291 PIG-291] || HOD param passing || No ||
[Pig Wiki] Update of "PigTypesBranchIntegration" by OlgaN
Dear Wiki user, You have subscribed to a wiki page or wiki category on "Pig Wiki" for change notification. The following page has been changed by OlgaN: http://wiki.apache.org/pig/PigTypesBranchIntegration -- || '''JIRA''' || '''Comments''' || '''Done'''|| || [http://issues.apache.org/jira/browse/PIG-246 PIG-246] || UDF repository creation - needs finalizing the EvalFunc.exec interface and changing DataAtom to String || No || || [http://issues.apache.org/jira/browse/PIG-18 PIG-18] || making Pig work with HOD 0.4; should be doable now - involves backend - HExecutionEngine.java - a little more involved and needs careful merging|| No || - || [http://issues.apache.org/jira/browse/PIG-106 PIG-106] || Optimize Pig by replacing String '+' and StringBuffer with StringBuilder; - there are changes in backend/Operators which have changed in structure now || No || + || [http://issues.apache.org/jira/browse/PIG-106 PIG-106] || Optimize Pig by replacing String '+' and StringBuffer with StringBuilder; - there are changes in backend/Operators which have changed in structure now || Yes || || [http://issues.apache.org/jira/browse/PIG-156 PIG-156] || unit tests broken under windows can be merged now - depends on streaming and other test cases being merged first|| No || || [http://issues.apache.org/jira/browse/PIG-85 PIG-85] || making PigStorage work with control characters || No || || [http://issues.apache.org/jira/browse/PIG-250 PIG-250] || speculative execution is broken; || Patch supplied - needs to be committed || - || [http://issues.apache.org/jira/browse/PIG-198 PIG-198] || integration with hadoop 0.17; Files not patched: * src/org/apache/pig/backend/executionengine/PigSlicer.java - not present yet in types * src/org/apache/pig/backend/hadoop/executionengine/HExecutionEngine.java - changes depend on Config properties change * src/org/apache/pig/backend/hadoop/executionengine/mapreduceExec/PigMapReduce.java - this is present in ../impl/mapReduceLayer now and is different in content - depends on some other change going in first which will cause PigMapReduce to "implement" MapRunnable and Reducer (in the types branch it currently does not implement any interface) * src/org/apache/pig/backend/hadoop/executionengine/mapreduceExec/PigCombine.java - not present in types || Patch supplied - needs to be committed || + || [http://issues.apache.org/jira/browse/PIG-198 PIG-198] || integration with hadoop 0.17; Files not patched: * src/org/apache/pig/backend/executionengine/PigSlicer.java - not present yet in types * src/org/apache/pig/backend/hadoop/executionengine/HExecutionEngine.java - changes depend on Config properties change * src/org/apache/pig/backend/hadoop/executionengine/mapreduceExec/PigMapReduce.java - this is present in ../impl/mapReduceLayer now and is different in content - depends on some other change going in first which will cause PigMapReduce to "implement" MapRunnable and Reducer (in the types branch it currently does not implement any interface) * src/org/apache/pig/backend/hadoop/executionengine/mapreduceExec/PigCombine.java - not present in types || Yes || || [http://issues.apache.org/jira/browse/PIG-266 PIG-266] || Warnings with hadoop 17 || No || || [http://issues.apache.org/jira/browse/PIG-291 PIG-291] || HOD param passing || No || || [http://issues.apache.org/jira/browse/PIG-176 PIG-176] || memory management; depends on PIG-111; || No ||
svn commit: r677638 - in /incubator/pig/branches/types: src/org/apache/pig/builtin/Utf8StorageConverter.java src/org/apache/pig/impl/mapReduceLayer/PigMapReduce.java test/org/apache/pig/test/TestConve
Author: gates
Date: Thu Jul 17 10:05:06 2008
New Revision: 677638
URL: http://svn.apache.org/viewvc?rev=677638&view=rev
Log:
PIG-312 When casting to an int or long, if the cast fails, try to cast to a
double and then to int or long.
Modified:
incubator/pig/branches/types/src/org/apache/pig/builtin/Utf8StorageConverter.java
incubator/pig/branches/types/src/org/apache/pig/impl/mapReduceLayer/PigMapReduce.java
incubator/pig/branches/types/test/org/apache/pig/test/TestConversions.java
Modified:
incubator/pig/branches/types/src/org/apache/pig/builtin/Utf8StorageConverter.java
URL:
http://svn.apache.org/viewvc/incubator/pig/branches/types/src/org/apache/pig/builtin/Utf8StorageConverter.java?rev=677638&r1=677637&r2=677638&view=diff
==
---
incubator/pig/branches/types/src/org/apache/pig/builtin/Utf8StorageConverter.java
(original)
+++
incubator/pig/branches/types/src/org/apache/pig/builtin/Utf8StorageConverter.java
Thu Jul 17 10:05:06 2008
@@ -50,6 +50,9 @@
protected BagFactory mBagFactory = BagFactory.getInstance();
protected TupleFactory mTupleFactory = TupleFactory.getInstance();
protected final Log mLog = LogFactory.getLog(getClass());
+
+private Integer mMaxInt = new Integer(Integer.MAX_VALUE);
+private Long mMaxLong = new Long(Long.MAX_VALUE);
public Utf8StorageConverter() {
}
@@ -86,24 +89,54 @@
}
public Integer bytesToInteger(byte[] b) throws IOException {
+String s = new String(b);
try {
-return Integer.valueOf(new String(b));
+return Integer.valueOf(s);
} catch (NumberFormatException nfe) {
-mLog.warn("Unable to interpret value " + b + " in field being " +
-"converted to int, caught NumberFormatException <" +
-nfe.getMessage() + "> field discarded");
-return null;
+// It's possible that this field can be interpreted as a double.
+// Unfortunately Java doesn't handle this in Integer.valueOf. So
+// we need to try to convert it to a double and if that works then
+// go to an int.
+try {
+Double d = Double.valueOf(s);
+// Need to check for an overflow error
+if (d.doubleValue() > mMaxInt.doubleValue() + 1.0) {
+mLog.warn("Value " + d + " too large for integer");
+return null;
+}
+return new Integer(d.intValue());
+} catch (NumberFormatException nfe2) {
+mLog.warn("Unable to interpret value " + b + " in field being
" +
+"converted to int, caught NumberFormatException <" +
+nfe.getMessage() + "> field discarded");
+return null;
+}
}
}
public Long bytesToLong(byte[] b) throws IOException {
+String s = new String(b);
try {
-return Long.valueOf(new String(b));
+return Long.valueOf(s);
} catch (NumberFormatException nfe) {
-mLog.warn("Unable to interpret value " + b + " in field being " +
-"converted to long, caught NumberFormatException <" +
-nfe.getMessage() + "> field discarded");
-return null;
+// It's possible that this field can be interpreted as a double.
+// Unfortunately Java doesn't handle this in Long.valueOf. So
+// we need to try to convert it to a double and if that works then
+// go to an long.
+try {
+Double d = Double.valueOf(s);
+// Need to check for an overflow error
+if (d.doubleValue() > mMaxLong.doubleValue() + 1.0) {
+mLog.warn("Value " + d + " too large for integer");
+return null;
+}
+return new Long(d.longValue());
+} catch (NumberFormatException nfe2) {
+mLog.warn("Unable to interpret value " + b + " in field being
" +
+"converted to long, caught NumberFormatException <" +
+nfe.getMessage() + "> field discarded");
+return null;
+}
}
}
Modified:
incubator/pig/branches/types/src/org/apache/pig/impl/mapReduceLayer/PigMapReduce.java
URL:
http://svn.apache.org/viewvc/incubator/pig/branches/types/src/org/apache/pig/impl/mapReduceLayer/PigMapReduce.java?rev=677638&r1=677637&r2=677638&view=diff
==
---
incubator/pig/branches/types/src/org/apache/pig/impl/mapReduceLayer/PigMapReduce.java
(original)
+++
incubator/pig/branches/types/src/org/apache/pig/impl/mapReduceLayer/PigMapReduce.java
Thu Jul 17 10:05:06 2008
@@ -46,7 +46,7 @@
import org.
svn commit: r677637 - in /incubator/pig/branches/types: ./ src/org/apache/pig/ src/org/apache/pig/backend/datastorage/ src/org/apache/pig/backend/hadoop/datastorage/ src/org/apache/pig/backend/local/d
Author: olga Date: Thu Jul 17 10:04:07 2008 New Revision: 677637 URL: http://svn.apache.org/viewvc?rev=677637&view=rev Log: PIG-198, and 106 Modified: incubator/pig/branches/types/build.xml incubator/pig/branches/types/src/org/apache/pig/ExecType.java incubator/pig/branches/types/src/org/apache/pig/backend/datastorage/ElementDescriptor.java incubator/pig/branches/types/src/org/apache/pig/backend/hadoop/datastorage/HDataStorage.java incubator/pig/branches/types/src/org/apache/pig/backend/hadoop/datastorage/HFile.java incubator/pig/branches/types/src/org/apache/pig/backend/hadoop/datastorage/HPath.java incubator/pig/branches/types/src/org/apache/pig/backend/local/datastorage/LocalPath.java incubator/pig/branches/types/src/org/apache/pig/data/IndexedTuple.java incubator/pig/branches/types/src/org/apache/pig/impl/io/FileLocalizer.java incubator/pig/branches/types/src/org/apache/pig/impl/mapReduceLayer/JobControlCompiler.java incubator/pig/branches/types/src/org/apache/pig/impl/mapReduceLayer/PigInputFormat.java incubator/pig/branches/types/src/org/apache/pig/impl/mapReduceLayer/PigOutputFormat.java incubator/pig/branches/types/src/org/apache/pig/impl/mapReduceLayer/PigSplit.java incubator/pig/branches/types/src/org/apache/pig/impl/mapReduceLayer/SortPartitioner.java incubator/pig/branches/types/test/org/apache/pig/test/MiniCluster.java incubator/pig/branches/types/test/org/apache/pig/test/TestCombiner.java Modified: incubator/pig/branches/types/build.xml URL: http://svn.apache.org/viewvc/incubator/pig/branches/types/build.xml?rev=677637&r1=677636&r2=677637&view=diff == --- incubator/pig/branches/types/build.xml (original) +++ incubator/pig/branches/types/build.xml Thu Jul 17 10:04:07 2008 @@ -18,14 +18,6 @@ - - - - - - - - @@ -35,7 +27,18 @@ - + + + + + + + + + + + + @@ -150,28 +153,62 @@ *** Building Main Sources *** +*** To compile with all warnings enabled, supply -Dall.warnings=1 on command line *** +*** If all.warnings property is supplied, compile-sources-all-warnings target will be executed *** +*** Else, compile-sources (which only warns about deprecations) target will be executed *** + + + + + + + + *** Building Test Sources *** +*** To compile with all warnings enabled, supply -Dall.warnings=1 on command line *** +*** If all.warnings property is supplied, compile-sources-all-warnings target will be executed *** +*** Else, compile-sources (which only warns about deprecations) target will be executed *** + + + + + + + + - + + - + + + + + + + + + Modified: incubator/pig/branches/types/src/org/apache/pig/ExecType.java URL: http://svn.apache.org/viewvc/incubator/pig/branches/types/src/org/apache/pig/ExecType.java?rev=677637&r1=677636&r2=677637&view=diff == --- incubator/pig/branches/types/src/org/apache/pig/ExecType.java (original) +++ incubator/pig/branches/types/src/org/apache/pig/ExecType.java Thu Jul 17 10:04:07 2008 @@ -15,6 +15,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ + package org.apache.pig; /** Modified: incubator/pig/branches/types/src/org/apache/pig/backend/datastorage/ElementDescriptor.java URL: http://svn.apache.org/viewvc/incubator/pig/branches/types/src/org/apache/pig/backend/datastorage/ElementDescriptor.java?rev=677637&r1=677636&r2=677637&view=diff == --- incubator/pig/branches/types/src/org/apache/pig/backend/datastorage/ElementDescriptor.java (original) +++ incubator/pig/branches/types/src/org/apache/pig/backend/datastorage/ElementDescriptor.java Thu Jul 17 10:04:07 2008 @@ -140,6 +140,13 @@ */ public void updateConfiguration(Properties newConfig) throws IOException; + +/** + * Defines whether the element is visible to users or + * contains system's metadata + * @return true if this is system file; false otherwise + */ +public boolean systemElement(); /** * List entity statistics Modified: incubator/pig/branches/types/src/org/apache/pig/backend/hadoop/data
