[jira] Assigned: (PIG-1446) OOME in a query having a bincond in the inner plan of a Foreach.
[ https://issues.apache.org/jira/browse/PIG-1446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan reassigned PIG-1446: - Assignee: Ashutosh Chauhan OOME in a query having a bincond in the inner plan of a Foreach. Key: PIG-1446 URL: https://issues.apache.org/jira/browse/PIG-1446 Project: Pig Issue Type: Bug Affects Versions: 0.7.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: pig-1446.patch This is seen when For Each is following a group-by and there is a bin cond as an inner plan of For Each. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1446) OOME in a query having a bincond in the inner plan of a Foreach.
[ https://issues.apache.org/jira/browse/PIG-1446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated PIG-1446: -- Attachment: pig-1446.patch Sequence of event is as follows: 1) MultiQuery optimizer combined 30 group-bys in one reducer. So, there are 30 pipelines in a reducer. 2) Each of these group-by has a ForEach after them. 3) ForEach has a bincond in its own plan. 4) Group-by resulted in large bags (10s of million of records). 5) Tuple containing group and bag is attached to the roots of inner plan of FE. 6) FE pulled the tuples through its leaves. 7) Due to short-circuiting in bin-cond, one branch of the plan is never pulled resulting in stray reference of bag which actually was not needed. 8) Due to MQ optimized 30 group-bys, we had many such bags now hanging in there, eating up all the memory. Fix: Detach tuples from the roots once you are done in FE. OOME in a query having a bincond in the inner plan of a Foreach. Key: PIG-1446 URL: https://issues.apache.org/jira/browse/PIG-1446 Project: Pig Issue Type: Bug Affects Versions: 0.7.0 Reporter: Ashutosh Chauhan Attachments: pig-1446.patch This is seen when For Each is following a group-by and there is a bin cond as an inner plan of For Each. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1446) OOME in a query having a bincond in the inner plan of a Foreach.
[ https://issues.apache.org/jira/browse/PIG-1446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated PIG-1446: -- Status: Patch Available (was: Open) OOME in a query having a bincond in the inner plan of a Foreach. Key: PIG-1446 URL: https://issues.apache.org/jira/browse/PIG-1446 Project: Pig Issue Type: Bug Affects Versions: 0.7.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: pig-1446.patch This is seen when For Each is following a group-by and there is a bin cond as an inner plan of For Each. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1438) [Performance] MultiQueryOptimizer should also merge DISTINCT jobs
[ https://issues.apache.org/jira/browse/PIG-1438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Richard Ding updated PIG-1438: -- Status: Resolved (was: Patch Available) Hadoop Flags: [Reviewed] Resolution: Fixed Committed to both trunk and 0.7 branch. [Performance] MultiQueryOptimizer should also merge DISTINCT jobs - Key: PIG-1438 URL: https://issues.apache.org/jira/browse/PIG-1438 Project: Pig Issue Type: Improvement Components: impl Affects Versions: 0.7.0 Reporter: Richard Ding Assignee: Richard Ding Fix For: 0.8.0 Attachments: PIG-1438.patch, PIG-1438_1.patch Current implementation doesn't merge jobs derived from DISTINCT statements. The reason is that DISTINCT jobs are implemented using a special combiner (DistinctCombiner). But we should be able to merge jobs that have the same type of combiner (e.g. merge multiple DISTINCT jobs into one). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1302) Include zebra's pigtest ant target as a part of pig's ant test target
[ https://issues.apache.org/jira/browse/PIG-1302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giridharan Kesavan updated PIG-1302: Status: Patch Available (was: Open) Include zebra's pigtest ant target as a part of pig's ant test target --- Key: PIG-1302 URL: https://issues.apache.org/jira/browse/PIG-1302 Project: Pig Issue Type: Improvement Affects Versions: 0.7.0 Reporter: Pradeep Kamath Assignee: Giridharan Kesavan Attachments: PIG-1302.patch There are changes made in Pig interfaces which break zebra loaders/storers. It would be good to run the pig tests in the zebra unit tests as part of running pig's core-test for each patch submission. So essentially in the test ant target in pig, we would need to invoke zebra's pigtest target. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1428) Add getPigStatusReporter() to PigHadoopLogger
[ https://issues.apache.org/jira/browse/PIG-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12877616#action_12877616 ] Ashutosh Chauhan commented on PIG-1428: --- I propose a slightly different approach here. Instead of adding getPigStatusReporter() to PigLogger() interface and the corresponding implementation in PigHadoopLogger, we can add a static singleton method in PigStatusReporter and also add a setContext( TaskInputOutputContext context) We can then set the context in map() and reduce() functions and users will have full access of the reporter object through the static method. This will allow us to keep error logging different then status reporting. Add getPigStatusReporter() to PigHadoopLogger - Key: PIG-1428 URL: https://issues.apache.org/jira/browse/PIG-1428 Project: Pig Issue Type: Bug Affects Versions: 0.7.0 Reporter: Ashutosh Chauhan Assignee: Dmitriy V. Ryaboy Fix For: 0.8.0 Attachments: PIG-1428.patch, PIG-1428.patch Without this getter method, its not possible to get counters, report progress etc. from UDFs. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1446) OOME in a query having a bincond in the inner plan of a Foreach.
[ https://issues.apache.org/jira/browse/PIG-1446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12877540#action_12877540 ] Daniel Dai commented on PIG-1446: - +1. Please commit after test and hudson pass. OOME in a query having a bincond in the inner plan of a Foreach. Key: PIG-1446 URL: https://issues.apache.org/jira/browse/PIG-1446 Project: Pig Issue Type: Bug Affects Versions: 0.7.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: pig-1446.patch This is seen when For Each is following a group-by and there is a bin cond as an inner plan of For Each. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1405) Need to move many standard functions from piggybank into Pig
[ https://issues.apache.org/jira/browse/PIG-1405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12877619#action_12877619 ] Aniket Mokashi commented on PIG-1405: - Currently, we have COR(elation) and COV(ariance) functions in piggybank as part of stats package. But, there is no existing implementation of VAR(iance). Need to move many standard functions from piggybank into Pig Key: PIG-1405 URL: https://issues.apache.org/jira/browse/PIG-1405 Project: Pig Issue Type: Improvement Reporter: Alan Gates Assignee: Aniket Mokashi Fix For: 0.8.0 There are currently a number of functions in Piggybank that represent features commonly supported by languages and database engines. We need to decide which of these Pig should support as built in functions and put them in org.apache.pig.builtin. This will also mean adding unit tests and javadocs for some UDFs. The existing classes will be left in Piggybank for some time for backward compatibility. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1429) Add Boolean Data Type to Pig
[ https://issues.apache.org/jira/browse/PIG-1429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12877544#action_12877544 ] Russell Jurney commented on PIG-1429: - The patch needs more work. Should knock it out in the next couple weeks. Add Boolean Data Type to Pig Key: PIG-1429 URL: https://issues.apache.org/jira/browse/PIG-1429 Project: Pig Issue Type: New Feature Components: data Affects Versions: 0.7.0 Reporter: Russell Jurney Assignee: Russell Jurney Fix For: 0.8.0 Attachments: working_boolean.patch Original Estimate: 8h Remaining Estimate: 8h Pig needs a Boolean data type. Pig-1097 is dependent on doing this. I volunteer. Is there anything beyond the work in src/org/apache/pig/data/ plus unit tests to make this work? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1333) API interface to Pig
[
https://issues.apache.org/jira/browse/PIG-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12877611#action_12877611
]
Olga Natkovich commented on PIG-1333:
-
+1 to that! :)
API interface to Pig
Key: PIG-1333
URL: https://issues.apache.org/jira/browse/PIG-1333
Project: Pig
Issue Type: Improvement
Reporter: Olga Natkovich
Assignee: Richard Ding
Fix For: 0.8.0
Attachments: PIG-1333.patch
It would be nice to make Pig more friendly for applications like workflow
that would be executing pig scripts on user behalf.
Currently, they would have to use pig command line to execute the code;
however, this has limitation on the kind of output that would be delivered.
For instance, it is hard to produce error information that is easy to use
programatically or collect statistics.
The proposal is to create a class that mimics the behavior of the Main but
gives users a status object back. The the main code of pig would look
somethig like:
public static void main(String args[])
{
PigStatus ps = PigMain.exec(args);
exit (PigStatus.rc);
}
We need to define the following:
- Content of PigStatus. It should at least include
* return code
* error string
* exception
* statistics
- A way to propagate the status class through pig code
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1333) API interface to Pig
[
https://issues.apache.org/jira/browse/PIG-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12877588#action_12877588
]
Richard Ding commented on PIG-1333:
---
Thanks Dmitriy for the suggestion on enums. It will simplify the code.
Right now the job counters we retrieve are
{code}
MAP_INPUT_RECORDS
MAP_OUTPUT_RECORDS
REDUCE_INPUT_RECORDS
REDUCE_OUTPUT_RECORDS
HDFS_BYTES_WRITTEN
SPILLABLE_MEMORY_MANAGER_SPILL_COUNT
PROACTIVE_SPILL_COUNT
{code}
plus Pig MultiStore counters.
I'm not sure we should make all Hadoop counters available through the new API.
How useful will it be to the users? I'm open to suggestions.
API interface to Pig
Key: PIG-1333
URL: https://issues.apache.org/jira/browse/PIG-1333
Project: Pig
Issue Type: Improvement
Reporter: Olga Natkovich
Assignee: Richard Ding
Fix For: 0.8.0
Attachments: PIG-1333.patch
It would be nice to make Pig more friendly for applications like workflow
that would be executing pig scripts on user behalf.
Currently, they would have to use pig command line to execute the code;
however, this has limitation on the kind of output that would be delivered.
For instance, it is hard to produce error information that is easy to use
programatically or collect statistics.
The proposal is to create a class that mimics the behavior of the Main but
gives users a status object back. The the main code of pig would look
somethig like:
public static void main(String args[])
{
PigStatus ps = PigMain.exec(args);
exit (PigStatus.rc);
}
We need to define the following:
- Content of PigStatus. It should at least include
* return code
* error string
* exception
* statistics
- A way to propagate the status class through pig code
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1428) Add getPigStatusReporter() to PigHadoopLogger
[
https://issues.apache.org/jira/browse/PIG-1428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12877591#action_12877591
]
Ashutosh Chauhan commented on PIG-1428:
---
So, I read through PIG-889. It seems that there never was a documented way to
use counters, reporters etc from UDFs, Load/Store Funcs. Actually, there is a
hacky way to do it, which exists in DefaultAbstractBag.java
{code}
protected void incSpillCount(Enum counter) {
// Increment the spill count
// warn is a misnomer. The function updates the counter. If the update
// fails, it dumps a warning
PigHadoopLogger.getInstance().warn(this, Spill counter incremented,
counter);
}
{code}
But in PIG-889 Santhosh has argued against for this (mis)use of PigLogger. I
think we need to provide a formal way to Pig users to access counters,
reporters from our interfaces (UDFs, L/S) as PigHadoopLogger is designed for
error-handling (warning aggregation in particular) and not for this purpose.
And we shall mark this class as Internal only, before some one starts using it.
With the same argument, above method where Pig is internally making use of its
own Counters is flawed and needs to be corrected.
Add getPigStatusReporter() to PigHadoopLogger
-
Key: PIG-1428
URL: https://issues.apache.org/jira/browse/PIG-1428
Project: Pig
Issue Type: Bug
Affects Versions: 0.7.0
Reporter: Ashutosh Chauhan
Assignee: Dmitriy V. Ryaboy
Fix For: 0.8.0
Attachments: PIG-1428.patch, PIG-1428.patch
Without this getter method, its not possible to get counters, report progress
etc. from UDFs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1333) API interface to Pig
[
https://issues.apache.org/jira/browse/PIG-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12877607#action_12877607
]
Richard Ding commented on PIG-1333:
---
Now I also think this patch is too large :-) I'll break it into two: one patch
for this jira that concentrates on implementing the new API, the other patch
will address the code cleanup in a different jira.
API interface to Pig
Key: PIG-1333
URL: https://issues.apache.org/jira/browse/PIG-1333
Project: Pig
Issue Type: Improvement
Reporter: Olga Natkovich
Assignee: Richard Ding
Fix For: 0.8.0
Attachments: PIG-1333.patch
It would be nice to make Pig more friendly for applications like workflow
that would be executing pig scripts on user behalf.
Currently, they would have to use pig command line to execute the code;
however, this has limitation on the kind of output that would be delivered.
For instance, it is hard to produce error information that is easy to use
programatically or collect statistics.
The proposal is to create a class that mimics the behavior of the Main but
gives users a status object back. The the main code of pig would look
somethig like:
public static void main(String args[])
{
PigStatus ps = PigMain.exec(args);
exit (PigStatus.rc);
}
We need to define the following:
- Content of PigStatus. It should at least include
* return code
* error string
* exception
* statistics
- A way to propagate the status class through pig code
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1333) API interface to Pig
[
https://issues.apache.org/jira/browse/PIG-1333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12877585#action_12877585
]
Richard Ding commented on PIG-1333:
---
Thanks for reviewing the patch. My comments are inline.
bq. (1) General comment. This patch is very large and combines multiple
different issues that could have been separated into multiple patches to make
it easier to review and test
In this patch JobStats and OutputStats replaced the HJob as the return value of
execution of a pig script. This results in several deprecated/removed classes.
I also go through the unit tests to replace most of the deprecated methods used
there and this causes modification of many test classes.
bq. (2) We are missing script level feature collection. (I see the one at job
level.) For each script, we want to collect overall script features such as
different operators: join, order by, etc., is it a multiquery, does it have
UDF. Also, we would want to know if combiner was used and whether the script
spilled but maybe both of those can be at the job level.
I'll add the script level features, in addition to the job level feature. The
value of job level feature (pig.job.feature is the key in job xml) will be a
string, but the value of script level feature (pig.script.features) will be a
bit set represented by a long.
bq. (3) We need to add separate comment to the JIRA marked as documentation
that describes PigRunner since it is a new interface that we need to include in
0.8.0 documentation.
Will do.
bq. (4) MapReduceLauncher. Why was exception handling and temp store handling
code removed?
Exception handling is still there. Handling of temp stores for failed jobs is
moved to JobStats.
bq. (5) OutputStats assumes that location is a path which might not be true for
non-file stores.
It isn't necessary to use path in OutputStats (just trying to get a more
readable short name for location). I'll move it.
bq. (6) ScriptState: There are maps/hashes optimized for enums
(http://java.sun.com/j2se/1.5.0/docs/api/java/util/EnumMap.html)
Dmitriy also commented on this. I'll modified the usage of emums based on the
comments.
bq. (7) Why JobStats is derived from an operator?
The JobGraph of the script is a derived class from BaseOperatorPlan (in
experimental package) whose elements are Operators. JobStats is derived from
Operator in order to use the methods on the base class.
bq. (8) Why did JOB_NAME_PREFIX got removed from PigContext?
JOB_NAME_PREFIX is still there. What is removed is the private member 'jobName'
which isn't used in the class and whose getter and setter had been removed long
time ago.
bq. (9) Why do we need to synchronize getTemporaryFile?
I'll remove the synchronize on this method. It was copied from the
corresponding deprecated method.
API interface to Pig
Key: PIG-1333
URL: https://issues.apache.org/jira/browse/PIG-1333
Project: Pig
Issue Type: Improvement
Reporter: Olga Natkovich
Assignee: Richard Ding
Fix For: 0.8.0
Attachments: PIG-1333.patch
It would be nice to make Pig more friendly for applications like workflow
that would be executing pig scripts on user behalf.
Currently, they would have to use pig command line to execute the code;
however, this has limitation on the kind of output that would be delivered.
For instance, it is hard to produce error information that is easy to use
programatically or collect statistics.
The proposal is to create a class that mimics the behavior of the Main but
gives users a status object back. The the main code of pig would look
somethig like:
public static void main(String args[])
{
PigStatus ps = PigMain.exec(args);
exit (PigStatus.rc);
}
We need to define the following:
- Content of PigStatus. It should at least include
* return code
* error string
* exception
* statistics
- A way to propagate the status class through pig code
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-928) UDFs in scripting languages
[
https://issues.apache.org/jira/browse/PIG-928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12877667#action_12877667
]
Aniket Mokashi commented on PIG-928:
I support above comment.
Also, in favor of not breaking old code. I think, we should avoid introducing
new keywords.
In the above proposal, by adding python as a lang-keyword I meant to hide
extensibility of ScriptEngine interface by natively supporting python. If we
have to allow users add support for other languages. we need to allow using
org.apache.pig.scripting.jython.JythonScriptEngine. But this will need us to
document the scriptengine interface.
Following seems to be more suitable choice. Comments?
{code}
-- register all UDFs inside test.py using custom (or builtin) ScriptEngine
register 'test.py' using org.apache.pig.scripting.jython.JythonScriptEngine
ship ('1.py', '2.py');
-- namespace? test.helloworld?
b = foreach a generate helloworld(a.$0), complex(a.$1);
-- register helloworld UDF as hello using JythonScriptEngine
define hello using org.apache.pig.scripting.jython.JythonScriptEngine from
'test.py'#helloworld ship ('1.py', '2.py');
b = foreach a generate helloworld(a.$0);
{code}
Also, register scalascript.jar would not be necessary if
getStandardScriptJarPath() returns the path of the jar.
UDFs in scripting languages
---
Key: PIG-928
URL: https://issues.apache.org/jira/browse/PIG-928
Project: Pig
Issue Type: New Feature
Reporter: Alan Gates
Assignee: Aniket Mokashi
Fix For: 0.8.0
Attachments: calltrace.png, package.zip, pig-greek.tgz,
pig.scripting.patch.arnab, pyg.tgz, RegisterPythonUDF2.patch,
RegisterScriptUDFDefineParse.patch, scripting.tgz, scripting.tgz, test.zip
It should be possible to write UDFs in scripting languages such as python,
ruby, etc. This frees users from needing to compile Java, generate a jar,
etc. It also opens Pig to programmers who prefer scripting languages over
Java.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1405) Need to move many standard functions from piggybank into Pig
[ https://issues.apache.org/jira/browse/PIG-1405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aniket Mokashi updated PIG-1405: Attachment: StandardUDFtoPig.patch Initial patch.. ToDo- Check for documentation errors Need to move many standard functions from piggybank into Pig Key: PIG-1405 URL: https://issues.apache.org/jira/browse/PIG-1405 Project: Pig Issue Type: Improvement Reporter: Alan Gates Assignee: Aniket Mokashi Fix For: 0.8.0 Attachments: StandardUDFtoPig.patch There are currently a number of functions in Piggybank that represent features commonly supported by languages and database engines. We need to decide which of these Pig should support as built in functions and put them in org.apache.pig.builtin. This will also mean adding unit tests and javadocs for some UDFs. The existing classes will be left in Piggybank for some time for backward compatibility. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1444) [Zebra] Zebra build should have a test-smoke target
[ https://issues.apache.org/jira/browse/PIG-1444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12877687#action_12877687 ] Yan Zhou commented on PIG-1444: --- Hudson server appears to be hanging. Following is the result from internal run: [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 1 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [Zebra] Zebra build should have a test-smoke target --- Key: PIG-1444 URL: https://issues.apache.org/jira/browse/PIG-1444 Project: Pig Issue Type: Task Components: build Affects Versions: 0.8.0 Reporter: Gaurav Jain Priority: Minor Fix For: 0.8.0 Attachments: PIG-1444.patch Zebra build should have a test-smoke target that should atleast use minicluster for its test-cases -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
