[jira] Commented: (PIG-1085) Pass JobConf and UDF specific configuration information to UDFs
[ https://issues.apache.org/jira/browse/PIG-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12776897#action_12776897 ] Hadoop QA commented on PIG-1085: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12424686/udfconf-2.patch against trunk revision 835189. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 12 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 204 javac compiler warnings (more than the trunk's current 199 warnings). +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/150/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/150/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/150/console This message is automatically generated. > Pass JobConf and UDF specific configuration information to UDFs > --- > > Key: PIG-1085 > URL: https://issues.apache.org/jira/browse/PIG-1085 > Project: Pig > Issue Type: New Feature > Components: impl >Reporter: Alan Gates >Assignee: Alan Gates > Attachments: udfconf-2.patch, udfconf.patch > > > Users have long asked for a way to get the JobConf structure in their UDFs. > It would also be nice to have a way to pass properties between the front end > and back end so that UDFs can store state during parse time and use it at > runtime. > This patch does part of what is proposed in PIG-602, but not all of it. It > does not provide a way to give user specified configuration files to UDFs. > So I will mark 602 as depending on this bug, but it isn't a duplicate. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-979) Acummulator Interface for UDFs
[ https://issues.apache.org/jira/browse/PIG-979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12776960#action_12776960 ] Hadoop QA commented on PIG-979: --- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12424621/PIG-979.patch against trunk revision 835284. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. -1 release audit. The applied patch generated 350 release audit warnings (more than the trunk's current 318 warnings). +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/151/testReport/ Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/151/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/151/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/151/console This message is automatically generated. > Acummulator Interface for UDFs > -- > > Key: PIG-979 > URL: https://issues.apache.org/jira/browse/PIG-979 > Project: Pig > Issue Type: New Feature >Affects Versions: 0.4.0 >Reporter: Alan Gates >Assignee: Ying He > Fix For: 0.6.0 > > Attachments: PIG-979.patch, PIG-979.patch > > > Add an accumulator interface for UDFs that would allow them to take a set > number of records at a time instead of the entire bag. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-979) Acummulator Interface for UDFs
[ https://issues.apache.org/jira/browse/PIG-979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12777090#action_12777090 ] Ying He commented on PIG-979: - the release audit warnings are all from html files. > Acummulator Interface for UDFs > -- > > Key: PIG-979 > URL: https://issues.apache.org/jira/browse/PIG-979 > Project: Pig > Issue Type: New Feature >Affects Versions: 0.4.0 >Reporter: Alan Gates >Assignee: Ying He > Fix For: 0.6.0 > > Attachments: PIG-979.patch, PIG-979.patch > > > Add an accumulator interface for UDFs that would allow them to take a set > number of records at a time instead of the entire bag. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1089) Pig 0.6.0 Documentation
[ https://issues.apache.org/jira/browse/PIG-1089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Corinne Chandel updated PIG-1089: - Attachment: Pig-6-Beta-2.patch Pig 0.6 docs - beta patch #2 > DONE One thing that I think we need to clarify is that only left outer join is supported in the case of replicated outer join. > DONE Also, exec and run commands are misclasified as file commands in grant. They should be moved to the utility commands and they are not depricated > Pig 0.6.0 Documentation > --- > > Key: PIG-1089 > URL: https://issues.apache.org/jira/browse/PIG-1089 > Project: Pig > Issue Type: Task > Components: documentation >Affects Versions: 0.6.0 >Reporter: Corinne Chandel >Assignee: Corinne Chandel >Priority: Blocker > Fix For: 0.6.0 > > Attachments: Pig-6-Beta-2.patch, Pig-6-Beta.patch > > > Pig 0.6.0 documentation: > > Ability to use Hadoop dfs commands from Pig > > Replicated left outer join > > Skewed outer join > > Map-side group > > Accumulate Interface for UDFs > > Improved Memory Mgt > > Integration with Zebra -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1060) MultiQuery optimization throws error for multi-level splits
[ https://issues.apache.org/jira/browse/PIG-1060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12777096#action_12777096 ] Richard Ding commented on PIG-1060: --- The release audit warnings are all from html files. > MultiQuery optimization throws error for multi-level splits > --- > > Key: PIG-1060 > URL: https://issues.apache.org/jira/browse/PIG-1060 > Project: Pig > Issue Type: Bug >Affects Versions: 0.5.0 >Reporter: Ankur >Assignee: Richard Ding > Attachments: PIG-1060.patch > > > Consider the following scenario :- > 1. Multi-level splits in the map plan. > 2. Each split branch further progressing across a local-global rearrange. > 3. Output of each of these finally merged via a UNION. > MultiQuery optimizer throws the following error in such a case: > "ERROR 2146: Internal Error. Inconsistency in key index found during > optimization." -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-979) Acummulator Interface for UDFs
[ https://issues.apache.org/jira/browse/PIG-979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated PIG-979: --- Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) Patch committed, thanks Ying, Alan! > Acummulator Interface for UDFs > -- > > Key: PIG-979 > URL: https://issues.apache.org/jira/browse/PIG-979 > Project: Pig > Issue Type: New Feature >Affects Versions: 0.4.0 >Reporter: Alan Gates >Assignee: Ying He > Fix For: 0.6.0 > > Attachments: PIG-979.patch, PIG-979.patch > > > Add an accumulator interface for UDFs that would allow them to take a set > number of records at a time instead of the entire bag. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1089) Pig 0.6.0 Documentation
[ https://issues.apache.org/jira/browse/PIG-1089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12777113#action_12777113 ] Hadoop QA commented on PIG-1089: +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12424687/Pig-6-Beta.patch against trunk revision 835284. +1 @author. The patch does not contain any @author tags. +0 tests included. The patch appears to be a documentation patch that doesn't require tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/152/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/152/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/152/console This message is automatically generated. > Pig 0.6.0 Documentation > --- > > Key: PIG-1089 > URL: https://issues.apache.org/jira/browse/PIG-1089 > Project: Pig > Issue Type: Task > Components: documentation >Affects Versions: 0.6.0 >Reporter: Corinne Chandel >Assignee: Corinne Chandel >Priority: Blocker > Fix For: 0.6.0 > > Attachments: Pig-6-Beta-2.patch, Pig-6-Beta.patch > > > Pig 0.6.0 documentation: > > Ability to use Hadoop dfs commands from Pig > > Replicated left outer join > > Skewed outer join > > Map-side group > > Accumulate Interface for UDFs > > Improved Memory Mgt > > Integration with Zebra -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
RE: [VOTE] Branch for Pig 0.6.0 release
We have 4 +1 votes for this and no -1s. I am about to create 0.6.0 branch. Please, hold off any commits to the trunk until I am done. Thanks, Olga -Original Message- From: Alan Gates [mailto:ga...@yahoo-inc.com] Sent: Monday, November 09, 2009 12:43 PM To: pig-dev@hadoop.apache.org Subject: Re: [VOTE] Branch for Pig 0.6.0 release +1. In addition to the new features we've added, our change to use Hadoop's LineRecordReader brought Pig to parity with Hadoop in the PigMix tests, about a 30% average performance improvement. This should be huge for our users. Alan. On Nov 9, 2009, at 12:26 PM, Olga Natkovich wrote: > Hi, > > > > I would like to propose to branch for Pig 0.6.0 release with the > intent > to have a release before the end of the year. We have done a lot of > work > since branching for Pig 0.5.0 that we would like to share with users. > This includes changing how bags are spilled onto disk (PIG-975, > PIG-1037), skewed and fragment-replicated outer join plus many other > performance improvements and bug fixes. > > > > Please vote by Thursday. > > > > Thanks, > > > > Olga > > > > >
RE: [VOTE] Branch for Pig 0.6.0 release
I created the branch. Trunk is open for commits. Olga -Original Message- From: Olga Natkovich Sent: Thursday, November 12, 2009 10:46 AM To: 'pig-dev@hadoop.apache.org' Subject: RE: [VOTE] Branch for Pig 0.6.0 release We have 4 +1 votes for this and no -1s. I am about to create 0.6.0 branch. Please, hold off any commits to the trunk until I am done. Thanks, Olga -Original Message- From: Alan Gates [mailto:ga...@yahoo-inc.com] Sent: Monday, November 09, 2009 12:43 PM To: pig-dev@hadoop.apache.org Subject: Re: [VOTE] Branch for Pig 0.6.0 release +1. In addition to the new features we've added, our change to use Hadoop's LineRecordReader brought Pig to parity with Hadoop in the PigMix tests, about a 30% average performance improvement. This should be huge for our users. Alan. On Nov 9, 2009, at 12:26 PM, Olga Natkovich wrote: > Hi, > > > > I would like to propose to branch for Pig 0.6.0 release with the > intent > to have a release before the end of the year. We have done a lot of > work > since branching for Pig 0.5.0 that we would like to share with users. > This includes changing how bags are spilled onto disk (PIG-975, > PIG-1037), skewed and fragment-replicated outer join plus many other > performance improvements and bug fixes. > > > > Please vote by Thursday. > > > > Thanks, > > > > Olga > > > > >
[jira] Commented: (PIG-966) Proposed rework for LoadFunc, StoreFunc, and Slice/r interfaces
[ https://issues.apache.org/jira/browse/PIG-966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12777130#action_12777130 ] Richard Ding commented on PIG-966: -- Since two methods on LoadFunc interface have default implementation (massageFilename for relativeToAbsolutePath, Utf8StorageConverter for getLoadCaster) that many implementers of LoadFunc can use, we propose to change LoadFunc to an abstract class. Use of abstract class also allows us to make backward-compatible changes to the API in later releases. One drawback is that the corresponding StoreFunc can't be an abstract class, since PigStorage now implements both interfaces and Java doesn't allow multiple inheritance. But we think it worthwhile to make the change. Another proposal is to make both LoadFunc and StoreFunc generic types. Both are using Hadoop's generic types (e.g. InputFormat, OutputFormat, RecordReader, RecordWriter) in their methods . Change to generic types will provide compile-time type checking, reduce Javac warnings, and force the implementers to specify the input/output types. Any comments? > Proposed rework for LoadFunc, StoreFunc, and Slice/r interfaces > --- > > Key: PIG-966 > URL: https://issues.apache.org/jira/browse/PIG-966 > Project: Pig > Issue Type: Improvement > Components: impl >Reporter: Alan Gates >Assignee: Alan Gates > > I propose that we rework the LoadFunc, StoreFunc, and Slice/r interfaces > significantly. See http://wiki.apache.org/pig/LoadStoreRedesignProposal for > full details -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1064) Behvaiour of COGROUP with and without schema when using "*" operator
[
https://issues.apache.org/jira/browse/PIG-1064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pradeep Kamath updated PIG-1064:
Attachment: PIG-1064-2.patch
Attached patch address unit test failures - the failures were in other tests
wherein cogroup * without schema would be valid in the front end. With the
changes in the patch, this is no longer the case. I have removed these
testcases and in one case retained it since it tests with different loadfuncs.
> Behvaiour of COGROUP with and without schema when using "*" operator
>
>
> Key: PIG-1064
> URL: https://issues.apache.org/jira/browse/PIG-1064
> Project: Pig
> Issue Type: Bug
> Components: impl
>Affects Versions: 0.6.0
>Reporter: Viraj Bhat
>Assignee: Pradeep Kamath
> Fix For: 0.6.0
>
> Attachments: PIG-1064-2.patch, PIG-1064.patch
>
>
> I have 2 tab separated files, "1.txt" and "2.txt"
> $ cat 1.txt
>
> 1 2
> 2 3
>
> $ cat 2.txt
> 1 2
> 2 3
> I use COGROUP feature of Pig in the following way:
> $java -cp pig.jar:$HADOOP_HOME org.apache.pig.Main
> {code}
> grunt> A = load '1.txt';
> grunt> B = load '2.txt' as (b0, b1);
> grunt> C = cogroup A by *, B by *;
> {code}
> 2009-10-29 12:46:04,150 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR
> 1012: Each COGroup input has to have the same number of inner plans
> Details at logfile: pig_1256845224752.log
> ==
> If I reverse, the order of the schema's
> {code}
> grunt> A = load '1.txt' as (a0, a1);
> grunt> B = load '2.txt';
> grunt> C = cogroup A by *, B by *;
> {code}
> 2009-10-29 12:49:27,869 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR
> 1013: Grouping attributes can either be star (*) or a list of expressions,
> but not both.
> Details at logfile: pig_1256845224752.log
> ==
> Now running without schema??
> {code}
> grunt> A = load '1.txt';
> grunt> B = load '2.txt';
> grunt> C = cogroup A by *, B by *;
> grunt> dump C;
> {code}
> 2009-10-29 12:55:37,202 [main] INFO
> org.apache.pig.backend.local.executionengine.LocalPigLauncher - Successfully
> stored result in: "file:/tmp/temp-319926700/tmp-1990275961"
> 2009-10-29 12:55:37,202 [main] INFO
> org.apache.pig.backend.local.executionengine.LocalPigLauncher - Records
> written : 2
> 2009-10-29 12:55:37,202 [main] INFO
> org.apache.pig.backend.local.executionengine.LocalPigLauncher - Bytes written
> : 154
> 2009-10-29 12:55:37,202 [main] INFO
> org.apache.pig.backend.local.executionengine.LocalPigLauncher - 100% complete!
> 2009-10-29 12:55:37,202 [main] INFO
> org.apache.pig.backend.local.executionengine.LocalPigLauncher - Success!!
> ((1,2),{(1,2)},{(1,2)})
> ((2,3),{(2,3)},{(2,3)})
> ==
> Is this a bug or a feature?
> Viraj
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1064) Behvaiour of COGROUP with and without schema when using "*" operator
[
https://issues.apache.org/jira/browse/PIG-1064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pradeep Kamath updated PIG-1064:
Status: Open (was: Patch Available)
> Behvaiour of COGROUP with and without schema when using "*" operator
>
>
> Key: PIG-1064
> URL: https://issues.apache.org/jira/browse/PIG-1064
> Project: Pig
> Issue Type: Bug
> Components: impl
>Affects Versions: 0.6.0
>Reporter: Viraj Bhat
>Assignee: Pradeep Kamath
> Fix For: 0.6.0
>
> Attachments: PIG-1064-2.patch, PIG-1064.patch
>
>
> I have 2 tab separated files, "1.txt" and "2.txt"
> $ cat 1.txt
>
> 1 2
> 2 3
>
> $ cat 2.txt
> 1 2
> 2 3
> I use COGROUP feature of Pig in the following way:
> $java -cp pig.jar:$HADOOP_HOME org.apache.pig.Main
> {code}
> grunt> A = load '1.txt';
> grunt> B = load '2.txt' as (b0, b1);
> grunt> C = cogroup A by *, B by *;
> {code}
> 2009-10-29 12:46:04,150 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR
> 1012: Each COGroup input has to have the same number of inner plans
> Details at logfile: pig_1256845224752.log
> ==
> If I reverse, the order of the schema's
> {code}
> grunt> A = load '1.txt' as (a0, a1);
> grunt> B = load '2.txt';
> grunt> C = cogroup A by *, B by *;
> {code}
> 2009-10-29 12:49:27,869 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR
> 1013: Grouping attributes can either be star (*) or a list of expressions,
> but not both.
> Details at logfile: pig_1256845224752.log
> ==
> Now running without schema??
> {code}
> grunt> A = load '1.txt';
> grunt> B = load '2.txt';
> grunt> C = cogroup A by *, B by *;
> grunt> dump C;
> {code}
> 2009-10-29 12:55:37,202 [main] INFO
> org.apache.pig.backend.local.executionengine.LocalPigLauncher - Successfully
> stored result in: "file:/tmp/temp-319926700/tmp-1990275961"
> 2009-10-29 12:55:37,202 [main] INFO
> org.apache.pig.backend.local.executionengine.LocalPigLauncher - Records
> written : 2
> 2009-10-29 12:55:37,202 [main] INFO
> org.apache.pig.backend.local.executionengine.LocalPigLauncher - Bytes written
> : 154
> 2009-10-29 12:55:37,202 [main] INFO
> org.apache.pig.backend.local.executionengine.LocalPigLauncher - 100% complete!
> 2009-10-29 12:55:37,202 [main] INFO
> org.apache.pig.backend.local.executionengine.LocalPigLauncher - Success!!
> ((1,2),{(1,2)},{(1,2)})
> ((2,3),{(2,3)},{(2,3)})
> ==
> Is this a bug or a feature?
> Viraj
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1064) Behvaiour of COGROUP with and without schema when using "*" operator
[
https://issues.apache.org/jira/browse/PIG-1064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pradeep Kamath updated PIG-1064:
Status: Patch Available (was: Open)
> Behvaiour of COGROUP with and without schema when using "*" operator
>
>
> Key: PIG-1064
> URL: https://issues.apache.org/jira/browse/PIG-1064
> Project: Pig
> Issue Type: Bug
> Components: impl
>Affects Versions: 0.6.0
>Reporter: Viraj Bhat
>Assignee: Pradeep Kamath
> Fix For: 0.6.0
>
> Attachments: PIG-1064-2.patch, PIG-1064.patch
>
>
> I have 2 tab separated files, "1.txt" and "2.txt"
> $ cat 1.txt
>
> 1 2
> 2 3
>
> $ cat 2.txt
> 1 2
> 2 3
> I use COGROUP feature of Pig in the following way:
> $java -cp pig.jar:$HADOOP_HOME org.apache.pig.Main
> {code}
> grunt> A = load '1.txt';
> grunt> B = load '2.txt' as (b0, b1);
> grunt> C = cogroup A by *, B by *;
> {code}
> 2009-10-29 12:46:04,150 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR
> 1012: Each COGroup input has to have the same number of inner plans
> Details at logfile: pig_1256845224752.log
> ==
> If I reverse, the order of the schema's
> {code}
> grunt> A = load '1.txt' as (a0, a1);
> grunt> B = load '2.txt';
> grunt> C = cogroup A by *, B by *;
> {code}
> 2009-10-29 12:49:27,869 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR
> 1013: Grouping attributes can either be star (*) or a list of expressions,
> but not both.
> Details at logfile: pig_1256845224752.log
> ==
> Now running without schema??
> {code}
> grunt> A = load '1.txt';
> grunt> B = load '2.txt';
> grunt> C = cogroup A by *, B by *;
> grunt> dump C;
> {code}
> 2009-10-29 12:55:37,202 [main] INFO
> org.apache.pig.backend.local.executionengine.LocalPigLauncher - Successfully
> stored result in: "file:/tmp/temp-319926700/tmp-1990275961"
> 2009-10-29 12:55:37,202 [main] INFO
> org.apache.pig.backend.local.executionengine.LocalPigLauncher - Records
> written : 2
> 2009-10-29 12:55:37,202 [main] INFO
> org.apache.pig.backend.local.executionengine.LocalPigLauncher - Bytes written
> : 154
> 2009-10-29 12:55:37,202 [main] INFO
> org.apache.pig.backend.local.executionengine.LocalPigLauncher - 100% complete!
> 2009-10-29 12:55:37,202 [main] INFO
> org.apache.pig.backend.local.executionengine.LocalPigLauncher - Success!!
> ((1,2),{(1,2)},{(1,2)})
> ((2,3),{(2,3)},{(2,3)})
> ==
> Is this a bug or a feature?
> Viraj
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Created: (PIG-1090) Update sources to reflect recent changes in load-store interfaces
Update sources to reflect recent changes in load-store interfaces - Key: PIG-1090 URL: https://issues.apache.org/jira/browse/PIG-1090 Project: Pig Issue Type: Sub-task Reporter: Pradeep Kamath Assignee: Pradeep Kamath -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1090) Update sources to reflect recent changes in load-store interfaces
[ https://issues.apache.org/jira/browse/PIG-1090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pradeep Kamath updated PIG-1090: Description: There have been some changes (as recorded in the Changes Section, Nov 2 2009 sub section of http://wiki.apache.org/pig/LoadStoreRedesignProposal) in the load/store interfaces - this jira is to track the task of making those changes under src. Changes under test will be addresses in a different jira. > Update sources to reflect recent changes in load-store interfaces > - > > Key: PIG-1090 > URL: https://issues.apache.org/jira/browse/PIG-1090 > Project: Pig > Issue Type: Sub-task >Reporter: Pradeep Kamath >Assignee: Pradeep Kamath > > There have been some changes (as recorded in the Changes Section, Nov 2 2009 > sub section of http://wiki.apache.org/pig/LoadStoreRedesignProposal) in the > load/store interfaces - this jira is to track the task of making those > changes under src. Changes under test will be addresses in a different jira. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (PIG-879) Pig should provide a way for input location string in load statement to be passed as-is to the Loader
[ https://issues.apache.org/jira/browse/PIG-879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Richard Ding reassigned PIG-879: Assignee: Richard Ding > Pig should provide a way for input location string in load statement to be > passed as-is to the Loader > - > > Key: PIG-879 > URL: https://issues.apache.org/jira/browse/PIG-879 > Project: Pig > Issue Type: Bug >Affects Versions: 0.3.0 >Reporter: Pradeep Kamath >Assignee: Richard Ding > > Due to multiquery optimization, Pig always converts the filenames to > absolute URIs (see > http://wiki.apache.org/pig/PigMultiQueryPerformanceSpecification - section > about Incompatible Changes - Path Names and Schemes). This is necessary since > the script may have "cd .." statements between load or store statements and > if the load statements have relative paths, we would need to convert to > absolute paths to know where to load/store from. To do this > QueryParser.massageFilename() has the code below[1] which basically gives the > fully qualified hdfs path > > However the issue with this approach is that if the filename string is > something like > "hdfs://localhost.localdomain:39125/user/bla/1,hdfs://localhost.localdomain:39125/user/bla/2", > the code below[1] actually translates this to > hdfs://localhost.localdomain:38264/user/bla/1,hdfs://localhost.localdomain:38264/user/bla/2 > and throws an exception that it is an incorrect path. > > Some loaders may want to interpret the filenames (the input location string > in the load statement) in any way they wish and may want Pig to not make > absolute paths out of them. > > There are a few options to address this: > 1)A command line switch to indicate to Pig that pathnames in the script > are all absolute and hence Pig should not alter them and pass them as-is to > Loaders and Storers. > 2)A keyword in the load and store statements to indicate the same intent > to pig > 3)A property which users can supply on cmdline or in pig.properties to > indicate the same intent. > 4)A method in LoadFunc - relativeToAbsolutePath(String filename, String > curDir) which does the conversion to absolute - this way Loader can chose to > implement it as a noop. > Thoughts? > -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1064) Behvaiour of COGROUP with and without schema when using "*" operator
[
https://issues.apache.org/jira/browse/PIG-1064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12777265#action_12777265
]
Hadoop QA commented on PIG-1064:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12424755/PIG-1064-2.patch
against trunk revision 835499.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 12 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/153/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/153/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/153/console
This message is automatically generated.
> Behvaiour of COGROUP with and without schema when using "*" operator
>
>
> Key: PIG-1064
> URL: https://issues.apache.org/jira/browse/PIG-1064
> Project: Pig
> Issue Type: Bug
> Components: impl
>Affects Versions: 0.6.0
>Reporter: Viraj Bhat
>Assignee: Pradeep Kamath
> Fix For: 0.6.0
>
> Attachments: PIG-1064-2.patch, PIG-1064.patch
>
>
> I have 2 tab separated files, "1.txt" and "2.txt"
> $ cat 1.txt
>
> 1 2
> 2 3
>
> $ cat 2.txt
> 1 2
> 2 3
> I use COGROUP feature of Pig in the following way:
> $java -cp pig.jar:$HADOOP_HOME org.apache.pig.Main
> {code}
> grunt> A = load '1.txt';
> grunt> B = load '2.txt' as (b0, b1);
> grunt> C = cogroup A by *, B by *;
> {code}
> 2009-10-29 12:46:04,150 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR
> 1012: Each COGroup input has to have the same number of inner plans
> Details at logfile: pig_1256845224752.log
> ==
> If I reverse, the order of the schema's
> {code}
> grunt> A = load '1.txt' as (a0, a1);
> grunt> B = load '2.txt';
> grunt> C = cogroup A by *, B by *;
> {code}
> 2009-10-29 12:49:27,869 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR
> 1013: Grouping attributes can either be star (*) or a list of expressions,
> but not both.
> Details at logfile: pig_1256845224752.log
> ==
> Now running without schema??
> {code}
> grunt> A = load '1.txt';
> grunt> B = load '2.txt';
> grunt> C = cogroup A by *, B by *;
> grunt> dump C;
> {code}
> 2009-10-29 12:55:37,202 [main] INFO
> org.apache.pig.backend.local.executionengine.LocalPigLauncher - Successfully
> stored result in: "file:/tmp/temp-319926700/tmp-1990275961"
> 2009-10-29 12:55:37,202 [main] INFO
> org.apache.pig.backend.local.executionengine.LocalPigLauncher - Records
> written : 2
> 2009-10-29 12:55:37,202 [main] INFO
> org.apache.pig.backend.local.executionengine.LocalPigLauncher - Bytes written
> : 154
> 2009-10-29 12:55:37,202 [main] INFO
> org.apache.pig.backend.local.executionengine.LocalPigLauncher - 100% complete!
> 2009-10-29 12:55:37,202 [main] INFO
> org.apache.pig.backend.local.executionengine.LocalPigLauncher - Success!!
> ((1,2),{(1,2)},{(1,2)})
> ((2,3),{(2,3)},{(2,3)})
> ==
> Is this a bug or a feature?
> Viraj
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-598) Parameter substitution ($PARAMETER) should not be performed in comments
[
https://issues.apache.org/jira/browse/PIG-598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12777268#action_12777268
]
Olga Natkovich commented on PIG-598:
The code looks reasonable.
When I tried to run tests with comments (both kind) in the middle of the line -
they are not recognized
> Parameter substitution ($PARAMETER) should not be performed in comments
> ---
>
> Key: PIG-598
> URL: https://issues.apache.org/jira/browse/PIG-598
> Project: Pig
> Issue Type: Bug
> Components: impl
>Affects Versions: 0.2.0
>Reporter: David Ciemiewicz
>Assignee: Thejas M Nair
> Attachments: PIG-598.patch
>
>
> Compiling the following code example will generate an error that
> $NOT_A_PARAMETER is an Undefined Parameter.
> This is problematic as sometimes you want to comment out parts of your code,
> including parameters so that you don't have to define them.
> This I think it would be really good if parameter substitution was not
> performed in comments.
> {code}
> -- $NOT_A_PARAMETER
> {code}
> {code}
> -bash-3.00$ pig -exectype local -latest comment.pig
> USING: /grid/0/gs/pig/current
> java.lang.RuntimeException: Undefined parameter : NOT_A_PARAMETER
> at
> org.apache.pig.tools.parameters.PreprocessorContext.substitute(PreprocessorContext.java:221)
> at
> org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor.parsePigFile(ParameterSubstitutionPreprocessor.java:106)
> at
> org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor.genSubstitutedFile(ParameterSubstitutionPreprocessor.java:86)
> at org.apache.pig.Main.runParamPreprocessor(Main.java:394)
> at org.apache.pig.Main.main(Main.java:296)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1090) Update sources to reflect recent changes in load-store interfaces
[
https://issues.apache.org/jira/browse/PIG-1090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pradeep Kamath updated PIG-1090:
Attachment: PIG-1090.patch
Attached patch to make changes under src directory of the code base to comply
with interface changes. Changes have not been made under test since cleaning up
tests will be undertaken in a separate jira.
Here are results from a local 'test-patch' run:
{noformat}
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] -1 tests included. The patch doesn't appear to include any new
or modified tests.
[exec] Please justify why no tests are needed for
this patch.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning
messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number
of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs
warnings.
[exec]
[exec] -1 release audit. The applied patch generated 331 release
audit warnings (more than the trunk's current 330 warnings).
[exec]
[exec]
[exec]
{noformat}
* Since this patch only makes changes to comply with recent interface changes
there are no new tests added.
* The release audit warning is due to a difference in a html file:
{noformat}
cat /homes/pradeepk/tmp/releaseAuditDiffWarnings.txt
73d72
< [java] !?
/tmp/PIG-1090/load-store-redesign/build/pig-0.6.0-dev/docs/jdiff/changes/org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POCast.html
{noformat}
> Update sources to reflect recent changes in load-store interfaces
> -
>
> Key: PIG-1090
> URL: https://issues.apache.org/jira/browse/PIG-1090
> Project: Pig
> Issue Type: Sub-task
>Reporter: Pradeep Kamath
>Assignee: Pradeep Kamath
> Attachments: PIG-1090.patch
>
>
> There have been some changes (as recorded in the Changes Section, Nov 2 2009
> sub section of http://wiki.apache.org/pig/LoadStoreRedesignProposal) in the
> load/store interfaces - this jira is to track the task of making those
> changes under src. Changes under test will be addresses in a different jira.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1090) Update sources to reflect recent changes in load-store interfaces
[ https://issues.apache.org/jira/browse/PIG-1090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pradeep Kamath updated PIG-1090: Status: Patch Available (was: Open) > Update sources to reflect recent changes in load-store interfaces > - > > Key: PIG-1090 > URL: https://issues.apache.org/jira/browse/PIG-1090 > Project: Pig > Issue Type: Sub-task >Reporter: Pradeep Kamath >Assignee: Pradeep Kamath > Attachments: PIG-1090.patch > > > There have been some changes (as recorded in the Changes Section, Nov 2 2009 > sub section of http://wiki.apache.org/pig/LoadStoreRedesignProposal) in the > load/store interfaces - this jira is to track the task of making those > changes under src. Changes under test will be addresses in a different jira. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1090) Update sources to reflect recent changes in load-store interfaces
[ https://issues.apache.org/jira/browse/PIG-1090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12777293#action_12777293 ] Hadoop QA commented on PIG-1090: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12424793/PIG-1090.patch against trunk revision 835499. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/47/console This message is automatically generated. > Update sources to reflect recent changes in load-store interfaces > - > > Key: PIG-1090 > URL: https://issues.apache.org/jira/browse/PIG-1090 > Project: Pig > Issue Type: Sub-task >Reporter: Pradeep Kamath >Assignee: Pradeep Kamath > Attachments: PIG-1090.patch > > > There have been some changes (as recorded in the Changes Section, Nov 2 2009 > sub section of http://wiki.apache.org/pig/LoadStoreRedesignProposal) in the > load/store interfaces - this jira is to track the task of making those > changes under src. Changes under test will be addresses in a different jira. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1064) Behvaiour of COGROUP with and without schema when using "*" operator
[
https://issues.apache.org/jira/browse/PIG-1064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pradeep Kamath updated PIG-1064:
Status: Patch Available (was: Open)
> Behvaiour of COGROUP with and without schema when using "*" operator
>
>
> Key: PIG-1064
> URL: https://issues.apache.org/jira/browse/PIG-1064
> Project: Pig
> Issue Type: Bug
> Components: impl
>Affects Versions: 0.6.0
>Reporter: Viraj Bhat
>Assignee: Pradeep Kamath
> Fix For: 0.6.0
>
> Attachments: PIG-1064-2.patch, PIG-1064-3.patch, PIG-1064.patch
>
>
> I have 2 tab separated files, "1.txt" and "2.txt"
> $ cat 1.txt
>
> 1 2
> 2 3
>
> $ cat 2.txt
> 1 2
> 2 3
> I use COGROUP feature of Pig in the following way:
> $java -cp pig.jar:$HADOOP_HOME org.apache.pig.Main
> {code}
> grunt> A = load '1.txt';
> grunt> B = load '2.txt' as (b0, b1);
> grunt> C = cogroup A by *, B by *;
> {code}
> 2009-10-29 12:46:04,150 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR
> 1012: Each COGroup input has to have the same number of inner plans
> Details at logfile: pig_1256845224752.log
> ==
> If I reverse, the order of the schema's
> {code}
> grunt> A = load '1.txt' as (a0, a1);
> grunt> B = load '2.txt';
> grunt> C = cogroup A by *, B by *;
> {code}
> 2009-10-29 12:49:27,869 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR
> 1013: Grouping attributes can either be star (*) or a list of expressions,
> but not both.
> Details at logfile: pig_1256845224752.log
> ==
> Now running without schema??
> {code}
> grunt> A = load '1.txt';
> grunt> B = load '2.txt';
> grunt> C = cogroup A by *, B by *;
> grunt> dump C;
> {code}
> 2009-10-29 12:55:37,202 [main] INFO
> org.apache.pig.backend.local.executionengine.LocalPigLauncher - Successfully
> stored result in: "file:/tmp/temp-319926700/tmp-1990275961"
> 2009-10-29 12:55:37,202 [main] INFO
> org.apache.pig.backend.local.executionengine.LocalPigLauncher - Records
> written : 2
> 2009-10-29 12:55:37,202 [main] INFO
> org.apache.pig.backend.local.executionengine.LocalPigLauncher - Bytes written
> : 154
> 2009-10-29 12:55:37,202 [main] INFO
> org.apache.pig.backend.local.executionengine.LocalPigLauncher - 100% complete!
> 2009-10-29 12:55:37,202 [main] INFO
> org.apache.pig.backend.local.executionengine.LocalPigLauncher - Success!!
> ((1,2),{(1,2)},{(1,2)})
> ((2,3),{(2,3)},{(2,3)})
> ==
> Is this a bug or a feature?
> Viraj
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1064) Behvaiour of COGROUP with and without schema when using "*" operator
[
https://issues.apache.org/jira/browse/PIG-1064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pradeep Kamath updated PIG-1064:
Status: Open (was: Patch Available)
> Behvaiour of COGROUP with and without schema when using "*" operator
>
>
> Key: PIG-1064
> URL: https://issues.apache.org/jira/browse/PIG-1064
> Project: Pig
> Issue Type: Bug
> Components: impl
>Affects Versions: 0.6.0
>Reporter: Viraj Bhat
>Assignee: Pradeep Kamath
> Fix For: 0.6.0
>
> Attachments: PIG-1064-2.patch, PIG-1064-3.patch, PIG-1064.patch
>
>
> I have 2 tab separated files, "1.txt" and "2.txt"
> $ cat 1.txt
>
> 1 2
> 2 3
>
> $ cat 2.txt
> 1 2
> 2 3
> I use COGROUP feature of Pig in the following way:
> $java -cp pig.jar:$HADOOP_HOME org.apache.pig.Main
> {code}
> grunt> A = load '1.txt';
> grunt> B = load '2.txt' as (b0, b1);
> grunt> C = cogroup A by *, B by *;
> {code}
> 2009-10-29 12:46:04,150 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR
> 1012: Each COGroup input has to have the same number of inner plans
> Details at logfile: pig_1256845224752.log
> ==
> If I reverse, the order of the schema's
> {code}
> grunt> A = load '1.txt' as (a0, a1);
> grunt> B = load '2.txt';
> grunt> C = cogroup A by *, B by *;
> {code}
> 2009-10-29 12:49:27,869 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR
> 1013: Grouping attributes can either be star (*) or a list of expressions,
> but not both.
> Details at logfile: pig_1256845224752.log
> ==
> Now running without schema??
> {code}
> grunt> A = load '1.txt';
> grunt> B = load '2.txt';
> grunt> C = cogroup A by *, B by *;
> grunt> dump C;
> {code}
> 2009-10-29 12:55:37,202 [main] INFO
> org.apache.pig.backend.local.executionengine.LocalPigLauncher - Successfully
> stored result in: "file:/tmp/temp-319926700/tmp-1990275961"
> 2009-10-29 12:55:37,202 [main] INFO
> org.apache.pig.backend.local.executionengine.LocalPigLauncher - Records
> written : 2
> 2009-10-29 12:55:37,202 [main] INFO
> org.apache.pig.backend.local.executionengine.LocalPigLauncher - Bytes written
> : 154
> 2009-10-29 12:55:37,202 [main] INFO
> org.apache.pig.backend.local.executionengine.LocalPigLauncher - 100% complete!
> 2009-10-29 12:55:37,202 [main] INFO
> org.apache.pig.backend.local.executionengine.LocalPigLauncher - Success!!
> ((1,2),{(1,2)},{(1,2)})
> ((2,3),{(2,3)},{(2,3)})
> ==
> Is this a bug or a feature?
> Viraj
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1064) Behvaiour of COGROUP with and without schema when using "*" operator
[
https://issues.apache.org/jira/browse/PIG-1064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pradeep Kamath updated PIG-1064:
Attachment: PIG-1064-3.patch
There were a couple of new tests added by a recent patch (PIG-1038) which had
group by star and broke the tests with this patch - attached patch with fix in
the tests.
> Behvaiour of COGROUP with and without schema when using "*" operator
>
>
> Key: PIG-1064
> URL: https://issues.apache.org/jira/browse/PIG-1064
> Project: Pig
> Issue Type: Bug
> Components: impl
>Affects Versions: 0.6.0
>Reporter: Viraj Bhat
>Assignee: Pradeep Kamath
> Fix For: 0.6.0
>
> Attachments: PIG-1064-2.patch, PIG-1064-3.patch, PIG-1064.patch
>
>
> I have 2 tab separated files, "1.txt" and "2.txt"
> $ cat 1.txt
>
> 1 2
> 2 3
>
> $ cat 2.txt
> 1 2
> 2 3
> I use COGROUP feature of Pig in the following way:
> $java -cp pig.jar:$HADOOP_HOME org.apache.pig.Main
> {code}
> grunt> A = load '1.txt';
> grunt> B = load '2.txt' as (b0, b1);
> grunt> C = cogroup A by *, B by *;
> {code}
> 2009-10-29 12:46:04,150 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR
> 1012: Each COGroup input has to have the same number of inner plans
> Details at logfile: pig_1256845224752.log
> ==
> If I reverse, the order of the schema's
> {code}
> grunt> A = load '1.txt' as (a0, a1);
> grunt> B = load '2.txt';
> grunt> C = cogroup A by *, B by *;
> {code}
> 2009-10-29 12:49:27,869 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR
> 1013: Grouping attributes can either be star (*) or a list of expressions,
> but not both.
> Details at logfile: pig_1256845224752.log
> ==
> Now running without schema??
> {code}
> grunt> A = load '1.txt';
> grunt> B = load '2.txt';
> grunt> C = cogroup A by *, B by *;
> grunt> dump C;
> {code}
> 2009-10-29 12:55:37,202 [main] INFO
> org.apache.pig.backend.local.executionengine.LocalPigLauncher - Successfully
> stored result in: "file:/tmp/temp-319926700/tmp-1990275961"
> 2009-10-29 12:55:37,202 [main] INFO
> org.apache.pig.backend.local.executionengine.LocalPigLauncher - Records
> written : 2
> 2009-10-29 12:55:37,202 [main] INFO
> org.apache.pig.backend.local.executionengine.LocalPigLauncher - Bytes written
> : 154
> 2009-10-29 12:55:37,202 [main] INFO
> org.apache.pig.backend.local.executionengine.LocalPigLauncher - 100% complete!
> 2009-10-29 12:55:37,202 [main] INFO
> org.apache.pig.backend.local.executionengine.LocalPigLauncher - Success!!
> ((1,2),{(1,2)},{(1,2)})
> ((2,3),{(2,3)},{(2,3)})
> ==
> Is this a bug or a feature?
> Viraj
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1085) Pass JobConf and UDF specific configuration information to UDFs
[ https://issues.apache.org/jira/browse/PIG-1085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12777307#action_12777307 ] Daniel Dai commented on PIG-1085: - +1. Patch looks good. > Pass JobConf and UDF specific configuration information to UDFs > --- > > Key: PIG-1085 > URL: https://issues.apache.org/jira/browse/PIG-1085 > Project: Pig > Issue Type: New Feature > Components: impl >Reporter: Alan Gates >Assignee: Alan Gates > Attachments: udfconf-2.patch, udfconf.patch > > > Users have long asked for a way to get the JobConf structure in their UDFs. > It would also be nice to have a way to pass properties between the front end > and back end so that UDFs can store state during parse time and use it at > runtime. > This patch does part of what is proposed in PIG-602, but not all of it. It > does not provide a way to give user specified configuration files to UDFs. > So I will mark 602 as depending on this bug, but it isn't a duplicate. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1090) Update sources to reflect recent changes in load-store interfaces
[ https://issues.apache.org/jira/browse/PIG-1090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12777309#action_12777309 ] Pradeep Kamath commented on PIG-1090: - The above issue with Hadoop QA is because this patch is for load-store-redesign branch and not for trunk > Update sources to reflect recent changes in load-store interfaces > - > > Key: PIG-1090 > URL: https://issues.apache.org/jira/browse/PIG-1090 > Project: Pig > Issue Type: Sub-task >Reporter: Pradeep Kamath >Assignee: Pradeep Kamath > Attachments: PIG-1090.patch > > > There have been some changes (as recorded in the Changes Section, Nov 2 2009 > sub section of http://wiki.apache.org/pig/LoadStoreRedesignProposal) in the > load/store interfaces - this jira is to track the task of making those > changes under src. Changes under test will be addresses in a different jira. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-879) Pig should provide a way for input location string in load statement to be passed as-is to the Loader
[
https://issues.apache.org/jira/browse/PIG-879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12777316#action_12777316
]
Richard Ding commented on PIG-879:
--
As a related issue, there is a feature in Pig right now that allows user to
specify a local file in a load statement even if Pig is running in MapReduce
mode. Namely,
{code}
A = load 'file:test/org/apache/pig/test/data/passwd' using PigStorage(':');
{code}
is a valid Pig statement. Internally Pig moves the file from the local file
system to the HDFS file system and gives the corresponding HDFS URI to the
loader:
{code}
hdfs://localhost.localdomain:37575/tmp/temp957104276/tmp124591329
{code}
As we move to the new Load/Store API, there are two options:
1. Stop supporting this feature. The above load statement can be replaced by
the following statements:
{code}
copyFromLocal ./test/org/apache/pig/test/data/passw ./passw
A = load './passw' using PigStorage(':');
{code}
2. The default implementation of _relativeToAbsolutePath(String location, Path
curDir)_ method will move the local file (specified by location) to the HDFS
and return the corresponding HDFS URI.
Any comments on these options? Especially does anyone have strong opinion
against choosing option 1?
> Pig should provide a way for input location string in load statement to be
> passed as-is to the Loader
> -
>
> Key: PIG-879
> URL: https://issues.apache.org/jira/browse/PIG-879
> Project: Pig
> Issue Type: Bug
>Affects Versions: 0.3.0
>Reporter: Pradeep Kamath
>Assignee: Richard Ding
>
> Due to multiquery optimization, Pig always converts the filenames to
> absolute URIs (see
> http://wiki.apache.org/pig/PigMultiQueryPerformanceSpecification - section
> about Incompatible Changes - Path Names and Schemes). This is necessary since
> the script may have "cd .." statements between load or store statements and
> if the load statements have relative paths, we would need to convert to
> absolute paths to know where to load/store from. To do this
> QueryParser.massageFilename() has the code below[1] which basically gives the
> fully qualified hdfs path
>
> However the issue with this approach is that if the filename string is
> something like
> "hdfs://localhost.localdomain:39125/user/bla/1,hdfs://localhost.localdomain:39125/user/bla/2",
> the code below[1] actually translates this to
> hdfs://localhost.localdomain:38264/user/bla/1,hdfs://localhost.localdomain:38264/user/bla/2
> and throws an exception that it is an incorrect path.
>
> Some loaders may want to interpret the filenames (the input location string
> in the load statement) in any way they wish and may want Pig to not make
> absolute paths out of them.
>
> There are a few options to address this:
> 1)A command line switch to indicate to Pig that pathnames in the script
> are all absolute and hence Pig should not alter them and pass them as-is to
> Loaders and Storers.
> 2)A keyword in the load and store statements to indicate the same intent
> to pig
> 3)A property which users can supply on cmdline or in pig.properties to
> indicate the same intent.
> 4)A method in LoadFunc - relativeToAbsolutePath(String filename, String
> curDir) which does the conversion to absolute - this way Loader can chose to
> implement it as a noop.
> Thoughts?
>
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-598) Parameter substitution ($PARAMETER) should not be performed in comments
[
https://issues.apache.org/jira/browse/PIG-598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12777317#action_12777317
]
Ashutosh Chauhan commented on PIG-598:
--
One issue I faced while working on PIG-928 was when trying to name variables in
ruby bound to java variables. Ruby variable names (in BSF) needs to be
prepended with $ as shown below.
{code}
define myudf org.apache.pig.scriptedUDFs.GenericEvalFunc('wordtokens','return
$input.split();');
{code}
Now this $ appearing in the script should not be substituted. But parser tries
to substitute it and fails (with error undefined parameter) since param is not
specified while invoking Pig. My hack was to 'escape' this $ and then provide
param-substiution while invoking the script as -p input=input
{code}
define myudf org.apache.pig.scriptedUDFs.GenericEvalFunc('wordtokens','return
$$input.split();');
{code}
Note extra $. Obviously, if we are thinking of adding pig-928 and accept udfs
written in ruby, this will come up as an issue. One possibility is Pig should
not try to do substitution if param is not specified on cmd line. I think way
it currently is parser scans the script for $ and then tries to do substiution.
If substitution was not defined it error outs. I think if it is not specified
on cmd line, then parser should ignore it and continue instead of failing.
> Parameter substitution ($PARAMETER) should not be performed in comments
> ---
>
> Key: PIG-598
> URL: https://issues.apache.org/jira/browse/PIG-598
> Project: Pig
> Issue Type: Bug
> Components: impl
>Affects Versions: 0.2.0
>Reporter: David Ciemiewicz
>Assignee: Thejas M Nair
> Attachments: PIG-598.patch
>
>
> Compiling the following code example will generate an error that
> $NOT_A_PARAMETER is an Undefined Parameter.
> This is problematic as sometimes you want to comment out parts of your code,
> including parameters so that you don't have to define them.
> This I think it would be really good if parameter substitution was not
> performed in comments.
> {code}
> -- $NOT_A_PARAMETER
> {code}
> {code}
> -bash-3.00$ pig -exectype local -latest comment.pig
> USING: /grid/0/gs/pig/current
> java.lang.RuntimeException: Undefined parameter : NOT_A_PARAMETER
> at
> org.apache.pig.tools.parameters.PreprocessorContext.substitute(PreprocessorContext.java:221)
> at
> org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor.parsePigFile(ParameterSubstitutionPreprocessor.java:106)
> at
> org.apache.pig.tools.parameters.ParameterSubstitutionPreprocessor.genSubstitutedFile(ParameterSubstitutionPreprocessor.java:86)
> at org.apache.pig.Main.runParamPreprocessor(Main.java:394)
> at org.apache.pig.Main.main(Main.java:296)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1038) Optimize nested distinct/sort to use secondary key
[
https://issues.apache.org/jira/browse/PIG-1038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12777318#action_12777318
]
Daniel Dai commented on PIG-1038:
-
Couple of limitations for the currently implementation and will be addressed
later:
1. If sort is not nested inside foreach plan, it will not optimized. That's the
issue with merge join indexing which Ashutosh find.
2. All the distinct keys are assumed to be ascending sort. Actually for the
distinct key, the order of sort is flexible, if descending sort is cheaper, we
shall use descending sort. Eg:
{code}
C = foreach B { C1 = order A by $0 desc; C2 = C1.$0; C3 = distinct C2;
generate group, C3; }
{code}
Both order by and distinct is on the same key A.$0; however, order by use
descending order. If we use descending A.$0 as secondary key, we shall able to
remove both order by and distinct. This is not the case now. We can only remove
order by and leave distinct.
3. Main key has the same issue. Main key is group key and order does not
matters. However, in current implementation, we assume ascending order. Eg:
{code}
B = group a by (a0, a1);
C = foreach B { C1 = order A by a0 desc; generate group, C1; }
{code}
We use (a0, a1) as the main key, and nested order by cannot be removed.
However, if we use reverse the order of main key, then we can remove the order
by.
> Optimize nested distinct/sort to use secondary key
> --
>
> Key: PIG-1038
> URL: https://issues.apache.org/jira/browse/PIG-1038
> Project: Pig
> Issue Type: Improvement
> Components: impl
>Affects Versions: 0.4.0
>Reporter: Olga Natkovich
>Assignee: Daniel Dai
> Fix For: 0.6.0
>
> Attachments: PIG-1038-1.patch, PIG-1038-2.patch, PIG-1038-3.patch,
> PIG-1038-4.patch, PIG-1038-5.patch
>
>
> If nested foreach plan contains sort/distinct, it is possible to use hadoop
> secondary sort instead of SortedDataBag and DistinctDataBag to optimize the
> query.
> Eg1:
> A = load 'mydata';
> B = group A by $0;
> C = foreach B {
> D = order A by $1;
> generate group, D;
> }
> store C into 'myresult';
> We can specify a secondary sort on A.$1, and drop "order A by $1".
> Eg2:
> A = load 'mydata';
> B = group A by $0;
> C = foreach B {
> D = A.$1;
> E = distinct D;
> generate group, E;
> }
> store C into 'myresult';
> We can specify a secondary sort key on A.$1, and simplify "D=A.$1; E=distinct
> D" to a special version of distinct, which does not do the sorting.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1062) load-store-redesign branch: change SampleLoader and subclasses to work with new LoadFunc interface
[ https://issues.apache.org/jira/browse/PIG-1062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated PIG-1062: --- Attachment: PIG-1062.patch Patch for order-by sampling and skew-join sampling changes. > load-store-redesign branch: change SampleLoader and subclasses to work with > new LoadFunc interface > --- > > Key: PIG-1062 > URL: https://issues.apache.org/jira/browse/PIG-1062 > Project: Pig > Issue Type: Sub-task >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Attachments: PIG-1062.patch > > > This is part of the effort to implement new load store interfaces as laid out > in http://wiki.apache.org/pig/LoadStoreRedesignProposal . > PigStorage and BinStorage are now working. > SampleLoader and subclasses -RandomSampleLoader, PoissonSampleLoader need to > be changed to work with new LoadFunc interface. > Fixing SampleLoader and RandomSampleLoader will get order-by queries working. > PoissonSampleLoader is used by skew join. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1062) load-store-redesign branch: change SampleLoader and subclasses to work with new LoadFunc interface
[ https://issues.apache.org/jira/browse/PIG-1062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated PIG-1062: --- Status: Patch Available (was: Open) For load-store-redesign branch, hudson might not be able to apply patch to trunk. > load-store-redesign branch: change SampleLoader and subclasses to work with > new LoadFunc interface > --- > > Key: PIG-1062 > URL: https://issues.apache.org/jira/browse/PIG-1062 > Project: Pig > Issue Type: Sub-task >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Attachments: PIG-1062.patch > > > This is part of the effort to implement new load store interfaces as laid out > in http://wiki.apache.org/pig/LoadStoreRedesignProposal . > PigStorage and BinStorage are now working. > SampleLoader and subclasses -RandomSampleLoader, PoissonSampleLoader need to > be changed to work with new LoadFunc interface. > Fixing SampleLoader and RandomSampleLoader will get order-by queries working. > PoissonSampleLoader is used by skew join. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1062) load-store-redesign branch: change SampleLoader and subclasses to work with new LoadFunc interface
[ https://issues.apache.org/jira/browse/PIG-1062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12777352#action_12777352 ] Hadoop QA commented on PIG-1062: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12424797/PIG-1062.patch against trunk revision 835499. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no tests are needed for this patch. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h8.grid.sp2.yahoo.net/48/console This message is automatically generated. > load-store-redesign branch: change SampleLoader and subclasses to work with > new LoadFunc interface > --- > > Key: PIG-1062 > URL: https://issues.apache.org/jira/browse/PIG-1062 > Project: Pig > Issue Type: Sub-task >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Attachments: PIG-1062.patch > > > This is part of the effort to implement new load store interfaces as laid out > in http://wiki.apache.org/pig/LoadStoreRedesignProposal . > PigStorage and BinStorage are now working. > SampleLoader and subclasses -RandomSampleLoader, PoissonSampleLoader need to > be changed to work with new LoadFunc interface. > Fixing SampleLoader and RandomSampleLoader will get order-by queries working. > PoissonSampleLoader is used by skew join. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1064) Behvaiour of COGROUP with and without schema when using "*" operator
[
https://issues.apache.org/jira/browse/PIG-1064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12777392#action_12777392
]
Hadoop QA commented on PIG-1064:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12424792/PIG-1064-3.patch
against trunk revision 835499.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 15 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/154/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/154/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/154/console
This message is automatically generated.
> Behvaiour of COGROUP with and without schema when using "*" operator
>
>
> Key: PIG-1064
> URL: https://issues.apache.org/jira/browse/PIG-1064
> Project: Pig
> Issue Type: Bug
> Components: impl
>Affects Versions: 0.6.0
>Reporter: Viraj Bhat
>Assignee: Pradeep Kamath
> Fix For: 0.6.0
>
> Attachments: PIG-1064-2.patch, PIG-1064-3.patch, PIG-1064.patch
>
>
> I have 2 tab separated files, "1.txt" and "2.txt"
> $ cat 1.txt
>
> 1 2
> 2 3
>
> $ cat 2.txt
> 1 2
> 2 3
> I use COGROUP feature of Pig in the following way:
> $java -cp pig.jar:$HADOOP_HOME org.apache.pig.Main
> {code}
> grunt> A = load '1.txt';
> grunt> B = load '2.txt' as (b0, b1);
> grunt> C = cogroup A by *, B by *;
> {code}
> 2009-10-29 12:46:04,150 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR
> 1012: Each COGroup input has to have the same number of inner plans
> Details at logfile: pig_1256845224752.log
> ==
> If I reverse, the order of the schema's
> {code}
> grunt> A = load '1.txt' as (a0, a1);
> grunt> B = load '2.txt';
> grunt> C = cogroup A by *, B by *;
> {code}
> 2009-10-29 12:49:27,869 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR
> 1013: Grouping attributes can either be star (*) or a list of expressions,
> but not both.
> Details at logfile: pig_1256845224752.log
> ==
> Now running without schema??
> {code}
> grunt> A = load '1.txt';
> grunt> B = load '2.txt';
> grunt> C = cogroup A by *, B by *;
> grunt> dump C;
> {code}
> 2009-10-29 12:55:37,202 [main] INFO
> org.apache.pig.backend.local.executionengine.LocalPigLauncher - Successfully
> stored result in: "file:/tmp/temp-319926700/tmp-1990275961"
> 2009-10-29 12:55:37,202 [main] INFO
> org.apache.pig.backend.local.executionengine.LocalPigLauncher - Records
> written : 2
> 2009-10-29 12:55:37,202 [main] INFO
> org.apache.pig.backend.local.executionengine.LocalPigLauncher - Bytes written
> : 154
> 2009-10-29 12:55:37,202 [main] INFO
> org.apache.pig.backend.local.executionengine.LocalPigLauncher - 100% complete!
> 2009-10-29 12:55:37,202 [main] INFO
> org.apache.pig.backend.local.executionengine.LocalPigLauncher - Success!!
> ((1,2),{(1,2)},{(1,2)})
> ((2,3),{(2,3)},{(2,3)})
> ==
> Is this a bug or a feature?
> Viraj
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
