Re: Creating a branch for Pig on Spark (PIG-4059)
Praveen me should be good enough for us. We will have 1-2 more folks contributing from Sigmoid but we can manage by merging patches from them. Initially it will be good to structure patches through us as we would like to avoid breaking architecture flow. Regards Mayur On Tue, Aug 26, 2014 at 8:27 AM, Cheolsoo Park piaozhe...@gmail.com wrote: Hi guys, I asked about branch committership to the infra mailing list, and here is the reply- Many projects have what they consider 'partial committers' that is folks who have access to specific parts of a projects svn tree. Some projects do this for GSoC participants, others as a mechanism for moving to 'full committership' within the project. Do note though that in the eyes of the ASF someone with an ICLA and an account with any permissions to commit code anywhere in the public svn tree is a committer. IOW, you would vote, have ICLAs filed, and request account creation as per normal, and then merely adjust the karma in asf-authorization-template (and or LDAP) Looks like we need to vote and follow the normal process just like any other new committer. @Praveen, Jacec, I think Mayur and Praveen from Sigmoid Analytics need branch committership. Will anyone else work on Pig-on-Spark? Please reply. Once I have a full list of people, I will open a vote for Pig PMCs. Thanks, Cheolsoo On Mon, Aug 25, 2014 at 11:51 AM, Cheolsoo Park piaozhe...@gmail.com wrote: Additionally, I will give branch-specific commit permission to people who will work on Pig on Spark (assuming it is possible). Please let me know if you have any objection on this too. On Mon, Aug 25, 2014 at 10:25 AM, Jarek Jarcec Cecho jar...@apache.org wrote: No objections from my side, thank you for creating the branch Cheolsoo and kudos to the Sigmoid Analytics team for the great work! Jarcec On Aug 25, 2014, at 7:14 PM, Cheolsoo Park piaozhe...@gmail.com wrote: Hi devs, Sigmoid Analytics has been working on Pig-on-Spark (PIG-4059), and they want to merge their work into Apache. I am going to create a Spark branch for them. Please let me know if you have any concerns. Thanks, Cheolsoo -- Regards, Mayur Rustagi Ph: +1 (760) 203 3257 http://www.sigmoidanalytics.com @mayur_rustagi http://www.twitter.com/mayur_rustagi
Re: Creating a branch for Pig on Spark (PIG-4059)
I would second Mayur - as he and Praveen are most familiar with all the changes, I would let them be the only additional branch committers. I’m expecting that they will review other contributor's patches similarly as normal committers and hence the contributors will start building Karma to get the “branch committership” later as well (if the branch will be around for that long). Jarcec On Aug 26, 2014, at 9:04 AM, Mayur Rustagi ma...@sigmoidanalytics.com wrote: Praveen me should be good enough for us. We will have 1-2 more folks contributing from Sigmoid but we can manage by merging patches from them. Initially it will be good to structure patches through us as we would like to avoid breaking architecture flow. Regards Mayur On Tue, Aug 26, 2014 at 8:27 AM, Cheolsoo Park piaozhe...@gmail.com wrote: Hi guys, I asked about branch committership to the infra mailing list, and here is the reply- Many projects have what they consider 'partial committers' that is folks who have access to specific parts of a projects svn tree. Some projects do this for GSoC participants, others as a mechanism for moving to 'full committership' within the project. Do note though that in the eyes of the ASF someone with an ICLA and an account with any permissions to commit code anywhere in the public svn tree is a committer. IOW, you would vote, have ICLAs filed, and request account creation as per normal, and then merely adjust the karma in asf-authorization-template (and or LDAP) Looks like we need to vote and follow the normal process just like any other new committer. @Praveen, Jacec, I think Mayur and Praveen from Sigmoid Analytics need branch committership. Will anyone else work on Pig-on-Spark? Please reply. Once I have a full list of people, I will open a vote for Pig PMCs. Thanks, Cheolsoo On Mon, Aug 25, 2014 at 11:51 AM, Cheolsoo Park piaozhe...@gmail.com wrote: Additionally, I will give branch-specific commit permission to people who will work on Pig on Spark (assuming it is possible). Please let me know if you have any objection on this too. On Mon, Aug 25, 2014 at 10:25 AM, Jarek Jarcec Cecho jar...@apache.org wrote: No objections from my side, thank you for creating the branch Cheolsoo and kudos to the Sigmoid Analytics team for the great work! Jarcec On Aug 25, 2014, at 7:14 PM, Cheolsoo Park piaozhe...@gmail.com wrote: Hi devs, Sigmoid Analytics has been working on Pig-on-Spark (PIG-4059), and they want to merge their work into Apache. I am going to create a Spark branch for them. Please let me know if you have any concerns. Thanks, Cheolsoo -- Regards, Mayur Rustagi Ph: +1 (760) 203 3257 http://www.sigmoidanalytics.com @mayur_rustagi http://www.twitter.com/mayur_rustagi
Re: Creating a branch for Pig on Spark (PIG-4059)
Ok. I will start a vote today and get back to you in few days. On Tue, Aug 26, 2014 at 1:28 AM, Jarek Jarcec Cecho jar...@apache.org wrote: I would second Mayur - as he and Praveen are most familiar with all the changes, I would let them be the only additional branch committers. I’m expecting that they will review other contributor's patches similarly as normal committers and hence the contributors will start building Karma to get the “branch committership” later as well (if the branch will be around for that long). Jarcec On Aug 26, 2014, at 9:04 AM, Mayur Rustagi ma...@sigmoidanalytics.com wrote: Praveen me should be good enough for us. We will have 1-2 more folks contributing from Sigmoid but we can manage by merging patches from them. Initially it will be good to structure patches through us as we would like to avoid breaking architecture flow. Regards Mayur On Tue, Aug 26, 2014 at 8:27 AM, Cheolsoo Park piaozhe...@gmail.com wrote: Hi guys, I asked about branch committership to the infra mailing list, and here is the reply- Many projects have what they consider 'partial committers' that is folks who have access to specific parts of a projects svn tree. Some projects do this for GSoC participants, others as a mechanism for moving to 'full committership' within the project. Do note though that in the eyes of the ASF someone with an ICLA and an account with any permissions to commit code anywhere in the public svn tree is a committer. IOW, you would vote, have ICLAs filed, and request account creation as per normal, and then merely adjust the karma in asf-authorization-template (and or LDAP) Looks like we need to vote and follow the normal process just like any other new committer. @Praveen, Jacec, I think Mayur and Praveen from Sigmoid Analytics need branch committership. Will anyone else work on Pig-on-Spark? Please reply. Once I have a full list of people, I will open a vote for Pig PMCs. Thanks, Cheolsoo On Mon, Aug 25, 2014 at 11:51 AM, Cheolsoo Park piaozhe...@gmail.com wrote: Additionally, I will give branch-specific commit permission to people who will work on Pig on Spark (assuming it is possible). Please let me know if you have any objection on this too. On Mon, Aug 25, 2014 at 10:25 AM, Jarek Jarcec Cecho jar...@apache.org wrote: No objections from my side, thank you for creating the branch Cheolsoo and kudos to the Sigmoid Analytics team for the great work! Jarcec On Aug 25, 2014, at 7:14 PM, Cheolsoo Park piaozhe...@gmail.com wrote: Hi devs, Sigmoid Analytics has been working on Pig-on-Spark (PIG-4059), and they want to merge their work into Apache. I am going to create a Spark branch for them. Please let me know if you have any concerns. Thanks, Cheolsoo -- Regards, Mayur Rustagi Ph: +1 (760) 203 3257 http://www.sigmoidanalytics.com @mayur_rustagi http://www.twitter.com/mayur_rustagi
[DISCUSS] Re: Dropping support for JDK6 in Apache Hadoop
Pig has support for jdk7 from Pig 0.10. I think we should drop support for JDK6 from Pig 0.14 and also publish maven binaries with jdk 1.7 from Pig 0.14. Also it is high time to drop support for Hadoop 0.20. None of the other hadoop projects officially support Hadoop 0.20 anymore. I would like to get rid of the reflection in code w.r.t to UGI, be able to add support for fetching Credentials in UDFs, Load and StoreFunc, etc. If there are no major objections, will start two separate voting threads for that. Regards, Rohini On Tue, Aug 19, 2014 at 10:52 AM, Arun C Murthy a...@hortonworks.com wrote: [Apologies for the wide distribution.] Dear HBase/Hive/Pig/Oozie communities, We, over at Hadoop are considering dropping support for JDK6 this year. As you maybe aware we just released hadoop-2.5.0 and are now considering making the next release i.e. hadoop-2.6.0 the *last* release of Apache Hadoop which supports JDK6. This means, from hadoop-2.7.0 onwards we will not support JDK6 anymore and we *may* start relying on JDK7-specific apis. Now, the above releases a proposal and we do not want to pull the trigger without talking to projects downstream - hence the request for you feedback. Please feel free to forward this to other communities you might deem to be at risk from this too. thanks, Arun -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
Re: [DISCUSS] Re: Dropping support for JDK6 in Apache Hadoop
I'm +1 on both of these. But as a side note Hive actually still supports Hadoop 0.20, so you're statement below isn't quite true. Alan. Rohini Palaniswamy mailto:rohini.adi...@gmail.com August 26, 2014 at 9:36 Pig has support for jdk7 from Pig 0.10. I think we should drop support for JDK6 from Pig 0.14 and also publish maven binaries with jdk 1.7 from Pig 0.14. Also it is high time to drop support for Hadoop 0.20. None of the other hadoop projects officially support Hadoop 0.20 anymore. I would like to get rid of the reflection in code w.r.t to UGI, be able to add support for fetching Credentials in UDFs, Load and StoreFunc, etc. If there are no major objections, will start two separate voting threads for that. Regards, Rohini -- Sent with Postbox http://www.getpostbox.com -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[jira] [Commented] (PIG-3087) Refactor TestLogicalPlanBuilder to be meaningful
[ https://issues.apache.org/jira/browse/PIG-3087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14111354#comment-14111354 ] Hao Gao commented on PIG-3087: -- Can I try this? Refactor TestLogicalPlanBuilder to be meaningful Key: PIG-3087 URL: https://issues.apache.org/jira/browse/PIG-3087 Project: Pig Issue Type: Bug Reporter: Jonathan Coveney Labels: newbie Fix For: 0.14.0 Attachments: PIG-3087-0.patch I started doing this as part of another patch, but there are some bigger issues, and I don't have the time to dig in atm. That said, a lot of the tests as written don't test anything. I used more modern junit patterns, and discovered we had a lot of tests that weren't functioning properly. Making them function properly unveiled that the general buildLp pattern doesn't work properly anymore for many cases where it would throw an error in grunt, but for whatever reason no error is thrown in the tests. Any test with _1 is a test that previous failed, that now doesn't. Some, however, don't make sense so I think what really needs to be done is figure out which should be failing, which shouldn't, and then fix buildLp accordingly. I will attach my pass at it, but it is incomplete and needs work. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (PIG-3087) Refactor TestLogicalPlanBuilder to be meaningful
[ https://issues.apache.org/jira/browse/PIG-3087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14111401#comment-14111401 ] Jonathan Coveney commented on PIG-3087: --- Please do! Refactor TestLogicalPlanBuilder to be meaningful Key: PIG-3087 URL: https://issues.apache.org/jira/browse/PIG-3087 Project: Pig Issue Type: Bug Reporter: Jonathan Coveney Labels: newbie Fix For: 0.14.0 Attachments: PIG-3087-0.patch I started doing this as part of another patch, but there are some bigger issues, and I don't have the time to dig in atm. That said, a lot of the tests as written don't test anything. I used more modern junit patterns, and discovered we had a lot of tests that weren't functioning properly. Making them function properly unveiled that the general buildLp pattern doesn't work properly anymore for many cases where it would throw an error in grunt, but for whatever reason no error is thrown in the tests. Any test with _1 is a test that previous failed, that now doesn't. Some, however, don't make sense so I think what really needs to be done is figure out which should be failing, which shouldn't, and then fix buildLp accordingly. I will attach my pass at it, but it is incomplete and needs work. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (PIG-4141) Ship UDF/LoadFunc/StoreFunc dependent jar automatically
[ https://issues.apache.org/jira/browse/PIG-4141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated PIG-4141: Attachment: PIG-4141-1.patch Ship UDF/LoadFunc/StoreFunc dependent jar automatically --- Key: PIG-4141 URL: https://issues.apache.org/jira/browse/PIG-4141 Project: Pig Issue Type: New Feature Components: impl Reporter: Daniel Dai Assignee: Daniel Dai Fix For: 0.14.0 Attachments: PIG-4141-1.patch When user use AvroStorage/JsonStorage/OrcStorage, they need to register dependent jars manually. It would be much convenient if we can provide a mechanism for UDF/LoadFunc/StoreFunc to claim the dependency and ship jars automatically. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (PIG-4141) Ship UDF/LoadFunc/StoreFunc dependent jar automatically
Daniel Dai created PIG-4141: --- Summary: Ship UDF/LoadFunc/StoreFunc dependent jar automatically Key: PIG-4141 URL: https://issues.apache.org/jira/browse/PIG-4141 Project: Pig Issue Type: New Feature Components: impl Reporter: Daniel Dai Assignee: Daniel Dai Fix For: 0.14.0 Attachments: PIG-4141-1.patch When user use AvroStorage/JsonStorage/OrcStorage, they need to register dependent jars manually. It would be much convenient if we can provide a mechanism for UDF/LoadFunc/StoreFunc to claim the dependency and ship jars automatically. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Review Request 24789: New logical optimizer rule: ConstantCalculator
On Aug. 25, 2014, 8:24 p.m., Thejas Nair wrote: trunk/src/org/apache/pig/newplan/logical/rules/ConstantCalculator.java, line 53 https://reviews.apache.org/r/24789/diff/2/?file=664437#file664437line53 There is no processedOperators.add happening. Is this variable needed ? Yes, we should add processed operators. - Daniel --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24789/#review51430 --- On Aug. 19, 2014, 5:41 p.m., Daniel Dai wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24789/ --- (Updated Aug. 19, 2014, 5:41 p.m.) Review request for pig. Repository: pig Description --- See PIG-4128 Diffs - trunk/src/org/apache/pig/EvalFunc.java 1618727 trunk/src/org/apache/pig/Main.java 1618727 trunk/src/org/apache/pig/builtin/ABS.java 1618727 trunk/src/org/apache/pig/builtin/ARITY.java 1618727 trunk/src/org/apache/pig/builtin/AddDuration.java 1618727 trunk/src/org/apache/pig/builtin/Assert.java 1618727 trunk/src/org/apache/pig/builtin/BagSize.java 1618727 trunk/src/org/apache/pig/builtin/BagToString.java 1618727 trunk/src/org/apache/pig/builtin/BagToTuple.java 1618727 trunk/src/org/apache/pig/builtin/Base.java 1618727 trunk/src/org/apache/pig/builtin/BigDecimalAbs.java 1618727 trunk/src/org/apache/pig/builtin/BigIntegerAbs.java 1618727 trunk/src/org/apache/pig/builtin/CONCAT.java 1618727 trunk/src/org/apache/pig/builtin/ConstantSize.java 1618727 trunk/src/org/apache/pig/builtin/CubeDimensions.java 1618727 trunk/src/org/apache/pig/builtin/CurrentTime.java 1618727 trunk/src/org/apache/pig/builtin/DIFF.java 1618727 trunk/src/org/apache/pig/builtin/DaysBetween.java 1618727 trunk/src/org/apache/pig/builtin/DoubleRound.java 1618727 trunk/src/org/apache/pig/builtin/DoubleRoundTo.java 1618727 trunk/src/org/apache/pig/builtin/ENDSWITH.java 1618727 trunk/src/org/apache/pig/builtin/EqualsIgnoreCase.java 1618727 trunk/src/org/apache/pig/builtin/FloatAbs.java 1618727 trunk/src/org/apache/pig/builtin/FloatRound.java 1618727 trunk/src/org/apache/pig/builtin/FloatRoundTo.java 1618727 trunk/src/org/apache/pig/builtin/GetDay.java 1618727 trunk/src/org/apache/pig/builtin/GetHour.java 1618727 trunk/src/org/apache/pig/builtin/GetMilliSecond.java 1618727 trunk/src/org/apache/pig/builtin/GetMinute.java 1618727 trunk/src/org/apache/pig/builtin/GetMonth.java 1618727 trunk/src/org/apache/pig/builtin/GetSecond.java 1618727 trunk/src/org/apache/pig/builtin/GetWeek.java 1618727 trunk/src/org/apache/pig/builtin/GetWeekYear.java 1618727 trunk/src/org/apache/pig/builtin/GetYear.java 1618727 trunk/src/org/apache/pig/builtin/HoursBetween.java 1618727 trunk/src/org/apache/pig/builtin/INDEXOF.java 1618727 trunk/src/org/apache/pig/builtin/INVERSEMAP.java 1618727 trunk/src/org/apache/pig/builtin/IntAbs.java 1618727 trunk/src/org/apache/pig/builtin/IsEmpty.java 1618727 trunk/src/org/apache/pig/builtin/KEYSET.java 1618727 trunk/src/org/apache/pig/builtin/LAST_INDEX_OF.java 1618727 trunk/src/org/apache/pig/builtin/LCFIRST.java 1618727 trunk/src/org/apache/pig/builtin/LOWER.java 1618727 trunk/src/org/apache/pig/builtin/LTRIM.java 1618727 trunk/src/org/apache/pig/builtin/LongAbs.java 1618727 trunk/src/org/apache/pig/builtin/MapSize.java 1618727 trunk/src/org/apache/pig/builtin/MilliSecondsBetween.java 1618727 trunk/src/org/apache/pig/builtin/MinutesBetween.java 1618727 trunk/src/org/apache/pig/builtin/MonthsBetween.java 1618727 trunk/src/org/apache/pig/builtin/PluckTuple.java 1618727 trunk/src/org/apache/pig/builtin/REGEX_EXTRACT.java 1618727 trunk/src/org/apache/pig/builtin/REGEX_EXTRACT_ALL.java 1618727 trunk/src/org/apache/pig/builtin/REPLACE.java 1618727 trunk/src/org/apache/pig/builtin/ROUND.java 1618727 trunk/src/org/apache/pig/builtin/ROUND_TO.java 1618727 trunk/src/org/apache/pig/builtin/RTRIM.java 1618727 trunk/src/org/apache/pig/builtin/RollupDimensions.java 1618727 trunk/src/org/apache/pig/builtin/SIZE.java 1618727 trunk/src/org/apache/pig/builtin/SPRINTF.java 1618727 trunk/src/org/apache/pig/builtin/STARTSWITH.java 1618727 trunk/src/org/apache/pig/builtin/STRSPLIT.java 1618727 trunk/src/org/apache/pig/builtin/SUBSTRING.java 1618727 trunk/src/org/apache/pig/builtin/SUBTRACT.java 1618727 trunk/src/org/apache/pig/builtin/SecondsBetween.java 1618727 trunk/src/org/apache/pig/builtin/StringConcat.java 1618727 trunk/src/org/apache/pig/builtin/StringSize.java 1618727
Re: Review Request 24789: New logical optimizer rule: ConstantCalculator
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24789/ --- (Updated Aug. 26, 2014, 10:35 p.m.) Review request for pig. Repository: pig Description --- See PIG-4128 Diffs (updated) - trunk/src/org/apache/pig/EvalFunc.java 1618727 trunk/src/org/apache/pig/Main.java 1618727 trunk/src/org/apache/pig/builtin/ABS.java 1618727 trunk/src/org/apache/pig/builtin/ARITY.java 1618727 trunk/src/org/apache/pig/builtin/AddDuration.java 1618727 trunk/src/org/apache/pig/builtin/Assert.java 1618727 trunk/src/org/apache/pig/builtin/BagSize.java 1618727 trunk/src/org/apache/pig/builtin/BagToString.java 1618727 trunk/src/org/apache/pig/builtin/BagToTuple.java 1618727 trunk/src/org/apache/pig/builtin/Base.java 1618727 trunk/src/org/apache/pig/builtin/BigDecimalAbs.java 1618727 trunk/src/org/apache/pig/builtin/BigIntegerAbs.java 1618727 trunk/src/org/apache/pig/builtin/CONCAT.java 1618727 trunk/src/org/apache/pig/builtin/ConstantSize.java 1618727 trunk/src/org/apache/pig/builtin/CubeDimensions.java 1618727 trunk/src/org/apache/pig/builtin/CurrentTime.java 1618727 trunk/src/org/apache/pig/builtin/DIFF.java 1618727 trunk/src/org/apache/pig/builtin/DaysBetween.java 1618727 trunk/src/org/apache/pig/builtin/DoubleRound.java 1618727 trunk/src/org/apache/pig/builtin/DoubleRoundTo.java 1618727 trunk/src/org/apache/pig/builtin/ENDSWITH.java 1618727 trunk/src/org/apache/pig/builtin/EqualsIgnoreCase.java 1618727 trunk/src/org/apache/pig/builtin/FloatAbs.java 1618727 trunk/src/org/apache/pig/builtin/FloatRound.java 1618727 trunk/src/org/apache/pig/builtin/FloatRoundTo.java 1618727 trunk/src/org/apache/pig/builtin/GetDay.java 1618727 trunk/src/org/apache/pig/builtin/GetHour.java 1618727 trunk/src/org/apache/pig/builtin/GetMilliSecond.java 1618727 trunk/src/org/apache/pig/builtin/GetMinute.java 1618727 trunk/src/org/apache/pig/builtin/GetMonth.java 1618727 trunk/src/org/apache/pig/builtin/GetSecond.java 1618727 trunk/src/org/apache/pig/builtin/GetWeek.java 1618727 trunk/src/org/apache/pig/builtin/GetWeekYear.java 1618727 trunk/src/org/apache/pig/builtin/GetYear.java 1618727 trunk/src/org/apache/pig/builtin/HoursBetween.java 1618727 trunk/src/org/apache/pig/builtin/INDEXOF.java 1618727 trunk/src/org/apache/pig/builtin/INVERSEMAP.java 1618727 trunk/src/org/apache/pig/builtin/IntAbs.java 1618727 trunk/src/org/apache/pig/builtin/IsEmpty.java 1618727 trunk/src/org/apache/pig/builtin/KEYSET.java 1618727 trunk/src/org/apache/pig/builtin/LAST_INDEX_OF.java 1618727 trunk/src/org/apache/pig/builtin/LCFIRST.java 1618727 trunk/src/org/apache/pig/builtin/LOWER.java 1618727 trunk/src/org/apache/pig/builtin/LTRIM.java 1618727 trunk/src/org/apache/pig/builtin/LongAbs.java 1618727 trunk/src/org/apache/pig/builtin/MapSize.java 1618727 trunk/src/org/apache/pig/builtin/MilliSecondsBetween.java 1618727 trunk/src/org/apache/pig/builtin/MinutesBetween.java 1618727 trunk/src/org/apache/pig/builtin/MonthsBetween.java 1618727 trunk/src/org/apache/pig/builtin/PluckTuple.java 1618727 trunk/src/org/apache/pig/builtin/REGEX_EXTRACT.java 1618727 trunk/src/org/apache/pig/builtin/REGEX_EXTRACT_ALL.java 1618727 trunk/src/org/apache/pig/builtin/REPLACE.java 1618727 trunk/src/org/apache/pig/builtin/ROUND.java 1618727 trunk/src/org/apache/pig/builtin/ROUND_TO.java 1618727 trunk/src/org/apache/pig/builtin/RTRIM.java 1618727 trunk/src/org/apache/pig/builtin/RollupDimensions.java 1618727 trunk/src/org/apache/pig/builtin/SIZE.java 1618727 trunk/src/org/apache/pig/builtin/SPRINTF.java 1618727 trunk/src/org/apache/pig/builtin/STARTSWITH.java 1618727 trunk/src/org/apache/pig/builtin/STRSPLIT.java 1618727 trunk/src/org/apache/pig/builtin/SUBSTRING.java 1618727 trunk/src/org/apache/pig/builtin/SUBTRACT.java 1618727 trunk/src/org/apache/pig/builtin/SecondsBetween.java 1618727 trunk/src/org/apache/pig/builtin/StringConcat.java 1618727 trunk/src/org/apache/pig/builtin/StringSize.java 1618727 trunk/src/org/apache/pig/builtin/SubtractDuration.java 1618727 trunk/src/org/apache/pig/builtin/TOBAG.java 1618727 trunk/src/org/apache/pig/builtin/TOKENIZE.java 1618727 trunk/src/org/apache/pig/builtin/TOMAP.java 1618727 trunk/src/org/apache/pig/builtin/TOTUPLE.java 1618727 trunk/src/org/apache/pig/builtin/TRIM.java 1618727 trunk/src/org/apache/pig/builtin/ToDate.java 1618727 trunk/src/org/apache/pig/builtin/ToDate2ARGS.java 1618727 trunk/src/org/apache/pig/builtin/ToDate3ARGS.java 1618727 trunk/src/org/apache/pig/builtin/ToDateISO.java 1618727 trunk/src/org/apache/pig/builtin/ToMilliSeconds.java 1618727 trunk/src/org/apache/pig/builtin/ToString.java 1618727
[jira] [Updated] (PIG-4128) New logical optimizer rule: ConstantCalculator
[ https://issues.apache.org/jira/browse/PIG-4128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated PIG-4128: Attachment: PIG-4128-3.patch Addressing Thejas' review comments. New logical optimizer rule: ConstantCalculator -- Key: PIG-4128 URL: https://issues.apache.org/jira/browse/PIG-4128 Project: Pig Issue Type: New Feature Components: impl Reporter: Daniel Dai Assignee: Daniel Dai Fix For: 0.14.0 Attachments: PIG-4128-1.patch, PIG-4128-2.patch, PIG-4128-3.patch Pig used to have a LogicExpressionSimplifier to simplify expression which also calculates constant expression. The optimizer rule is buggy and we disable it by default in PIG-2316. However, we do need this feature especially in partition/predicate push down, since both does not deal with complex constant expression, we'd like to replace the expression with constant before the actual push down. Yes, user may manually do the calculation and rewrite the query, but even rewrite is sometimes not possible. Consider the case user want to push a datetime predicate, user have to write a ToDate udf since Pig does not have datetime constant. In this Jira, I provide a new rule: ConstantCalculator, which is much simpler and much less error prone, to replace LogicExpressionSimplifier. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Review Request 24789: New logical optimizer rule: ConstantCalculator
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24789/#review51603 --- Ship it! Ship It! - Thejas Nair On Aug. 26, 2014, 10:35 p.m., Daniel Dai wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24789/ --- (Updated Aug. 26, 2014, 10:35 p.m.) Review request for pig. Repository: pig Description --- See PIG-4128 Diffs - trunk/src/org/apache/pig/EvalFunc.java 1618727 trunk/src/org/apache/pig/Main.java 1618727 trunk/src/org/apache/pig/builtin/ABS.java 1618727 trunk/src/org/apache/pig/builtin/ARITY.java 1618727 trunk/src/org/apache/pig/builtin/AddDuration.java 1618727 trunk/src/org/apache/pig/builtin/Assert.java 1618727 trunk/src/org/apache/pig/builtin/BagSize.java 1618727 trunk/src/org/apache/pig/builtin/BagToString.java 1618727 trunk/src/org/apache/pig/builtin/BagToTuple.java 1618727 trunk/src/org/apache/pig/builtin/Base.java 1618727 trunk/src/org/apache/pig/builtin/BigDecimalAbs.java 1618727 trunk/src/org/apache/pig/builtin/BigIntegerAbs.java 1618727 trunk/src/org/apache/pig/builtin/CONCAT.java 1618727 trunk/src/org/apache/pig/builtin/ConstantSize.java 1618727 trunk/src/org/apache/pig/builtin/CubeDimensions.java 1618727 trunk/src/org/apache/pig/builtin/CurrentTime.java 1618727 trunk/src/org/apache/pig/builtin/DIFF.java 1618727 trunk/src/org/apache/pig/builtin/DaysBetween.java 1618727 trunk/src/org/apache/pig/builtin/DoubleRound.java 1618727 trunk/src/org/apache/pig/builtin/DoubleRoundTo.java 1618727 trunk/src/org/apache/pig/builtin/ENDSWITH.java 1618727 trunk/src/org/apache/pig/builtin/EqualsIgnoreCase.java 1618727 trunk/src/org/apache/pig/builtin/FloatAbs.java 1618727 trunk/src/org/apache/pig/builtin/FloatRound.java 1618727 trunk/src/org/apache/pig/builtin/FloatRoundTo.java 1618727 trunk/src/org/apache/pig/builtin/GetDay.java 1618727 trunk/src/org/apache/pig/builtin/GetHour.java 1618727 trunk/src/org/apache/pig/builtin/GetMilliSecond.java 1618727 trunk/src/org/apache/pig/builtin/GetMinute.java 1618727 trunk/src/org/apache/pig/builtin/GetMonth.java 1618727 trunk/src/org/apache/pig/builtin/GetSecond.java 1618727 trunk/src/org/apache/pig/builtin/GetWeek.java 1618727 trunk/src/org/apache/pig/builtin/GetWeekYear.java 1618727 trunk/src/org/apache/pig/builtin/GetYear.java 1618727 trunk/src/org/apache/pig/builtin/HoursBetween.java 1618727 trunk/src/org/apache/pig/builtin/INDEXOF.java 1618727 trunk/src/org/apache/pig/builtin/INVERSEMAP.java 1618727 trunk/src/org/apache/pig/builtin/IntAbs.java 1618727 trunk/src/org/apache/pig/builtin/IsEmpty.java 1618727 trunk/src/org/apache/pig/builtin/KEYSET.java 1618727 trunk/src/org/apache/pig/builtin/LAST_INDEX_OF.java 1618727 trunk/src/org/apache/pig/builtin/LCFIRST.java 1618727 trunk/src/org/apache/pig/builtin/LOWER.java 1618727 trunk/src/org/apache/pig/builtin/LTRIM.java 1618727 trunk/src/org/apache/pig/builtin/LongAbs.java 1618727 trunk/src/org/apache/pig/builtin/MapSize.java 1618727 trunk/src/org/apache/pig/builtin/MilliSecondsBetween.java 1618727 trunk/src/org/apache/pig/builtin/MinutesBetween.java 1618727 trunk/src/org/apache/pig/builtin/MonthsBetween.java 1618727 trunk/src/org/apache/pig/builtin/PluckTuple.java 1618727 trunk/src/org/apache/pig/builtin/REGEX_EXTRACT.java 1618727 trunk/src/org/apache/pig/builtin/REGEX_EXTRACT_ALL.java 1618727 trunk/src/org/apache/pig/builtin/REPLACE.java 1618727 trunk/src/org/apache/pig/builtin/ROUND.java 1618727 trunk/src/org/apache/pig/builtin/ROUND_TO.java 1618727 trunk/src/org/apache/pig/builtin/RTRIM.java 1618727 trunk/src/org/apache/pig/builtin/RollupDimensions.java 1618727 trunk/src/org/apache/pig/builtin/SIZE.java 1618727 trunk/src/org/apache/pig/builtin/SPRINTF.java 1618727 trunk/src/org/apache/pig/builtin/STARTSWITH.java 1618727 trunk/src/org/apache/pig/builtin/STRSPLIT.java 1618727 trunk/src/org/apache/pig/builtin/SUBSTRING.java 1618727 trunk/src/org/apache/pig/builtin/SUBTRACT.java 1618727 trunk/src/org/apache/pig/builtin/SecondsBetween.java 1618727 trunk/src/org/apache/pig/builtin/StringConcat.java 1618727 trunk/src/org/apache/pig/builtin/StringSize.java 1618727 trunk/src/org/apache/pig/builtin/SubtractDuration.java 1618727 trunk/src/org/apache/pig/builtin/TOBAG.java 1618727 trunk/src/org/apache/pig/builtin/TOKENIZE.java 1618727 trunk/src/org/apache/pig/builtin/TOMAP.java 1618727 trunk/src/org/apache/pig/builtin/TOTUPLE.java 1618727 trunk/src/org/apache/pig/builtin/TRIM.java 1618727
[jira] [Commented] (PIG-4128) New logical optimizer rule: ConstantCalculator
[ https://issues.apache.org/jira/browse/PIG-4128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14111505#comment-14111505 ] Thejas M Nair commented on PIG-4128: +1 New logical optimizer rule: ConstantCalculator -- Key: PIG-4128 URL: https://issues.apache.org/jira/browse/PIG-4128 Project: Pig Issue Type: New Feature Components: impl Reporter: Daniel Dai Assignee: Daniel Dai Fix For: 0.14.0 Attachments: PIG-4128-1.patch, PIG-4128-2.patch, PIG-4128-3.patch Pig used to have a LogicExpressionSimplifier to simplify expression which also calculates constant expression. The optimizer rule is buggy and we disable it by default in PIG-2316. However, we do need this feature especially in partition/predicate push down, since both does not deal with complex constant expression, we'd like to replace the expression with constant before the actual push down. Yes, user may manually do the calculation and rewrite the query, but even rewrite is sometimes not possible. Consider the case user want to push a datetime predicate, user have to write a ToDate udf since Pig does not have datetime constant. In this Jira, I provide a new rule: ConstantCalculator, which is much simpler and much less error prone, to replace LogicExpressionSimplifier. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (PIG-4128) New logical optimizer rule: ConstantCalculator
[ https://issues.apache.org/jira/browse/PIG-4128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai resolved PIG-4128. - Resolution: Fixed Hadoop Flags: Reviewed Patch committed to trunk. Thanks Thejas for review! New logical optimizer rule: ConstantCalculator -- Key: PIG-4128 URL: https://issues.apache.org/jira/browse/PIG-4128 Project: Pig Issue Type: New Feature Components: impl Reporter: Daniel Dai Assignee: Daniel Dai Fix For: 0.14.0 Attachments: PIG-4128-1.patch, PIG-4128-2.patch, PIG-4128-3.patch Pig used to have a LogicExpressionSimplifier to simplify expression which also calculates constant expression. The optimizer rule is buggy and we disable it by default in PIG-2316. However, we do need this feature especially in partition/predicate push down, since both does not deal with complex constant expression, we'd like to replace the expression with constant before the actual push down. Yes, user may manually do the calculation and rewrite the query, but even rewrite is sometimes not possible. Consider the case user want to push a datetime predicate, user have to write a ToDate udf since Pig does not have datetime constant. In this Jira, I provide a new rule: ConstantCalculator, which is much simpler and much less error prone, to replace LogicExpressionSimplifier. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] Subscription: PIG patch available
Issue Subscription Filter: PIG patch available (15 issues) Subscriber: pigdaily Key Summary PIG-4111Make Pig compiles with avro-1.7.7 https://issues.apache.org/jira/browse/PIG-4111 PIG-4103Fix TestRegisteredJarVisibility(after PIG-4083) https://issues.apache.org/jira/browse/PIG-4103 PIG-4066An optimization for ROLLUP operation in Pig https://issues.apache.org/jira/browse/PIG-4066 PIG-4004Upgrade the Pigmix queries from the (old) mapred API to mapreduce https://issues.apache.org/jira/browse/PIG-4004 PIG-4002Disable combiner when map-side aggregation is used https://issues.apache.org/jira/browse/PIG-4002 PIG-3952PigStorage accepts '-tagSplit' to return full split information https://issues.apache.org/jira/browse/PIG-3952 PIG-3911Define unique fields with @OutputSchema https://issues.apache.org/jira/browse/PIG-3911 PIG-3877Getting Geo Latitude/Longitude from Address Lines https://issues.apache.org/jira/browse/PIG-3877 PIG-3873Geo distance calculation using Haversine https://issues.apache.org/jira/browse/PIG-3873 PIG-3866Create ThreadLocal classloader per PigContext https://issues.apache.org/jira/browse/PIG-3866 PIG-3861duplicate jars get added to distributed cache https://issues.apache.org/jira/browse/PIG-3861 PIG-3668COR built-in function when atleast one of the coefficient values is NaN https://issues.apache.org/jira/browse/PIG-3668 PIG-3635Fix e2e tests for Hadoop 2.X on Windows https://issues.apache.org/jira/browse/PIG-3635 PIG-3587add functionality for rolling over dates https://issues.apache.org/jira/browse/PIG-3587 PIG-3441Allow Pig to use default resources from Configuration objects https://issues.apache.org/jira/browse/PIG-3441 You may edit this subscription at: https://issues.apache.org/jira/secure/FilterSubscription!default.jspa?subId=13225filterId=12322384
