Re: Pig 0.11

2012-10-28 Thread Rohini Palaniswamy
Hi all, Tried to take a look at the jira's Olga mentioned. Thought it would be easier to compile a list instead of reviewing and commenting on each jira. Here is a initial list that I compiled as a kick starter for the discussions. Query used:

Re: Are people ok with putting PIG-3017 into pig 11?

2012-10-30 Thread Rohini Palaniswamy
+1. Agree with Julien. Error on hitting Configuration's size limit is a bug. On Tue, Oct 30, 2012 at 1:40 PM, Julien Le Dem jul...@twitter.com wrote: To me in falls in the category of bug fixes we can add to an existing version. It is not adding any feature, it is fixing a bug/issue It is

Re: Review Request: PIG-2978 TestLoadStoreFuncLifeCycle fails with hadoop-2.0.x

2012-11-19 Thread Rohini Palaniswamy
that the additional call is fine. LocalJobRunner gets the outputcommitter to call setupJob (MAPREDUCE-3563) which was not done in H20's LocalJobRunner. - Rohini Palaniswamy On Nov. 19, 2012, 5:28 a.m., Cheolsoo Park wrote

Review Request: [PIG-2907] Publish pig 0.23 jars to maven

2012-11-20 Thread Rohini Palaniswamy
ant clean mvn-deploy -Dasfrepo=http://localhost:8089/nexus Thanks, Rohini Palaniswamy

Re: Review Request: [PIG-2907] Publish pig 0.23 jars to maven

2012-11-30 Thread Rohini Palaniswamy
/build.xml 1415689 Diff: https://reviews.apache.org/r/8157/diff/ Testing --- Tested with a local nexus repository using the command ant clean mvn-deploy -Dasfrepo=http://localhost:8089/nexus Thanks, Rohini Palaniswamy

Re: Prepare for Pig 0.10.1 release

2012-12-19 Thread Rohini Palaniswamy
Daniel, Since 0.10.1 is going out sooner and has support for hadoop 23, if it is acceptable, I would like to pull in PIG-2907 into 0.10.1. Other projects are blocked (hcat, oozie) on running unit tests with hadoop 23 because we don't have a pig jar for 23 in maven. Thoughts/objections?

Re: Prepare for Pig 0.10.1 release

2012-12-20 Thread Rohini Palaniswamy
, 2012 at 9:55 PM, Julien Le Dem jul...@twitter.com wrote: no objection to PIG-2907 in 0.10.1 However I do think new features should go in trunk. Julien On Wed, Dec 19, 2012 at 1:20 PM, Rohini Palaniswamy rohini.adi...@gmail.com wrote: Daniel, Since 0.10.1 is going out sooner

Re: [VOTE] Release Pig 0.10.1 (candidate 0)

2012-12-26 Thread Rohini Palaniswamy
Daniel, The tarball still has references to 0.10.0 in the documentation. RELEASE_NOTES.txt is not updated with 0.10.1 version number and Highlights section details are that of 0.10.0. docs/index.html in tarball still says Pig 0.10.0 documentation Regards, Rohini On Fri, Dec 21, 2012 at

Re: [VOTE] Release Pig 0.10.1 (candidate 3)

2013-01-03 Thread Rohini Palaniswamy
+1. Downloaded the tar binary, checked signature, ran unit tests, piggybank unit tests, checked docs/release notes, ran a simple script locally and against a cluster. On Mon, Dec 31, 2012 at 8:41 AM, Alan Gates ga...@hortonworks.com wrote: +1, yet again :). Checked the key signature and

Re: Run a job async

2013-01-25 Thread Rohini Palaniswamy
Jon, Those are good areas to check. Few things I have seen regarding those are 1) JythonScriptEngine -PythonInterpreter is static and is not suitable for multiple runs if the script names are same (hit this issue in PIG-2433 unit tests). 2) QueryParserDriver - There is a static cache with

Re: Pig 11.0

2013-01-25 Thread Rohini Palaniswamy
Thats good :). Unit tests have all been passing. I haven't run e2e tests on pig 0.11 for sometime. Will kick off one this weekend. It would be nice if Cheolsoo and Daniel can also kick off one run. Regards, Rohini On Fri, Jan 25, 2013 at 4:08 PM, Julien Le Dem jul...@twitter.com wrote: It

Re: Pig 11.0

2013-01-28 Thread Rohini Palaniswamy
will also run e2e on Hadoop-1.x and Hadoop-2.x. On Fri, Jan 25, 2013 at 5:02 PM, Daniel Dai da...@hortonworks.com wrote: I will run e2e tests on Hadoop 1.x over the weekend. Thanks, Daniel On Fri, Jan 25, 2013 at 4:27 PM, Rohini Palaniswamy rohini.adi...@gmail.com

Re: Pig 11 release notes - draft

2013-02-07 Thread Rohini Palaniswamy
Boolean data type is available from 0.10 Regards, Rohini On Thu, Feb 7, 2013 at 1:24 PM, Bill Graham billgra...@gmail.com wrote: Hi, In the Pig 11 release I'd like to include some specifics in the RELEASE_NOTES.txt file. Julien compiled this list (which I've tweaked a bit). Please give it

Re: Pig 11 release notes - draft

2013-02-07 Thread Rohini Palaniswamy
This is something Daniel mentioned in his talk in pig meetup and is missing in this list PIG-2482 - Integrate HCat DDL command into Pig Also there are a lot of UDFs added for DateTime datatype. A mention of that would be good. Regards, Rohini On Thu, Feb 7, 2013 at 2:27 PM, Rohini Palaniswamy

Re: Missing ANTLR dependency in Pig 0.10.1

2013-02-19 Thread Rohini Palaniswamy
You mean it is not pulled as a transitive dependency? Currently you have to manually specify that as a dependency in your pom. Can you file a jira to make that part of the pig pom? Regards, Rohini On Tue, Feb 19, 2013 at 10:42 AM, Minh LĂȘ ngocminh@gmail.com wrote: I tried to run PigServer

Review Request: [PIG-3173] - Partition filter pushdown does not happen if partition keys condition include a AND and OR construct

2013-03-19 Thread Rohini Palaniswamy
/newplan/PColFilterExtractor.java 1458047 http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/TestPartitionFilterPushDown.java 1458047 Diff: https://reviews.apache.org/r/10035/diff/ Testing --- Unit tests added and tested few cases manually with hcat. Thanks, Rohini

Re: Review Request: [PIG-3173] - Partition filter pushdown does not happen if partition keys condition include a AND and OR construct

2013-03-20 Thread Rohini Palaniswamy
., Rohini Palaniswamy wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/10035/ --- (Updated March 20, 2013, 12:16 a.m.) Review

Re: Anybody using custom Serializer/Deserializer in Pig Streaming?

2013-03-20 Thread Rohini Palaniswamy
Nice summarization Koji. Wish we had some object that has byte[] and length instead of byte[] as the return type of serialize() and method param of deserialize(). That would enable reuse and cut down on some of the copy. At least there is one copy we can cut down without any API changes by having

Re: Review Request: [PIG-3173] - Partition filter pushdown does not happen if partition keys condition include a AND and OR construct

2013-04-29 Thread Rohini Palaniswamy
/apache/pig/test/TestPartitionFilterPushDown.java 1467858 Diff: https://reviews.apache.org/r/10035/diff/ Testing --- Unit tests added and tested few cases manually with hcat. Thanks, Rohini Palaniswamy

Re: Review Request: PIG-3223 AvroStorage does not handle comma separated input paths

2013-04-30 Thread Rohini Palaniswamy
/fs/FileSystem.html#globStatus(org.apache.hadoop.fs.Path). Found this method in pig which would take care of the logic - LoadFunc.getPathStrings() . Use this for splitting paths. This should simplify the whole change - Rohini Palaniswamy On April 8, 2013, 10:03 p.m., Johnny

Re: Welcome our newest committer Prashant Kommireddi

2013-05-02 Thread Rohini Palaniswamy
Congrats Prashant !! On Thu, May 2, 2013 at 3:58 PM, Ashutosh Chauhan ashutosh.chau...@gmail.com wrote: Congrats Prashant. Hopefully your contributions to Pig will keep flowing in :) Ashutosh On Thu, May 2, 2013 at 3:41 PM, Mike Sukmanowsky m...@parsely.com wrote: Congrats! On

Re: Review Request: PIG-3223 AvroStorage does not handle comma separated input paths

2013-05-03 Thread Rohini Palaniswamy
is redundant - Rohini Palaniswamy On May 3, 2013, 12:33 a.m., Johnny Zhang wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/10351

Re: Review Request: PIG-3223 AvroStorage does not handle comma separated input paths

2013-05-03 Thread Rohini Palaniswamy
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/10351/#review20138 --- Ship it! Thanks Johnny. Looks good. - Rohini Palaniswamy On May

CHANGES.txt in trunk

2013-05-03 Thread Rohini Palaniswamy
Hi, I see lot of patches that went into 0.11 are under trunk in the CHANGES.txt. Should we sync the file with the CHANGES.txt in branch-0.11 and remove those jiras from trunk that went into 0.11? What is the usual process of updating CHANGES.txt when a jira is checked both into a branch and

Re: CHANGES.txt in trunk

2013-05-03 Thread Rohini Palaniswamy
I will put up the patch Daniel. Thanks, Rohini On Fri, May 3, 2013 at 1:38 PM, Daniel Dai da...@hortonworks.com wrote: Sure, I used to clean this up before release, but not strictly follow this rule. Patch welcome. Thanks, Daniel On Fri, May 3, 2013 at 1:34 PM, Rohini Palaniswamy

Re: CHANGES.txt in trunk

2013-05-05 Thread Rohini Palaniswamy
. But the trunk CHANGES file should include all changes that are on trunk. Alan. On May 3, 2013, at 1:34 PM, Rohini Palaniswamy wrote: Hi, I see lot of patches that went into 0.11 are under trunk in the CHANGES.txt. Should we sync the file with the CHANGES.txt in branch-0.11 and remove

Re: Pig package supporting both hadoop 1 and 2

2013-05-06 Thread Rohini Palaniswamy
Hi Julien, We use a perl script internally instead of bin/pig shell script which has some Y! deployment stuff. That is why changes have not been done to bin/pig to support both versions already. We have two lib directories - one for hadoop20, other for hadoop23(hadoop2) and choose the one to

Re: Unit test classpath trouble

2013-05-13 Thread Rohini Palaniswamy
Johnny, You need to increase PermGen space for 2.x else TestGrunt will hang with the OOM. Run ant with -DdebugArgs=-XX:+CMSClassUnloadingEnabled -XX:MaxPermSize=128M Regards, Rohini On Mon, May 13, 2013 at 10:22 AM, Johnny Zhang xiao...@cloudera.com wrote: I run unit test against

Re: Review Request: Changes for PIG-3321

2013-05-14 Thread Rohini Palaniswamy
/branch-0.11/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/storage/avro/PigAvroRecordReader.java https://reviews.apache.org/r/11155/#comment42392 writerSchema = readerSchema - Rohini Palaniswamy On May 14, 2013, 10:18 p.m., Harvey Chong wrote

Re: Review Request: Patch to address default values when schemas are merged in AvroStorage. It does this for Records containing primitive values

2013-05-14 Thread Rohini Palaniswamy
spaces. This patch will conflict with PIG-3321. Can you merge the changes once that is committed and upload a new patch? - Rohini Palaniswamy On May 14, 2013, 1:09 a.m., Viraj Bhat wrote: --- This is an automatically generated e-mail

Re: Review Request: Changes for PIG-3321

2013-05-15 Thread Rohini Palaniswamy
On May 14, 2013, 11:45 p.m., Rohini Palaniswamy wrote: http://svn.apache.org/repos/asf/pig/branches/branch-0.11/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/storage/avro/AvroStorage.java, line 92 https://reviews.apache.org/r/11155/diff/1/?file=291819#file291819line92

Re: Review Request: Changes for PIG-3321

2013-05-15 Thread Rohini Palaniswamy
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/11155/#review20613 --- Ship it! Ship It! - Rohini Palaniswamy On May 15, 2013, 7:01

Re: Review Request: PIG-3318 Patch to address default values when schemas are merged in AvroStorage. It does this for Records containing primitive values

2013-05-26 Thread Rohini Palaniswamy
schema, has the default value won't it be automatically set? - Rohini Palaniswamy On May 23, 2013, 12:12 a.m., Viraj Bhat wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/11135

Re: Review Request: PIG-3322 Fix the issue where NPE is thrown when reading a union which has nulls and add a testcase

2013-06-03 Thread Rohini Palaniswamy
://reviews.apache.org/r/11333/#comment44211 golden output http://svn.apache.org/repos/asf/pig/trunk/contrib/piggybank/java/src/test/java/org/apache/pig/piggybank/test/storage/avro/TestAvroStorage.java https://reviews.apache.org/r/11333/#comment44212 fileOutput - Rohini Palaniswamy

Re: Review Request: PIG-3322 Fix the issue where NPE is thrown when reading a union which has nulls and add a testcase

2013-06-03 Thread Rohini Palaniswamy
/src/test/java/org/apache/pig/piggybank/test/storage/avro/TestAvroStorage.java https://reviews.apache.org/r/11333/#comment44214 Isn't a load and store enough to reproduce the test case? Why such a long pig script? - Rohini Palaniswamy On May 29, 2013, 11:07 p.m., Viraj Bhat wrote

Re: Review Request: PIG-3331 Default values not written to Schema when specified in the output schema

2013-06-03 Thread Rohini Palaniswamy
/java/org/apache/pig/piggybank/test/storage/avro/TestAvroStorage.java https://reviews.apache.org/r/11355/#comment44215 Isn't a load and store enough to reproduce the test case? Why such a long pig script? Please try to keep the unit tests simple. - Rohini Palaniswamy On May 30, 2013, 2:29

Re: Review Request: PIG-3322 Fix the issue where NPE is thrown when reading a union which has nulls and add a testcase

2013-06-03 Thread Rohini Palaniswamy
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/11333/#review21383 --- Ship it! Ship It! - Rohini Palaniswamy On June 4, 2013, 12:15

Re: Review Request: PIG-3342 Allow conditions in case statement

2013-06-04 Thread Rohini Palaniswamy
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/11613/#review21419 --- Ship it! Ship It! - Rohini Palaniswamy On June 4, 2013, 1:46

Re: Review Request: PIG-3331 Default values not written to Schema when specified in the output schema

2013-06-04 Thread Rohini Palaniswamy
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/11355/#review21455 --- Ship it! Ship It! - Rohini Palaniswamy On June 4, 2013, 11:23

Uploading patches for review

2013-06-05 Thread Rohini Palaniswamy
Hi, Reviewing uploaded patches for few lines of change is easy. But when the change is more it is hard to read, review is more time consuming and at times you have to switch between the patch and eclipse to get more context. Without the surrounding code it is also easy to miss things on

Re: Review Request: PIG-3318 Patch to address default values when schemas are merged in AvroStorage. It does this for Records containing primitive values

2013-06-10 Thread Rohini Palaniswamy
/pig/trunk/contrib/piggybank/java/src/test/java/org/apache/pig/piggybank/test/storage/avro/TestAvroStorage.java https://reviews.apache.org/r/11135/#comment44778 testMultipleSchemasWithDefaultValue - Rohini Palaniswamy On May 30, 2013, 2:28 a.m., Viraj Bhat wrote

Re: Review Request: PIG-3318 Patch to address default values when schemas are merged in AvroStorage. It does this for Records containing primitive values

2013-06-11 Thread Rohini Palaniswamy
/#comment44867 formatting. space after , http://svn.apache.org/repos/asf/pig/trunk/contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/storage/avro/PigAvroRecordReader.java https://reviews.apache.org/r/11135/#comment44868 formatting. Code in next line - Rohini Palaniswamy On June 11

Re: Review Request: PIG-3318 Patch to address default values when schemas are merged in AvroStorage. It does this for Records containing primitive values

2013-06-12 Thread Rohini Palaniswamy
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/11135/#review21792 --- Ship it! Ship It! - Rohini Palaniswamy On June 12, 2013, 2:05

Review Request: [PIG-3341] Strict datetime parsing and improve performance of loading datetime values

2013-06-12 Thread Rohini Palaniswamy
Thanks, Rohini Palaniswamy

Re: Review Request: [PIG-3341] Strict datetime parsing and improve performance of loading datetime values

2013-06-12 Thread Rohini Palaniswamy
Diff: https://reviews.apache.org/r/11834/diff/ Testing --- Unit tests modified Thanks, Rohini Palaniswamy

Re: Improving macros

2013-06-19 Thread Rohini Palaniswamy
Jon is right. I am trying to ensure that each line is mostly parsed only once in https://issues.apache.org/jira/browse/PIG-3204. Have few issues with other commands in pig script like fs, shell, cd, illustrate, error messages not showing line numbers properly, etc which I have not got to solving

Re: Review Request 13122: PIG-3400 FS commands do not work with S3 paths

2013-07-31 Thread Rohini Palaniswamy
should be documented as part of http://pig.apache.org/docs/r0.11.1/cmds.html#fs and not http://pig.apache.org/docs/r0.11.1/cmds.html#utillity-cmds. Utility cmds are grunt commands. - Rohini Palaniswamy On July 31, 2013, 7:37 p.m., Cheolsoo Park wrote

Re: Review Request 13122: PIG-3400 FS commands do not work with S3 paths

2013-07-31 Thread Rohini Palaniswamy
system implementation. So can mention that. Same for other fs commands. src/docs/src/documentation/content/xdocs/cmds.xml https://reviews.apache.org/r/13122/#comment48263 White spaces in many places - Rohini Palaniswamy On July 31, 2013, 7:37 p.m., Cheolsoo Park wrote

Re: Review Request 13186: PIG-3395 Large filter expression makes Pig hang

2013-08-01 Thread Rohini Palaniswamy
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/13186/#review24473 --- Ship it! Ship It! - Rohini Palaniswamy On Aug. 1, 2013, 4:14

Re: Review Request 13122: PIG-3400 FS commands do not work with S3 paths

2013-08-01 Thread Rohini Palaniswamy
Getting rid of 2) may break existing user scripts. We can keep your s3 fix, but can continue to not document it. Probably also add a deprecated note to documentation, in the fs commands section saying cat, cd, ls, move, etc commands now work even without specify fs, but that is deprecated and will

Review Request 13535: [PIG-3204] Reduce the number of getSchema calls during script parsing

2013-08-13 Thread Rohini Palaniswamy
to track the number of times a line is parsed. TestGrunt and TestShortcuts test failures fixed. Thanks, Rohini Palaniswamy

Re: Review Request 13551: PIG-3414 Utils.getSchemaFromString() silently returns a wrong result when a comma is missing in the schema definition

2013-08-14 Thread Rohini Palaniswamy
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/13551/#review25150 --- Ship it! Ship It! - Rohini Palaniswamy On Aug. 14, 2013, 5:27

Re: Review Request 13535: [PIG-3204] Reduce the number of getSchema calls during script parsing

2013-08-14 Thread Rohini Palaniswamy
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/13535/#review25098 --- On Aug. 13, 2013, 2:34 p.m., Rohini Palaniswamy wrote

Re: Review Request 13535: [PIG-3204] Reduce the number of getSchema calls during script parsing

2013-08-14 Thread Rohini Palaniswamy
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/13535/#review25173 --- On Aug. 13, 2013, 2:34 p.m., Rohini Palaniswamy wrote: --- This is an automatically

Re: Review Request 13535: [PIG-3204] Reduce the number of getSchema calls during script parsing

2013-08-14 Thread Rohini Palaniswamy
, but numTimesInitiated is set to 10. _testSkipParseInRegisterForBatch(false, 10, 4); Rohini Palaniswamy wrote: It is 10 (4 + 6). Will change the hyphen to full stop so that it is not mistaken as minus. Cheolsoo Park wrote: I see. Thanks! Btw, why are there 7 function

Re: Review Request 13601: AvroStorage Failed to read paths separated by commas

2013-08-15 Thread Rohini Palaniswamy
/org/apache/pig/builtin/AvroStorage.java https://reviews.apache.org/r/13601/#comment49519 Can we remove this method? - Rohini Palaniswamy On Aug. 15, 2013, 7:09 p.m., Yuanli Dong Yuanli Dong wrote: --- This is an automatically

Hadoop 2.3 and pig

2013-09-03 Thread Rohini Palaniswamy
I know many of you are trying out Hadoop 2.x. Just FYI for those to save time if they hit the following issue when they are building directly off the branch. pig joins (replication, skewed and merge joins) are broken with a recent checkin - HADOOP-9877. For details refer

Review Request 14030: [PIG-3255] Avoid extra byte array copy in streaming deserialize

2013-09-08 Thread Rohini Palaniswamy
1518333 http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/TestStreaming.java 1518333 Diff: https://reviews.apache.org/r/14030/diff/ Testing --- No new unit tests. Only perf changes. TestStreaming tests passes Thanks, Rohini Palaniswamy

Re: Review Request 14030: [PIG-3255] Avoid extra byte array copy in streaming deserialize

2013-09-13 Thread Rohini Palaniswamy
/src/org/apache/pig/impl/util/StorageUtil.java 1523070 Diff: https://reviews.apache.org/r/14030/diff/ Testing (updated) --- No new unit tests. TestStreaming and TestStreamingLocal cover the changes and the tests passes. Thanks, Rohini Palaniswamy

Re: Review Request 14274: PIG-2672 Optimize the use of DistributedCache

2013-09-24 Thread Rohini Palaniswamy
On Sept. 25, 2013, 12:13 a.m., Rohini Palaniswamy wrote: trunk/src/org/apache/pig/backend/hadoop/executionengine/mapReduceLayer/JobControlCompiler.java, line 1495 https://reviews.apache.org/r/14274/diff/1/?file=355174#file355174line1495 Since the name of the file on hdfs

Re: [Discussion] Any thoughts on PIG-3457?

2013-09-30 Thread Rohini Palaniswamy
+1. I was already asking for keeping the new API changes only in Tez branch till it evolves and is finalized, so I have no objections to reverting it. Regards, Rohini On Mon, Sep 30, 2013 at 1:28 PM, Alan Gates ga...@hortonworks.com wrote: We should separate out two separate concerns. If I

Review Request: [PIG-2908] Fix unit tests to work with jdk7

2012-09-19 Thread Rohini Palaniswamy
1387698 http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/junit/OrderedJUnit4Runner.java PRE-CREATION Diff: https://reviews.apache.org/r/7176/diff/ Testing --- This patch fixes tests Thanks, Rohini Palaniswamy

Re: Review Request: PIG-2898: allow to run pig e2e tests in parallel mode.

2012-09-20 Thread Rohini Palaniswamy
of IPC::Run (which internally does fork and 1 sec sleep). - Rohini Palaniswamy On Sept. 12, 2012, 8:31 a.m., Ivan Veselovsky wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/7053

Re: How to specify temporary dirs in Pig local mode?

2012-09-21 Thread Rohini Palaniswamy
Ilya, Could you try passing -Dmapred.local.dir=random generated tmp location ( -Dyarn.nodemanager.local-dirs=random generated tmp location in case of hadoop 23) when launching pig local mode tests and see if that works. TestDriver.pm already has a block that passes additional java_params to

Re: Review Request: PIG-2898: allow to run pig e2e tests in parallel mode.

2012-10-02 Thread Rohini Palaniswamy
to be fixed before 0.11 is released. The test runs fine in 0.10 in seq mode. Removing this will also make the patch apply cleanly on 0.10 else we need to create a patch for 0.10. - Rohini Palaniswamy On Sept. 28, 2012, 12:20 p.m., Ivan Veselovsky wrote

Re: Review Request: PIG-2898: allow to run pig e2e tests in parallel mode.

2012-10-04 Thread Rohini Palaniswamy
to upload this patch in the jira. - Rohini Palaniswamy On Oct. 3, 2012, 12:53 p.m., Ivan Veselovsky wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/7053

Re: Unit test failures

2012-10-15 Thread Rohini Palaniswamy
have access to the build right now. Daniel seemed to know more about that. Julien On Mon, Oct 15, 2012 at 11:15 AM, Rohini Palaniswamy rohini.adi...@gmail.com wrote: Hi all, Quite a few unit tests are broken in trunk/0.11. Now that 0.11 is branched it is more

Re: Unit test failures

2012-10-15 Thread Rohini Palaniswamy
/jira/browse/PIG-2793 Alan. On Oct 15, 2012, at 1:26 PM, Rohini Palaniswamy wrote: Me and Cheolsoo are kicking off a new run for the tests to check the current failures. Will update the jira with those. Interested folks can pick up the sub-task jiras. Should we create another umbrella

Re: [VOTE] Release Pig 0.12.0 (candidate 2)

2013-10-08 Thread Rohini Palaniswamy
Daniel, Can we have https://issues.apache.org/jira/browse/PIG-3492https://issues.apache.org/jira/browse/PIG-3492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel included ? We have many scripts failing in 0.11 because of that issue. Regards, Rohini On Tue, Oct 8, 2013 at

Re: [VOTE] Release Pig 0.12.0 (candidate 2)

2013-10-08 Thread Rohini Palaniswamy
I guess it is ok. Regards, Rohini On Tue, Oct 8, 2013 at 3:18 PM, Daniel Dai da...@hortonworks.com wrote: Hi, Rohini, Is it Ok to put it in 0.12.1? It will take sometime to get it in, and redo the RC. Thanks, Daniel On Tue, Oct 8, 2013 at 3:09 PM, Rohini Palaniswamy rohini.adi

Re: Review Request 14801: PIG-3531 Add TezMiniCluster for unit tests

2013-10-22 Thread Rohini Palaniswamy
that to MiniGenericCluster.buildCluster(); to be cleaner. - Rohini Palaniswamy On Oct. 22, 2013, 1:02 a.m., Cheolsoo Park wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/14801

Re: Review Request 15261: PIG-3555 Initial implementation of Tez combiner optimization

2013-11-06 Thread Rohini Palaniswamy
. - Rohini Palaniswamy On Nov. 6, 2013, 11:04 a.m., Cheolsoo Park wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15261/ --- (Updated

Re: Review Request 15261: PIG-3555 Initial implementation of Tez combiner optimization

2013-11-06 Thread Rohini Palaniswamy
On Nov. 6, 2013, 7:46 p.m., Rohini Palaniswamy wrote: src/org/apache/pig/backend/hadoop/executionengine/tez/TezDagBuilder.java, line 156 https://reviews.apache.org/r/15261/diff/1/?file=379002#file379002line156 This + newEdges method does not seem to be right to me. Going

Re: Review Request 15261: PIG-3555 Initial implementation of Tez combiner optimization

2013-11-08 Thread Rohini Palaniswamy
/TezPrinter.java https://reviews.apache.org/r/15261/#comment55442 Can we print the combine plan after the plan on the vertex for easy readability? - Rohini Palaniswamy On Nov. 8, 2013, 7:57 a.m., Cheolsoo Park wrote

Re: Review Request 15634: PIG-3525 PigStats.get() and ScriptState.get() shouldn't return MR-specific objects

2013-11-17 Thread Rohini Palaniswamy
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/15634/#review29032 --- Ship it! Ship It! - Rohini Palaniswamy On Nov. 18, 2013, 1:23

Re: Review Request 15931: PIG-3585 Implement union in Tez

2013-12-01 Thread Rohini Palaniswamy
that in a single vertex to be more optimal. We can handle that in a separate jira though. - Rohini Palaniswamy On Dec. 1, 2013, 7 a.m., Cheolsoo Park wrote: --- This is an automatically generated e-mail. To reply, visit: https

Re: Review Request 15931: PIG-3585 Implement union in Tez

2013-12-01 Thread Rohini Palaniswamy
On Dec. 1, 2013, 5:06 p.m., Rohini Palaniswamy wrote: The code is fine if we have union after some processing. But for simple load and union case as below, this will create 3 vertices - 2 load vertices and one union vertex. a = load 'a' b = load 'b' c = union a, b In MR

Review Request 15949: PIG-3565 Make split work with Tez

2013-12-02 Thread Rohini Palaniswamy
. Will update tez.conf with e2e tests in a separate jira later. Thanks, Rohini Palaniswamy

Re: Review Request 15949: PIG-3565 Make split work with Tez

2013-12-03 Thread Rohini Palaniswamy
. Thanks, Rohini Palaniswamy

Review Request 16272: [PIG-3620] TezCompiler adds duplicate predecessors of blocking operators to TezPlan

2013-12-15 Thread Rohini Palaniswamy
/TestTezCompiler.java 1550856 Diff: https://reviews.apache.org/r/16272/diff/ Testing --- - Added tests to TestTezCompiler - Will add the e2e tests for Split with PIG-3626. MR multi-query is also broken now. Need to fix that as well for e2e to work. Thanks, Rohini Palaniswamy

Re: Review Request 16313: PIG-3604 Implement replicated join in Tez

2013-12-17 Thread Rohini Palaniswamy
vertex (POLocalRearrange) just changing the edge type to broadcast. Don't think the TezCompiler handles this now. - Rohini Palaniswamy On Dec. 17, 2013, 3:51 a.m., Cheolsoo Park wrote: --- This is an automatically generated e-mail

Re: Review Request 16313: PIG-3604 Implement replicated join in Tez

2013-12-17 Thread Rohini Palaniswamy
On Dec. 17, 2013, 3:52 p.m., Rohini Palaniswamy wrote: The approach is good when the replicate join is not the first vertex of the DAG (i.e in case of a MR, replicate join is part of a reduce). If it is the first vertex of the DAG, we need to compare and see that with this approach

Re: Review Request 16313: PIG-3604 Implement replicated join in Tez

2013-12-18 Thread Rohini Palaniswamy
On Dec. 17, 2013, 3:52 p.m., Rohini Palaniswamy wrote: test/org/apache/pig/tez/TestTezCompiler.java, line 216 https://reviews.apache.org/r/16313/diff/1/?file=398711#file398711line216 Can we add cases for - three or four way join? - replicated table is part of a reduce

Re: Review Request 16313: PIG-3604 Implement replicated join in Tez

2013-12-18 Thread Rohini Palaniswamy
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16313/#review30622 --- Ship it! Ship It! - Rohini Palaniswamy On Dec. 18, 2013, 3:04

Re: Review Request 16463: PIG-3636 Implement accumulator optimization in Tez

2013-12-26 Thread Rohini Palaniswamy
mentioning the issues that you faced so far with session reuse so that we come back later and fix them some time. - Rohini Palaniswamy On Dec. 26, 2013, 6:11 a.m., Cheolsoo Park wrote: --- This is an automatically generated e-mail. To reply

Re: Review Request 16309: PIG-3629 Implement STREAM operator in Tez

2013-12-26 Thread Rohini Palaniswamy
in TezResourceManager, TezOperPlan that have hdfs in to say defaultFS, srcFS or remoteFS as appropriate. - Rohini Palaniswamy On Dec. 24, 2013, 1:34 a.m., Alex Bain wrote: --- This is an automatically generated e-mail. To reply, visit: https

Re: Review Request 16463: PIG-3636 Implement accumulator optimization in Tez

2013-12-26 Thread Rohini Palaniswamy
On Dec. 26, 2013, 4:45 p.m., Rohini Palaniswamy wrote: test/org/apache/pig/test/TestAccumulator.java, line 114 https://reviews.apache.org/r/16463/diff/1/?file=403084#file403084line114 Why do we specifically need homeDirOnDfs? Isn't that the default current directory? Cheolsoo

Re: Review Request 16463: PIG-3636 Implement accumulator optimization in Tez

2013-12-26 Thread Rohini Palaniswamy
On Dec. 26, 2013, 4:45 p.m., Rohini Palaniswamy wrote: test/org/apache/pig/test/TestAccumulator.java, line 189 https://reviews.apache.org/r/16463/diff/1/?file=403084#file403084line189 Any reason for adding this newly within tests? tearDown() already has it. Cheolsoo Park wrote

Re: Review Request 16463: PIG-3636 Implement accumulator optimization in Tez

2013-12-26 Thread Rohini Palaniswamy
On Dec. 26, 2013, 4:45 p.m., Rohini Palaniswamy wrote: test/org/apache/pig/test/TestAccumulator.java, line 189 https://reviews.apache.org/r/16463/diff/1/?file=403084#file403084line189 Any reason for adding this newly within tests? tearDown() already has it. Cheolsoo Park wrote

Re: Review Request 16463: PIG-3636 Implement accumulator optimization in Tez

2013-12-26 Thread Rohini Palaniswamy
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16463/#review30905 --- Ship it! Ship It! - Rohini Palaniswamy On Dec. 27, 2013, 6:31

Pig User Group Meetup at LinkedIn on Fri Mar 14

2014-01-14 Thread Rohini Palaniswamy
Please join us for the Pig User Group Meetup this quarter at LinkedIn on Fri Mar 14. We have some interesting talks lined up on the recent developments in Pig. RSVP at http://www.meetup.com/PigUser/events/160604192/ Tentative lineup for this meetup: Pig on Tez Pig on Storm Intel Graph Builder

Re: Welcome to the new Pig PMC member Aniket Mokashi

2014-01-15 Thread Rohini Palaniswamy
Congrats Aniket! On Wed, Jan 15, 2014 at 10:12 AM, Mona Chitnis m...@apache.org wrote: Congrats Aniket! Good work! -- Mona Chitnis Software Engineer, Hadoop Team Yahoo! On Wednesday, January 15, 2014 9:17 AM, Xuefu Zhang xzh...@cloudera.com wrote: Congratulations, Aniket!

Review Request 17191: [PIG-3626] Make combiners, custom partitioners and secondary key sort work for multiple outputs

2014-01-22 Thread Rohini Palaniswamy
with this patch. Others pass. Will try and fix that shortly or in a different jira. Would like to get this patch in quickly as it has been long pending and rebasing is a pain. Thanks, Rohini Palaniswamy

Re: Review Request 17191: [PIG-3626] Make combiners, custom partitioners and secondary key sort work for multiple outputs

2014-01-22 Thread Rohini Palaniswamy
generated e-mail. To reply, visit: https://reviews.apache.org/r/17191/#review32581 --- On Jan. 22, 2014, 6:18 p.m., Rohini Palaniswamy wrote: --- This is an automatically generated e-mail

Re: Review Request 17266: [PIG-3661] Piggybank AvroStorage fails if used in more than one load or store statement

2014-01-24 Thread Rohini Palaniswamy
in git but failing when run in svn code base due to hidden .svn files(PIG-3717). That passes as well. Thanks, Rohini Palaniswamy

Review Request 17379: [PIG-3658] Use Tez ObjectRegistry to cache FRJoin map and WeightedRangePartitioner map

2014-01-26 Thread Rohini Palaniswamy
/diff/ Testing --- Operators and Join e2e pass. TEZ-711 is a issue when running on scale. Thanks, Rohini Palaniswamy

Re: Review Request 17379: [PIG-3658] Use Tez ObjectRegistry to cache FRJoin map and WeightedRangePartitioner map

2014-01-26 Thread Rohini Palaniswamy
1561449 Diff: https://reviews.apache.org/r/17379/diff/ Testing --- Operators and Join e2e pass. TEZ-711 is a issue when running on scale. Thanks, Rohini Palaniswamy

Re: Review Request 17439: PIG-3728: Fix TestSkewedJoin unit test in tez mode

2014-01-27 Thread Rohini Palaniswamy
Jan. 28, 2014, 3:44 a.m.) Review request for pig, Alex Bain, Daniel Dai, Mark Wagner, and Rohini Palaniswamy. Bugs: PIG-3728 https://issues.apache.org/jira/browse/PIG-3728 Repository: pig-git Description --- The patch fixes TestSkewedJoin in tez mode. Note I changed

Re: Review Request 17439: PIG-3728: Fix TestSkewedJoin unit test in tez mode

2014-01-29 Thread Rohini Palaniswamy
/WeightedRangePartitionerTez.java https://reviews.apache.org/r/17439/#comment62402 In what scenario is quantiles map empty? If we don't initialize weightedParts, quantiles, etc won't we encounter NPE later? - Rohini Palaniswamy On Jan. 28, 2014, 3:44 a.m., Cheolsoo Park wrote

Review Request 17529: [PIG-3732] Use ONE_TO_ONE edge and IdentityInOut in orderby intermediate vertex

2014-01-30 Thread Rohini Palaniswamy
/branches/tez/test/org/apache/pig/test/data/GoldenFiles/TEZC7.gld 1562426 Diff: https://reviews.apache.org/r/17529/diff/ Testing --- test-tez and tez.conf e2e tests pass Thanks, Rohini Palaniswamy

  1   2   3   4   5   6   7   8   9   10   >