Review Request: PIG-2973 TestStreaming test times out

2012-10-22 Thread Cheolsoo Park
, Cheolsoo Park

Re: Review Request: TestJobSumission and TestHBaseStorage don't work with HBase 0.94 and ZK 3.4.3

2012-10-22 Thread Cheolsoo Park
=TestHBaseStorage -Dhadoopversion=20 ant clean test -Dtestcase=TestJobSumission -Dhadoopversion=20 I also manually tested pig.jar with hbase 0.90 and 0.94. Once HBASE_HOME and ZOOKEEPER_HOME are set, HBaseStorage works fine with both versions. Thanks, Cheolsoo Park

Re: Review Request: TestJobSumission and TestHBaseStorage don't work with HBase 0.94 and ZK 3.4.3

2012-10-23 Thread Cheolsoo Park
--- On Oct. 22, 2012, 6:50 a.m., Cheolsoo Park wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/7676

Re: Review Request: TestJobSumission and TestHBaseStorage don't work with HBase 0.94 and ZK 3.4.3

2012-10-23 Thread Cheolsoo Park
, HBaseStorage works fine with both versions. Thanks, Cheolsoo Park

Re: Review Request: TestJobSumission and TestHBaseStorage don't work with HBase 0.94 and ZK 3.4.3

2012-10-23 Thread Cheolsoo Park
--- On Oct. 23, 2012, 4:48 p.m., Cheolsoo Park wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/7676/ --- (Updated Oct. 23

Re: Review Request: PIG-2973 TestStreaming test times out

2012-10-25 Thread Cheolsoo Park
: testAddJobConfToEnvironmentWithASynchInput. ant test -Dtestcase=TestStreaming passes with both hadoop 20 and 23. Thanks, Cheolsoo Park

Re: [ANNOUNCE] Welcome new Apache Pig Committers Rohini Palaniswamy

2012-10-26 Thread Cheolsoo Park
Congrat! Well deserved! On Fri, Oct 26, 2012 at 4:37 PM, Daniel Dai da...@hortonworks.com wrote: Here is another Pig committer announcement today. Please welcome Rohini Palaniswamy to be a Pig committer! Thanks, Daniel

Re: Review Request: Modernize a chunk of the tests

2012-10-27 Thread Cheolsoo Park
Please delete this line? - Cheolsoo Park On Oct. 25, 2012, 6:05 p.m., Jonathan Coveney wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/7734

Re: Are people ok with putting PIG-3017 into pig 11?

2012-10-30 Thread Cheolsoo Park
+1 On Tue, Oct 30, 2012 at 2:02 PM, Rohini Palaniswamy rohini.adi...@gmail.com wrote: +1. Agree with Julien. Error on hitting Configuration's size limit is a bug. On Tue, Oct 30, 2012 at 1:40 PM, Julien Le Dem jul...@twitter.com wrote: To me in falls in the category of bug fixes we can

Adding new test cases to TestBuiltin.java

2012-10-30 Thread Cheolsoo Park
Hi all, While reviewing PIG-2881 (Add SUBTRACT eval func), I had 2 questions: 1) How do we decide whether an eval func be a built-in func? For example, should SUBTRACT be added to the o.a.pig.builtin or piggybank? 2) Do we want to continue to add test cases to TestBuiltin.java for new built-in

Re: [DISCUSS] Remove Penny from contrib

2012-10-31 Thread Cheolsoo Park
+1. I agree. On Wed, Oct 31, 2012 at 2:54 PM, Alan Gates ga...@hortonworks.com wrote: I propose we remove Penny from contrib. Currently it does not compile in trunk. Looking through the commit logs no significant work has been done on it since it was initially committed. There are 3 open

Re: Review Request: PIG-2405: some unit test case failed with open JDK

2012-11-06 Thread Cheolsoo Park
://reviews.apache.org/r/7898/#comment28353 Please remove the trailing white space. - Cheolsoo Park On Nov. 6, 2012, 12:57 p.m., Fang Fang Chen wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r

Re: Build failed in Jenkins: Pig-trunk #1355

2012-11-07 Thread Cheolsoo Park
I know that we discussed about fixing the jenkins build. Any updates? I am looking at our build history. Basically, our build runs on hadoop1, hadoop2, and hadoop6. a) When it runs on hadoop1, it passes! b) When it runs on hadoop2, it fails with a clover license error! c) When it runs on

Re: Review Request: PIG-2405: some unit test case failed with open JDK

2012-11-07 Thread Cheolsoo Park
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/7898/#review13222 --- Ship it! Looks good. Thanks Fangfang! - Cheolsoo Park On Nov. 7

Re: Review Request: PIG-2924 PigStats should not be assuming all Storage classes to be file-based storage

2012-11-18 Thread Cheolsoo Park
--- On Nov. 19, 2012, 5:32 a.m., Cheolsoo Park wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/8122

Review Request: PIG-3066 Fix TestPigRunner in trunk

2012-12-02 Thread Cheolsoo Park
/TestPigRunner.java 69cd242 Diff: https://reviews.apache.org/r/8313/diff/ Testing --- ant test -Dtestcase=TestPigRunner -Dhadoopversion=[20|23] Thanks, Cheolsoo Park

Re: Review Request: PIG-3015 Rewrite of AvroStorage

2012-12-03 Thread Cheolsoo Park
log.debug(..., e)? - Cheolsoo Park On Nov. 17, 2012, 5:28 a.m., Joseph Adler wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/8104

Review Request: PIG-3096 Make PigUnit thread safe

2012-12-16 Thread Cheolsoo Park
/PigTest.java 50a5c79 Diff: https://reviews.apache.org/r/8631/diff/ Testing --- ant test -Dtestcase=TestPigTest I also tested it by running multiple PigUnit cases in parallel with tempus-fugit (http://tempusfugitlibrary.org/documentation/junit/parallel/) on a real cluster. Thanks, Cheolsoo

Review Request: PIG-3050 Fix FindBugs multithreading warnings

2012-12-17 Thread Cheolsoo Park
that both unit test and e2e test pass. Thanks, Cheolsoo Park

Re: [VOTE] Release Pig 0.10.1 (candidate 2)

2012-12-28 Thread Cheolsoo Park
tarball. I am not sure if this is a block or not. though Thanks, Cheolsoo On Fri, Dec 28, 2012 at 2:09 PM, Cheolsoo Park cheol...@cloudera.comwrote: +1. Downloaded the source tarball, built hadoop 20/23 binaries, and tested them against hadoop-1.0.x/2.0.x clusters. Btw, ant clean

Re: Review Request: PIG-3059 Global configurable minimum 'bad record' thresholds

2012-12-30 Thread Cheolsoo Park
=TestAvroStorage Thanks, Cheolsoo Park

Re: [VOTE] Release Pig 0.10.1 (candidate 3)

2013-01-04 Thread Cheolsoo Park
+1. Tested against hadoop 1.0.x and 2.0.x clusters. On Fri, Jan 4, 2013 at 12:14 AM, Jarek Jarcec Cecho jar...@apache.orgwrote: +1 (non-binding) * Verified checksum * Verified signatures * Tests seems to be passing * Checked top level files (NOTICE, LICENSE) Note: I personally prefer

Re: Review Request: Introduce a syntax to be able to easily refer to the previously defined relation

2013-01-20 Thread Cheolsoo Park
/r/9019/#comment33511 Java warning here: The serializable class does not declare a static final serialVersionUID field of type long. - Cheolsoo Park On Jan. 18, 2013, 7:05 p.m., Jonathan Coveney wrote

Re: Review Request: Add BigInteger and BigDecimal to Pig

2013-01-20 Thread Cheolsoo Park
://reviews.apache.org/r/9012/#comment33538 Shouldn't it be bytesToBigDecimal? src/org/apache/pig/data/DataType.java https://reviews.apache.org/r/9012/#comment33539 Typo: BigInteger = BigDecimal. - Cheolsoo Park On Jan. 18, 2013, 10:11 p.m., Jonathan Coveney wrote

Re: Pig 11.0

2013-01-25 Thread Cheolsoo Park
I will also run e2e on Hadoop-1.x and Hadoop-2.x. On Fri, Jan 25, 2013 at 5:02 PM, Daniel Dai da...@hortonworks.com wrote: I will run e2e tests on Hadoop 1.x over the weekend. Thanks, Daniel On Fri, Jan 25, 2013 at 4:27 PM, Rohini Palaniswamy rohini.adi...@gmail.com wrote: Thats good

Re: Run a job async

2013-01-25 Thread Cheolsoo Park
on. 2013/1/25 Cheolsoo Park cheol...@cloudera.com if you have multiple threads that run a query via PigServer, there is a great chance of the internals clashing because of the use of static variable within Pig. Recently, I spent some time on this, and what I found

Re: Pig 11.0

2013-01-28 Thread Cheolsoo Park
FAILED DEPENDENCY: 0 - The failures seem due to issues in my cluster rather than Pig issues. I will re-run them to verify. On Fri, Jan 25, 2013 at 5:31 PM, Cheolsoo Park cheol...@cloudera.comwrote: I will also run e2e on Hadoop-1.x and Hadoop-2.x. On Fri, Jan 25, 2013 at 5:02 PM, Daniel Dai

Failing unit tests in trunk

2013-02-07 Thread Cheolsoo Park
Hello, There was a discussion about failing unit tests in trunk at today's meet-up. I just wanted to let you know that we have jiras for all failing test cases in trunk: TestPigRunner PIG-3138 TestScriptUDF PIG-3153 TestPackage PIG-3154 TestTypeCheckingValidatorNewLP PIG-3155 TestSchemaTuple

Review Request: PIG-3156 TestSchemaTuple fails in trunk

2013-02-09 Thread Cheolsoo Park
/SchemaTupleClassGenerator.java 633c8d8 test/org/apache/pig/data/TestSchemaTuple.java 3e89996 Diff: https://reviews.apache.org/r/9395/diff/ Testing --- ant clean test -Dtestcase=TestSchema passes. Thanks, Cheolsoo Park

Re: Failing unit tests in trunk

2013-02-09 Thread Cheolsoo Park
! 2013/2/7 Cheolsoo Park cheol...@cloudera.com Hello, There was a discussion about failing unit tests in trunk at today's meet-up. I just wanted to let you know that we have jiras for all failing test cases in trunk: TestPigRunner PIG-3138 TestScriptUDF PIG-3153 TestPackage PIG

Re: Pig standalone jar

2013-02-14 Thread Cheolsoo Park
We already generate core pig jar even though it's not published on Maven repo. If you run ant clean jar, you will find two jars under the build dir: -rw-r--r-- 1 cheolsoo cheolsoo 3411312 Feb 14 09:41 pig-0.12.0-SNAPSHOT.jar -rw-r--r-- 1 cheolsoo cheolsoo 16225627 Feb 14 09:41

Re: Pig 0.12 3 months after Pig 0.11?

2013-02-17 Thread Cheolsoo Park
I don't mind driving it if everyone agrees with the time frame. One thing that I want to clarify is that the new AvroStorage is not backward compatible[1]. In addition, there are a couple of features that the new AvroStorage is missing at the moment[2]. So I was thinking that we keep the old

Re: Build failed in Jenkins: Pig-trunk #1414

2013-02-19 Thread Cheolsoo Park
This is broken by PIG-3131. Please review the hotfix: https://issues.apache.org/jira/browse/PIG-3131 On Tue, Feb 19, 2013 at 2:06 PM, Apache Jenkins Server jenk...@builds.apache.org wrote: See https://builds.apache.org/job/Pig-trunk/1414/changes Changes: [cheolsoo] PIG-3138: Decouple

Re: [ANNOUNCE] Welcome Bill Graham to join Pig PMC

2013-02-19 Thread Cheolsoo Park
Congratulations! On Tue, Feb 19, 2013 at 2:35 PM, Prasanth J buckeye.prasa...@gmail.comwrote: Congrats Bill! Thanks -- Prasanth On Feb 19, 2013, at 4:52 PM, Prashant Kommireddi prash1...@gmail.com wrote: Congrats Bill! On Tue, Feb 19, 2013 at 1:48 PM, Daniel Dai

Re: [VOTE] Release Pig 0.11.0 (candidate 2)

2013-02-20 Thread Cheolsoo Park
+1 (non-binding) I downloaded and compiled source tarball. I tested jars against Hadoop 1.x and 2.x based clusters. On Wed, Feb 20, 2013 at 5:10 PM, Julien Le Dem jul...@twitter.com wrote: +1 I've run a subset of the tests on the src tar run some jobs in local mode on the binary tar

Re: Review Request: Introduce a syntax making declared aliases optional

2013-03-01 Thread Cheolsoo Park
: -- tab after equal a = load '1.txt'; dump a; - Cheolsoo Park On March 1, 2013, 10:32 a.m., Jonathan Coveney wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/9496

Re: Review Request: PIG-3215 [piggybank] Add LTSVLoader to load LTSV files

2013-03-03 Thread Cheolsoo Park
On March 1, 2013, 2:54 p.m., Jonathan Coveney wrote: contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/storage/LTSVLoader.java, line 157 https://reviews.apache.org/r/9685/diff/1/?file=263710#file263710line157 In the case where they do not give it a schema, I think

Re: Welcome our new PMC chair, Julien Le Dem

2013-03-10 Thread Cheolsoo Park
Congrats Julien! On Sun, Mar 10, 2013 at 9:20 PM, Prasanth J buckeye.prasa...@gmail.comwrote: Congratulations! Thanks -- Prasanth On Mar 11, 2013, at 12:00 AM, Jarek Jarcec Cecho jar...@apache.org wrote: Congratulations sir! Jarcec On Sun, Mar 10, 2013 at 08:55:55PM -0700,

Can we commit PIG-3015 (Rewrite of AvroStorage) to trunk?

2013-03-18 Thread Cheolsoo Park
Hello, Thanks to Joseph Adler's contribution, we have a new AvroStorage ready. Although there are additional requests that we would like to address, I think we can implement them incrementally after we commit the current patches. As of now, - The core features are fully implemented. - All the

Re: Review Request: PIG-3141 [piggybank] Giving CSVExcelStorage an option to handle header rows

2013-03-20 Thread Cheolsoo Park
On March 20, 2013, 7:05 p.m., Cheolsoo Park wrote: contrib/piggybank/java/src/main/java/org/apache/pig/piggybank/storage/CSVExcelStorage.java, line 538 https://reviews.apache.org/r/9697/diff/2/?file=263987#file263987line538 Can you move this line to inside the if block? That's

Re: Review Request: PIG-3141 [piggybank] Giving CSVExcelStorage an option to handle header rows

2013-03-25 Thread Cheolsoo Park
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/9697/#review18379 --- Ship it! Ship It! - Cheolsoo Park On March 25, 2013, 3:17 p.m

Review Request: PIG-3269 In operator support

2013-04-08 Thread Cheolsoo Park
-CREATION Diff: https://reviews.apache.org/r/10337/diff/ Testing --- Added two new test suites: - TestIn.java - TestInUdf.java Also ran ant test-commit. Thanks, Cheolsoo Park

Review Request: PIG-3268 Case statement support

2013-04-08 Thread Cheolsoo Park
-CREATION Diff: https://reviews.apache.org/r/10341/diff/ Testing --- Added two new test suites: - TestCase.java - TestCaseUdf.java Also ran ant test-commit. Thanks, Cheolsoo Park

Re: Review Request: PIG-3269 In operator support

2013-04-09 Thread Cheolsoo Park
. Thanks, Cheolsoo Park

Re: Review Request: PIG-3268 Case statement support

2013-04-09 Thread Cheolsoo Park
://reviews.apache.org/r/10341/diff/ Testing --- Added two new test suites: - TestCase.java - TestCaseUdf.java Also ran ant test-commit. Thanks, Cheolsoo Park

Re: Review Request: PIG-3268 Case statement support

2013-04-09 Thread Cheolsoo Park
/pig/test/TestCase.java PRE-CREATION Diff: https://reviews.apache.org/r/10341/diff/ Testing --- Added two new test suites: - TestCase.java - TestCaseUdf.java Also ran ant test-commit. Thanks, Cheolsoo Park

Re: Review Request: PIG-3268 Case statement support

2013-04-11 Thread Cheolsoo Park
Diff: https://reviews.apache.org/r/10341/diff/ Testing --- Added two new test suites: - TestCase.java - TestCaseUdf.java Also ran ant test-commit. Thanks, Cheolsoo Park

Re: Review Request: PIG-3268 Case statement support

2013-04-11 Thread Cheolsoo Park
/commit-tests 5081fbc test/org/apache/pig/test/TestCase.java PRE-CREATION Diff: https://reviews.apache.org/r/10341/diff/ Testing --- Added two new test suites: - TestCase.java - TestCaseUdf.java Also ran ant test-commit. Thanks, Cheolsoo Park

Re: Build failed in Jenkins: Pig-trunk #1463

2013-04-23 Thread Cheolsoo Park
PIG-3290 is tracking this failure: https://issues.apache.org/jira/browse/PIG-3290 In addition, PIG-3286 is tracking another failing unit test: https://issues.apache.org/jira/browse/PIG-3286 On Mon, Apr 22, 2013 at 3:32 PM, Apache Jenkins Server jenk...@builds.apache.org wrote: See

Re: Review Request: [PIG-3173] - Partition filter pushdown does not happen if partition keys condition include a AND and OR construct

2013-04-29 Thread Cheolsoo Park
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/10035/#review19891 --- Ship it! Looks good to me. - Cheolsoo Park On April 29, 2013, 8

Re: Want to contribute

2013-04-30 Thread Cheolsoo Park
Welcome to Pig. There are hundreds of open jiras: https://issues.apache.org/jira/issues/?jql=project%20%3D%20PIG%20AND%20status%20%3D%20Open%20ORDER%20BY%20created%20DESC%2C%20priority%20DESC Please feel free to submit patches. Thanks, Cheolsoo On Tue, Apr 30, 2013 at 4:16 PM, Vineet Nair

Re: Want to contribute

2013-04-30 Thread Cheolsoo Park
how the framework is written. I can help in fixing some minor bugs/jira issues. Can some one help me how to get the source code ? Regards, Naidu On Wed, May 1, 2013 at 9:30 AM, Cheolsoo Park piaozhe...@gmail.com wrote: Welcome to Pig. There are hundreds of open jiras: https

Re: Welcome our newest committer Prashant Kommireddi

2013-05-02 Thread Cheolsoo Park
Congrats Prashant! On Thu, May 2, 2013 at 12:56 PM, Julien Le Dem jul...@ledem.net wrote: All, Please join me in welcoming Prashant Kommireddi as our newest Pig committer. He's been contributing to Pig for a while now. We look forward to him being a part of the project. Julien

Re: Unit test classpath trouble

2013-05-12 Thread Cheolsoo Park
Hi Andrew, I'm trying HEAD of branch-0.11. You need PIG-3248: https://issues.apache.org/jira/browse/PIG-3248. It's not in branch-0.11 but in trunk. In particular, you need to add the following lines to ivy.xml: +dependency org=org.apache.hadoop name=hadoop-yarn-client +

Re: Unit test classpath trouble

2013-05-13 Thread Cheolsoo Park
at 11:45 PM, Konstantin Boudnik c...@apache.org wrote: If unit tests can't be run - which seems to be the case - it sound like a release blocker to me. Any plans to back port this fix into 0.11.2? 0.11.1 doesn't have it, right? Cos On Sun, May 12, 2013 at 09:12PM, Cheolsoo Park wrote: Hi

Re: Review Request: Refactor physical operators to remove methods parameters that are always null

2013-05-17 Thread Cheolsoo Park
This isn't what you introduced, but I think this is incorrect. Shouldn't in.getNextBigDecimal() be in.getNextBigInteger() since we're casting BI to BD here? - Cheolsoo Park On May 16, 2013, 9:35 p.m., Julien Le Dem wrote

Re: Review Request: PIG-3331 Default values not written to Schema when specified in the output schema

2013-06-02 Thread Cheolsoo Park
and test files in a single patch and attach it in the jira? It would be very helpful if I could apply everything with a single patch command. Thank you! - Cheolsoo Park On May 30, 2013, 2:29 a.m., Viraj Bhat wrote

Re: Review Request: PIG-3322 Fix the issue where NPE is thrown when reading a union which has nulls and add a testcase

2013-06-02 Thread Cheolsoo Park
see org.apache.pig.builtin.mock.Storage.java. - Cheolsoo Park On May 29, 2013, 11:07 p.m., Viraj Bhat wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/11333

Review Request: PIG-3342 Allow conditions in case statement

2013-06-03 Thread Cheolsoo Park
/QueryParser.g 2b84c86 test/org/apache/pig/test/TestCase.java dbee495 Diff: https://reviews.apache.org/r/11613/diff/ Testing --- All unit tests pass. Thanks, Cheolsoo Park

Re: Uploading patches for review

2013-06-06 Thread Cheolsoo Park
I agree that we should use the RB more often. On Wed, Jun 5, 2013 at 6:50 PM, Jarek Jarcec Cecho jar...@apache.orgwrote: I can definitely recommend using the review board. We are using it already in all other projects that I'm participating and it's really helping us out. Jarcec On Wed,

Review Request: PIG-3346 New property that controls the number of combined splits

2013-06-10 Thread Cheolsoo Park
. A new unit test case is added for the new property. Thanks, Cheolsoo Park

Review Request: PIG-3288 Kill jobs if the number of output files is over a configurable limit

2013-06-10 Thread Cheolsoo Park
64465af test/e2e/pig/udfs/java/org/apache/pig/test/udf/storefunc/PigStorageWithFileCount.java e69de29 Diff: https://reviews.apache.org/r/11719/diff/ Testing --- All unit tests pass. A new e2e test case is added for the new property. Thanks, Cheolsoo Park

Re: Review Request: [PIG-3341] Strict datetime parsing and improve performance of loading datetime values

2013-06-12 Thread Cheolsoo Park
https://reviews.apache.org/r/11834/#comment44979 Unused import. http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/data/DataType.java https://reviews.apache.org/r/11834/#comment44978 Unused import. - Cheolsoo Park On June 12, 2013, 1:21 p.m., Rohini Palaniswamy wrote

Re: Review Request: [PIG-3341] Strict datetime parsing and improve performance of loading datetime values

2013-06-12 Thread Cheolsoo Park
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/11834/#review21812 --- Ship it! - Cheolsoo Park On June 12, 2013, 6:42 p.m., Rohini

Re: Status of JSON load/store funcs

2013-06-22 Thread Cheolsoo Park
You can upload a new patch to PIG-2641. I agree that we should improve the built-in JsonLoader rather than adding a new one to piggybank. You might want to ask Russell whether it is okay for you to take over his jira. I can review/commit your patch whenever it's ready. On Thu, Jun 20, 2013 at

Re: Review Request: PIG-3288 Kill jobs if the number of output files is over a configurable limit

2013-06-22 Thread Cheolsoo Park
e2e test case is added for the new property. Thanks, Cheolsoo Park

Review Request 12290: CASE and IN fail when expression includes dereferencing operator

2013-07-05 Thread Cheolsoo Park
, Cheolsoo Park

Review Request 12290: CASE and IN fail when expression includes dereferencing operator

2013-07-05 Thread Cheolsoo Park
, Cheolsoo Park

Review Request 12321: PIG-3021 Split results missing records when there is null values in the column comparison

2013-07-07 Thread Cheolsoo Park
. ant test -Dtestcase=TestSplit passes. Thanks, Cheolsoo Park

Re: Review Request 12290: CASE and IN fail when expression includes dereferencing operator

2013-07-16 Thread Cheolsoo Park
/ Testing --- Added new test cases to TestIn and TestCase. ant clean test -Dtestcase=TestIn ant clean test -Dtestcase=TestCase Thanks, Cheolsoo Park

Review Request 13122: PIG-3400 FS commands do not work with S3 paths

2013-07-31 Thread Cheolsoo Park
://path2' pig -e 'copyToLocal s3://path/file .' pig -e 'copyFromLocal file s3://path' Thanks, Cheolsoo Park

Re: Review Request 13122: PIG-3400 FS commands do not work with S3 paths

2013-07-31 Thread Cheolsoo Park
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/13122/#review24393 --- On July 31, 2013, 7:37 p.m., Cheolsoo Park wrote

Re: Review Request 13122: PIG-3400 FS commands do not work with S3 paths

2013-07-31 Thread Cheolsoo Park
-mail. To reply, visit: https://reviews.apache.org/r/13122/#review24376 --- On July 31, 2013, 7:37 p.m., Cheolsoo Park wrote: --- This is an automatically generated e-mail. To reply

Re: Review Request 13122: PIG-3400 FS commands do not work with S3 paths

2013-07-31 Thread Cheolsoo Park
we get rid of 2)? Thanks, Cheolsoo On Wed, Jul 31, 2013 at 1:01 PM, Aniket Mokashi aniket...@gmail.com wrote: I am not 100% sure but IIRC, fs commands were deprecated. On Wed, Jul 31, 2013 at 12:37 PM, Cheolsoo Park piaozhe...@gmail.comwrote

Review Request 13186: PIG-3395 Large filter expression makes Pig hang

2013-07-31 Thread Cheolsoo Park
/newplan/PColFilterExtractor.java 32ba788 test/org/apache/pig/test/TestPartitionFilterPushDown.java b6b26f4 Diff: https://reviews.apache.org/r/13186/diff/ Testing --- Adds new test cases to TestPartitionFilterPushDown. All unit tests pass. Thanks, Cheolsoo Park

Review Request 13210: PIG-3374 CASE and IN fail when expression includes dereferencing operator

2013-08-01 Thread Cheolsoo Park
://reviews.apache.org/r/13210/diff/ Testing --- Added new test cases. All the unit tests pass. Thanks, Cheolsoo Park

Re: Review Request 13535: [PIG-3204] Reduce the number of getSchema calls during script parsing

2013-08-13 Thread Cheolsoo Park
call them, so it will be easier to identify the difference when skipParseInRegisterForBatch is on and off? - Cheolsoo Park On Aug. 13, 2013, 2:34 p.m., Rohini Palaniswamy wrote: --- This is an automatically generated e-mail. To reply

Review Request 13551: PIG-3414 Utils.getSchemaFromString() silently returns a wrong result when a comma is missing in the schema definition

2013-08-13 Thread Cheolsoo Park
bfe76c4 Diff: https://reviews.apache.org/r/13551/diff/ Testing --- Added a new test case to TestSchema. All the unit tests pass. Thanks, Cheolsoo Park

Re: schema definition and subschema

2013-08-14 Thread Cheolsoo Park
Hi Keren, Hope this is too late. I am wondering why is LogicalFieldShema containing a LogicalSchema member? That's for nested tuple fields. For example, consider ( i:int, t:tuple(j:int) ). The field t:tuple needs to contain a list of field schemas, so you need a LogicalSchema. Here is how you

Re: Review Request 13535: [PIG-3204] Reduce the number of getSchema calls during script parsing

2013-08-14 Thread Cheolsoo Park
On Aug. 14, 2013, 2:13 a.m., Cheolsoo Park wrote: http://svn.apache.org/repos/asf/pig/trunk/test/org/apache/pig/test/TestGrunt.java, line 1083 https://reviews.apache.org/r/13535/diff/2/?file=340613#file340613line1083 I believe we shouldn't remove xargs. It was added by PIG-3099

Re: Review Request 13535: [PIG-3204] Reduce the number of getSchema calls during script parsing

2013-08-14 Thread Cheolsoo Park
! - Cheolsoo Park On Aug. 13, 2013, 2:34 p.m., Rohini Palaniswamy wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/13535

Re: Specifying a User Defined LoadStoreCaster in PigStorage

2013-08-16 Thread Cheolsoo Park
Hi Saj, Wouldn't it be easier if you extend PigStorage by overriding getLoadCaster() and let the user use that storage when they need to interact the legacy data? Thanks, Cheolsoo On Fri, Aug 16, 2013 at 1:40 PM, Sajid Raza windcl...@gmail.com wrote: I have a legacy dataset that has its own

Re: Slow Group By operator

2013-08-21 Thread Cheolsoo Park
. Real time runtime is measured using the time -p command. Best Regards, Benjamin On 20 August 2013 19:56, Cheolsoo Park piaozhe...@gmail.com wrote: Hi Benjarmin, Can you describe which step of group by is slow? Mapper side or reducer side? What's your query like? Can you share

Re: Slow Group By operator

2013-08-22 Thread Cheolsoo Park
performs a sort then the arithmetic operation explain the slowdown? (Apologies, I'm quite new to Pig/Hive - just my guesses). Regards, Benjamin On 22 August 2013 01:07, Cheolsoo Park piaozhe...@gmail.com wrote: Hi Benjamin, Thank you very much for sharing detailed information! 1

Re: Slow Group By operator

2013-08-24 Thread Cheolsoo Park
is faster: With combiner: real 668.44 real 663.10 real 665.05 Without combiner: real 795.97 real 810.51 real 810.16 Best Regards, Ben On 22 August 2013 16:33, Cheolsoo Park piaozhe...@gmail.com wrote: Hi Benjamin, To answer your question, how the Hadoop combiner works

Re: Slow Group By operator

2013-08-25 Thread Cheolsoo Park
I have no more suggestion. If you find anything, please share with us. I would be interested in understanding what you're seeing. On Sun, Aug 25, 2013 at 11:14 AM, Benjamin Jakobus jakobusbe...@gmail.comwrote: combiner + mapPartAgg set to true - yup! On 25 August 2013 18:57, Cheolsoo Park

Re: Creating a Tez branch

2013-09-02 Thread Cheolsoo Park
Created the tez branch: http://svn.apache.org/viewvc?view=revisionrevision=1519486 On Sat, Aug 31, 2013 at 3:54 PM, Cheolsoo Park piaozhe...@gmail.com wrote: Hello Pig devs, As discussed, I'd like to create a Tez branch. I think we can start with Achal's work (pigeon) and take a similar

Re: Are we ready for Pig 0.12.0 release?

2013-09-03 Thread Cheolsoo Park
+1. I will go through my jiras this week. On Tue, Sep 3, 2013 at 10:34 AM, Daniel Dai da...@hortonworks.com wrote: Hi, All, It has been more than half a year since initial Pig 0.11 release. I'd like roll a Pig 0.12 release around the end of September or the beginning of October. Let me know

Review Request 13950: Tez backend layout

2013-09-03 Thread Cheolsoo Park
/TezPrinter.java PRE-CREATION Diff: https://reviews.apache.org/r/13950/diff/ Testing --- Thanks, Cheolsoo Park

Re: Review Request 13950: Tez backend layout

2013-09-03 Thread Cheolsoo Park
/#review25860 --- On Sept. 3, 2013, 9:14 p.m., Cheolsoo Park wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/13950

Re: Review Request 13950: Tez backend layout

2013-09-03 Thread Cheolsoo Park
/TezPrinter.java PRE-CREATION Diff: https://reviews.apache.org/r/13950/diff/ Testing --- Thanks, Cheolsoo Park

Re: Welcome new Pig Committer - Koji Noguchi

2013-09-10 Thread Cheolsoo Park
Congratulations Koji! On Wed, Sep 11, 2013 at 7:32 AM, Prashant Kommireddi prash1...@gmail.comwrote: Congrats Koji! On Tue, Sep 10, 2013 at 10:01 AM, Xuefu Zhang xzh...@cloudera.com wrote: Congratulations, Koji. Looking forward to more of your contributions. --Xuefu On Tue, Sep

Review Request 14206: PIG-3466 Race Conditions in InternalDistinctBag during proactive spill

2013-09-18 Thread Cheolsoo Park
/browse/PIG-3466 Repository: pig-git Description --- See https://issues.apache.org/jira/browse/PIG-3466 Diffs - src/org/apache/pig/data/InternalDistinctBag.java 3f0140c Diff: https://reviews.apache.org/r/14206/diff/ Testing --- All unit tests pass. Thanks, Cheolsoo Park

Re: Review Request 14274: PIG-2672 Optimize the use of DistributedCache

2013-09-24 Thread Cheolsoo Park
://reviews.apache.org/r/14274/#comment51439 The following line is missing in the RB diff but it's in the attached the patch: properties.setProperty(PigConstants.PIG_SHARED_CACHE_ENABLED_KEY, true); Just pointing it out. - Cheolsoo Park On Sept. 21, 2013, 1:21 a.m., Aniket Mokashi

[Discussion] Any thoughts on PIG-3457?

2013-09-30 Thread Cheolsoo Park
Hi devs, PIG-3419 https://issues.apache.org/jira/browse/PIG-3419 broke backward compatibility for downstream applications such as Oozie, and PIG-3457https://issues.apache.org/jira/browse/PIG-3457 is trying to fix it. In summary, we need to keep the old MR-specific JobStats and PigStats for

Re: [Discussion] Any thoughts on PIG-3457?

2013-09-30 Thread Cheolsoo Park
To be specific, we will need to revert all the following commits in order: commit ad1b87d4ba073680ad0a7fc8c76baeb8b611c982 Author: Cheolsoo Park cheol...@apache.org Date: Fri Sep 20 22:47:29 2013 + PIG-3471: Add a base abstract class for ExecutionEngine (cheolsoo) git-svn-id

Re: [Discussion] Any thoughts on PIG-3457?

2013-09-30 Thread Cheolsoo Park
this might not work everywhere. On Mon, Sep 30, 2013 at 1:43 PM, Cheolsoo Park piaozhe...@gmail.com wrote: To be specific, we will need to revert all the following commits in order: commit ad1b87d4ba073680ad0a7fc8c76baeb8b611c982 Author: Cheolsoo Park cheol...@apache.org Date: Fri Sep

Re: [Discussion] Any thoughts on PIG-3457?

2013-09-30 Thread Cheolsoo Park
(deprecated APIs or new APIs). Alan. On Sep 30, 2013, at 11:45 AM, Cheolsoo Park wrote: Hi Jeremy, What you're saying makes sense, and patch is welcome. ;-) But complexity comes from that there are many classes that are associated with one another, and it seems necessary

Re: [Discussion] Any thoughts on PIG-3457?

2013-09-30 Thread Cheolsoo Park
. On Mon, Sep 30, 2013 at 2:42 PM, Cheolsoo Park piaozhe...@gmail.com wrote: I am waiting for +1 from Twitter. Like Alan suggested, let's revert PIG-3419 et al in 0.12 first. Then, we can decide what to do in trunk. I volunteer to do grunt work since I am the one who committed them

Re: [Discussion] Any thoughts on PIG-3457?

2013-09-30 Thread Cheolsoo Park
break anything, but please let me know if I made any mistake. Test-commit passes. Thank you, Cheolsoo On Mon, Sep 30, 2013 at 3:52 PM, Cheolsoo Park piaozhe...@gmail.com wrote: Thanks Aniket. I'll revert the aforementioned commits in 0.12 tonight. I will leave them in trunk until we decide

  1   2   3   4   5   6   7   8   9   10   >