date:20121029

[jira] [Updated] (PIG-2968) ColumnMapKeyPrune fails to prune a subtree inside foreach

2012-10-29 Thread Koji Noguchi (JIRA)


 [ 
https://issues.apache.org/jira/browse/PIG-2968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Koji Noguchi updated PIG-2968:
--

Fix Version/s: 0.11

I'd like to see this fixed in 0.11.

 ColumnMapKeyPrune fails to prune a subtree inside foreach
 -

 Key: PIG-2968
 URL: https://issues.apache.org/jira/browse/PIG-2968
 Project: Pig
  Issue Type: Bug
  Components: parser
Reporter: Koji Noguchi
Assignee: Koji Noguchi
Priority: Minor
 Fix For: 0.11

 Attachments: pig-2968-trunk_v01.txt, pig-2968-trunk_v02.txt


 Sample code 
 {noformat}
 $ cat test/foreach.pig 
 daily = load 'nyse' as (exchange, symbol);
 grpd = group daily by exchange;
 uniquecnt = foreach grpd {
 sym = daily.symbol;
 uniq_sym = distinct sym;
 generate group, uniq_sym;
 };
 another = FOREACH uniquecnt GENERATE group;
 explain another;
 {noformat}
 This breaks when it tries to prune uniq_sym-sym-innerload_daily
 bq. 2012-10-12 14:54:11,031 [main] ERROR org.apache.pig.tools.grunt.Grunt - 
 ERROR 2000: Error processing rule ColumnMapKeyPrune. Try -t ColumnMapKeyPrune

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-3008) Fix whitespace in Pig code

2012-10-29 Thread Julien Le Dem (JIRA)


[ 
https://issues.apache.org/jira/browse/PIG-3008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13486153#comment-13486153
 ] 

Julien Le Dem commented on PIG-3008:


I support a whitespace policy as defined by Cheolsoo above : No tabs, no 
trailing whitespace, 4 spaces. As RB as a hide whitespace changes feature we 
should just use it more.


 Fix whitespace in Pig code
 --

 Key: PIG-3008
 URL: https://issues.apache.org/jira/browse/PIG-3008
 Project: Pig
  Issue Type: Improvement
Reporter: Jonathan Coveney
 Fix For: 0.12

 Attachments: checkstyle.xml


 This JIRA exists mainly to get a conversation started. We've talked about it 
 before, and it's a tricky issue. That said, some of the Pig code is super, 
 super gnarly. We need some sort of path that will let it eventually be 
 fix-able.
 I posit: any file that hasn't been touched for over 6 months is eligible for 
 a whitespace patch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-3008) Fix whitespace in Pig code

2012-10-29 Thread Jonathan Coveney (JIRA)


[ 
https://issues.apache.org/jira/browse/PIG-3008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13486160#comment-13486160
 ] 

Jonathan Coveney commented on PIG-3008:
---

I agree that we should make sure that all new code passes a checkstyle (and we 
should have a checkstyle.xml and we should enforce it). But when will be able 
to update old files?

 Fix whitespace in Pig code
 --

 Key: PIG-3008
 URL: https://issues.apache.org/jira/browse/PIG-3008
 Project: Pig
  Issue Type: Improvement
Reporter: Jonathan Coveney
 Fix For: 0.12

 Attachments: checkstyle.xml


 This JIRA exists mainly to get a conversation started. We've talked about it 
 before, and it's a tricky issue. That said, some of the Pig code is super, 
 super gnarly. We need some sort of path that will let it eventually be 
 fix-able.
 I posit: any file that hasn't been touched for over 6 months is eligible for 
 a whitespace patch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-2970) Nested foreach getting incorrect schema when having unrelated inner query

2012-10-29 Thread Rohini Palaniswamy (JIRA)


 [ 
https://issues.apache.org/jira/browse/PIG-2970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohini Palaniswamy updated PIG-2970:


Affects Version/s: 0.10.0
Fix Version/s: 0.12
   0.11

 Nested foreach getting incorrect schema when having unrelated inner query
 -

 Key: PIG-2970
 URL: https://issues.apache.org/jira/browse/PIG-2970
 Project: Pig
  Issue Type: Bug
  Components: parser
Affects Versions: 0.10.0
Reporter: Koji Noguchi
Assignee: Koji Noguchi
Priority: Minor
 Fix For: 0.11, 0.12

 Attachments: pig-2970-trunk-v01.txt


 While looking at PIG-2968, hit a weird error message.
 {noformat}
 $ cat -n test/foreach2.pig
  1  daily = load 'nyse' as (exchange, symbol);
  2  grpd = group daily by exchange;
  3  unique = foreach grpd {
  4  sym = daily.symbol;
  5  uniq_sym = distinct sym;
  6  --ignoring uniq_sym result
  7  generate group, daily;
  8  };
  9  describe unique;
 10  zzz = foreach unique generate group;
 11  explain zzz;
 % pig -x local -t ColumnMapKeyPrune test/foreach2.pig
 ...
 unique: {symbol: bytearray}
 2012-10-12 16:55:44,226 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 
 1025: 
 file test/foreach2.pig, line 10, column 30 Invalid field projection. 
 Projected field [group] does not exist in schema: symbol:bytearray.
 ...
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-3008) Fix whitespace in Pig code

2012-10-29 Thread Prashant Kommireddi (JIRA)


[ 
https://issues.apache.org/jira/browse/PIG-3008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13486221#comment-13486221
 ] 

Prashant Kommireddi commented on PIG-3008:
--

Here's a formatter that hbase project uses 
http://svn.apache.org/repos/asf/hbase/trunk/dev-support/hbase_eclipse_formatter.xml

 Fix whitespace in Pig code
 --

 Key: PIG-3008
 URL: https://issues.apache.org/jira/browse/PIG-3008
 Project: Pig
  Issue Type: Improvement
Reporter: Jonathan Coveney
 Fix For: 0.12

 Attachments: checkstyle.xml


 This JIRA exists mainly to get a conversation started. We've talked about it 
 before, and it's a tricky issue. That said, some of the Pig code is super, 
 super gnarly. We need some sort of path that will let it eventually be 
 fix-able.
 I posit: any file that hasn't been touched for over 6 months is eligible for 
 a whitespace patch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (PIG-3014) CurrentTime() UDF has undesirable characteristics

2012-10-29 Thread Jonathan Coveney (JIRA)

Jonathan Coveney created PIG-3014:
-

 Summary: CurrentTime() UDF has undesirable characteristics
 Key: PIG-3014
 URL: https://issues.apache.org/jira/browse/PIG-3014
 Project: Pig
  Issue Type: Bug
Reporter: Jonathan Coveney


As part of the explanation of the new DateTime datatype I noticed that we had 
added a CurrentTime() UDF. The issue with this UDF is that it returns the 
current time _of every exec invocation_, which can lead to confusing results. 
In PIG-1431 I proposed a way such that every instance of the same NOW() will 
return the same time, which I think is better. Would enjoy thoughts.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-2924) PigStats should not be assuming all Storage classes to be file-based storage

2012-10-29 Thread Bill Graham (JIRA)


 [ 
https://issues.apache.org/jira/browse/PIG-2924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bill Graham updated PIG-2924:
-

Status: Open  (was: Patch Available)

This looks great, thanks for taking this one. I think we need to make a few 
changes to the pattern used PIG-2574 though, because we could have a case where 
we have multiple store funcs that each write to a different data source.

* Instead of registering a single new computer it would be ideal if we could 
register a list of computers.
* Each computer could have a {{boolean supports(POStore poStore)}} method that 
returns whether this class supports a given POStore. This can often be done by 
inspecting the output path. A default URI-based abstract class could help with 
that part.
* The computers would then be consulted in order, where the first to support 
the POStore wins.
* If a computer can't determine a size for some reason (i.e., it doesn't 
support it or an exception occurred), it shouldn't return 0. Instead maybe we 
reserve -1 for this case and document it as such. 
* Having the word Computer in the interface name and configs could cause 
confusion, due to how it's an overloaded term. I don't have any great 
suggestions though. {{PigStatsOutputSizeReader}}?

Thoughts? 
 


 PigStats should not be assuming all Storage classes to be file-based storage
 

 Key: PIG-2924
 URL: https://issues.apache.org/jira/browse/PIG-2924
 Project: Pig
  Issue Type: Bug
  Components: tools
Affects Versions: 0.10.0, 0.9.2
Reporter: Harsh J
Assignee: Cheolsoo Park
 Attachments: PIG-2924-2.patch, PIG-2924.patch


 Using PigStatsUtil (like Oozie does) to collect JobStats for jobs that use a 
 HBaseStorage blows up when the stats are asked to be accumulated.
 This is because JobStats (which adds stuff up) is assuming all storages are 
 file based and that it can do listStatus/etc. operations on their 
 filespec-provided filename. For HBaseStorage, this is set to the tablename 
 and there's no such file, leading to an exception (FileNotFound or Invalid 
 URI - depending on using 'tablename' or 'hbase://tablename').

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-3008) Fix whitespace in Pig code

2012-10-29 Thread Daniel Dai (JIRA)


[ 
https://issues.apache.org/jira/browse/PIG-3008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13486237#comment-13486237
 ] 

Daniel Dai commented on PIG-3008:
-

Agree with Cheolsoo, we shall start with a limited scope. A giant patch would 
will make code tracing (svn blame) harder. Adding to the list:
- Use Unix new line style

 Fix whitespace in Pig code
 --

 Key: PIG-3008
 URL: https://issues.apache.org/jira/browse/PIG-3008
 Project: Pig
  Issue Type: Improvement
Reporter: Jonathan Coveney
 Fix For: 0.12

 Attachments: checkstyle.xml


 This JIRA exists mainly to get a conversation started. We've talked about it 
 before, and it's a tricky issue. That said, some of the Pig code is super, 
 super gnarly. We need some sort of path that will let it eventually be 
 fix-able.
 I posit: any file that hasn't been touched for over 6 months is eligible for 
 a whitespace patch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: [ANNOUNCE] Welcome new Apache Pig Committers Rohini Palaniswamy

2012-10-29 Thread Julien Le Dem

Congrats Rohini !


On Sun, Oct 28, 2012 at 9:42 AM, Bill Graham billgra...@gmail.com wrote:
 Congrats Rohini! Great news indeed.

 On Saturday, October 27, 2012, Jon Coveney wrote:

 Wonderful news!

 On Oct 26, 2012, at 9:51 PM, Gianmarco De Francisci Morales 
 g...@apache.org javascript:; wrote:

  Congratulations Rohini!
  Welcome onboard :)
  --
  Gianmarco
 
 
  On Fri, Oct 26, 2012 at 7:32 PM, Prasanth J 
  buckeye.prasa...@gmail.comjavascript:;
 wrote:
  Congrats Rohini!
 
  Thanks
  -- Prasanth
 
  On Oct 26, 2012, at 10:21 PM, Santhosh Srinivasan 
 santhosh_mut...@yahoo.com javascript:; wrote:
 
  Congrats Rohini! Full speed ahead now :)
 
  On Oct 26, 2012, at 4:37 PM, Daniel Dai 
  da...@hortonworks.comjavascript:;
 wrote:
 
  Here is another Pig committer announcement today. Please welcome
  Rohini Palaniswamy to be a Pig committer!
 
  Thanks,
  Daniel
 



 --
 Sent from Gmail Mobile

[jira] [Updated] (PIG-2409) Tracking URL for hadoop 23 does not show up

2012-10-29 Thread Daniel Dai (JIRA)


 [ 
https://issues.apache.org/jira/browse/PIG-2409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Dai updated PIG-2409:


Fix Version/s: (was: 0.11)
   0.12

 Tracking URL for hadoop 23 does not show up
 ---

 Key: PIG-2409
 URL: https://issues.apache.org/jira/browse/PIG-2409
 Project: Pig
  Issue Type: Bug
  Components: impl
Affects Versions: 0.9.2, 0.10.0, 0.11
Reporter: Daniel Dai
Assignee: Daniel Dai
Priority: Minor
  Labels: hadoop023
 Fix For: 0.12


 Pig used to show a tracking url for hadoop job:
 More information at: 
 http://localhost:50030/jobdetails.jsp?jobid=job_201112071119_0001
 This information does not show up in hadoop 23.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (PIG-3011) Add Cheolsoo to list of Apache Pig committers

2012-10-29 Thread Cheolsoo Park (JIRA)


 [ 
https://issues.apache.org/jira/browse/PIG-3011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cheolsoo Park resolved PIG-3011.


Resolution: Fixed

 Add Cheolsoo to list of Apache Pig committers
 -

 Key: PIG-3011
 URL: https://issues.apache.org/jira/browse/PIG-3011
 Project: Pig
  Issue Type: Bug
Reporter: Jonathan Coveney
Assignee: Cheolsoo Park
Priority: Critical

 It's always fun for someone's first commit to be adding themselves to this 
 page: http://pig.apache.org/whoweare.html (it's also a good chance to make 
 sure your dev setup is properly configured to allow committing)
 Welcome aboard!

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-3008) Fix whitespace in Pig code

2012-10-29 Thread Jonathan Coveney (JIRA)


[ 
https://issues.apache.org/jira/browse/PIG-3008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13486302#comment-13486302
 ] 

Jonathan Coveney commented on PIG-3008:
---

2 questions:

1. Is there a way to run checkstyle just on the files you changed?
2. Is there a way to make an IDE (or just the commandline) fix the indents for 
you? I can do it by hand but some if it is really bad and would be tedious to 
fix.

I like the idea that if you touch a file for other reasons, then we can 
automate the flow to try and get the whitespace formatting good.

 Fix whitespace in Pig code
 --

 Key: PIG-3008
 URL: https://issues.apache.org/jira/browse/PIG-3008
 Project: Pig
  Issue Type: Improvement
Reporter: Jonathan Coveney
 Fix For: 0.12

 Attachments: checkstyle.xml


 This JIRA exists mainly to get a conversation started. We've talked about it 
 before, and it's a tricky issue. That said, some of the Pig code is super, 
 super gnarly. We need some sort of path that will let it eventually be 
 fix-able.
 I posit: any file that hasn't been touched for over 6 months is eligible for 
 a whitespace patch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-2968) ColumnMapKeyPrune fails to prune a subtree inside foreach

2012-10-29 Thread Cheolsoo Park (JIRA)


[ 
https://issues.apache.org/jira/browse/PIG-2968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13486329#comment-13486329
 ] 

Cheolsoo Park commented on PIG-2968:


+1.

I will commit it after running tests. Thanks Koji!

 ColumnMapKeyPrune fails to prune a subtree inside foreach
 -

 Key: PIG-2968
 URL: https://issues.apache.org/jira/browse/PIG-2968
 Project: Pig
  Issue Type: Bug
  Components: parser
Reporter: Koji Noguchi
Assignee: Koji Noguchi
Priority: Minor
 Fix For: 0.11

 Attachments: pig-2968-trunk_v01.txt, pig-2968-trunk_v02.txt


 Sample code 
 {noformat}
 $ cat test/foreach.pig 
 daily = load 'nyse' as (exchange, symbol);
 grpd = group daily by exchange;
 uniquecnt = foreach grpd {
 sym = daily.symbol;
 uniq_sym = distinct sym;
 generate group, uniq_sym;
 };
 another = FOREACH uniquecnt GENERATE group;
 explain another;
 {noformat}
 This breaks when it tries to prune uniq_sym-sym-innerload_daily
 bq. 2012-10-12 14:54:11,031 [main] ERROR org.apache.pig.tools.grunt.Grunt - 
 ERROR 2000: Error processing rule ColumnMapKeyPrune. Try -t ColumnMapKeyPrune

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-2968) ColumnMapKeyPrune fails to prune a subtree inside foreach

2012-10-29 Thread Cheolsoo Park (JIRA)


[ 
https://issues.apache.org/jira/browse/PIG-2968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13486342#comment-13486342
 ] 

Cheolsoo Park commented on PIG-2968:


Actually, can you please fix the indentation of testPruneSubTreeForEach() using 
4 spaces? We're trying to clean up white spaces in code base.

Thanks!


 ColumnMapKeyPrune fails to prune a subtree inside foreach
 -

 Key: PIG-2968
 URL: https://issues.apache.org/jira/browse/PIG-2968
 Project: Pig
  Issue Type: Bug
  Components: parser
Reporter: Koji Noguchi
Assignee: Koji Noguchi
Priority: Minor
 Fix For: 0.11

 Attachments: pig-2968-trunk_v01.txt, pig-2968-trunk_v02.txt


 Sample code 
 {noformat}
 $ cat test/foreach.pig 
 daily = load 'nyse' as (exchange, symbol);
 grpd = group daily by exchange;
 uniquecnt = foreach grpd {
 sym = daily.symbol;
 uniq_sym = distinct sym;
 generate group, uniq_sym;
 };
 another = FOREACH uniquecnt GENERATE group;
 explain another;
 {noformat}
 This breaks when it tries to prune uniq_sym-sym-innerload_daily
 bq. 2012-10-12 14:54:11,031 [main] ERROR org.apache.pig.tools.grunt.Grunt - 
 ERROR 2000: Error processing rule ColumnMapKeyPrune. Try -t ColumnMapKeyPrune

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-2968) ColumnMapKeyPrune fails to prune a subtree inside foreach

2012-10-29 Thread Rohini Palaniswamy (JIRA)


[ 
https://issues.apache.org/jira/browse/PIG-2968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13486344#comment-13486344
 ] 

Rohini Palaniswamy commented on PIG-2968:
-

Koji/Cheolsoo,
bq. But I couldn't come up with expected raw query that matches the pruned 
result.
   I missed commenting on this in the initial review. Should we take a crack at 
this before committing this. Or at least some Assert? 

 ColumnMapKeyPrune fails to prune a subtree inside foreach
 -

 Key: PIG-2968
 URL: https://issues.apache.org/jira/browse/PIG-2968
 Project: Pig
  Issue Type: Bug
  Components: parser
Reporter: Koji Noguchi
Assignee: Koji Noguchi
Priority: Minor
 Fix For: 0.11

 Attachments: pig-2968-trunk_v01.txt, pig-2968-trunk_v02.txt


 Sample code 
 {noformat}
 $ cat test/foreach.pig 
 daily = load 'nyse' as (exchange, symbol);
 grpd = group daily by exchange;
 uniquecnt = foreach grpd {
 sym = daily.symbol;
 uniq_sym = distinct sym;
 generate group, uniq_sym;
 };
 another = FOREACH uniquecnt GENERATE group;
 explain another;
 {noformat}
 This breaks when it tries to prune uniq_sym-sym-innerload_daily
 bq. 2012-10-12 14:54:11,031 [main] ERROR org.apache.pig.tools.grunt.Grunt - 
 ERROR 2000: Error processing rule ColumnMapKeyPrune. Try -t ColumnMapKeyPrune

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-2990) the -secretDebugCmd shouldn't be a secret and should just be...a command

2012-10-29 Thread Jonathan Coveney (JIRA)


 [ 
https://issues.apache.org/jira/browse/PIG-2990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Coveney updated PIG-2990:
--

   Resolution: Fixed
Fix Version/s: 0.12
   Status: Resolved  (was: Patch Available)

Thanks Rohini! Committed.

 the -secretDebugCmd shouldn't be a secret and should just be...a command
 

 Key: PIG-2990
 URL: https://issues.apache.org/jira/browse/PIG-2990
 Project: Pig
  Issue Type: Improvement
Reporter: Jonathan Coveney
Assignee: Jonathan Coveney
Priority: Minor
  Labels: newbie
 Fix For: 0.12

 Attachments: PIG-2990-0.patch


 It's a useful command, and it's weird that it's not in -help

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-3008) Fix whitespace in Pig code

2012-10-29 Thread Gianmarco De Francisci Morales (JIRA)

[
https://issues.apache.org/jira/browse/PIG-3008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13486350#comment-13486350
]

Gianmarco De Francisci Morales commented on PIG-3008:
-

1. I am assuming we will integrate this with Ant. The checkstyle ant target
wants a list of file to run on. So if you are able to identify the list of
files you changed automatically, then yes. How do you define changed?
2. Yes, as long as there is a compatible formatter profile (i.e. it is not too
complex). For our use case it should be fine. I know it can be done with
Eclipse. For other IDEs I guess it can be done as well but don't know how.

We also need integration with Jenkins for automatic builds.

Fix whitespace in Pig code
--

Key: PIG-3008
URL: https://issues.apache.org/jira/browse/PIG-3008
Project: Pig
Issue Type: Improvement
Reporter: Jonathan Coveney
Fix For: 0.12

Attachments: checkstyle.xml

This JIRA exists mainly to get a conversation started. We've talked about it
before, and it's a tricky issue. That said, some of the Pig code is super,
super gnarly. We need some sort of path that will let it eventually be
fix-able.
I posit: any file that hasn't been touched for over 6 months is eligible for
a whitespace patch.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-2898) Parallel execution of e2e tests

2012-10-29 Thread Daniel Dai (JIRA)


[ 
https://issues.apache.org/jira/browse/PIG-2898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13486352#comment-13486352
 ] 

Daniel Dai commented on PIG-2898:
-

+1. Rohini, can you commit once you get permission?

 Parallel execution of e2e tests
 ---

 Key: PIG-2898
 URL: https://issues.apache.org/jira/browse/PIG-2898
 Project: Pig
  Issue Type: Improvement
  Components: e2e harness
Affects Versions: 0.10.0
Reporter: Andrey Klochkov
Assignee: Ivan A. Veselovsky
  Labels: test
 Fix For: 0.11, 0.12

 Attachments: PIG-2898-branch-0.10-6-final.patch, 
 PIG-2898-branch-0.10-7.patch, PIG-2898-trunk-3.patch, 
 PIG-2898-trunk-6-final.patch, PIG-2898-trunk-7.patch


 Today it takes ~19 hours to run the full set of e2e tests in mapred mode. The 
 bottleneck here is the client side, and per our observations it can help a 
 lot if the e2e harness would be able to run tests in parallel threads.
 We prototyped changes in e2e harness allowing to run tests in a configurable 
 number of threads. Preliminary results show more than 6x reduction in 
 execution time when using a small 3-nodes M/R cluster with modest 
 configuration. Going to share a patch shortly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-2968) ColumnMapKeyPrune fails to prune a subtree inside foreach

2012-10-29 Thread Cheolsoo Park (JIRA)


[ 
https://issues.apache.org/jira/browse/PIG-2968?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13486365#comment-13486365
 ] 

Cheolsoo Park commented on PIG-2968:


Hi Rohini, sure I can wait until your concern is addressed.

If Koji can improve the test case, that will be of course great. But since I 
don't have a better suggestion, I won't insist.

Thanks!

 ColumnMapKeyPrune fails to prune a subtree inside foreach
 -

 Key: PIG-2968
 URL: https://issues.apache.org/jira/browse/PIG-2968
 Project: Pig
  Issue Type: Bug
  Components: parser
Reporter: Koji Noguchi
Assignee: Koji Noguchi
Priority: Minor
 Fix For: 0.11

 Attachments: pig-2968-trunk_v01.txt, pig-2968-trunk_v02.txt


 Sample code 
 {noformat}
 $ cat test/foreach.pig 
 daily = load 'nyse' as (exchange, symbol);
 grpd = group daily by exchange;
 uniquecnt = foreach grpd {
 sym = daily.symbol;
 uniq_sym = distinct sym;
 generate group, uniq_sym;
 };
 another = FOREACH uniquecnt GENERATE group;
 explain another;
 {noformat}
 This breaks when it tries to prune uniq_sym-sym-innerload_daily
 bq. 2012-10-12 14:54:11,031 [main] ERROR org.apache.pig.tools.grunt.Grunt - 
 ERROR 2000: Error processing rule ColumnMapKeyPrune. Try -t ColumnMapKeyPrune

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (PIG-3015) Rewrite of AvroStorage

2012-10-29 Thread Joseph Adler (JIRA)

Joseph Adler created PIG-3015:
-

 Summary: Rewrite of AvroStorage
 Key: PIG-3015
 URL: https://issues.apache.org/jira/browse/PIG-3015
 Project: Pig
  Issue Type: Improvement
  Components: piggybank
Reporter: Joseph Adler


The current AvroStorage implementation has a lot of issues: it requires old 
versions of Avro, it copies data much more than needed, and it's verbose and 
complicated. (One pet peeve of mine is that old versions of Avro don't support 
Snappy compression.)

I rewrote AvroStorage from scratch to fix these issues. In early tests, the new 
implementation is significantly faster, and the code is a lot simpler. 
Rewriting AvroStorage also enabled me to implement support for Trevni.

I'm opening this ticket to facilitate discussion while I figure out the best 
way to contribute the changes back to Apache.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-2582) Store size in bytes (not mbytes) in ResourceStatistics

2012-10-29 Thread Jonathan Coveney (JIRA)


[ 
https://issues.apache.org/jira/browse/PIG-2582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13486396#comment-13486396
 ] 

Jonathan Coveney commented on PIG-2582:
---

Given it is unstable, I say just go ahead with the plan to make it private, and 
have the deprecated getter/setter

 Store size in bytes (not mbytes) in ResourceStatistics
 --

 Key: PIG-2582
 URL: https://issues.apache.org/jira/browse/PIG-2582
 Project: Pig
  Issue Type: Bug
Reporter: Travis Crawford
Assignee: Prashant Kommireddi
Priority: Minor
 Attachments: PIG-2582.patch


 In 
 [ResourceStatistics.java|http://svn.apache.org/viewvc/pig/trunk/src/org/apache/pig/ResourceStatistics.java?view=markup]
  we see mBytes is public, and has a public getter/setter.
 {code}
 47public Long mBytes; // size in megabytes
 196   public Long getmBytes() {
 197   return mBytes;
 198   }
 199   public ResourceStatistics setmBytes(Long mBytes) {
 200   this.mBytes = mBytes;
 201   return this;
 202   }
 {code}
 Typically sizes are stored as bytes, potentially having convenience functions 
 to return with different units.
 If mBytes can be marked private without causing woes it might be worth 
 storing size as bytes instead.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-2582) Store size in bytes (not mbytes) in ResourceStatistics

2012-10-29 Thread Prashant Kommireddi (JIRA)


[ 
https://issues.apache.org/jira/browse/PIG-2582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13486410#comment-13486410
 ] 

Prashant Kommireddi commented on PIG-2582:
--

Also, there is a common theme of returning the object return this; on all 
setters. I don't think this should exist, but we should probably tackle that in 
the next release to make sure we are fine with the existing changes at first.


 Store size in bytes (not mbytes) in ResourceStatistics
 --

 Key: PIG-2582
 URL: https://issues.apache.org/jira/browse/PIG-2582
 Project: Pig
  Issue Type: Bug
Reporter: Travis Crawford
Assignee: Prashant Kommireddi
Priority: Minor
 Attachments: PIG-2582.patch


 In 
 [ResourceStatistics.java|http://svn.apache.org/viewvc/pig/trunk/src/org/apache/pig/ResourceStatistics.java?view=markup]
  we see mBytes is public, and has a public getter/setter.
 {code}
 47public Long mBytes; // size in megabytes
 196   public Long getmBytes() {
 197   return mBytes;
 198   }
 199   public ResourceStatistics setmBytes(Long mBytes) {
 200   this.mBytes = mBytes;
 201   return this;
 202   }
 {code}
 Typically sizes are stored as bytes, potentially having convenience functions 
 to return with different units.
 If mBytes can be marked private without causing woes it might be worth 
 storing size as bytes instead.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-3015) Rewrite of AvroStorage

2012-10-29 Thread Cheolsoo Park (JIRA)


[ 
https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13486427#comment-13486427
 ] 

Cheolsoo Park commented on PIG-3015:


Hi Joseph,

Thank you very much for opening the jira. I have recently worked on AvroStorage 
by myself, and I totally agree with you. Since you already have code to 
contribute, this is even better. :-)

As part of re-write, I would also like to propose to migrate AvroStorge from 
Piggybank to the core Pig. I have 2 reasons for this:
# AvroStorage is widely used, so it makes sense to include it in the core Pig 
rather than in Piggybank.
# Until migration is complete, we can maintain both versions (new one in core 
Pig and old one in Piggybank) to avoid breaking backward compatibility. Another 
motivation for re-write to me is to clean up funny options that the current 
AvroStorage has. So I think that it's unavoidable to break backward 
compatibility.

I asked this question on the [user mailing 
list|http://mail-archives.apache.org/mod_mbox/pig-user/201208.mbox/%3C27EE5059-F811-4E19-B1A3-951B4BB3BDDF%40hortonworks.com%3E]
 a while ago, and nobody disagreed. But please let me know if anyone has 
objections.


To start with, I am wondering if you can post your code as a patch to this jira 
and the review board. Assuming that we're going to move AvroStorage to the core 
Pig, you can probably create a new package called 
org.apache.pig.backend.hadoop.avro and add your code there. If you could 
break your patch into smaller pieces and attach them to sub-tasks of this jira, 
that would be helpful too. 

Please let me know what you think.

Thanks!

 Rewrite of AvroStorage
 --

 Key: PIG-3015
 URL: https://issues.apache.org/jira/browse/PIG-3015
 Project: Pig
  Issue Type: Improvement
  Components: piggybank
Reporter: Joseph Adler

 The current AvroStorage implementation has a lot of issues: it requires old 
 versions of Avro, it copies data much more than needed, and it's verbose and 
 complicated. (One pet peeve of mine is that old versions of Avro don't 
 support Snappy compression.)
 I rewrote AvroStorage from scratch to fix these issues. In early tests, the 
 new implementation is significantly faster, and the code is a lot simpler. 
 Rewriting AvroStorage also enabled me to implement support for Trevni.
 I'm opening this ticket to facilitate discussion while I figure out the best 
 way to contribute the changes back to Apache.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Build failed in Jenkins: Pig-trunk #1349

2012-10-29 Thread Apache Jenkins Server

See https://builds.apache.org/job/Pig-trunk/1349/changes

Changes:

[jcoveney] PIG-2990: the -secretDebugCmd shouldnt be a secret and should just 
be...a command (jcoveney)

--
[...truncated 6641 lines...]
 [findbugs]   org.apache.hadoop.io.file.tfile.TFile$Reader$Scanner$Entry
 [findbugs]   org.apache.hadoop.fs.FSDataInputStream
 [findbugs]   org.python.core.PyObject
 [findbugs]   jline.History
 [findbugs]   org.jruby.embed.internal.LocalContextProvider
 [findbugs]   org.apache.hadoop.io.BooleanWritable
 [findbugs]   org.apache.log4j.Logger
 [findbugs]   org.apache.hadoop.hbase.filter.FamilyFilter
 [findbugs]   org.codehaus.jackson.annotate.JsonPropertyOrder
 [findbugs]   groovy.lang.Tuple
 [findbugs]   org.antlr.runtime.IntStream
 [findbugs]   org.apache.hadoop.util.ReflectionUtils
 [findbugs]   org.apache.hadoop.fs.ContentSummary
 [findbugs]   org.jruby.runtime.builtin.IRubyObject
 [findbugs]   org.jruby.RubyInteger
 [findbugs]   org.python.core.PyTuple
 [findbugs]   org.mortbay.log.Log
 [findbugs]   org.apache.hadoop.conf.Configuration
 [findbugs]   com.google.common.base.Joiner
 [findbugs]   org.apache.hadoop.mapreduce.lib.input.FileSplit
 [findbugs]   org.apache.hadoop.mapred.Counters$Counter
 [findbugs]   com.jcraft.jsch.Channel
 [findbugs]   org.apache.hadoop.mapred.JobPriority
 [findbugs]   org.apache.commons.cli.Options
 [findbugs]   org.apache.hadoop.mapred.JobID
 [findbugs]   org.apache.hadoop.util.bloom.BloomFilter
 [findbugs]   org.python.core.PyFrame
 [findbugs]   org.apache.hadoop.hbase.filter.CompareFilter
 [findbugs]   org.apache.hadoop.util.VersionInfo
 [findbugs]   org.python.core.PyString
 [findbugs]   org.apache.hadoop.io.Text$Comparator
 [findbugs]   org.jruby.runtime.Block
 [findbugs]   org.antlr.runtime.MismatchedSetException
 [findbugs]   org.apache.hadoop.io.BytesWritable
 [findbugs]   org.apache.hadoop.fs.FsShell
 [findbugs]   org.joda.time.Months
 [findbugs]   org.mozilla.javascript.ImporterTopLevel
 [findbugs]   org.apache.hadoop.hbase.mapreduce.TableOutputFormat
 [findbugs]   org.apache.hadoop.mapred.TaskReport
 [findbugs]   org.apache.hadoop.security.UserGroupInformation
 [findbugs]   org.antlr.runtime.tree.RewriteRuleSubtreeStream
 [findbugs]   org.apache.commons.cli.HelpFormatter
 [findbugs]   com.google.common.collect.Maps
 [findbugs]   org.joda.time.ReadableInstant
 [findbugs]   org.mozilla.javascript.NativeObject
 [findbugs]   org.apache.hadoop.hbase.HConstants
 [findbugs]   org.apache.hadoop.io.serializer.Deserializer
 [findbugs]   org.antlr.runtime.FailedPredicateException
 [findbugs]   org.apache.hadoop.io.compress.CompressionCodec
 [findbugs]   org.jruby.RubyNil
 [findbugs]   org.apache.hadoop.fs.FileStatus
 [findbugs]   org.apache.hadoop.hbase.client.Result
 [findbugs]   org.apache.hadoop.mapreduce.JobContext
 [findbugs]   org.codehaus.jackson.JsonGenerator
 [findbugs]   org.apache.hadoop.mapreduce.TaskAttemptContext
 [findbugs]   org.apache.hadoop.io.LongWritable$Comparator
 [findbugs]   org.codehaus.jackson.map.util.LRUMap
 [findbugs]   org.apache.hadoop.hbase.util.Bytes
 [findbugs]   org.antlr.runtime.MismatchedTokenException
 [findbugs]   org.codehaus.jackson.JsonParser
 [findbugs]   com.jcraft.jsch.UserInfo
 [findbugs]   org.python.core.PyException
 [findbugs]   org.apache.commons.cli.ParseException
 [findbugs]   org.apache.hadoop.io.compress.CompressionOutputStream
 [findbugs]   org.apache.hadoop.hbase.filter.WritableByteArrayComparable
 [findbugs]   org.antlr.runtime.tree.CommonTreeNodeStream
 [findbugs]   org.apache.log4j.Level
 [findbugs]   org.apache.hadoop.hbase.client.Scan
 [findbugs]   org.jruby.anno.JRubyMethod
 [findbugs]   org.apache.hadoop.mapreduce.Job
 [findbugs]   com.google.common.util.concurrent.Futures
 [findbugs]   org.apache.commons.logging.LogFactory
 [findbugs]   org.apache.commons.collections.IteratorUtils
 [findbugs]   org.apache.commons.codec.binary.Base64
 [findbugs]   org.codehaus.jackson.map.ObjectMapper
 [findbugs]   org.apache.hadoop.fs.FileSystem
 [findbugs]   org.jruby.embed.LocalContextScope
 [findbugs]   org.apache.hadoop.hbase.filter.FilterList$Operator
 [findbugs]   org.jruby.RubySymbol
 [findbugs]   org.apache.hadoop.hbase.io.ImmutableBytesWritable
 [findbugs]   org.apache.hadoop.io.serializer.SerializationFactory
 [findbugs]   org.antlr.runtime.tree.TreeAdaptor
 [findbugs]   org.apache.hadoop.mapred.RunningJob
 [findbugs]   org.antlr.runtime.CommonTokenStream
 [findbugs]   org.apache.hadoop.io.DataInputBuffer
 [findbugs]   org.apache.hadoop.io.file.tfile.TFile
 [findbugs]   org.apache.commons.cli.GnuParser
 [findbugs]   org.mozilla.javascript.Context
 [findbugs]   org.apache.hadoop.io.FloatWritable
 [findbugs]   org.antlr.runtime.tree.RewriteEarlyExitException
 [findbugs]   org.apache.hadoop.hbase.HBaseConfiguration
 [findbugs]   org.codehaus.jackson.JsonGenerationException
 [findbugs]   org.apache.hadoop.mapreduce.TaskInputOutputContext
 [findbugs]

[jira] [Updated] (PIG-2997) Provide a convenience constructor on PigServer that accepts Configuration

2012-10-29 Thread Prashant Kommireddi (JIRA)


 [ 
https://issues.apache.org/jira/browse/PIG-2997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Prashant Kommireddi updated PIG-2997:
-

Attachment: PIG-2997_1.patch

Thanks for the review Rohini. 

Adding a test case. Also, I have moved the Configuration-Properties switch 
into PigContext so stuff other than PigServer can use it too. 

 Provide a convenience constructor on PigServer that accepts Configuration
 -

 Key: PIG-2997
 URL: https://issues.apache.org/jira/browse/PIG-2997
 Project: Pig
  Issue Type: Improvement
Reporter: Prashant Kommireddi
Assignee: Prashant Kommireddi
 Fix For: 0.11

 Attachments: PIG-2997_1.patch, PIG-2997.patch


 PigServer currently has Properties based constructor. Hadoop in general deals 
 with Configuration and it would be good to have a PigServer constructor that 
 accepts the same. With this, user does not have to worry about creating 
 Properties object out of conf and can simply invoke this new constructor.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

79 matches

Mail list logo