from:"Joseph Adler"

Joseph Adler created PIG-3120:
-

 Summary: setStoreFuncUDFContextSignature called with null signature
 Key: PIG-3120
 URL: https://issues.apache.org/jira/browse/PIG-3120
 Project: Pig
  Issue Type: Bug
  Components: impl
Affects Versions: 0.12
Reporter: Joseph Adler
Priority: Critical
 Fix For: 0.12


I'm currently working on PIG-3015 and am having trouble passing the 
UDFContextSignature to the store func. It looks like the store func on the head 
end is being set to a non-null value, but a null value is being passed to 
setStoreFuncUDFContextSignature on the back end. I'm opening this ticket to 
track this issue; I'll follow up with a reproducible test case when I have a 
clean one.

I suspect this problem occurs when running on a real cluster, but may not occur 
in the standard unit tests.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-3120) setStoreFuncUDFContextSignature called with null signature


[ 
https://issues.apache.org/jira/browse/PIG-3120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13553352#comment-13553352
 ] 

Joseph Adler commented on PIG-3120:
---

OK, tracked down the issue. It looks like the UDFContextSignature is not 
getting propagated if there is a LIMIT statement in the pig code.

Very specifically, in 
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.LimitAdjuster.adjust,
 it looks like Pig was creating a new POStore object but not copying the 
signature. Here is the offending code:

{code}
// this is line 132...
POStore st = new POStore(new 
OperatorKey(scope,nig.getNextNodeId(scope)));
st.setSFile(oldSpec);
st.setIsTmpStore(oldIsTmpStore);
st.setSchema(((POStore)mpLeaf).getSchema());

limitAdjustMROp.reducePlan.addAsLeaf(st);
{code}

This is easily fixable by inserting this statement at line 137:

{code}
st.setSignature(((POStore)mpLeaf).getSignature());
{code}
I'll follow up with a path for this issue.

 setStoreFuncUDFContextSignature called with null signature
 --

 Key: PIG-3120
 URL: https://issues.apache.org/jira/browse/PIG-3120
 Project: Pig
  Issue Type: Bug
  Components: impl
Affects Versions: 0.12
Reporter: Joseph Adler
Priority: Critical
 Fix For: 0.12


 I'm currently working on PIG-3015 and am having trouble passing the 
 UDFContextSignature to the store func. It looks like the store func on the 
 head end is being set to a non-null value, but a null value is being passed 
 to setStoreFuncUDFContextSignature on the back end. I'm opening this ticket 
 to track this issue; I'll follow up with a reproducible test case when I have 
 a clean one.
 I suspect this problem occurs when running on a real cluster, but may not 
 occur in the standard unit tests.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-3120) setStoreFuncUDFContextSignature called with null signature


 [ 
https://issues.apache.org/jira/browse/PIG-3120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Adler updated PIG-3120:
--

Status: Patch Available  (was: Open)

 setStoreFuncUDFContextSignature called with null signature
 --

 Key: PIG-3120
 URL: https://issues.apache.org/jira/browse/PIG-3120
 Project: Pig
  Issue Type: Bug
  Components: impl
Affects Versions: 0.12
Reporter: Joseph Adler
Priority: Critical
 Fix For: 0.12


 I'm currently working on PIG-3015 and am having trouble passing the 
 UDFContextSignature to the store func. It looks like the store func on the 
 head end is being set to a non-null value, but a null value is being passed 
 to setStoreFuncUDFContextSignature on the back end. I'm opening this ticket 
 to track this issue; I'll follow up with a reproducible test case when I have 
 a clean one.
 I suspect this problem occurs when running on a real cluster, but may not 
 occur in the standard unit tests.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-3120) setStoreFuncUDFContextSignature called with null signature


 [ 
https://issues.apache.org/jira/browse/PIG-3120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Adler updated PIG-3120:
--

Status: Open  (was: Patch Available)

 setStoreFuncUDFContextSignature called with null signature
 --

 Key: PIG-3120
 URL: https://issues.apache.org/jira/browse/PIG-3120
 Project: Pig
  Issue Type: Bug
  Components: impl
Affects Versions: 0.12
Reporter: Joseph Adler
Priority: Critical
 Fix For: 0.12


 I'm currently working on PIG-3015 and am having trouble passing the 
 UDFContextSignature to the store func. It looks like the store func on the 
 head end is being set to a non-null value, but a null value is being passed 
 to setStoreFuncUDFContextSignature on the back end. I'm opening this ticket 
 to track this issue; I'll follow up with a reproducible test case when I have 
 a clean one.
 I suspect this problem occurs when running on a real cluster, but may not 
 occur in the standard unit tests.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-3120) setStoreFuncUDFContextSignature called with null signature


 [ 
https://issues.apache.org/jira/browse/PIG-3120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Adler updated PIG-3120:
--

Attachment: PIG-3120.patch

This patch resolves an issue with UDF StoreFunc signatures when using LIMIT 
statements

 setStoreFuncUDFContextSignature called with null signature
 --

 Key: PIG-3120
 URL: https://issues.apache.org/jira/browse/PIG-3120
 Project: Pig
  Issue Type: Bug
  Components: impl
Affects Versions: 0.12
Reporter: Joseph Adler
Priority: Critical
 Fix For: 0.12

 Attachments: PIG-3120.patch


 I'm currently working on PIG-3015 and am having trouble passing the 
 UDFContextSignature to the store func. It looks like the store func on the 
 head end is being set to a non-null value, but a null value is being passed 
 to setStoreFuncUDFContextSignature on the back end. I'm opening this ticket 
 to track this issue; I'll follow up with a reproducible test case when I have 
 a clean one.
 I suspect this problem occurs when running on a real cluster, but may not 
 occur in the standard unit tests.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-3015) Rewrite of AvroStorage

2013-01-11 Thread Joseph Adler (JIRA)

[
https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13551502#comment-13551502
]

Joseph Adler commented on PIG-3015:
---

Just got bitten by PIG-2266 while doing some performance testing with this
ticket. I'm going to add that fix to this patch so that AvroStorage and
TrevniStorage actually work.

Rewrite of AvroStorage
--

[jira] [Commented] (PIG-3015) Rewrite of AvroStorage

2013-01-07 Thread Joseph Adler (JIRA)


[ 
https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546164#comment-13546164
 ] 

Joseph Adler commented on PIG-3015:
---

Hi Cheolsoo:

What size file are you using? You can configure the sync interval with the 
parameter avro.mapred.sync.interval (defined in 
org.apache.avro.mapred.AvroOutputFormat), and implemented in my latest patch 
(the one from last week).

-- Joe

 Rewrite of AvroStorage
 --

 Key: PIG-3015
 URL: https://issues.apache.org/jira/browse/PIG-3015
 Project: Pig
  Issue Type: Improvement
  Components: piggybank
Reporter: Joseph Adler
Assignee: Joseph Adler
 Attachments: PIG-3015-2.patch, PIG-3015-3.patch, PIG-3015-4.patch, 
 PIG-3015-5.patch, Test.tar.gz


 The current AvroStorage implementation has a lot of issues: it requires old 
 versions of Avro, it copies data much more than needed, and it's verbose and 
 complicated. (One pet peeve of mine is that old versions of Avro don't 
 support Snappy compression.)
 I rewrote AvroStorage from scratch to fix these issues. In early tests, the 
 new implementation is significantly faster, and the code is a lot simpler. 
 Rewriting AvroStorage also enabled me to implement support for Trevni (as 
 TrevniStorage).
 I'm opening this ticket to facilitate discussion while I figure out the best 
 way to contribute the changes back to Apache.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-3015) Rewrite of AvroStorage


 [ 
https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Adler updated PIG-3015:
--

Attachment: PIG-3015-5.patch

Added fixes for compression (and other metadata)

 Rewrite of AvroStorage
 --

 Key: PIG-3015
 URL: https://issues.apache.org/jira/browse/PIG-3015
 Project: Pig
  Issue Type: Improvement
  Components: piggybank
Reporter: Joseph Adler
Assignee: Joseph Adler
 Attachments: PIG-3015-2.patch, PIG-3015-3.patch, PIG-3015-4.patch, 
 PIG-3015-5.patch, PIG-3015.patch


 The current AvroStorage implementation has a lot of issues: it requires old 
 versions of Avro, it copies data much more than needed, and it's verbose and 
 complicated. (One pet peeve of mine is that old versions of Avro don't 
 support Snappy compression.)
 I rewrote AvroStorage from scratch to fix these issues. In early tests, the 
 new implementation is significantly faster, and the code is a lot simpler. 
 Rewriting AvroStorage also enabled me to implement support for Trevni (as 
 TrevniStorage).
 I'm opening this ticket to facilitate discussion while I figure out the best 
 way to contribute the changes back to Apache.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-3015) Rewrite of AvroStorage


 [ 
https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Adler updated PIG-3015:
--

Attachment: (was: PIG-3015.patch)

 Rewrite of AvroStorage
 --

 Key: PIG-3015
 URL: https://issues.apache.org/jira/browse/PIG-3015
 Project: Pig
  Issue Type: Improvement
  Components: piggybank
Reporter: Joseph Adler
Assignee: Joseph Adler
 Attachments: PIG-3015-2.patch, PIG-3015-3.patch, PIG-3015-4.patch


 The current AvroStorage implementation has a lot of issues: it requires old 
 versions of Avro, it copies data much more than needed, and it's verbose and 
 complicated. (One pet peeve of mine is that old versions of Avro don't 
 support Snappy compression.)
 I rewrote AvroStorage from scratch to fix these issues. In early tests, the 
 new implementation is significantly faster, and the code is a lot simpler. 
 Rewriting AvroStorage also enabled me to implement support for Trevni (as 
 TrevniStorage).
 I'm opening this ticket to facilitate discussion while I figure out the best 
 way to contribute the changes back to Apache.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-3015) Rewrite of AvroStorage


 [ 
https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Adler updated PIG-3015:
--

Attachment: (was: PIG-3015-5.patch)

 Rewrite of AvroStorage
 --

 Key: PIG-3015
 URL: https://issues.apache.org/jira/browse/PIG-3015
 Project: Pig
  Issue Type: Improvement
  Components: piggybank
Reporter: Joseph Adler
Assignee: Joseph Adler
 Attachments: PIG-3015-2.patch, PIG-3015-3.patch, PIG-3015-4.patch


 The current AvroStorage implementation has a lot of issues: it requires old 
 versions of Avro, it copies data much more than needed, and it's verbose and 
 complicated. (One pet peeve of mine is that old versions of Avro don't 
 support Snappy compression.)
 I rewrote AvroStorage from scratch to fix these issues. In early tests, the 
 new implementation is significantly faster, and the code is a lot simpler. 
 Rewriting AvroStorage also enabled me to implement support for Trevni (as 
 TrevniStorage).
 I'm opening this ticket to facilitate discussion while I figure out the best 
 way to contribute the changes back to Apache.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-3015) Rewrite of AvroStorage


 [ 
https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Adler updated PIG-3015:
--

Attachment: PIG-3015-5.patch

Oops, this one contains the changes.

 Rewrite of AvroStorage
 --

 Key: PIG-3015
 URL: https://issues.apache.org/jira/browse/PIG-3015
 Project: Pig
  Issue Type: Improvement
  Components: piggybank
Reporter: Joseph Adler
Assignee: Joseph Adler
 Attachments: PIG-3015-2.patch, PIG-3015-3.patch, PIG-3015-4.patch, 
 PIG-3015-5.patch


 The current AvroStorage implementation has a lot of issues: it requires old 
 versions of Avro, it copies data much more than needed, and it's verbose and 
 complicated. (One pet peeve of mine is that old versions of Avro don't 
 support Snappy compression.)
 I rewrote AvroStorage from scratch to fix these issues. In early tests, the 
 new implementation is significantly faster, and the code is a lot simpler. 
 Rewriting AvroStorage also enabled me to implement support for Trevni (as 
 TrevniStorage).
 I'm opening this ticket to facilitate discussion while I figure out the best 
 way to contribute the changes back to Apache.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Review Request: PIG-3015 Rewrite of AvroStorage

2013-01-04 Thread Joseph Adler

/recordsWithDoubleUnderscores.avsc 
PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/recordsWithEnums.avsc PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/recordsWithFixed.avsc PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/recordsWithMaps.avsc PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/recordsWithMapsOfRecords.avsc 
PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/recordsWithNullableUnions.avsc 
PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/recursiveRecord.avsc PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/simpleRecordsTrevni.avsc PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/testDirectory.avsc PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/testDirectoryCounts.avsc PRE-CREATION 
  test/unit-tests 7cede06 

Diff: https://reviews.apache.org/r/8104/diff/


Testing
---


Thanks,

Joseph Adler

Re: Review Request: PIG-3015 Rewrite of AvroStorage

2013-01-04 Thread Joseph Adler

/pig/builtin/avro/schema/recordsWithEnums.avsc PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/recordsWithFixed.avsc PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/recordsWithMaps.avsc PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/recordsWithMapsOfRecords.avsc 
PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/recordsWithNullableUnions.avsc 
PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/recursiveRecord.avsc PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/simpleRecordsTrevni.avsc PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/testDirectory.avsc PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/testDirectoryCounts.avsc PRE-CREATION 
  test/unit-tests 7cede06 

Diff: https://reviews.apache.org/r/8104/diff/


Testing
---


Thanks,

Joseph Adler

[jira] [Commented] (PIG-3015) Rewrite of AvroStorage


[ 
https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13544248#comment-13544248
 ] 

Joseph Adler commented on PIG-3015:
---

Hi Russ,

I think you're right... it looks like you could do something like this in 
AvroRecordReader.nextKeyValue:

{code}
  @Override
  public boolean nextKeyValue() throws IOException, InterruptedException {

if (reader.pastSync(end)) {
  return false;
}

try {
  currentRecord = reader.next(new GenericData.Record(schema));
} catch (NoSuchElementException e) {
  return false;
} catch (IOException ioe) {
  reader.sync(reader.tell()+1);
  throw ioe;
}

return true;
  }
{code}

Let me test this out to make sure it runs correctly on uncorrupted files. Would 
you mind creating a corrupted test file that I can use for testing?

 Rewrite of AvroStorage
 --

 Key: PIG-3015
 URL: https://issues.apache.org/jira/browse/PIG-3015
 Project: Pig
  Issue Type: Improvement
  Components: piggybank
Reporter: Joseph Adler
Assignee: Joseph Adler
 Attachments: PIG-3015-2.patch, PIG-3015-3.patch, PIG-3015-4.patch, 
 PIG-3015-5.patch


 The current AvroStorage implementation has a lot of issues: it requires old 
 versions of Avro, it copies data much more than needed, and it's verbose and 
 complicated. (One pet peeve of mine is that old versions of Avro don't 
 support Snappy compression.)
 I rewrote AvroStorage from scratch to fix these issues. In early tests, the 
 new implementation is significantly faster, and the code is a lot simpler. 
 Rewriting AvroStorage also enabled me to implement support for Trevni (as 
 TrevniStorage).
 I'm opening this ticket to facilitate discussion while I figure out the best 
 way to contribute the changes back to Apache.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-3015) Rewrite of AvroStorage

2013-01-03 Thread Joseph Adler (JIRA)

[
https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13543455#comment-13543455
]

Joseph Adler commented on PIG-3015:
---

Hi Cheolsoo,

You're totally right; I don't check the compression properties. I know that the
avro mapred library does check those parameters
(org.apache.avro.mapred.AvroOutPutFormat), but I don't use that output format.
Fixing and testing, will follow up with a patch.

-- Joe

Rewrite of AvroStorage
--

[jira] [Commented] (PIG-3059) Global configurable minimum 'bad record' thresholds

2013-01-02 Thread Joseph Adler (JIRA)

[
https://issues.apache.org/jira/browse/PIG-3059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13542718#comment-13542718
]

Joseph Adler commented on PIG-3059:
---

Sorry to take so long to get back to this. It was a long break from work...

Thanks so much for taking this over. I like the way you've implemented this.

Global configurable minimum 'bad record' thresholds
---

Key: PIG-3059
URL: https://issues.apache.org/jira/browse/PIG-3059
Project: Pig
Issue Type: New Feature
Components: impl
Affects Versions: 0.11
Reporter: Russell Jurney
Assignee: Cheolsoo Park
Fix For: 0.12

Attachments: avro_test_files-2.tar.gz, PIG-3059-2.patch,
PIG-3059.patch

See PIG-2614.
Pig dies when one record in a LOAD of a billion records fails to parse. This
is almost certainly not the desired behavior. elephant-bird and some other
storage UDFs have minimum thresholds in terms of percent and count that must
be exceeded before a job will fail outright.
We need these limits to be configurable for Pig, globally. I've come to
realize what a major problem Pig's crashing on bad records is for new Pig
users. I believe this feature can greatly improve Pig.
An example of a config would look like:
pig.storage.bad.record.threshold=0.01
pig.storage.bad.record.min=100
A thorough discussion of this issue is available here:
http://www.quora.com/Big-Data/In-Big-Data-ETL-how-many-records-are-an-acceptable-loss

[jira] [Updated] (PIG-3015) Rewrite of AvroStorage

2012-12-20 Thread Joseph Adler (JIRA)


 [ 
https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Adler updated PIG-3015:
--

Attachment: (was: PIG-3015.patch)

 Rewrite of AvroStorage
 --

 Key: PIG-3015
 URL: https://issues.apache.org/jira/browse/PIG-3015
 Project: Pig
  Issue Type: Improvement
  Components: piggybank
Reporter: Joseph Adler
Assignee: Joseph Adler
 Attachments: PIG-3015.patch


 The current AvroStorage implementation has a lot of issues: it requires old 
 versions of Avro, it copies data much more than needed, and it's verbose and 
 complicated. (One pet peeve of mine is that old versions of Avro don't 
 support Snappy compression.)
 I rewrote AvroStorage from scratch to fix these issues. In early tests, the 
 new implementation is significantly faster, and the code is a lot simpler. 
 Rewriting AvroStorage also enabled me to implement support for Trevni (as 
 TrevniStorage).
 I'm opening this ticket to facilitate discussion while I figure out the best 
 way to contribute the changes back to Apache.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Review Request: PIG-3015 Rewrite of AvroStorage

2012-12-20 Thread Joseph Adler

/recordsOfArrays.avsc PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/recordsOfArraysOfRecords.avsc 
PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/recordsSubSchema.avsc PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/recordsSubSchemaNullable.avsc 
PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/recordsWithDoubleUnderscores.avsc 
PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/recordsWithEnums.avsc PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/recordsWithFixed.avsc PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/recordsWithMaps.avsc PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/recordsWithMapsOfRecords.avsc 
PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/recordsWithNullableUnions.avsc 
PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/recursiveRecord.avsc PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/simpleRecordsTrevni.avsc PRE-CREATION 
  test/unit-tests 7cede06 

Diff: https://reviews.apache.org/r/8104/diff/


Testing
---


Thanks,

Joseph Adler

[jira] [Updated] (PIG-3015) Rewrite of AvroStorage

2012-12-17 Thread Joseph Adler (JIRA)


 [ 
https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Adler updated PIG-3015:
--

Attachment: (was: PIG-3015.patch)

 Rewrite of AvroStorage
 --

 Key: PIG-3015
 URL: https://issues.apache.org/jira/browse/PIG-3015
 Project: Pig
  Issue Type: Improvement
  Components: piggybank
Reporter: Joseph Adler
Assignee: Joseph Adler
 Attachments: PIG-3015.patch


 The current AvroStorage implementation has a lot of issues: it requires old 
 versions of Avro, it copies data much more than needed, and it's verbose and 
 complicated. (One pet peeve of mine is that old versions of Avro don't 
 support Snappy compression.)
 I rewrote AvroStorage from scratch to fix these issues. In early tests, the 
 new implementation is significantly faster, and the code is a lot simpler. 
 Rewriting AvroStorage also enabled me to implement support for Trevni (as 
 TrevniStorage).
 I'm opening this ticket to facilitate discussion while I figure out the best 
 way to contribute the changes back to Apache.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-3015) Rewrite of AvroStorage

2012-12-17 Thread Joseph Adler (JIRA)


 [ 
https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Adler updated PIG-3015:
--

Attachment: PIG-3015.patch

 Rewrite of AvroStorage
 --

 Key: PIG-3015
 URL: https://issues.apache.org/jira/browse/PIG-3015
 Project: Pig
  Issue Type: Improvement
  Components: piggybank
Reporter: Joseph Adler
Assignee: Joseph Adler
 Attachments: PIG-3015.patch


 The current AvroStorage implementation has a lot of issues: it requires old 
 versions of Avro, it copies data much more than needed, and it's verbose and 
 complicated. (One pet peeve of mine is that old versions of Avro don't 
 support Snappy compression.)
 I rewrote AvroStorage from scratch to fix these issues. In early tests, the 
 new implementation is significantly faster, and the code is a lot simpler. 
 Rewriting AvroStorage also enabled me to implement support for Trevni (as 
 TrevniStorage).
 I'm opening this ticket to facilitate discussion while I figure out the best 
 way to contribute the changes back to Apache.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-3015) Rewrite of AvroStorage

2012-12-17 Thread Joseph Adler (JIRA)


[ 
https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13534212#comment-13534212
 ] 

Joseph Adler commented on PIG-3015:
---

My apologies; forgot to add those to the patch. Replaced the patch version.

 Rewrite of AvroStorage
 --

 Key: PIG-3015
 URL: https://issues.apache.org/jira/browse/PIG-3015
 Project: Pig
  Issue Type: Improvement
  Components: piggybank
Reporter: Joseph Adler
Assignee: Joseph Adler
 Attachments: PIG-3015.patch


 The current AvroStorage implementation has a lot of issues: it requires old 
 versions of Avro, it copies data much more than needed, and it's verbose and 
 complicated. (One pet peeve of mine is that old versions of Avro don't 
 support Snappy compression.)
 I rewrote AvroStorage from scratch to fix these issues. In early tests, the 
 new implementation is significantly faster, and the code is a lot simpler. 
 Rewriting AvroStorage also enabled me to implement support for Trevni (as 
 TrevniStorage).
 I'm opening this ticket to facilitate discussion while I figure out the best 
 way to contribute the changes back to Apache.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Review Request: PIG-3015 Rewrite of AvroStorage

2012-12-17 Thread Joseph Adler

 
  test/org/apache/pig/builtin/avro/schema/recordsSubSchema.avsc PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/recordsSubSchemaNullable.avsc 
PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/recordsWithDoubleUnderscores.avsc 
PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/recordsWithEnums.avsc PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/recordsWithFixed.avsc PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/recordsWithMaps.avsc PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/recordsWithMapsOfRecords.avsc 
PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/recordsWithNullableUnions.avsc 
PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/recursiveRecord.avsc PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/simpleRecordsTrevni.avsc PRE-CREATION 
  test/unit-tests 7cede06 

Diff: https://reviews.apache.org/r/8104/diff/


Testing
---


Thanks,

Joseph Adler

[jira] [Commented] (PIG-3015) Rewrite of AvroStorage

2012-12-11 Thread Joseph Adler (JIRA)


[ 
https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13529293#comment-13529293
 ] 

Joseph Adler commented on PIG-3015:
---

Ivy should be able to pull the jar from a maven repo. Do you need to build your 
own Avro jar from source?

 Rewrite of AvroStorage
 --

 Key: PIG-3015
 URL: https://issues.apache.org/jira/browse/PIG-3015
 Project: Pig
  Issue Type: Improvement
  Components: piggybank
Reporter: Joseph Adler
Assignee: Joseph Adler
 Attachments: PIG-3015.patch


 The current AvroStorage implementation has a lot of issues: it requires old 
 versions of Avro, it copies data much more than needed, and it's verbose and 
 complicated. (One pet peeve of mine is that old versions of Avro don't 
 support Snappy compression.)
 I rewrote AvroStorage from scratch to fix these issues. In early tests, the 
 new implementation is significantly faster, and the code is a lot simpler. 
 Rewriting AvroStorage also enabled me to implement support for Trevni (as 
 TrevniStorage).
 I'm opening this ticket to facilitate discussion while I figure out the best 
 way to contribute the changes back to Apache.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-3015) Rewrite of AvroStorage

2012-12-06 Thread Joseph Adler (JIRA)


 [ 
https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Adler updated PIG-3015:
--

Attachment: (was: PIG-3015.patch)

 Rewrite of AvroStorage
 --

 Key: PIG-3015
 URL: https://issues.apache.org/jira/browse/PIG-3015
 Project: Pig
  Issue Type: Improvement
  Components: piggybank
Reporter: Joseph Adler
Assignee: Joseph Adler
 Attachments: PIG-3015.patch


 The current AvroStorage implementation has a lot of issues: it requires old 
 versions of Avro, it copies data much more than needed, and it's verbose and 
 complicated. (One pet peeve of mine is that old versions of Avro don't 
 support Snappy compression.)
 I rewrote AvroStorage from scratch to fix these issues. In early tests, the 
 new implementation is significantly faster, and the code is a lot simpler. 
 Rewriting AvroStorage also enabled me to implement support for Trevni.
 I'm opening this ticket to facilitate discussion while I figure out the best 
 way to contribute the changes back to Apache.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-3015) Rewrite of AvroStorage

2012-12-06 Thread Joseph Adler (JIRA)


 [ 
https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Adler updated PIG-3015:
--

Attachment: PIG-3015.patch

Added test cases for TrevniStorage (and made sure the test cases all pass)

 Rewrite of AvroStorage
 --

 Key: PIG-3015
 URL: https://issues.apache.org/jira/browse/PIG-3015
 Project: Pig
  Issue Type: Improvement
  Components: piggybank
Reporter: Joseph Adler
Assignee: Joseph Adler
 Attachments: PIG-3015.patch


 The current AvroStorage implementation has a lot of issues: it requires old 
 versions of Avro, it copies data much more than needed, and it's verbose and 
 complicated. (One pet peeve of mine is that old versions of Avro don't 
 support Snappy compression.)
 I rewrote AvroStorage from scratch to fix these issues. In early tests, the 
 new implementation is significantly faster, and the code is a lot simpler. 
 Rewriting AvroStorage also enabled me to implement support for Trevni.
 I'm opening this ticket to facilitate discussion while I figure out the best 
 way to contribute the changes back to Apache.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-3015) Rewrite of AvroStorage

2012-12-06 Thread Joseph Adler (JIRA)

[
https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Joseph Adler updated PIG-3015:
--

Description:
The current AvroStorage implementation has a lot of issues: it requires old
versions of Avro, it copies data much more than needed, and it's verbose and
complicated. (One pet peeve of mine is that old versions of Avro don't support
Snappy compression.)

I rewrote AvroStorage from scratch to fix these issues. In early tests, the new
implementation is significantly faster, and the code is a lot simpler.
Rewriting AvroStorage also enabled me to implement support for Trevni (as
TrevniStorage).

I'm opening this ticket to facilitate discussion while I figure out the best
way to contribute the changes back to Apache.

was:
The current AvroStorage implementation has a lot of issues: it requires old
versions of Avro, it copies data much more than needed, and it's verbose and
complicated. (One pet peeve of mine is that old versions of Avro don't support
Snappy compression.)

I'm opening this ticket to facilitate discussion while I figure out the best
way to contribute the changes back to Apache.

Rewrite of AvroStorage
--

Key: PIG-3015
URL: https://issues.apache.org/jira/browse/PIG-3015
Project: Pig
Issue Type: Improvement
Components: piggybank
Reporter: Joseph Adler
Assignee: Joseph Adler
Attachments: PIG-3015.patch

[jira] [Commented] (PIG-3015) Rewrite of AvroStorage

2012-12-05 Thread Joseph Adler (JIRA)

[
https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13510629#comment-13510629
]

Joseph Adler commented on PIG-3015:
---

Hi Johannes,

As you probably know, the Avro specification limits the set of valid characters
in names (see http://avro.apache.org/docs/current/spec.html#Names). Names must

- start with [A-Za-z_]
- subsequently contain only [A-Za-z0-9_]

So double colons aren't allowed. PIG-2684 proposes using namespaces as the
solution. I think that's a poor choice; namespaces are often used for other
purposes. Specifically, names spaces are essential if you are writing
complicated data processing software that processes multiple types of avro
serialized objects. In my experience, the avro schema and protocol compilers
produce much better, more usable code if you use name spaces.

There are two good workarounds:

- The Pig user can rename variables in a bag before storing the bag using
AvroStorage
- The Pig user can manually specify the output schema before storing the bag
with AvroStorage

So, here's a specific suggestion:

- By default, throw an exception if the pig schema contains a name with a
double-colon and the user does not specify an output schema
- Add an option to AvroStorage to transform double colons to something else.
(Maybe double underscores? Maybe storing them in the namespace?)

What do you think?

Rewrite of AvroStorage
--

Key: PIG-3015
URL: https://issues.apache.org/jira/browse/PIG-3015
Project: Pig
Issue Type: Improvement
Components: piggybank
Reporter: Joseph Adler
Assignee: Joseph Adler
Attachments: PIG-3015.patch

The current AvroStorage implementation has a lot of issues: it requires old
versions of Avro, it copies data much more than needed, and it's verbose and
complicated. (One pet peeve of mine is that old versions of Avro don't
support Snappy compression.)
I rewrote AvroStorage from scratch to fix these issues. In early tests, the
new implementation is significantly faster, and the code is a lot simpler.
Rewriting AvroStorage also enabled me to implement support for Trevni.
I'm opening this ticket to facilitate discussion while I figure out the best
way to contribute the changes back to Apache.

[jira] [Commented] (PIG-2684) :: in field name causes AvroStorage to fail

2012-12-05 Thread Joseph Adler (JIRA)


[ 
https://issues.apache.org/jira/browse/PIG-2684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13510645#comment-13510645
 ] 

Joseph Adler commented on PIG-2684:
---

I'm addressing this right now in PIG-3015. This isn't a bug; it's just a 
mismatch between the set of names that Avro allows and the names that Pig 
allows. (As a side note, there are good reasons why only some variable names 
are allowed in Avro: limiting the characters in names allows Avro to generate 
code to process Avro objects in a number of different languages. Colons in 
variable names would make it difficult to do this.)

First, there are two workaround for this problem right now:

- The user can rename variables before storing the bag
- The user can manually specify the output schema 

Second, I don't like the idea of using namespaces for this. Namespaces are 
important for specific record types in Avro; they are translated by the 
protocol and schema compiles into package names for java classes.

To make AvroStorage easier to user, I think it would make sense to add an 
option to AvroStorage to translate names with colons in some reasonable way: 
maybe translating the double colons to double underscores.

 :: in field name causes AvroStorage to fail
 ---

 Key: PIG-2684
 URL: https://issues.apache.org/jira/browse/PIG-2684
 Project: Pig
  Issue Type: Bug
  Components: piggybank
Reporter: Fabian Alenius

 There appears to be a bug in AvroStorage which causes it to fail when there 
 are field names that contain ::
 For example, the following will fail:
 data = load 'test.txt' as (one, two);
 grp = GROUP data by (one, two);
 result = foreach grp generate FLATTEN(group); 
   
 
 store result into 'test.avro' using 
 org.apache.pig.piggybank.storage.avro.AvroStorage();
 ERROR 2999: Unexpected internal error. Illegal character in: group::one
 While the following will succeed:
 data = load 'test.txt' as (one, two);
 grp = GROUP data by (one, two);
 result = foreach grp generate FLATTEN(group) as (one,two);
  
 store result into 'test.avro' using 
 org.apache.pig.piggybank.storage.avro.AvroStorage();
 Here is a minimal test case:
 data = load 'test.txt' as (one::two, three);  
   
 
 store data into 'test.avro' using 
 org.apache.pig.piggybank.storage.avro.AvroStorage();

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-3015) Rewrite of AvroStorage

2012-12-05 Thread Joseph Adler (JIRA)


 [ 
https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Adler updated PIG-3015:
--

Attachment: PIG-3015.patch

I added support for files that don't have records, added option for dealing 
with double colons in variable names.

 Rewrite of AvroStorage
 --

 Key: PIG-3015
 URL: https://issues.apache.org/jira/browse/PIG-3015
 Project: Pig
  Issue Type: Improvement
  Components: piggybank
Reporter: Joseph Adler
Assignee: Joseph Adler
 Attachments: PIG-3015.patch


 The current AvroStorage implementation has a lot of issues: it requires old 
 versions of Avro, it copies data much more than needed, and it's verbose and 
 complicated. (One pet peeve of mine is that old versions of Avro don't 
 support Snappy compression.)
 I rewrote AvroStorage from scratch to fix these issues. In early tests, the 
 new implementation is significantly faster, and the code is a lot simpler. 
 Rewriting AvroStorage also enabled me to implement support for Trevni.
 I'm opening this ticket to facilitate discussion while I figure out the best 
 way to contribute the changes back to Apache.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Review Request: PIG-3015 Rewrite of AvroStorage

2012-12-05 Thread Joseph Adler



 On Dec. 3, 2012, 7:22 p.m., Cheolsoo Park wrote:
  Overall looks great! I haven't gone through the test cases yet, but here 
  are my comments so far.
  
  
  1) I noticed that I cannot load .avro files that are not record types. For 
  example, I tried to load a .avro file whose schema is int as follows:
  
  [cheolsoo@cheolsoo-mr1-0 pig-svn]$ java -jar avro-tools-1.5.4.jar getschema 
  foo2/test_int.avro 
  int
  
  [cheolsoo@cheolsoo-mr1-0 pig-svn]$ java -jar avro-tools-1.5.4.jar tojson 
  foo2/test_int.avro 
  1
  
  in = LOAD 'foo2/test_int.avro' USING AvroStorage('int');
  DUMP in;
  
  This gives me the following error:
  
  Caused by: java.io.IOException: avroSchemaToResourceSchema only processes 
  records
  
  Can only Avro record type be loaded? Or am I doing something wrong?
  
  
  2) TestAvroStorage needs to be more automated. To run it, I had to run the 
  following commands:
  
  ant clean compile-test
  cd ./test/org/apache/pig/builtin/avro
  python createests.py
  cd -
  ant clean test -Dtestcase=TestAvroStorage
  
  Ideally, I should be able to run a single command: ant clean 
  -Dtestcase=TestAvroStorage. Please let me know if you need help for this.
  
  
  3) python createests.py fails with the following errors. I suppose that 
  some files are missing:
  
  creating data/avro/uncompressed/testDirectoryCounts.avro
  Exception in thread main java.io.FileNotFoundException: 
  data/json/testDirectoryCounts.json (No such file or directory)
  ...
  creating evenFileNameTestDirectoryCounts.avro
  Exception in thread main java.io.FileNotFoundException: 
  data/json/evenFileNameTestDirectoryCounts.json (No such file or directory)
  ...
  
  
  4) ant test -Dtestcase=TestAvroStorage fails with the following errors. I 
  suppose that this is due to the missing files:
  
  Testcase: testLoadDirectory took 0.005 sec
  FAILED
  Testcase: testLoadGlob took 0.004 sec
  FAILED
  Testcase: testPartialLoadGlob took 0.005 sec
  FAILED
  
  
  5) Typo in the name of createests.py. It should be createtests.py.
  
  
  6) Is createTests.bash needed at all? If not, can you remove it?
  
  
  I have more comments inline:

Sounds like the python script isn't working completely correctly. I'll debug 
that script and make sure it generates all the required files.

Can I take you up on your offer to help automate that build process? I'm not 
exactly sure what to modify to automatically run the python script to create 
the test files.


 On Dec. 3, 2012, 7:22 p.m., Cheolsoo Park wrote:
  src/org/apache/pig/builtin/AvroStorage.java, lines 296-305
  https://reviews.apache.org/r/8104/diff/1/?file=191564#file191564line296
 
  This won't work in the following case. Let's say p matches two dirs, 
  and one dir is empty.
  
  p = foo*
  
  foo1
  foo2/bar.avro
  
  I would expect the schema of bar.avro is returned, but I get an 
  IOException instead.

Added proper depth first search to find the first file. (I decided to sort by 
modification date, most recent first.)


- Joseph


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8104/#review13962
---


On Nov. 17, 2012, 5:28 a.m., Joseph Adler wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/8104/
 ---
 
 (Updated Nov. 17, 2012, 5:28 a.m.)
 
 
 Review request for pig and Cheolsoo Park.
 
 
 Description
 ---
 
 The current AvroStorage implementation has a lot of issues: it requires old 
 versions of Avro, it copies data much more than needed, and it's verbose and 
 complicated. (One pet peeve of mine is that old versions of Avro don't 
 support Snappy compression.)
 
 I rewrote AvroStorage from scratch to fix these issues. In early tests, the 
 new implementation is significantly faster, and the code is a lot simpler. 
 Rewriting AvroStorage also enabled me to implement support for Trevni.
 
 This is the latest version of the patch, complete with test cases and 
 TrevniStorage. (Test cases for TrevniStorage are still missing).
 
 
 This addresses bug PIG-3015.
 https://issues.apache.org/jira/browse/PIG-3015
 
 
 Diffs
 -
 
   build.xml 7d468a0 
   ivy.xml 70e8d50 
   ivy/libraries.properties 317564f 
   src/org/apache/pig/builtin/AvroStorage.java PRE-CREATION 
   src/org/apache/pig/builtin/TrevniStorage.java PRE-CREATION 
   src/org/apache/pig/impl/util/AvroBagWrapper.java PRE-CREATION 
   src/org/apache/pig/impl/util/AvroMapWrapper.java PRE-CREATION 
   src/org/apache/pig/impl/util/AvroRecordReader.java PRE-CREATION 
   src/org/apache/pig/impl/util/AvroRecordWriter.java PRE-CREATION 
   src/org/apache/pig/impl/util/AvroStorageDataConversionUtilities.java 
 PRE-CREATION 
   src/org/apache/pig

[jira] [Commented] (PIG-3015) Rewrite of AvroStorage

2012-12-04 Thread Joseph Adler (JIRA)


[ 
https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13509992#comment-13509992
 ] 

Joseph Adler commented on PIG-3015:
---

I think that approach makes sense; each object in a file should be wrapped in a 
Tuple. Suppose that a file example.avro contained the data:

  {[1, 2, 3, 4, 5]}
  {[6, 7, 8, 9, 10]}

and had this schema: {name : IntArray, type : array, items : int}, 
and we loaded this as

  A = LOAD 'example.avro' USING AvroStorage;

The bag A would have the Pig schema A:{(IntArray:{(int)})}; it would contain 
two tuples, which would in turn each contain one bag of integers. Does that 
sound correct? If so, I'll go implement that.


 Rewrite of AvroStorage
 --

 Key: PIG-3015
 URL: https://issues.apache.org/jira/browse/PIG-3015
 Project: Pig
  Issue Type: Improvement
  Components: piggybank
Reporter: Joseph Adler
Assignee: Joseph Adler
 Attachments: PIG-3015.patch


 The current AvroStorage implementation has a lot of issues: it requires old 
 versions of Avro, it copies data much more than needed, and it's verbose and 
 complicated. (One pet peeve of mine is that old versions of Avro don't 
 support Snappy compression.)
 I rewrote AvroStorage from scratch to fix these issues. In early tests, the 
 new implementation is significantly faster, and the code is a lot simpler. 
 Rewriting AvroStorage also enabled me to implement support for Trevni.
 I'm opening this ticket to facilitate discussion while I figure out the best 
 way to contribute the changes back to Apache.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Review Request: PIG-3015 Rewrite of AvroStorage

2012-12-03 Thread Joseph Adler



 On Dec. 3, 2012, 7:22 p.m., Cheolsoo Park wrote:
  src/org/apache/pig/builtin/AvroStorage.java, lines 171-172
  https://reviews.apache.org/r/8104/diff/1/?file=191564#file191564line171
 
  Same problem as above.

Fixing this one within getAvroSchema


 On Dec. 3, 2012, 7:22 p.m., Cheolsoo Park wrote:
  src/org/apache/pig/builtin/AvroStorage.java, lines 382-388
  https://reviews.apache.org/r/8104/diff/1/?file=191564#file191564line382
 
  Is this needed?
  
  In the constructor, schema is supposed to be set. If not, there must be 
  an error. Shouldn't we throw an exception instead of re-trying to set 
  schema?
  
  Please correct me if I am wrong.

Pretty sure you're right about this one (and that this code is redundant).


 On Dec. 3, 2012, 7:22 p.m., Cheolsoo Park wrote:
  src/org/apache/pig/builtin/TrevniStorage.java, line 160
  https://reviews.apache.org/r/8104/diff/1/?file=191565#file191565line160
 
  AvroStorage accepts files that do not end .avro. Shouldn't 
  TrevniStorage do the same?

Good point... though I realize that I've defined visible avro files and 
visible trevni files methods that are probably not useful. I should probably 
just drop the methods.


 On Dec. 3, 2012, 7:22 p.m., Cheolsoo Park wrote:
  src/org/apache/pig/impl/util/AvroRecordReader.java, lines 110-118
  https://reviews.apache.org/r/8104/diff/1/?file=191568#file191568line110
 
  I can't find where -ignoreErrors is used. I guess that error handling 
  for bad files is not implemented yet?

No, I haven't implemented it yet. I suspect that the best way to implement the 
error ignoring functionality is from within Pig, and should apply to all file 
types (not just Avro)... I'll add that discussion to the right JIRA thread


 On Dec. 3, 2012, 7:22 p.m., Cheolsoo Park wrote:
  src/org/apache/pig/impl/util/AvroStorageSchemaConversionUtilities.java, 
  lines 85-91
  https://reviews.apache.org/r/8104/diff/1/?file=191571#file191571line85
 
  How about a union type that contains a single data type (e.g. 
  [string])? They're currently supported.

Good point; that's a trivial change. Adding that


 On Dec. 3, 2012, 7:22 p.m., Cheolsoo Park wrote:
  src/org/apache/pig/impl/util/AvroTupleWrapper.java, line 163
  https://reviews.apache.org/r/8104/diff/1/?file=191572#file191572line163
 
  Can you instead use log.debug(..., e)?

Just added the exception to the line logging line above


- Joseph


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/8104/#review13962
---


On Nov. 17, 2012, 5:28 a.m., Joseph Adler wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/8104/
 ---
 
 (Updated Nov. 17, 2012, 5:28 a.m.)
 
 
 Review request for pig and Cheolsoo Park.
 
 
 Description
 ---
 
 The current AvroStorage implementation has a lot of issues: it requires old 
 versions of Avro, it copies data much more than needed, and it's verbose and 
 complicated. (One pet peeve of mine is that old versions of Avro don't 
 support Snappy compression.)
 
 I rewrote AvroStorage from scratch to fix these issues. In early tests, the 
 new implementation is significantly faster, and the code is a lot simpler. 
 Rewriting AvroStorage also enabled me to implement support for Trevni.
 
 This is the latest version of the patch, complete with test cases and 
 TrevniStorage. (Test cases for TrevniStorage are still missing).
 
 
 This addresses bug PIG-3015.
 https://issues.apache.org/jira/browse/PIG-3015
 
 
 Diffs
 -
 
   build.xml 7d468a0 
   ivy.xml 70e8d50 
   ivy/libraries.properties 317564f 
   src/org/apache/pig/builtin/AvroStorage.java PRE-CREATION 
   src/org/apache/pig/builtin/TrevniStorage.java PRE-CREATION 
   src/org/apache/pig/impl/util/AvroBagWrapper.java PRE-CREATION 
   src/org/apache/pig/impl/util/AvroMapWrapper.java PRE-CREATION 
   src/org/apache/pig/impl/util/AvroRecordReader.java PRE-CREATION 
   src/org/apache/pig/impl/util/AvroRecordWriter.java PRE-CREATION 
   src/org/apache/pig/impl/util/AvroStorageDataConversionUtilities.java 
 PRE-CREATION 
   src/org/apache/pig/impl/util/AvroStorageSchemaConversionUtilities.java 
 PRE-CREATION 
   src/org/apache/pig/impl/util/AvroTupleWrapper.java PRE-CREATION 
   test/commit-tests 5081fbc 
   test/org/apache/pig/builtin/TestAvroStorage.java PRE-CREATION 
   test/org/apache/pig/builtin/avro/code/pig/directory_test.pig PRE-CREATION 
   test/org/apache/pig/builtin/avro/code/pig/identity.pig PRE-CREATION 
   test/org/apache/pig/builtin/avro/code/pig/identity_ai1_ao2.pig PRE-CREATION 
   test/org/apache/pig/builtin/avro/code/pig/identity_ao2.pig PRE-CREATION 
   test/org/apache/pig/builtin/avro/code/pig/identity_codec.pig PRE-CREATION 
   test

[jira] [Commented] (PIG-3015) Rewrite of AvroStorage

[
https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13509296#comment-13509296
]

Joseph Adler commented on PIG-3015:
---

I made most of the recommended changes (thanks for looking this over), and have
a follow up question:

I have always assumed that AvroStorage was designed to be used with Hadoop
sequence files that contained a series of records, so I implemented AvroStorage
to only work with a file in this format. Are there cases where the highest
level schema for a file will be another type? If so... what does that mean for
pig? Is there one record per file?

Here's a specific example: suppose that we have this schema:

{name : IntArray, type : array, items : int}

Suppose that we have 3 files to load, each with this schema, each containing an
array of 10 integers. Should we load this into pig as a single bag with 30
integers? A bag containing three bags (each, in turn, containing 10 integers)?
Or reject this file entirely?

Rewrite of AvroStorage
--

Key: PIG-3015
URL: https://issues.apache.org/jira/browse/PIG-3015
Project: Pig
Issue Type: Improvement
Components: piggybank
Reporter: Joseph Adler
Assignee: Joseph Adler
Attachments: PIG-3015.patch

The current AvroStorage implementation has a lot of issues: it requires old
versions of Avro, it copies data much more than needed, and it's verbose and
complicated. (One pet peeve of mine is that old versions of Avro don't
support Snappy compression.)
I rewrote AvroStorage from scratch to fix these issues. In early tests, the
new implementation is significantly faster, and the code is a lot simpler.
Rewriting AvroStorage also enabled me to implement support for Trevni.
I'm opening this ticket to facilitate discussion while I figure out the best
way to contribute the changes back to Apache.

[jira] [Commented] (PIG-2614) AvroStorage crashes on LOADING a single bad error


[ 
https://issues.apache.org/jira/browse/PIG-2614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13509325#comment-13509325
 ] 

Joseph Adler commented on PIG-2614:
---

Could I propose an alternative? 

I like this functionality, but I don't think that this should be specific to 
Avro records. I think that is should be straightforward to modify 
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader to 
implement this functionality for ALL LoadFunc types. Specifically, it should be 
possible to count the number of Exceptions thrown by the getNext method in the 
underlying load function (inside PigRecordReader.nextKeyValue).



 AvroStorage crashes on LOADING a single bad error
 -

 Key: PIG-2614
 URL: https://issues.apache.org/jira/browse/PIG-2614
 Project: Pig
  Issue Type: Bug
  Components: piggybank
Affects Versions: 0.10.0, 0.11
Reporter: Russell Jurney
Assignee: Jonathan Coveney
  Labels: avro, avrostorage, bad, book, cutting, doug, for, my, 
 pig, sadism
 Fix For: 0.11, 0.10.1

 Attachments: PIG-2614_0.patch, PIG-2614_1.patch, PIG-2614_2.patch, 
 test_avro_files.tar.gz


 AvroStorage dies when a single bad record exists, such as one with missing 
 fields.  This is very bad on 'big data,' where bad records are inevitable.  
 See discussion at 
 http://www.quora.com/Big-Data/In-Big-Data-ETL-how-many-records-are-an-acceptable-loss
  for more theory.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-3015) Rewrite of AvroStorage


 [ 
https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Adler updated PIG-3015:
--

Status: Open  (was: Patch Available)

replacing with revised patch

 Rewrite of AvroStorage
 --

 Key: PIG-3015
 URL: https://issues.apache.org/jira/browse/PIG-3015
 Project: Pig
  Issue Type: Improvement
  Components: piggybank
Reporter: Joseph Adler
Assignee: Joseph Adler

 The current AvroStorage implementation has a lot of issues: it requires old 
 versions of Avro, it copies data much more than needed, and it's verbose and 
 complicated. (One pet peeve of mine is that old versions of Avro don't 
 support Snappy compression.)
 I rewrote AvroStorage from scratch to fix these issues. In early tests, the 
 new implementation is significantly faster, and the code is a lot simpler. 
 Rewriting AvroStorage also enabled me to implement support for Trevni.
 I'm opening this ticket to facilitate discussion while I figure out the best 
 way to contribute the changes back to Apache.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-3015) Rewrite of AvroStorage


 [ 
https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Adler updated PIG-3015:
--

Attachment: (was: PIG-3015.patch)

 Rewrite of AvroStorage
 --

 Key: PIG-3015
 URL: https://issues.apache.org/jira/browse/PIG-3015
 Project: Pig
  Issue Type: Improvement
  Components: piggybank
Reporter: Joseph Adler
Assignee: Joseph Adler

 The current AvroStorage implementation has a lot of issues: it requires old 
 versions of Avro, it copies data much more than needed, and it's verbose and 
 complicated. (One pet peeve of mine is that old versions of Avro don't 
 support Snappy compression.)
 I rewrote AvroStorage from scratch to fix these issues. In early tests, the 
 new implementation is significantly faster, and the code is a lot simpler. 
 Rewriting AvroStorage also enabled me to implement support for Trevni.
 I'm opening this ticket to facilitate discussion while I figure out the best 
 way to contribute the changes back to Apache.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-3015) Rewrite of AvroStorage


 [ 
https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Adler updated PIG-3015:
--

Status: Patch Available  (was: Open)

Revised patch; reflects comments and suggestions from review board

 Rewrite of AvroStorage
 --

 Key: PIG-3015
 URL: https://issues.apache.org/jira/browse/PIG-3015
 Project: Pig
  Issue Type: Improvement
  Components: piggybank
Reporter: Joseph Adler
Assignee: Joseph Adler
 Attachments: PIG-3015.patch


 The current AvroStorage implementation has a lot of issues: it requires old 
 versions of Avro, it copies data much more than needed, and it's verbose and 
 complicated. (One pet peeve of mine is that old versions of Avro don't 
 support Snappy compression.)
 I rewrote AvroStorage from scratch to fix these issues. In early tests, the 
 new implementation is significantly faster, and the code is a lot simpler. 
 Rewriting AvroStorage also enabled me to implement support for Trevni.
 I'm opening this ticket to facilitate discussion while I figure out the best 
 way to contribute the changes back to Apache.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-3015) Rewrite of AvroStorage


 [ 
https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Adler updated PIG-3015:
--

Attachment: PIG-3015.patch

Revised patch (compiles together all changes)

 Rewrite of AvroStorage
 --

 Key: PIG-3015
 URL: https://issues.apache.org/jira/browse/PIG-3015
 Project: Pig
  Issue Type: Improvement
  Components: piggybank
Reporter: Joseph Adler
Assignee: Joseph Adler
 Attachments: PIG-3015.patch


 The current AvroStorage implementation has a lot of issues: it requires old 
 versions of Avro, it copies data much more than needed, and it's verbose and 
 complicated. (One pet peeve of mine is that old versions of Avro don't 
 support Snappy compression.)
 I rewrote AvroStorage from scratch to fix these issues. In early tests, the 
 new implementation is significantly faster, and the code is a lot simpler. 
 Rewriting AvroStorage also enabled me to implement support for Trevni.
 I'm opening this ticket to facilitate discussion while I figure out the best 
 way to contribute the changes back to Apache.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-3015) Rewrite of AvroStorage

2012-11-28 Thread Joseph Adler (JIRA)


[ 
https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506099#comment-13506099
 ] 

Joseph Adler commented on PIG-3015:
---

Hi Timothy:

I have not tried the patch with Pig 0.10, but I don't know of any reason why it 
would not work. Give it a spin and let us know what happens.

-- Joe

 Rewrite of AvroStorage
 --

 Key: PIG-3015
 URL: https://issues.apache.org/jira/browse/PIG-3015
 Project: Pig
  Issue Type: Improvement
  Components: piggybank
Reporter: Joseph Adler
Assignee: Joseph Adler
 Attachments: PIG-3015.patch


 The current AvroStorage implementation has a lot of issues: it requires old 
 versions of Avro, it copies data much more than needed, and it's verbose and 
 complicated. (One pet peeve of mine is that old versions of Avro don't 
 support Snappy compression.)
 I rewrote AvroStorage from scratch to fix these issues. In early tests, the 
 new implementation is significantly faster, and the code is a lot simpler. 
 Rewriting AvroStorage also enabled me to implement support for Trevni.
 I'm opening this ticket to facilitate discussion while I figure out the best 
 way to contribute the changes back to Apache.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-2614) AvroStorage crashes on LOADING a single bad error

2012-11-28 Thread Joseph Adler (JIRA)


[ 
https://issues.apache.org/jira/browse/PIG-2614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506101#comment-13506101
 ] 

Joseph Adler commented on PIG-2614:
---

Repeating an old question: is there any reason that this patch is only for 
Avro? I think this could work for all storage types.

 AvroStorage crashes on LOADING a single bad error
 -

 Key: PIG-2614
 URL: https://issues.apache.org/jira/browse/PIG-2614
 Project: Pig
  Issue Type: Bug
  Components: piggybank
Affects Versions: 0.10.0, 0.11
Reporter: Russell Jurney
Assignee: Jonathan Coveney
  Labels: avro, avrostorage, bad, book, cutting, doug, for, my, 
 pig, sadism
 Fix For: 0.11, 0.10.1

 Attachments: PIG-2614_0.patch, PIG-2614_1.patch


 AvroStorage dies when a single bad record exists, such as one with missing 
 fields.  This is very bad on 'big data,' where bad records are inevitable.  
 See discussion at 
 http://www.quora.com/Big-Data/In-Big-Data-ETL-how-many-records-are-an-acceptable-loss
  for more theory.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: LOAD multiple files with glob

2012-11-26 Thread Joseph Adler

It's a total rewrite, so it hasn't exactly made it in.

But yes, file globs should work correctly. That's one of the unit tests.
(All of the unit tests pass, incidentally.)


On Mon, Nov 26, 2012 at 10:23 AM, Russell Jurney
russell.jur...@gmail.comwrote:

 Is the globbing feature making it into the AvroStorage rewrite?

 Russell Jurney twitter.com/rjurney


 On Nov 26, 2012, at 7:50 AM, Bart Verwilst li...@verwilst.be wrote:

  To answer myself again, I compiled Pig 0.11 and Piggybank, and it's
 working very well now, globbing seems to be fully supported!
 
  Bart Verwilst schreef op 26.11.2012 15:33:
  To answer myself, could this be part of the solution? :
 
  https://issues.apache.org/jira/browse/PIG-2492
 
  Guess I'll have to wait for 0.11 then?
 
  Bart Verwilst schreef op 26.11.2012 14:19:
  14:16:08  centos6-hadoop-hishiru  ~ $ cat avro-test.pig
  REGISTER 'hdfs:///lib/avro-1.7.2.jar';
  REGISTER 'hdfs:///lib/json-simple-1.1.1.jar';
  REGISTER 'hdfs:///lib/piggybank.jar';
 
  DEFINE AvroStorage org.apache.pig.piggybank.storage.avro.AvroStorage();
  avro = load '/test/*' USING AvroStorage();
  describe avro;
 
  14:16:09  centos6-hadoop-hishiru  ~ $ pig avro-test.pig
  Schema for avro unknown.
 
  14:16:17  centos6-hadoop-hishiru  ~ $ vim avro-test.pig
 
  14:16:25  centos6-hadoop-hishiru  ~ $ cat avro-test.pig
  REGISTER 'hdfs:///lib/avro-1.7.2.jar';
  REGISTER 'hdfs:///lib/json-simple-1.1.1.jar';
  REGISTER 'hdfs:///lib/piggybank.jar';
 
  DEFINE AvroStorage org.apache.pig.piggybank.storage.avro.AvroStorage();
  avro = load '/test/2012-11-25.avro' USING AvroStorage();
  describe avro;
 
  14:16:30  centos6-hadoop-hishiru  ~ $ pig avro-test.pig
  avro: {id: long,timestamp: long,latitude: int,longitude: int,speed:
  int,heading: int,terminalid: int,customerid: chararray,mileage:
  int,creationtime: long,tracetype: int,traceproperties: {ARRAY_ELEM:
  (id: long,value: chararray,pkey: chararray)}}
 
  14:16:55  centos6-hadoop-hishiru  ~ $ hadoop fs -ls /test/
  Found 1 items
  -rw-r--r--   3 hdfs supergroup   63140500 2012-11-26 14:13
 /test/2012-11-25.avro
 
  Cheolsoo Park schreef op 26.11.2012 10:45:
  Hi,
 
  Invalid field projection. Projected field [tracetype] does not
 exist.
 
  The error indicates that the tracetype doesn't exist in the Pig
 schema of
  the relation avro. What AvroStorage does is to automatically
 convert Avro
  schema to Pig schema during the load. Although you have tracetype
 in your
  Avro schema, tracetype doesn't exist in the generated Pig schema for
  whatever reason.
 
  Can you please try to describe avro? You can replace group and dump
  commands with describe in your Pig script. This will show you what
 the Pig
  schema of avro is. If tracetype indeed doesn't exist, you have to
 find
  out why it doesn't. It could be because the schema of .avro files is
 not
  the same or because there is a bug in AvroStorage, etc.
 
  Maybe globbing with [] doesnt work, but wildcard works?
 
  You're right. AvroStorage internally uses Hadoop path globing, and
 Hadoop
  path globing doesn't support '[ ]'. But the above error (Projected
 field
  [tracetype] does not exist) is not because of this.
 URISyntaxException is
  what you will get because of '[ ]'.
 
  Thanks,
  Cheolsoo
 
 
 
  On Sun, Nov 25, 2012 at 10:25 AM, Bart Verwilst li...@verwilst.be
 wrote:
 
  Just tried this:
 
 
  --**--
  REGISTER 'hdfs:///lib/avro-1.7.2.jar';
  REGISTER 'hdfs:///lib/json-simple-1.1.**1.jar';
  REGISTER 'hdfs:///lib/piggybank.jar';
 
  DEFINE AvroStorage
 org.apache.pig.piggybank.**storage.avro.AvroStorage();
 
  avro = load '/data/2012/trace_ejb3/2012-**01-0*.avro' USING
 AvroStorage();
 
  groups = group avro by tracetype;
 
  dump groups;
  --**--
 
  gave me:
 
  file avro-test.pig, line 10, column 23 Invalid field projection.
  Projected field [tracetype] does not exist.
 
  Pig Stack Trace
  ---
  ERROR 1025:
  file avro-test.pig, line 10, column 23 Invalid field projection.
  Projected field [tracetype] does not exist.
 
  org.apache.pig.impl.**logicalLayer.**FrontendException: ERROR 1066:
  Unable to open iterator for alias groups
 at
 org.apache.pig.PigServer.**openIterator(PigServer.java:**862)
 at org.apache.pig.tools.grunt.**GruntParser.processDump(**
  GruntParser.java:682)
 at org.apache.pig.tools.**pigscript.parser.**
  PigScriptParser.parse(**PigScriptParser.java:303)
 at
 org.apache.pig.tools.grunt.**GruntParser.parseStopOnError(**
  GruntParser.java:189)
 at
 org.apache.pig.tools.grunt.**GruntParser.parseStopOnError(**
  GruntParser.java:165)
 at org.apache.pig.tools.grunt.**Grunt.exec(Grunt.java:84)
 at org.apache.pig.Main.run(Main.**java:555)
 at org.apache.pig.Main.main(Main.**java:111)
 at sun.reflect.**NativeMethodAccessorImpl.**invoke0(Native
 Method)
 at

[jira] [Commented] (PIG-2614) AvroStorage crashes on LOADING a single bad error

2012-11-20 Thread Joseph Adler (JIRA)


[ 
https://issues.apache.org/jira/browse/PIG-2614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13501339#comment-13501339
 ] 

Joseph Adler commented on PIG-2614:
---

Just taking a look at this patch now

If I'm reading the code correctly, it should not be necesarry for the author of 
a UDF (specifically a LoadFunc) to do anything special to take advantage of 
this functionality. Is that correct?

 AvroStorage crashes on LOADING a single bad error
 -

 Key: PIG-2614
 URL: https://issues.apache.org/jira/browse/PIG-2614
 Project: Pig
  Issue Type: Bug
  Components: piggybank
Affects Versions: 0.10.0, 0.11
Reporter: Russell Jurney
Assignee: Jonathan Coveney
  Labels: avro, avrostorage, bad, book, cutting, doug, for, my, 
 pig, sadism
 Fix For: 0.11, 0.10.1

 Attachments: PIG-2614_0.patch, PIG-2614_1.patch


 AvroStorage dies when a single bad record exists, such as one with missing 
 fields.  This is very bad on 'big data,' where bad records are inevitable.  
 See discussion at 
 http://www.quora.com/Big-Data/In-Big-Data-ETL-how-many-records-are-an-acceptable-loss
  for more theory.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (PIG-3015) Rewrite of AvroStorage

2012-11-20 Thread Joseph Adler (JIRA)

[
https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13501340#comment-13501340
]

Joseph Adler commented on PIG-3015:
---

I just took at look at PIG-2614. It looks like the PIG-2614 patch will be
compatible with this patch; PIG-2614 simply counts errors as values are read
from a LoadFunc. Am I missing something? I'd be happy to drop the option to
ignore bad records; I think that would make the options for this function
cleaner and easier to understand.

Rewrite of AvroStorage
--

Key: PIG-3015
URL: https://issues.apache.org/jira/browse/PIG-3015
Project: Pig
Issue Type: Improvement
Components: piggybank
Reporter: Joseph Adler
Assignee: Joseph Adler
Attachments: PIG-3015.patch

The current AvroStorage implementation has a lot of issues: it requires old
versions of Avro, it copies data much more than needed, and it's verbose and
complicated. (One pet peeve of mine is that old versions of Avro don't
support Snappy compression.)
I rewrote AvroStorage from scratch to fix these issues. In early tests, the
new implementation is significantly faster, and the code is a lot simpler.
Rewriting AvroStorage also enabled me to implement support for Trevni.
I'm opening this ticket to facilitate discussion while I figure out the best
way to contribute the changes back to Apache.

Review Request: PIG-3015 Rewrite of AvroStorage

2012-11-16 Thread Joseph Adler

-CREATION 
  test/org/apache/pig/builtin/avro/data/json/recursiveRecord.json PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/recordWithRepeatedSubRecords.avsc 
PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/records.avsc PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/recordsAsOutputByPig.avsc 
PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/recordsOfArrays.avsc PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/recordsOfArraysOfRecords.avsc 
PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/recordsSubSchema.avsc PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/recordsSubSchemaNullable.avsc 
PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/recordsWithEnums.avsc PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/recordsWithFixed.avsc PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/recordsWithMaps.avsc PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/recordsWithMapsOfRecords.avsc 
PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/recordsWithNullableUnions.avsc 
PRE-CREATION 
  test/org/apache/pig/builtin/avro/schema/recursiveRecord.avsc PRE-CREATION 
  test/unit-tests 0f18a0e 

Diff: https://reviews.apache.org/r/8104/diff/


Testing
---


Thanks,

Joseph Adler

[jira] [Commented] (PIG-3015) Rewrite of AvroStorage

2012-11-16 Thread Joseph Adler (JIRA)


[ 
https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13499348#comment-13499348
 ] 

Joseph Adler commented on PIG-3015:
---

I have made all the changes that you suggested (including rewriting the script 
that builds test cases in Python) and have uploaded the new version to the RB: 
https://reviews.apache.org/r/8104/

 Rewrite of AvroStorage
 --

 Key: PIG-3015
 URL: https://issues.apache.org/jira/browse/PIG-3015
 Project: Pig
  Issue Type: Improvement
  Components: piggybank
Reporter: Joseph Adler
Assignee: Joseph Adler
 Attachments: PIG-3015.patch


 The current AvroStorage implementation has a lot of issues: it requires old 
 versions of Avro, it copies data much more than needed, and it's verbose and 
 complicated. (One pet peeve of mine is that old versions of Avro don't 
 support Snappy compression.)
 I rewrote AvroStorage from scratch to fix these issues. In early tests, the 
 new implementation is significantly faster, and the code is a lot simpler. 
 Rewriting AvroStorage also enabled me to implement support for Trevni.
 I'm opening this ticket to facilitate discussion while I figure out the best 
 way to contribute the changes back to Apache.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Work started] (PIG-3015) Rewrite of AvroStorage


 [ 
https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on PIG-3015 started by Joseph Adler.

 Rewrite of AvroStorage
 --

 Key: PIG-3015
 URL: https://issues.apache.org/jira/browse/PIG-3015
 Project: Pig
  Issue Type: Improvement
  Components: piggybank
Reporter: Joseph Adler
Assignee: Joseph Adler

 The current AvroStorage implementation has a lot of issues: it requires old 
 versions of Avro, it copies data much more than needed, and it's verbose and 
 complicated. (One pet peeve of mine is that old versions of Avro don't 
 support Snappy compression.)
 I rewrote AvroStorage from scratch to fix these issues. In early tests, the 
 new implementation is significantly faster, and the code is a lot simpler. 
 Rewriting AvroStorage also enabled me to implement support for Trevni.
 I'm opening this ticket to facilitate discussion while I figure out the best 
 way to contribute the changes back to Apache.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-3015) Rewrite of AvroStorage


 [ 
https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Adler updated PIG-3015:
--

Status: Patch Available  (was: In Progress)

Here is a patch with a working implementation (plus new unit tests and a bash 
script to generate the test data files; just run the bash script in the 
test/org/apache/pig/builtin/avro directory to generate all the avro files 
needed for testing)

 Rewrite of AvroStorage
 --

 Key: PIG-3015
 URL: https://issues.apache.org/jira/browse/PIG-3015
 Project: Pig
  Issue Type: Improvement
  Components: piggybank
Reporter: Joseph Adler
Assignee: Joseph Adler
 Attachments: PIG-3015.patch


 The current AvroStorage implementation has a lot of issues: it requires old 
 versions of Avro, it copies data much more than needed, and it's verbose and 
 complicated. (One pet peeve of mine is that old versions of Avro don't 
 support Snappy compression.)
 I rewrote AvroStorage from scratch to fix these issues. In early tests, the 
 new implementation is significantly faster, and the code is a lot simpler. 
 Rewriting AvroStorage also enabled me to implement support for Trevni.
 I'm opening this ticket to facilitate discussion while I figure out the best 
 way to contribute the changes back to Apache.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-3015) Rewrite of AvroStorage


 [ 
https://issues.apache.org/jira/browse/PIG-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joseph Adler updated PIG-3015:
--

Patch Info: Patch Available

 Rewrite of AvroStorage
 --

 Key: PIG-3015
 URL: https://issues.apache.org/jira/browse/PIG-3015
 Project: Pig
  Issue Type: Improvement
  Components: piggybank
Reporter: Joseph Adler
Assignee: Joseph Adler
 Attachments: PIG-3015.patch


 The current AvroStorage implementation has a lot of issues: it requires old 
 versions of Avro, it copies data much more than needed, and it's verbose and 
 complicated. (One pet peeve of mine is that old versions of Avro don't 
 support Snappy compression.)
 I rewrote AvroStorage from scratch to fix these issues. In early tests, the 
 new implementation is significantly faster, and the code is a lot simpler. 
 Rewriting AvroStorage also enabled me to implement support for Trevni.
 I'm opening this ticket to facilitate discussion while I figure out the best 
 way to contribute the changes back to Apache.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (PIG-3015) Rewrite of AvroStorage