[jira] [Commented] (AVRO-1068) Avro Java does not build on recent Eclipse versions with m2eclipse

2012-04-20 Thread Scott Carey (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13258286#comment-13258286
 ] 

Scott Carey commented on AVRO-1068:
---

+1 looks good.


 Avro Java does not build on recent Eclipse versions with m2eclipse
 --

 Key: AVRO-1068
 URL: https://issues.apache.org/jira/browse/AVRO-1068
 Project: Avro
  Issue Type: Improvement
  Components: java
Affects Versions: 1.7.0
Reporter: Thiruvalluvan M. G.
Assignee: Thiruvalluvan M. G.
 Fix For: 1.7.0

 Attachments: AVRO-1068.patch


 It is so because, in the recent versions of m2eclipse require manual mapping 
 of non-standard maven lifecycle tasks. The forthcoming fix addresses that 
 problem.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-1066) ArrayIndexOutOfBoundsException in ParsingEncoder when trying to use a json encoder to serialize a deep object graph

2012-04-19 Thread Thiruvalluvan M. G. (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13257601#comment-13257601
 ] 

Thiruvalluvan M. G. commented on AVRO-1066:
---

Good catch. Daniel Lord's fix would work.

The problem has never been seen because we have not had nested array/maps with 
10 levels or mores.

 ArrayIndexOutOfBoundsException in ParsingEncoder when trying to use a json 
 encoder to serialize a deep object graph
 ---

 Key: AVRO-1066
 URL: https://issues.apache.org/jira/browse/AVRO-1066
 Project: Avro
  Issue Type: Bug
Affects Versions: 1.6.2
Reporter: Daniel Lord

 I have an avro record that can have an array of children records of the same 
 type.  If this structure gets to be too deep then I continually get an 
 ArrayOutOfBoundsException in ParsingEncoder when trying to use a json 
 encoder.  This works fine when using a binary encoder.  My schema looks 
 something like this: 
 {
 name : MyRecord,
 type : record,
 fields : [
 {
 name : fooField,
 type : int
 },
 {
 name : childRecords,
 type : [ null, { type : array, items : MyRecord } ]
 }
 ]
 }
 The code I'm using to capture a JSON string for debugging looks like this: 
 ByteArrayOutputStream os = null;
 try {
 os = new ByteArrayOutputStream(2048);
 final Encoder jsonEncoder = 
 EncoderFactory.get().jsonEncoder(MyRecord.SCHEMA$, os);
 final DatumWriterMyRecord datumWriter = new 
 SpecificDatumWriterMyRecord(MyRecord.class);
 datumWriter.write(record, jsonEncoder);
 jsonEncoder.flush();
 return new String(os.toByteArray(), Charset.defaultCharset());
 } catch (IOException e) {
 return null;
 } finally {
 if (os != null) {
 try {
 os.close();
 } catch (IOException e) {
 }
 }
 }
 The error I get looks like this: 
 java.lang.ArrayIndexOutOfBoundsException: 10
   at org.apache.avro.io.ParsingEncoder.push(ParsingEncoder.java:55)
   at org.apache.avro.io.JsonEncoder.writeArrayStart(JsonEncoder.java:231)
   at 
 org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:125)
   at 
 org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:67)
   at 
 org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:72)
   at 
 org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:105)
   at 
 org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
   at 
 org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:105)
   at 
 org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
   at 
 org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:129)
   at 
 org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:67)
   at 
 org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:72)
   at 
 org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:105)
   at 
 org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
   at 
 org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:129)
   at 
 org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:67)
   at 
 org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:72)
   at 
 org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:105)
   at 
 org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
   at 
 org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:129)
   at 
 org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:67)
   at 
 org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:72)
   at 
 org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:105)
   at 
 org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
   at 
 org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:129)
   at 
 org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:67)
   at 
 org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:72)
   at 
 org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:105)
   at 
 org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65

[jira] [Commented] (AVRO-1064) Encoder.writeString throws NullPointerException when the charSequence is null

2012-04-19 Thread Thiruvalluvan M. G. (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13257609#comment-13257609
 ] 

Thiruvalluvan M. G. commented on AVRO-1064:
---

The patch makes Encoder treat {{null}} for Avro {{String}} as empty. There is 
no such requirement in Encoder's specification. This would make the the spec 
somewhat non-uniform. For example, if we assign the interpretation for {{null}} 
for Avro {{String}}, we should also add interpretation for {{null}} for Avro 
{{bytes}}, {{fixed}} and so on.

I think ensuring the the bean doesn't have {{null}} member or annotating the 
property {{@Nullable}} is a better way to handle this.

 Encoder.writeString throws NullPointerException when the charSequence is null
 -

 Key: AVRO-1064
 URL: https://issues.apache.org/jira/browse/AVRO-1064
 Project: Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.6.3
Reporter: Kevin Zhao
 Fix For: 1.7.0

 Attachments: Encoder.java.patch


 org.apache.avro.io.Encoder.writeString(CharSequence charSequence) throws 
 NullPointerException when dealing with a javabean which has a null value in 
 one of its fields, like the following one:
 Person person = new Person();
 person.setName(Kevin);
 person.setId(null);

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-806) add a column-major codec for data files

2012-04-18 Thread Doug Cutting (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13257077#comment-13257077
 ] 

Doug Cutting commented on AVRO-806:
---

I've implemented a new column-file format at:

  https://github.com/cutting/trevni

This supports writing Avro data.

If folks find this useful then I intend to contribute it to Apache.


 add a column-major codec for data files
 ---

 Key: AVRO-806
 URL: https://issues.apache.org/jira/browse/AVRO-806
 Project: Avro
  Issue Type: New Feature
  Components: java, spec
Reporter: Doug Cutting
Assignee: Doug Cutting
 Fix For: 1.7.0

 Attachments: AVRO-806-v2.patch, AVRO-806.patch, avro-file-columnar.pdf


 Define a codec that, when a data file's schema is a record schema, writes 
 blocks within the file in column-major order.  This would permit better 
 compression and also permit efficient skipping of fields that are not of 
 interest.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-1066) ArrayIndexOutOfBoundsException in ParsingEncoder when trying to use a json encoder to serialize a deep object graph

2012-04-18 Thread Daniel Lord (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13257162#comment-13257162
 ] 

Daniel Lord commented on AVRO-1066:
---

There was a post on the mailing list about this a few weeks ago.  I just tested 
the suggested fix there and it works for me.  Changing ParsingEncoder's push 
method to this worked for me.

  /** Push a new collection on to the stack. */
  protected final void push() {
if (++pos = counts.length) {
  counts = Arrays.copyOf(counts, pos + 10);
}
counts[pos] = 0;
  }

 ArrayIndexOutOfBoundsException in ParsingEncoder when trying to use a json 
 encoder to serialize a deep object graph
 ---

 Key: AVRO-1066
 URL: https://issues.apache.org/jira/browse/AVRO-1066
 Project: Avro
  Issue Type: Bug
Affects Versions: 1.6.2
Reporter: Daniel Lord

 I have an avro record that can have an array of children records of the same 
 type.  If this structure gets to be too deep then I continually get an 
 ArrayOutOfBoundsException in ParsingEncoder when trying to use a json 
 encoder.  This works fine when using a binary encoder.  My schema looks 
 something like this: 
 {
 name : MyRecord,
 type : record,
 fields : [
 {
 name : fooField,
 type : int
 },
 {
 name : childRecords,
 type : [ null, { type : array, items : MyRecord } ]
 }
 ]
 }
 The code I'm using to capture a JSON string for debugging looks like this: 
 ByteArrayOutputStream os = null;
 try {
 os = new ByteArrayOutputStream(2048);
 final Encoder jsonEncoder = 
 EncoderFactory.get().jsonEncoder(MyRecord.SCHEMA$, os);
 final DatumWriterMyRecord datumWriter = new 
 SpecificDatumWriterMyRecord(MyRecord.class);
 datumWriter.write(record, jsonEncoder);
 jsonEncoder.flush();
 return new String(os.toByteArray(), Charset.defaultCharset());
 } catch (IOException e) {
 return null;
 } finally {
 if (os != null) {
 try {
 os.close();
 } catch (IOException e) {
 }
 }
 }
 The error I get looks like this: 
 java.lang.ArrayIndexOutOfBoundsException: 10
   at org.apache.avro.io.ParsingEncoder.push(ParsingEncoder.java:55)
   at org.apache.avro.io.JsonEncoder.writeArrayStart(JsonEncoder.java:231)
   at 
 org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:125)
   at 
 org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:67)
   at 
 org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:72)
   at 
 org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:105)
   at 
 org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
   at 
 org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:105)
   at 
 org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
   at 
 org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:129)
   at 
 org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:67)
   at 
 org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:72)
   at 
 org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:105)
   at 
 org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
   at 
 org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:129)
   at 
 org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:67)
   at 
 org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:72)
   at 
 org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:105)
   at 
 org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
   at 
 org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:129)
   at 
 org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:67)
   at 
 org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:72)
   at 
 org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:105)
   at 
 org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
   at 
 org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:129)
   at 
 org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:67)
   at 
 org.apache.avro.generic.GenericDatumWriter.write

[jira] [Commented] (AVRO-1062) DataFileWriter uses java.rmi.server.UID to generate unique id,which causes avro compilation problem on Android Delvik

2012-04-17 Thread Kevin Zhao (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13255419#comment-13255419
 ] 

Kevin Zhao commented on AVRO-1062:
--

use java.util.UUID as the alternative of java.rmi.server.UID to generate unique 
id

 DataFileWriter uses java.rmi.server.UID to generate unique id,which causes 
 avro compilation problem on Android Delvik
 -

 Key: AVRO-1062
 URL: https://issues.apache.org/jira/browse/AVRO-1062
 Project: Avro
  Issue Type: Improvement
  Components: java
Affects Versions: 1.6.3
 Environment: Android 2.3.3-API level 10
Reporter: Kevin Zhao
 Fix For: 1.7.0

 Attachments: AVRO-1062.patch


 Because Android Delvik does not have java.rmi.* packages and 
 org.apache.avro.file.DataFileWriter has a reference of 
 java.rmi.server.UID,avro fails in compilation process on Android.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-1062) DataFileWriter uses java.rmi.server.UID to generate unique id,which causes avro compilation problem on Android Delvik

2012-04-17 Thread Kevin Zhao (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13255420#comment-13255420
 ] 

Kevin Zhao commented on AVRO-1062:
--

request for code reviewing~ :)

 DataFileWriter uses java.rmi.server.UID to generate unique id,which causes 
 avro compilation problem on Android Delvik
 -

 Key: AVRO-1062
 URL: https://issues.apache.org/jira/browse/AVRO-1062
 Project: Avro
  Issue Type: Improvement
  Components: java
Affects Versions: 1.6.3
 Environment: Android 2.3.3-API level 10
Reporter: Kevin Zhao
 Fix For: 1.7.0

 Attachments: AVRO-1062.patch


 Because Android Delvik does not have java.rmi.* packages and 
 org.apache.avro.file.DataFileWriter has a reference of 
 java.rmi.server.UID,avro fails in compilation process on Android.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-1056) ReflectData thinks the instance of a java.util.Map as a record type

2012-04-17 Thread Kevin Zhao (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13255421#comment-13255421
 ] 

Kevin Zhao commented on AVRO-1056:
--

request for code reviewing~ 

 ReflectData thinks the instance of a java.util.Map as a record type
 ---

 Key: AVRO-1056
 URL: https://issues.apache.org/jira/browse/AVRO-1056
 Project: Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.6.3
 Environment: MacOSX 10.7.2
 Eclipse 3.7
 JDK1.6
Reporter: Kevin Zhao
 Fix For: 1.6.3

 Attachments: ReflectData.java.patch


 the avro's java api org.apache.avro.reflect.ReflectData.isRecord() should add 
 the following code.
   if (datum instanceof Map) return false;

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-1063) Ruby client should use multi_json rather than being locked down to yajl

2012-04-17 Thread Paul Dlug (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13255516#comment-13255516
 ] 

Paul Dlug commented on AVRO-1063:
-

I implemented this and sent a pull request via Github: 
https://github.com/apache/avro/pull/8 All tests pass after swapping yajl for 
multi_json.

 Ruby client should use multi_json rather than being locked down to yajl
 ---

 Key: AVRO-1063
 URL: https://issues.apache.org/jira/browse/AVRO-1063
 Project: Avro
  Issue Type: Improvement
  Components: ruby
Reporter: Paul Dlug
Priority: Minor

 The avro ruby client uses yajl for JSON serialization which is just one of 
 many suitable JSON implementations for ruby. The multi_json gem provides a 
 wrapper for JSON serialization selecting the fastest library available (Oj is 
 now even faster than Yajl) and falling back to a pure ruby implementation 
 bundled with multi_json. Requiring yajl also precludes the ruby gem from 
 being used under jruby since it requires a C extension.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-1062) DataFileWriter uses java.rmi.server.UID to generate unique id,which causes avro compilation problem on Android Delvik

2012-04-17 Thread Doug Cutting (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13255802#comment-13255802
 ] 

Doug Cutting commented on AVRO-1062:


Looks good to me.  We could perhaps improve sync marker generation, but this 
issue should make it no worse and permits Avro to run on Android.  I'll commit 
this soon unless someone objects.

 DataFileWriter uses java.rmi.server.UID to generate unique id,which causes 
 avro compilation problem on Android Delvik
 -

 Key: AVRO-1062
 URL: https://issues.apache.org/jira/browse/AVRO-1062
 Project: Avro
  Issue Type: Improvement
  Components: java
Affects Versions: 1.6.3
 Environment: Android 2.3.3-API level 10
Reporter: Kevin Zhao
  Labels: patch
 Fix For: 1.7.0

 Attachments: AVRO-1062.patch


 Because Android Delvik does not have java.rmi.* packages and 
 org.apache.avro.file.DataFileWriter has a reference of 
 java.rmi.server.UID,avro fails in compilation process on Android.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-1064) Encoder.writeString throws NullPointerException when the charSequence is null

2012-04-17 Thread Kevin Zhao (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13256188#comment-13256188
 ] 

Kevin Zhao commented on AVRO-1064:
--

request for code reviewing~

 Encoder.writeString throws NullPointerException when the charSequence is null
 -

 Key: AVRO-1064
 URL: https://issues.apache.org/jira/browse/AVRO-1064
 Project: Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.6.3
Reporter: Kevin Zhao
 Fix For: 1.7.0

 Attachments: Encoder.java.patch


 org.apache.avro.io.Encoder.writeString(CharSequence charSequence) throws 
 NullPointerException when dealing with a javabean which has a null value in 
 one of its fields, like the following one:
 Person person = new Person();
 person.setName(Kevin);
 person.setId(null);

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-565) Investigate Python encoding error

2012-04-16 Thread Miki Tebeka (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13255064#comment-13255064
 ] 

Miki Tebeka commented on AVRO-565:
--

IMO it's not the library role. To quote [The Zen of 
Python|http://www.python.org/dev/peps/pep-0020/]: In the face of ambiguity, 
refuse the temptation to guess.

You can do the same in your code (try to encode, if fails convert to utf8)

 Investigate Python encoding error
 -

 Key: AVRO-565
 URL: https://issues.apache.org/jira/browse/AVRO-565
 Project: Avro
  Issue Type: Bug
  Components: python
Reporter: Jeff Hammerbacher
 Attachments: reproduce_avro_565.py


 Tyler B is seeing the following encoding error: 
 http://avro.pastebin.com/b4HSYjCz.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-1061) Add sync interval option to Avro commandline tools

2012-04-16 Thread Doug Cutting (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13255174#comment-13255174
 ] 

Doug Cutting commented on AVRO-1061:


This looks good to me.

We should probably add a test to TestDataFileTools that checks this.

Should we also increase the default sync interval to 64k?




 Add sync interval option to Avro commandline tools
 --

 Key: AVRO-1061
 URL: https://issues.apache.org/jira/browse/AVRO-1061
 Project: Avro
  Issue Type: Improvement
  Components: java
Affects Versions: 1.7.0
Reporter: Ari Pollak
Priority: Trivial
 Attachments: AVRO-1061.patch


 It would be nice to expose the sync interval to the avro commandline writer 
 tools, since I've seen a 20%+ decrease in file size using deflate compression 
 and a 64K+ sync interval instead of the default of 16K.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-1057) Java builder API fails when default value does not match the first type in a union

2012-04-13 Thread Doug Cutting (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13253666#comment-13253666
 ] 

Doug Cutting commented on AVRO-1057:


I think this is correct.  The specification says, Default values for union 
fields correspond to the first schema in the union  
(http://avro.apache.org/docs/current/spec.html).  So the default value for 
{boolean null} must be a boolean.

 Java builder API fails when default value does not match the first type in a 
 union
 --

 Key: AVRO-1057
 URL: https://issues.apache.org/jira/browse/AVRO-1057
 Project: Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.6.3
Reporter: Christophe Taton
Priority: Minor

 The following definition works fine with the builder:
 record Rec {
   union { boolean, null } field = false;
 }
 but this one fails:
 record Rec {
   union { boolean, null } field = null;
 }
 Rec.newBuilder().build() fails with this error:
 org.apache.avro.AvroRuntimeException: org.apache.avro.AvroTypeException: 
 Non-boolean default for boolean: null
   at Rec$Builder.build
 Caused by: org.apache.avro.AvroTypeException: Non-boolean default for 
 boolean: null
   at 
 org.apache.avro.io.parsing.ResolvingGrammarGenerator.encode(ResolvingGrammarGenerator.java:393)
   at 
 org.apache.avro.io.parsing.ResolvingGrammarGenerator.encode(ResolvingGrammarGenerator.java:350)
   at 
 org.apache.avro.data.RecordBuilderBase.defaultValue(RecordBuilderBase.java:178)
   at Rec$Builder.build

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-1057) Java builder API fails when default value does not match the first type in a union

2012-04-13 Thread Christophe Taton (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13253675#comment-13253675
 ] 

Christophe Taton commented on AVRO-1057:


Ah, you are right, I missed that bit.
I got confused as the IDL compiler and JSON schema currently allow bogus 
declarations.
Could the IDL compiler and the schema parser check this more aggressively?

 Java builder API fails when default value does not match the first type in a 
 union
 --

 Key: AVRO-1057
 URL: https://issues.apache.org/jira/browse/AVRO-1057
 Project: Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.6.3
Reporter: Christophe Taton
Priority: Minor

 The following definition works fine with the builder:
 record Rec {
   union { boolean, null } field = false;
 }
 but this one fails:
 record Rec {
   union { boolean, null } field = null;
 }
 Rec.newBuilder().build() fails with this error:
 org.apache.avro.AvroRuntimeException: org.apache.avro.AvroTypeException: 
 Non-boolean default for boolean: null
   at Rec$Builder.build
 Caused by: org.apache.avro.AvroTypeException: Non-boolean default for 
 boolean: null
   at 
 org.apache.avro.io.parsing.ResolvingGrammarGenerator.encode(ResolvingGrammarGenerator.java:393)
   at 
 org.apache.avro.io.parsing.ResolvingGrammarGenerator.encode(ResolvingGrammarGenerator.java:350)
   at 
 org.apache.avro.data.RecordBuilderBase.defaultValue(RecordBuilderBase.java:178)
   at Rec$Builder.build

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-1057) Java builder API fails when default value does not match the first type in a union

2012-04-13 Thread Doug Cutting (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13253692#comment-13253692
 ] 

Doug Cutting commented on AVRO-1057:


Yes, this could be checked more aggressively.  The existing logic for this is 
in RecordBuilderBase#defaultValue().  It writes the default JSON parse tree, 
then reads it into the appropriate Avro data structure (generic, specific, 
reflect).  The writing does the error checking, and that's in 
ResolvingGrammarGenerator#encode.  So the compiler and/or parser could call 
that.  We probably don't want to check it unconditionally in the schema parser, 
as schema parsing is performance sensitive.

 Java builder API fails when default value does not match the first type in a 
 union
 --

 Key: AVRO-1057
 URL: https://issues.apache.org/jira/browse/AVRO-1057
 Project: Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.6.3
Reporter: Christophe Taton
Priority: Minor

 The following definition works fine with the builder:
 record Rec {
   union { boolean, null } field = false;
 }
 but this one fails:
 record Rec {
   union { boolean, null } field = null;
 }
 Rec.newBuilder().build() fails with this error:
 org.apache.avro.AvroRuntimeException: org.apache.avro.AvroTypeException: 
 Non-boolean default for boolean: null
   at Rec$Builder.build
 Caused by: org.apache.avro.AvroTypeException: Non-boolean default for 
 boolean: null
   at 
 org.apache.avro.io.parsing.ResolvingGrammarGenerator.encode(ResolvingGrammarGenerator.java:393)
   at 
 org.apache.avro.io.parsing.ResolvingGrammarGenerator.encode(ResolvingGrammarGenerator.java:350)
   at 
 org.apache.avro.data.RecordBuilderBase.defaultValue(RecordBuilderBase.java:178)
   at Rec$Builder.build

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-593) Java: add support for Hadoop's new mapreduce APIs

2012-04-13 Thread Garrett Wu (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13253806#comment-13253806
 ] 

Garrett Wu commented on AVRO-593:
-

Thank you, Doug, for doing the hard part -- integration!

 Java: add support for Hadoop's new mapreduce APIs
 -

 Key: AVRO-593
 URL: https://issues.apache.org/jira/browse/AVRO-593
 Project: Avro
  Issue Type: New Feature
  Components: java
Affects Versions: 1.3.2, 1.3.3
 Environment: Avro 1.3.3, Hadoop 0.20.2
Reporter: Steve Severance
Assignee: Garrett Wu
 Fix For: 1.7.0

 Attachments: AVRO-593-test.tgz, AVRO-593.patch, AVRO-593.patch, 
 AVRO-593.patch, AVRO-593.patch


 Avro should work with Hadoop's newer org.apache.hadoop.mapreduce API, in 
 addition to the older org.apache.hadoop.mapred API.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-565) Investigate Python encoding error

2012-04-13 Thread Miki Tebeka (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13253964#comment-13253964
 ] 

Miki Tebeka commented on AVRO-565:
--

Russel, the problem is that you pass a *str* with unicode characters. Python 
has no way of knowing the encoding and the default if 'ascii'. If you prepend a 
'u' to make the body unicode:
{code}
email_hash = {'body': uVerit\xc3\xa1\r\nEstat\xc3\xadstica\r\n}
{code}
This will work since now it's a unicode string and it can be encoded.

As a side note - these kind of things were one of the reasons for Python 3. I 
also recommend viewing [this 
video|http://pyvideo.org/video/948/pragmatic-unicode-or-how-do-i-stop-the-pain] 
which personally helped me.

I'll give you time to respond before closing this.

 Investigate Python encoding error
 -

 Key: AVRO-565
 URL: https://issues.apache.org/jira/browse/AVRO-565
 Project: Avro
  Issue Type: Bug
  Components: python
Reporter: Jeff Hammerbacher
 Attachments: reproduce_avro_565.py


 Tyler B is seeing the following encoding error: 
 http://avro.pastebin.com/b4HSYjCz.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-565) Investigate Python encoding error

2012-04-13 Thread Russell Jurney (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13253967#comment-13253967
 ] 

Russell Jurney commented on AVRO-565:
-

Pardon me, I mean it would be neat if Avro worked with ugly strings :)

 Investigate Python encoding error
 -

 Key: AVRO-565
 URL: https://issues.apache.org/jira/browse/AVRO-565
 Project: Avro
  Issue Type: Bug
  Components: python
Reporter: Jeff Hammerbacher
 Attachments: reproduce_avro_565.py


 Tyler B is seeing the following encoding error: 
 http://avro.pastebin.com/b4HSYjCz.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-565) Investigate Python encoding error

2012-04-13 Thread Russell Jurney (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13253966#comment-13253966
 ] 

Russell Jurney commented on AVRO-565:
-

I create unicode objects when I know the codec, otherwise the decode crashes.

codeif charset:
  subject = subject.decode(charset)/code

It would be neat if Python worked with ugly strings.

 Investigate Python encoding error
 -

 Key: AVRO-565
 URL: https://issues.apache.org/jira/browse/AVRO-565
 Project: Avro
  Issue Type: Bug
  Components: python
Reporter: Jeff Hammerbacher
 Attachments: reproduce_avro_565.py


 Tyler B is seeing the following encoding error: 
 http://avro.pastebin.com/b4HSYjCz.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-593) Avro mapreduce apis incompatible with hadoop 0.20.2

2012-04-11 Thread Scott Carey (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13252047#comment-13252047
 ] 

Scott Carey commented on AVRO-593:
--

No objections, but I have not had time for a deep review and won't for more 
than a week.  I don't think we need to hold this up for my full review, I can 
always create another ticket for later changes.

 Avro mapreduce apis incompatible with hadoop 0.20.2
 ---

 Key: AVRO-593
 URL: https://issues.apache.org/jira/browse/AVRO-593
 Project: Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.3.2, 1.3.3
 Environment: Avro 1.3.3, Hadoop 0.20.2
Reporter: Steve Severance
Assignee: Garrett Wu
 Attachments: AVRO-593-test.tgz, AVRO-593.patch, AVRO-593.patch, 
 AVRO-593.patch, AVRO-593.patch


 The avro api's for hadoop use the hadoop mapreduce api that has been 
 deprecated. A new avro mapreduce api should be implemented for hadoop 0.20 
 and higher.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-1055) Race condition in Java fingerprinting code

2012-04-09 Thread Doug Cutting (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250094#comment-13250094
 ] 

Doug Cutting commented on AVRO-1055:


+1 Looks good to me.  You might add a comment saying something like, Nested 
class used so that table is not built unless it's used.

 Race condition in Java fingerprinting code
 --

 Key: AVRO-1055
 URL: https://issues.apache.org/jira/browse/AVRO-1055
 Project: Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.7.0
Reporter: Thiruvalluvan M. G.
Assignee: Thiruvalluvan M. G.
Priority: Minor
 Attachments: AVRO-1055.patch


 There is a subtle race condition. If the fpTable64 is not yet initialized and 
 two thread try to compute FP for two schemas (or the same schema) at the same 
 time, one thread will start initializing the table while the other can start 
 using the partially initialized table giving wrong result.
 The forthcoming patch fixes that issue

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-1052) Support MultipleOutputFormat in Avro

2012-04-05 Thread Ashish Nagavaram (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247344#comment-13247344
 ] 

Ashish Nagavaram commented on AVRO-1052:


I tried to find one but was not able to. I will start working on a Avro 
Specific MultipleOutputs and see how it goes. 

 Support MultipleOutputFormat in Avro
 

 Key: AVRO-1052
 URL: https://issues.apache.org/jira/browse/AVRO-1052
 Project: Avro
  Issue Type: New Feature
  Components: java
 Environment: Avro 1.6.1, Hadoop 0.20.205.0.3
Reporter: Ashish Nagavaram
  Labels: avro, newbie
   Original Estimate: 432h
  Remaining Estimate: 432h

 Api's for Avro to write records into multiple files similar to the 
 MultipleTextOutputFormat. All the files have the same output schema

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-551) C: Build and pass tests on Win32

2012-04-03 Thread Vivek Nadkarni (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13245626#comment-13245626
 ] 

Vivek Nadkarni commented on AVRO-551:
-

Hi - 

I am in the process of porting Avro-C, version 1.6.3, to Microsoft
Visual C++ 2008. I would like to submit my changes for inclusion in
the Avro-C project.

In the porting process, I needed to include a few header files that
are not present in the Microsoft environment. I have included the
following files:

1. inttypes.h  stdint.h
   Obtained from
   http://code.google.com/p/msinttypes/
   This project uses the New BSD License:
   http://www.opensource.org/licenses/bsd-license.php

2. dirent.h
   Obtained from
   http://www.softagalleria.net/dirent.php
   http://www.softagalleria.net/download/dirent/dirent-1.11.zip
   The file copyright header matches the MIT-License.

Can I just include these files in my patchset when I upload my
patches? Is there anything special I need to do to include these files
in the project?

Thanks,
Vivek


 C: Build and pass tests on Win32
 

 Key: AVRO-551
 URL: https://issues.apache.org/jira/browse/AVRO-551
 Project: Avro
  Issue Type: Improvement
  Components: c
Reporter: Bruce Mitchener

 Avro C does not currently build on Win32. We need to address that.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-551) C: Build and pass tests on Win32

2012-04-03 Thread Doug Cutting (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13245856#comment-13245856
 ] 

Doug Cutting commented on AVRO-551:
---

It's fine to include MIT and BSD licensed code, but the licenses for these 
files should be appended to the end of Avro's top-level LICENSE.txt file.  For 
more information see:

http://apache.org/legal/resolved.html

 C: Build and pass tests on Win32
 

 Key: AVRO-551
 URL: https://issues.apache.org/jira/browse/AVRO-551
 Project: Avro
  Issue Type: Improvement
  Components: c
Reporter: Bruce Mitchener
 Attachments: AVRO-551.patch


 Avro C does not currently build on Win32. We need to address that.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-565) Investigate Python encoding error

2012-03-27 Thread Russell Jurney (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239381#comment-13239381
 ] 

Russell Jurney commented on AVRO-565:
-

Oh man, this has been KILLING me for a couple weeks.

What am I to do?  What is the status of this?  No matter what I do to the input 
field, python avro cannot write the field.

Even though I do this: subject.decode(charset).encode('utf-8')

The write still dies.  Oh man this has been KILLING me.  Help :)



 Investigate Python encoding error
 -

 Key: AVRO-565
 URL: https://issues.apache.org/jira/browse/AVRO-565
 Project: Avro
  Issue Type: Bug
  Components: python
Reporter: Jeff Hammerbacher

 Tyler B is seeing the following encoding error: 
 http://avro.pastebin.com/b4HSYjCz.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-565) Investigate Python encoding error

2012-03-27 Thread Russell Jurney (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239401#comment-13239401
 ] 

Russell Jurney commented on AVRO-565:
-

On further reflection...

I have changed my code to have all strings become Unicode objects, which Avro 
can write: decode(mystring, mycharset)  

Before I was using UTF-8 encoded strings: 
mystring.decode(mycharset).encode('utf-8)

As outlined above, this was resulting in double UTF-age.  Suggest that the 
API/docs should only accept Unicode objects, not strings, since strings do not 
work well.

 Investigate Python encoding error
 -

 Key: AVRO-565
 URL: https://issues.apache.org/jira/browse/AVRO-565
 Project: Avro
  Issue Type: Bug
  Components: python
Reporter: Jeff Hammerbacher

 Tyler B is seeing the following encoding error: 
 http://avro.pastebin.com/b4HSYjCz.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-565) Investigate Python encoding error

2012-03-27 Thread Miki Tebeka (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239444#comment-13239444
 ] 

Miki Tebeka commented on AVRO-565:
--

Russel, can you provide an example input that causes the problem? Even better 
will be an example script that reproduces the problem.

 Investigate Python encoding error
 -

 Key: AVRO-565
 URL: https://issues.apache.org/jira/browse/AVRO-565
 Project: Avro
  Issue Type: Bug
  Components: python
Reporter: Jeff Hammerbacher

 Tyler B is seeing the following encoding error: 
 http://avro.pastebin.com/b4HSYjCz.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-1050) Avro PHP consumes too much memory due to code in io.php (AvroStringIO append_str)

2012-03-23 Thread A B (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236657#comment-13236657
 ] 

A B commented on AVRO-1050:
---

As I mentioned we have a fix in place as follows (patch will be forthcoming):

Original code (io.php, class AvroStringIO):
  private function append_str($str)
  {
$this-check_closed();
$ary = str_split($str);
$len = count($ary);
$this-buffer = array_merge($this-buffer, $ary);
$this-current_index += $len;
return $len;
  }

Fixed code (io.php, class AvroStringIO): 
  private function append_str($str)
  {
$this-check_closed();
$this-buffer .= $str;
$len = strlen($str);
$this-current_index += $len;
return $len;
  }


 Avro PHP consumes too much memory due to code in io.php (AvroStringIO 
 append_str)
 -

 Key: AVRO-1050
 URL: https://issues.apache.org/jira/browse/AVRO-1050
 Project: Avro
  Issue Type: Bug
  Components: php
Affects Versions: 1.6.2
Reporter: A B
 Fix For: 1.7.0


 While attempting to encode large messages, our team found that the PHP 
 run-time would exceed its memory limit and die. Profiling and use of 
 memory_get_usage showed us that we were spending a lot of time in append_str 
 and that the PHP process grew after every call to this method, resulting in 
 the leak-like behavior that caused the process to grow and then crash. We 
 rewrote append_str to use the string concatenation operator instead of the 
 somewhat elaborate scheme in place today (str_split, followed by the copying 
 of one byte at a time from the latter array to the final string) and the 
 memory usage issues disappeared with no loss of functionality. We would like 
 to provide the patch to fix this issue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-823) Support Avro data files in C#

2012-03-17 Thread Edward Wohlman (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13232072#comment-13232072
 ] 

Edward Wohlman commented on AVRO-823:
-

Any progress on this? I was looking into using Avro as a new file format and I 
work primarily in C#.
I tried adding File support by translating directly from the Java 
implementation, but I am not really familiar enough with the remainder of the 
library yet to get it to work.

I may have another go, but I could probably use a few pointers.

 Support Avro data files in C#
 -

 Key: AVRO-823
 URL: https://issues.apache.org/jira/browse/AVRO-823
 Project: Avro
  Issue Type: New Feature
  Components: csharp
Affects Versions: 1.5.1
Reporter: Patrick Angeles

 Currently, the C# implementation does not handle reading and writing of Avro 
 data files.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-823) Support Avro data files in C#

2012-03-17 Thread Thiruvalluvan M. G. (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13232142#comment-13232142
 ] 

Thiruvalluvan M. G. commented on AVRO-823:
--

If you can attach your work so far, someone may take up from there. I'm not 
sure if anyone would, but still one will be more motivated than if he has to 
start from scratch.

 Support Avro data files in C#
 -

 Key: AVRO-823
 URL: https://issues.apache.org/jira/browse/AVRO-823
 Project: Avro
  Issue Type: New Feature
  Components: csharp
Affects Versions: 1.5.1
Reporter: Patrick Angeles

 Currently, the C# implementation does not handle reading and writing of Avro 
 data files.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-1045) deepCopy of BYTES underflow exception

2012-03-15 Thread Scott Carey (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13230303#comment-13230303
 ] 

Scott Carey commented on AVRO-1045:
---


There are two choices that make sense to me:
* Copy the whole buffer and set all positions in the destination to the same 
thing as the source (limit, pos, mark, capacity), being cognizant of 
arrayOffset in case the source buffer was a slice of a larger array.
* Assume that the data of intereste is between pos and limit, copy that into a 
new byte buffer starting at index 0 with the new limit set to (limit - pos). 
In both cases, the original buffer needs to be returned to its original state.

Avro isn't currently doing either.

 deepCopy of BYTES underflow exception
 -

 Key: AVRO-1045
 URL: https://issues.apache.org/jira/browse/AVRO-1045
 Project: Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.6.2
Reporter: Jeremy Lewi
Priority: Minor
 Fix For: 1.6.3

 Attachments: AVRO-1045.patch


 In org.apache.avro.generic.GenericData.deepCopy - the code for copying a 
 ByteBuffer is
 ByteBuffer byteBufferValue = (ByteBuffer) value;
 byte[] bytesCopy = new byte[byteBufferValue.capacity()];
 byteBufferValue.rewind();
 byteBufferValue.get(bytesCopy);
 byteBufferValue.rewind();
 return ByteBuffer.wrap(bytesCopy);
 I think this is problematic because it will cause an UnderFlow exception to 
 be thrown if the ByteBuffer limit is less than the capacity of the byte 
 buffer.
 My use case is as follows. I have ByteBuffer's backed by large arrays so I 
 can avoid resizing the array every time I write data. So limit  capacity. 
 When the data is written, or copied
 I think avro should respect this. When data is serialized, avro should 
 automatically use the minimum number of bytes.
 When an object is copied, I think it makes sense to preserve the capacity of 
 the underlying buffer as opposed to compacting it.
 So I think the code could be fixed by replacing get with 
 byteBufferValue.get(bytesCopy, 0 , byteBufferValue.limit());

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-1045) deepCopy of BYTES underflow exception

2012-03-14 Thread Doug Cutting (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13229630#comment-13229630
 ] 

Doug Cutting commented on AVRO-1045:


It's a little odd to require that deepCopy() preserve more than is checked by 
equals().  Some folks may might reasonably expect deepCopy() to compact large 
ByteBuffers.  We could perhaps add a 'protected ByteBuffer 
GenericData#copyBytes(ByteBuffer)' method that could be overridden in a 
subclass?  Would that work in your case?  Am I being overly cautious?

 deepCopy of BYTES underflow exception
 -

 Key: AVRO-1045
 URL: https://issues.apache.org/jira/browse/AVRO-1045
 Project: Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.6.2
Reporter: Jeremy Lewi
Priority: Minor
 Fix For: 1.6.3

 Attachments: AVRO-1045.patch


 In org.apache.avro.generic.GenericData.deepCopy - the code for copying a 
 ByteBuffer is
 ByteBuffer byteBufferValue = (ByteBuffer) value;
 byte[] bytesCopy = new byte[byteBufferValue.capacity()];
 byteBufferValue.rewind();
 byteBufferValue.get(bytesCopy);
 byteBufferValue.rewind();
 return ByteBuffer.wrap(bytesCopy);
 I think this is problematic because it will cause an UnderFlow exception to 
 be thrown if the ByteBuffer limit is less than the capacity of the byte 
 buffer.
 My use case is as follows. I have ByteBuffer's backed by large arrays so I 
 can avoid resizing the array every time I write data. So limit  capacity. 
 When the data is written, or copied
 I think avro should respect this. When data is serialized, avro should 
 automatically use the minimum number of bytes.
 When an object is copied, I think it makes sense to preserve the capacity of 
 the underlying buffer as opposed to compacting it.
 So I think the code could be fixed by replacing get with 
 byteBufferValue.get(bytesCopy, 0 , byteBufferValue.limit());

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-1045) deepCopy of BYTES underflow exception

2012-03-14 Thread Jeremy Lewi (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13229718#comment-13229718
 ] 

Jeremy Lewi commented on AVRO-1045:
---

I think the issue is less my use case then maintaining compatibility with 
existing code. In my patch, I decided to preserve the capacity on copy rather 
than compact because this seemed slightly less likely to break existing code.
The current implementation creates a new buffer of equal capacity, so in theory 
its not compacting the buffer. However, if the buffer isn't full, i.e. limit  
capacity, then the current code causes an exception. So based on the current 
implementation, its unclear whether compacting or not compacting is the 
expected behavior.

With regards to my use case, compacting on copy is fine, so long as deep copy 
doesn't expect limit = capacity like it currently does. 

 deepCopy of BYTES underflow exception
 -

 Key: AVRO-1045
 URL: https://issues.apache.org/jira/browse/AVRO-1045
 Project: Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.6.2
Reporter: Jeremy Lewi
Priority: Minor
 Fix For: 1.6.3

 Attachments: AVRO-1045.patch


 In org.apache.avro.generic.GenericData.deepCopy - the code for copying a 
 ByteBuffer is
 ByteBuffer byteBufferValue = (ByteBuffer) value;
 byte[] bytesCopy = new byte[byteBufferValue.capacity()];
 byteBufferValue.rewind();
 byteBufferValue.get(bytesCopy);
 byteBufferValue.rewind();
 return ByteBuffer.wrap(bytesCopy);
 I think this is problematic because it will cause an UnderFlow exception to 
 be thrown if the ByteBuffer limit is less than the capacity of the byte 
 buffer.
 My use case is as follows. I have ByteBuffer's backed by large arrays so I 
 can avoid resizing the array every time I write data. So limit  capacity. 
 When the data is written, or copied
 I think avro should respect this. When data is serialized, avro should 
 automatically use the minimum number of bytes.
 When an object is copied, I think it makes sense to preserve the capacity of 
 the underlying buffer as opposed to compacting it.
 So I think the code could be fixed by replacing get with 
 byteBufferValue.get(bytesCopy, 0 , byteBufferValue.limit());

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-301) Handle non-reserved properties appropriately in the Python implementation

2012-03-14 Thread Miki Tebeka (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13229864#comment-13229864
 ] 

Miki Tebeka commented on AVRO-301:
--

Patch looks good, going to to commit tomorrow afternoon if there are no 
objections.

 Handle non-reserved properties appropriately in the Python implementation
 -

 Key: AVRO-301
 URL: https://issues.apache.org/jira/browse/AVRO-301
 Project: Avro
  Issue Type: New Feature
  Components: python
Affects Versions: 1.6.2
Reporter: Jeff Hammerbacher
Assignee: Marcio Silva
 Attachments: AVRO-301-patch-1.patch, AVRO-301-patch-2.patch, 
 AVRO-301-patch-3.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-1044) avro-maven-plugin requires dependency resolution which breaks multi-module projects

2012-03-13 Thread Arvind Prabhakar (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13228742#comment-13228742
 ] 

Arvind Prabhakar commented on AVRO-1044:


The tarball I attached does not reproduce this problem, it is complaining about 
something else. Hence I will remove it and work on getting a reproducible 
testcase.

 avro-maven-plugin requires dependency resolution which breaks multi-module 
 projects
 ---

 Key: AVRO-1044
 URL: https://issues.apache.org/jira/browse/AVRO-1044
 Project: Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.6.2
Reporter: Arvind Prabhakar
Priority: Critical
 Fix For: 1.6.3


 Use of avro-maven-plugin breaks multimodule projects since it forces the 
 dependency resolution of all of the dependencies, some of which may be from 
 within the reactor and not yet installed in the local cache.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-406) Support streaming RPC calls

2012-03-09 Thread Commented

[ 
https://issues.apache.org/jira/browse/AVRO-406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13226016#comment-13226016
 ] 

André Cruz commented on AVRO-406:
-

Has streaming support been definitely abandoned? It would be very useful for 
Cassandra.

 Support streaming RPC calls
 ---

 Key: AVRO-406
 URL: https://issues.apache.org/jira/browse/AVRO-406
 Project: Avro
  Issue Type: New Feature
  Components: java, spec
Reporter: Todd Lipcon
  Labels: rpc

 Avro nicely supports chunking of container types into multiple frames. We 
 need to expose this to RPC layer to facilitate use cases like the Hadoop 
 Datanode where a single RPC can yield far more data than should be buffered 
 in memory.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-987) Make Avro OSGi ready

2012-03-07 Thread Ioannis Canellos (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13224088#comment-13224088
 ] 

Ioannis Canellos commented on AVRO-987:
---

I'll rework on the patch to align it to the current trunk.

-- 
*Ioannis Canellos*
*
FuseSource http://fusesource.com

**
Blog: http://iocanel.blogspot.com
**
Twitter: iocanel
*


 Make Avro OSGi ready
 

 Key: AVRO-987
 URL: https://issues.apache.org/jira/browse/AVRO-987
 Project: Avro
  Issue Type: New Feature
  Components: java
Reporter: Ioannis Canellos
 Attachments: AVRO-987-patch-updated.txt, AVRO-987-patch.txt


 It would be really nice to be able to use Avro inside OSGi. To achieve this 
 two things are required:
 i) Provide proper MANIFEST.MF.
 ii) Deal with potential class loading issues. Avro uses Class.forName a lot 
 and that is not very OSGi friendly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-987) Make Avro OSGi ready

2012-03-06 Thread Doug Cutting (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13223408#comment-13223408
 ] 

Doug Cutting commented on AVRO-987:
---

The patch looks reasonable to me, but fails to apply cleanly.  Can someone 
please provide a version that applies to the current trunk?  Thanks!

 Make Avro OSGi ready
 

 Key: AVRO-987
 URL: https://issues.apache.org/jira/browse/AVRO-987
 Project: Avro
  Issue Type: New Feature
  Components: java
Reporter: Ioannis Canellos
 Attachments: AVRO-987-patch-updated.txt, AVRO-987-patch.txt


 It would be really nice to be able to use Avro inside OSGi. To achieve this 
 two things are required:
 i) Provide proper MANIFEST.MF.
 ii) Deal with potential class loading issues. Avro uses Class.forName a lot 
 and that is not very OSGi friendly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-1022) Error in validate name

2012-03-06 Thread Doug Cutting (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13223419#comment-13223419
 ] 

Doug Cutting commented on AVRO-1022:


Raymie, this is a good approach.  The spec language that requires ASCII should 
be changed from MUST to SHOULD.

One use case that Scott mentioned that your prose does not is transmitting 
schemas from other systems, e.g., Avro Schemas might often be generated 
automatically from Pig or SQL schemas.  In these cases accepting liberally 
permits schemas to pass through Avro losslessly.  Strict validation is really 
only useful when a developer is the schema author.  In many (most?) cases Avro 
might be an underlying tool, used indirectly through an application, and in 
these cases strict validation is probably not useful.

 Error in validate name
 --

 Key: AVRO-1022
 URL: https://issues.apache.org/jira/browse/AVRO-1022
 Project: Avro
  Issue Type: Bug
  Components: java
Reporter: Raymie Stata
Priority: Minor
 Attachments: AVRO-1022.patch, AVRO-1022.patch, 
 unicode-recommendation.html


 Fix schema.validateName to allow only ASCII letters, not Unicode letters.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas

2012-03-06 Thread Doug Cutting (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13223428#comment-13223428
 ] 

Doug Cutting commented on AVRO-1006:


Patch looks good. +1 I'll commit this unless there are objections.

 Fingerprints for Avro Schemas
 -

 Key: AVRO-1006
 URL: https://issues.apache.org/jira/browse/AVRO-1006
 Project: Avro
  Issue Type: New Feature
  Components: java
Reporter: Raymie Stata
Assignee: Raymie Stata
  Labels: features
 Attachments: AVRO-1006-prelim.patch, AVRO-1006.patch, 
 AVRO-1006.patch, AVRO-1006.patch, AVRO-1006.patch, AVRO-1006.patch, 
 schema-fingerprinting.html, schema-fingerprinting.html, 
 schema-fingerprinting.html


 Add function that returns a standardized, 64-bit fingerprint for schemas.  
 Fingerprints are designed such that the chances of collisions is very, very 
 low.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-784) SpecificCompiler should generate accessors

2012-03-05 Thread Doug Cutting (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13222847#comment-13222847
 ] 

Doug Cutting commented on AVRO-784:
---

Scott, good point.  Let's open a new issue to add unboxed accessors.

 SpecificCompiler should generate accessors
 --

 Key: AVRO-784
 URL: https://issues.apache.org/jira/browse/AVRO-784
 Project: Avro
  Issue Type: Improvement
  Components: java
Affects Versions: 1.5.0
Reporter: E. Sammer
  Labels: features
 Attachments: avro-784.diff, avro-784.diff


 Avro's Java SpecificCompiler should generate java bean style accessors.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-987) Make Avro OSGi ready

2012-03-04 Thread Claus Ibsen (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221862#comment-13221862
 ] 

Claus Ibsen commented on AVRO-987:
--

Any update on this?

 Make Avro OSGi ready
 

 Key: AVRO-987
 URL: https://issues.apache.org/jira/browse/AVRO-987
 Project: Avro
  Issue Type: New Feature
  Components: java
Reporter: Ioannis Canellos
 Attachments: AVRO-987-patch-updated.txt, AVRO-987-patch.txt


 It would be really nice to be able to use Avro inside OSGi. To achieve this 
 two things are required:
 i) Provide proper MANIFEST.MF.
 ii) Deal with potential class loading issues. Avro uses Class.forName a lot 
 and that is not very OSGi friendly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-784) SpecificCompiler should generate accessors

2012-03-04 Thread Scott Carey (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13222130#comment-13222130
 ] 

Scott Carey commented on AVRO-784:
--

I think there were some details that did not make it into AVRO-839.  In 
particular, all accessors in AVRO-839 appear to be boxed, whether the value is 
nullable or not.  I have not moved to using the API there because of the boxing 
overhead incurred.  These issues will need to go into another ticket for later.

 SpecificCompiler should generate accessors
 --

 Key: AVRO-784
 URL: https://issues.apache.org/jira/browse/AVRO-784
 Project: Avro
  Issue Type: Improvement
  Components: java
Affects Versions: 1.5.0
Reporter: E. Sammer
  Labels: features
 Attachments: avro-784.diff, avro-784.diff


 Avro's Java SpecificCompiler should generate java bean style accessors.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-1022) Error in validate name

2012-03-03 Thread Raymie Stata (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221634#comment-13221634
 ] 

Raymie Stata commented on AVRO-1022:


Based on the dialog above, I propose that we turn this into a set of 
recommendations for Avro implementers and app developers.  I've attached a 
draft of these recommendations, if folks like them, I can turn them into a 
patch for the spec.

The basic idea is as follows.  We cast this as an interoperability problem.  
On the one hand, the current spec for names facilitates language 
interoperability, and we shouldn't change it.  On the other hand, we should 
apply the Robustness principle: accept liberally, generate strictly.  (Looking 
closely at Scott's and Doug's comments, I think this is what they're really 
getting at.)

In our context, Avro implementations accept schemas, Avro app developers 
generate them.  Thus, we recommend that Avro implementations, by default, 
accept schemas liberally.  This means accepting arbitrary Unicode strings for 
names (but with no normalization or other Unicode processing).  At the same 
time, Avro developers for who want language interoperability are exhorted to 
generate strictly, and in particular, use only names that follow the strict 
definition in the Avro spec, because those names are easily handled by almost 
any language out there.

To help developers generate strictly, we recommend that Avro implementations 
provide optional mechanisms to perform schema validation, preferably as early 
in the dev process as possible (eg, unit-test or even compile time).  (For 
developers who don't care about language interop, they can ignore these 
mechanisms, and Avro implementations, being liberal by default, should happily 
consume their input.)

The file I just posted says the above in a little more detail.

 Error in validate name
 --

 Key: AVRO-1022
 URL: https://issues.apache.org/jira/browse/AVRO-1022
 Project: Avro
  Issue Type: Bug
  Components: java
Reporter: Raymie Stata
Priority: Minor
 Attachments: AVRO-1022.patch, AVRO-1022.patch, 
 unicode-recommendation.html


 Fix schema.validateName to allow only ASCII letters, not Unicode letters.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-1027) NettyTransceiver will deadlock when attempting transceive/disconnect on the same thread

2012-03-02 Thread James Baldassari (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221055#comment-13221055
 ] 

James Baldassari commented on AVRO-1027:


All my tests are passing with trunk + AVRO-1027

 NettyTransceiver will deadlock when attempting transceive/disconnect on the 
 same thread
 ---

 Key: AVRO-1027
 URL: https://issues.apache.org/jira/browse/AVRO-1027
 Project: Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.6.1
Reporter: Simon Wilkinson
Assignee: James Baldassari
 Fix For: 1.6.3

 Attachments: AVRO-1027-v2.patch, AVRO-1027.patch


 If an Exception is caught while trying to write to a Channel, Netty can 
 deliver the Exception to a ChannelUpstreamHandler on the same thread that 
 attempted to write to the Channel. If this occurs with the 
 NettyClientAvroHandler implementation of ChannelUpstreamHandler then the 
 thread will deadlock.
 Specifically, NettyClientAvroHandler overrides the 
 ChannelUpstreamHandler.exceptionCaught() method to perform a disconnect, 
 which requires the NettyTransceiver's write lock. However, in the above 
 situation, the thread will already have locked the NettyTransceiver's read 
 lock to write to the Channel. ReentrantReadWriteLock does not allow upgrading 
 from a read to a write lock, hence the thread deadlocks.
 Example stack trace (simplified):
 SessionManager-TimeoutPoller prio=10 tid=0x7b689c00 nid=0x375d waiting on 
 condition [0x7b0ad000..0x7b0ade70]
java.lang.Thread.State: WAITING (parking)
 at sun.misc.Unsafe.park(Native Method)
 - parking to wait for  0xf2a944d8 (a 
 java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
 at 
 java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807)
  [Acquire write lock] at 
  org.apache.avro.ipc.NettyTransceiver.disconnect(NettyTransceiver.java:285)
 at 
 org.apache.avro.ipc.NettyTransceiver.access$2(NettyTransceiver.java:281)
 at 
 org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.exceptionCaught(NettyTransceiver.java:499)
 at 
 org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
 at 
 org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.handleUpstream(NettyTransceiver.java:473)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783)
 at 
 org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:238)
 at 
 org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
 at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:432)
 at 
 org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:661)
 at 
 org.jboss.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:372)
 at 
 org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:117)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:771)
 at org.jboss.netty.channel.Channels.write(Channels.java:632)
 at 
 org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
 at org.jboss.netty.channel.Channels.write(Channels.java:611)
 at org.jboss.netty.channel.Channels.write(Channels.java:578)
 at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:251)
  [Acquire read lock] at 
  org.apache.avro.ipc.NettyTransceiver.writeDataPack(NettyTransceiver.java:413)
  [Acquire read lock] at 
  org.apache.avro.ipc.NettyTransceiver.transceive(NettyTransceiver.java:394

[jira] [Commented] (AVRO-1027) NettyTransceiver will deadlock when attempting transceive/disconnect on the same thread

2012-03-02 Thread Doug Cutting (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221291#comment-13221291
 ] 

Doug Cutting commented on AVRO-1027:


James, thanks for running your tests.

I guess I'll go ahead and commit this now and roll a release candidate today.

 NettyTransceiver will deadlock when attempting transceive/disconnect on the 
 same thread
 ---

 Key: AVRO-1027
 URL: https://issues.apache.org/jira/browse/AVRO-1027
 Project: Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.6.1
Reporter: Simon Wilkinson
Assignee: James Baldassari
 Fix For: 1.6.3

 Attachments: AVRO-1027-v2.patch, AVRO-1027.patch


 If an Exception is caught while trying to write to a Channel, Netty can 
 deliver the Exception to a ChannelUpstreamHandler on the same thread that 
 attempted to write to the Channel. If this occurs with the 
 NettyClientAvroHandler implementation of ChannelUpstreamHandler then the 
 thread will deadlock.
 Specifically, NettyClientAvroHandler overrides the 
 ChannelUpstreamHandler.exceptionCaught() method to perform a disconnect, 
 which requires the NettyTransceiver's write lock. However, in the above 
 situation, the thread will already have locked the NettyTransceiver's read 
 lock to write to the Channel. ReentrantReadWriteLock does not allow upgrading 
 from a read to a write lock, hence the thread deadlocks.
 Example stack trace (simplified):
 SessionManager-TimeoutPoller prio=10 tid=0x7b689c00 nid=0x375d waiting on 
 condition [0x7b0ad000..0x7b0ade70]
java.lang.Thread.State: WAITING (parking)
 at sun.misc.Unsafe.park(Native Method)
 - parking to wait for  0xf2a944d8 (a 
 java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
 at 
 java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807)
  [Acquire write lock] at 
  org.apache.avro.ipc.NettyTransceiver.disconnect(NettyTransceiver.java:285)
 at 
 org.apache.avro.ipc.NettyTransceiver.access$2(NettyTransceiver.java:281)
 at 
 org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.exceptionCaught(NettyTransceiver.java:499)
 at 
 org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
 at 
 org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.handleUpstream(NettyTransceiver.java:473)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783)
 at 
 org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:238)
 at 
 org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
 at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:432)
 at 
 org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:661)
 at 
 org.jboss.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:372)
 at 
 org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:117)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:771)
 at org.jboss.netty.channel.Channels.write(Channels.java:632)
 at 
 org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
 at org.jboss.netty.channel.Channels.write(Channels.java:611)
 at org.jboss.netty.channel.Channels.write(Channels.java:578)
 at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:251)
  [Acquire read lock] at 
  org.apache.avro.ipc.NettyTransceiver.writeDataPack(NettyTransceiver.java:413)
  [Acquire read lock

[jira] [Commented] (AVRO-1027) NettyTransceiver will deadlock when attempting transceive/disconnect on the same thread

2012-03-02 Thread George Fletcher (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221305#comment-13221305
 ] 

George Fletcher commented on AVRO-1027:
---

A release candidate today would be fantastic! Thanks!

 NettyTransceiver will deadlock when attempting transceive/disconnect on the 
 same thread
 ---

 Key: AVRO-1027
 URL: https://issues.apache.org/jira/browse/AVRO-1027
 Project: Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.6.1
Reporter: Simon Wilkinson
Assignee: James Baldassari
 Fix For: 1.6.3

 Attachments: AVRO-1027-v2.patch, AVRO-1027.patch


 If an Exception is caught while trying to write to a Channel, Netty can 
 deliver the Exception to a ChannelUpstreamHandler on the same thread that 
 attempted to write to the Channel. If this occurs with the 
 NettyClientAvroHandler implementation of ChannelUpstreamHandler then the 
 thread will deadlock.
 Specifically, NettyClientAvroHandler overrides the 
 ChannelUpstreamHandler.exceptionCaught() method to perform a disconnect, 
 which requires the NettyTransceiver's write lock. However, in the above 
 situation, the thread will already have locked the NettyTransceiver's read 
 lock to write to the Channel. ReentrantReadWriteLock does not allow upgrading 
 from a read to a write lock, hence the thread deadlocks.
 Example stack trace (simplified):
 SessionManager-TimeoutPoller prio=10 tid=0x7b689c00 nid=0x375d waiting on 
 condition [0x7b0ad000..0x7b0ade70]
java.lang.Thread.State: WAITING (parking)
 at sun.misc.Unsafe.park(Native Method)
 - parking to wait for  0xf2a944d8 (a 
 java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
 at 
 java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807)
  [Acquire write lock] at 
  org.apache.avro.ipc.NettyTransceiver.disconnect(NettyTransceiver.java:285)
 at 
 org.apache.avro.ipc.NettyTransceiver.access$2(NettyTransceiver.java:281)
 at 
 org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.exceptionCaught(NettyTransceiver.java:499)
 at 
 org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
 at 
 org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.handleUpstream(NettyTransceiver.java:473)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783)
 at 
 org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:238)
 at 
 org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
 at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:432)
 at 
 org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:661)
 at 
 org.jboss.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:372)
 at 
 org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:117)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:771)
 at org.jboss.netty.channel.Channels.write(Channels.java:632)
 at 
 org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
 at org.jboss.netty.channel.Channels.write(Channels.java:611)
 at org.jboss.netty.channel.Channels.write(Channels.java:578)
 at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:251)
  [Acquire read lock] at 
  org.apache.avro.ipc.NettyTransceiver.writeDataPack(NettyTransceiver.java:413)
  [Acquire read lock] at 
  org.apache.avro.ipc.NettyTransceiver.transceive(NettyTransceiver.java:394

[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas

2012-03-02 Thread Doug Cutting (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221342#comment-13221342
 ] 

Doug Cutting commented on AVRO-1006:


Raymie SchemaFingerprint.fingerprint seems unnecessarily long...

Now this becomes 
SchemaNormalization.fp(SchemaNormalization.toParsingForm(schema)).  The 'fp' 
might better be spelled out as 'fingerprint'.  Also a utility method like 
SchemaNormalization.parsingFingerprint(schema) might be useful.

Graham pass a Normalizer instance...

With the latest API, someone can already call SchemaNormalization.fingerprint() 
with a differently normalized schema, so I don't see the need for this.  As we 
add more normalizers to Avro we can add new methods, so I'm not (yet) seeing 
the advantage of adding a Normalization interface.

 Fingerprints for Avro Schemas
 -

 Key: AVRO-1006
 URL: https://issues.apache.org/jira/browse/AVRO-1006
 Project: Avro
  Issue Type: New Feature
  Components: java
Reporter: Raymie Stata
Assignee: Raymie Stata
  Labels: features
 Attachments: AVRO-1006-prelim.patch, AVRO-1006.patch, 
 AVRO-1006.patch, AVRO-1006.patch, schema-fingerprinting.html, 
 schema-fingerprinting.html, schema-fingerprinting.html


 Add function that returns a standardized, 64-bit fingerprint for schemas.  
 Fingerprints are designed such that the chances of collisions is very, very 
 low.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-1027) NettyTransceiver will deadlock when attempting transceive/disconnect on the same thread

2012-03-02 Thread James Baldassari (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221345#comment-13221345
 ] 

James Baldassari commented on AVRO-1027:


Doug, just a minor correction on the attribution for this patch.  The original 
patch was contributed by Simon Wilkinson.  I just updated it to apply cleanly 
with patch -p0, so I think Simon should probably be the contributor for this 
one.

 NettyTransceiver will deadlock when attempting transceive/disconnect on the 
 same thread
 ---

 Key: AVRO-1027
 URL: https://issues.apache.org/jira/browse/AVRO-1027
 Project: Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.6.1
Reporter: Simon Wilkinson
Assignee: James Baldassari
 Fix For: 1.6.3

 Attachments: AVRO-1027-v2.patch, AVRO-1027.patch


 If an Exception is caught while trying to write to a Channel, Netty can 
 deliver the Exception to a ChannelUpstreamHandler on the same thread that 
 attempted to write to the Channel. If this occurs with the 
 NettyClientAvroHandler implementation of ChannelUpstreamHandler then the 
 thread will deadlock.
 Specifically, NettyClientAvroHandler overrides the 
 ChannelUpstreamHandler.exceptionCaught() method to perform a disconnect, 
 which requires the NettyTransceiver's write lock. However, in the above 
 situation, the thread will already have locked the NettyTransceiver's read 
 lock to write to the Channel. ReentrantReadWriteLock does not allow upgrading 
 from a read to a write lock, hence the thread deadlocks.
 Example stack trace (simplified):
 SessionManager-TimeoutPoller prio=10 tid=0x7b689c00 nid=0x375d waiting on 
 condition [0x7b0ad000..0x7b0ade70]
java.lang.Thread.State: WAITING (parking)
 at sun.misc.Unsafe.park(Native Method)
 - parking to wait for  0xf2a944d8 (a 
 java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
 at 
 java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807)
  [Acquire write lock] at 
  org.apache.avro.ipc.NettyTransceiver.disconnect(NettyTransceiver.java:285)
 at 
 org.apache.avro.ipc.NettyTransceiver.access$2(NettyTransceiver.java:281)
 at 
 org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.exceptionCaught(NettyTransceiver.java:499)
 at 
 org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
 at 
 org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.handleUpstream(NettyTransceiver.java:473)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783)
 at 
 org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:238)
 at 
 org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
 at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:432)
 at 
 org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:661)
 at 
 org.jboss.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:372)
 at 
 org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:117)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:771)
 at org.jboss.netty.channel.Channels.write(Channels.java:632)
 at 
 org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
 at org.jboss.netty.channel.Channels.write(Channels.java:611)
 at org.jboss.netty.channel.Channels.write(Channels.java:578)
 at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:251)
  [Acquire read lock

[jira] [Commented] (AVRO-1027) NettyTransceiver will deadlock when attempting transceive/disconnect on the same thread

2012-03-02 Thread Simon Wilkinson (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221380#comment-13221380
 ] 

Simon Wilkinson commented on AVRO-1027:
---

No problem.

I've been thinking about how to construct the unit test for this. As James has 
said, it will be tricky.
Specifically, to force the disconnect, the server has to be closed after a 
client thread has acquired a NettyTransceiver's read lock in transceive(), but 
before the request gets very far in the NettyTransceiver's Netty pipeline.

There doesn't seem any way to make this happen consistently without adding some 
debug synchronization to NettyTransceiver.

 NettyTransceiver will deadlock when attempting transceive/disconnect on the 
 same thread
 ---

 Key: AVRO-1027
 URL: https://issues.apache.org/jira/browse/AVRO-1027
 Project: Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.6.1
Reporter: Simon Wilkinson
Assignee: Simon Wilkinson
 Fix For: 1.6.3

 Attachments: AVRO-1027-v2.patch, AVRO-1027.patch


 If an Exception is caught while trying to write to a Channel, Netty can 
 deliver the Exception to a ChannelUpstreamHandler on the same thread that 
 attempted to write to the Channel. If this occurs with the 
 NettyClientAvroHandler implementation of ChannelUpstreamHandler then the 
 thread will deadlock.
 Specifically, NettyClientAvroHandler overrides the 
 ChannelUpstreamHandler.exceptionCaught() method to perform a disconnect, 
 which requires the NettyTransceiver's write lock. However, in the above 
 situation, the thread will already have locked the NettyTransceiver's read 
 lock to write to the Channel. ReentrantReadWriteLock does not allow upgrading 
 from a read to a write lock, hence the thread deadlocks.
 Example stack trace (simplified):
 SessionManager-TimeoutPoller prio=10 tid=0x7b689c00 nid=0x375d waiting on 
 condition [0x7b0ad000..0x7b0ade70]
java.lang.Thread.State: WAITING (parking)
 at sun.misc.Unsafe.park(Native Method)
 - parking to wait for  0xf2a944d8 (a 
 java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
 at 
 java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807)
  [Acquire write lock] at 
  org.apache.avro.ipc.NettyTransceiver.disconnect(NettyTransceiver.java:285)
 at 
 org.apache.avro.ipc.NettyTransceiver.access$2(NettyTransceiver.java:281)
 at 
 org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.exceptionCaught(NettyTransceiver.java:499)
 at 
 org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
 at 
 org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.handleUpstream(NettyTransceiver.java:473)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783)
 at 
 org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:238)
 at 
 org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
 at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:432)
 at 
 org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:661)
 at 
 org.jboss.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:372)
 at 
 org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:117)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:771)
 at org.jboss.netty.channel.Channels.write(Channels.java:632)
 at 
 org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582

[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas

2012-03-02 Thread Doug Cutting (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221386#comment-13221386
 ] 

Doug Cutting commented on AVRO-1006:


A few minor nits:
 - the 'href' in one of the documentation links is missing its 'h'
 - the WHITESPACE comment should perhaps read, Eliminate all whitespace in 
JSON outside of string literals.
 - we might define a nested FingerprintAlgorithm Enum for the implemented 
fingerprint algorithm names.
 - SchemaNormalization should probably have a private constructor, e.g., 
'private SchemaNormalization() {}'
 - the #fingerprint link in the class documentation is broken.

Otherwise I'm +1 and look forward to committing this early next week unless 
there are objections.

 Fingerprints for Avro Schemas
 -

 Key: AVRO-1006
 URL: https://issues.apache.org/jira/browse/AVRO-1006
 Project: Avro
  Issue Type: New Feature
  Components: java
Reporter: Raymie Stata
Assignee: Raymie Stata
  Labels: features
 Attachments: AVRO-1006-prelim.patch, AVRO-1006.patch, 
 AVRO-1006.patch, AVRO-1006.patch, AVRO-1006.patch, 
 schema-fingerprinting.html, schema-fingerprinting.html, 
 schema-fingerprinting.html


 Add function that returns a standardized, 64-bit fingerprint for schemas.  
 Fingerprints are designed such that the chances of collisions is very, very 
 low.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas

2012-03-02 Thread Raymie Stata (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221388#comment-13221388
 ] 

Raymie Stata commented on AVRO-1006:


The string passed to the fingerprint method is in turn passed through to 
MessageDigest.getInstance.  Using an enum here would prevent folks from 
accessing digest algo's in the underlying library.  I suggest leaving that 
as-is.

Otherwise I'll fix the rest of these issues.

 Fingerprints for Avro Schemas
 -

 Key: AVRO-1006
 URL: https://issues.apache.org/jira/browse/AVRO-1006
 Project: Avro
  Issue Type: New Feature
  Components: java
Reporter: Raymie Stata
Assignee: Raymie Stata
  Labels: features
 Attachments: AVRO-1006-prelim.patch, AVRO-1006.patch, 
 AVRO-1006.patch, AVRO-1006.patch, AVRO-1006.patch, 
 schema-fingerprinting.html, schema-fingerprinting.html, 
 schema-fingerprinting.html


 Add function that returns a standardized, 64-bit fingerprint for schemas.  
 Fingerprints are designed such that the chances of collisions is very, very 
 low.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas

2012-03-02 Thread Raymie Stata (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221403#comment-13221403
 ] 

Raymie Stata commented on AVRO-1006:


Yes, this is meant to be a cross-language feature, and the definition of 
parsing normal form is in the Avro spec as a result.

To aid in assuring interoperability, this patch includes file-based test data 
that should be used across languages.  Test inputs and outputs can be found in 
share/test/data/schema-tests.txt

 Fingerprints for Avro Schemas
 -

 Key: AVRO-1006
 URL: https://issues.apache.org/jira/browse/AVRO-1006
 Project: Avro
  Issue Type: New Feature
  Components: java
Reporter: Raymie Stata
Assignee: Raymie Stata
  Labels: features
 Attachments: AVRO-1006-prelim.patch, AVRO-1006.patch, 
 AVRO-1006.patch, AVRO-1006.patch, AVRO-1006.patch, AVRO-1006.patch, 
 schema-fingerprinting.html, schema-fingerprinting.html, 
 schema-fingerprinting.html


 Add function that returns a standardized, 64-bit fingerprint for schemas.  
 Fingerprints are designed such that the chances of collisions is very, very 
 low.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-1027) NettyTransceiver will deadlock when attempting transceive/disconnect on the same thread

2012-03-02 Thread Simon Wilkinson (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221465#comment-13221465
 ] 

Simon Wilkinson commented on AVRO-1027:
---


Correction, the server must close after the Netty Channel has been checked for 
'readiness' in NettyTransceiver.writeDataPack(NettyDataPack dataPack), but 
before the request gets very far in the NettyTransceiver's Netty pipeline.

I.e.:

private void writeDataPack(NettyDataPack dataPack) throws IOException {
  Channel channel = getChannel(); // Channel appears ready
  
  // Server closes
  
  channel.write(dataPack); // Netty will notice the disconnection and handle on 
the calling thread
}

 NettyTransceiver will deadlock when attempting transceive/disconnect on the 
 same thread
 ---

 Key: AVRO-1027
 URL: https://issues.apache.org/jira/browse/AVRO-1027
 Project: Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.6.1
Reporter: Simon Wilkinson
Assignee: Simon Wilkinson
 Fix For: 1.6.3

 Attachments: AVRO-1027-v2.patch, AVRO-1027.patch


 If an Exception is caught while trying to write to a Channel, Netty can 
 deliver the Exception to a ChannelUpstreamHandler on the same thread that 
 attempted to write to the Channel. If this occurs with the 
 NettyClientAvroHandler implementation of ChannelUpstreamHandler then the 
 thread will deadlock.
 Specifically, NettyClientAvroHandler overrides the 
 ChannelUpstreamHandler.exceptionCaught() method to perform a disconnect, 
 which requires the NettyTransceiver's write lock. However, in the above 
 situation, the thread will already have locked the NettyTransceiver's read 
 lock to write to the Channel. ReentrantReadWriteLock does not allow upgrading 
 from a read to a write lock, hence the thread deadlocks.
 Example stack trace (simplified):
 SessionManager-TimeoutPoller prio=10 tid=0x7b689c00 nid=0x375d waiting on 
 condition [0x7b0ad000..0x7b0ade70]
java.lang.Thread.State: WAITING (parking)
 at sun.misc.Unsafe.park(Native Method)
 - parking to wait for  0xf2a944d8 (a 
 java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
 at 
 java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807)
  [Acquire write lock] at 
  org.apache.avro.ipc.NettyTransceiver.disconnect(NettyTransceiver.java:285)
 at 
 org.apache.avro.ipc.NettyTransceiver.access$2(NettyTransceiver.java:281)
 at 
 org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.exceptionCaught(NettyTransceiver.java:499)
 at 
 org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
 at 
 org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.handleUpstream(NettyTransceiver.java:473)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783)
 at 
 org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:238)
 at 
 org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
 at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:432)
 at 
 org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:661)
 at 
 org.jboss.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:372)
 at 
 org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:117)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:771)
 at org.jboss.netty.channel.Channels.write(Channels.java:632)
 at 
 org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream

[jira] [Commented] (AVRO-1027) NettyTransceiver will deadlock when attempting transceive/disconnect on the same thread

2012-03-01 Thread Doug Cutting (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13220296#comment-13220296
 ] 

Doug Cutting commented on AVRO-1027:


Simon, do you expect to supply a test today?

It would be good to include this in 1.6.3, but I don't think it's a 
showstopper, since it's not a regression, is it?

 NettyTransceiver will deadlock when attempting transceive/disconnect on the 
 same thread
 ---

 Key: AVRO-1027
 URL: https://issues.apache.org/jira/browse/AVRO-1027
 Project: Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.6.1
Reporter: Simon Wilkinson
Assignee: James Baldassari
 Fix For: 1.6.3

 Attachments: AVRO-1027-v2.patch, AVRO-1027.patch


 If an Exception is caught while trying to write to a Channel, Netty can 
 deliver the Exception to a ChannelUpstreamHandler on the same thread that 
 attempted to write to the Channel. If this occurs with the 
 NettyClientAvroHandler implementation of ChannelUpstreamHandler then the 
 thread will deadlock.
 Specifically, NettyClientAvroHandler overrides the 
 ChannelUpstreamHandler.exceptionCaught() method to perform a disconnect, 
 which requires the NettyTransceiver's write lock. However, in the above 
 situation, the thread will already have locked the NettyTransceiver's read 
 lock to write to the Channel. ReentrantReadWriteLock does not allow upgrading 
 from a read to a write lock, hence the thread deadlocks.
 Example stack trace (simplified):
 SessionManager-TimeoutPoller prio=10 tid=0x7b689c00 nid=0x375d waiting on 
 condition [0x7b0ad000..0x7b0ade70]
java.lang.Thread.State: WAITING (parking)
 at sun.misc.Unsafe.park(Native Method)
 - parking to wait for  0xf2a944d8 (a 
 java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
 at 
 java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807)
  [Acquire write lock] at 
  org.apache.avro.ipc.NettyTransceiver.disconnect(NettyTransceiver.java:285)
 at 
 org.apache.avro.ipc.NettyTransceiver.access$2(NettyTransceiver.java:281)
 at 
 org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.exceptionCaught(NettyTransceiver.java:499)
 at 
 org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
 at 
 org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.handleUpstream(NettyTransceiver.java:473)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783)
 at 
 org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:238)
 at 
 org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
 at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:432)
 at 
 org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:661)
 at 
 org.jboss.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:372)
 at 
 org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:117)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:771)
 at org.jboss.netty.channel.Channels.write(Channels.java:632)
 at 
 org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
 at org.jboss.netty.channel.Channels.write(Channels.java:611)
 at org.jboss.netty.channel.Channels.write(Channels.java:578)
 at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:251)
  [Acquire read lock] at 
  org.apache.avro.ipc.NettyTransceiver.writeDataPack(NettyTransceiver.java:413

[jira] [Commented] (AVRO-1027) NettyTransceiver will deadlock when attempting transceive/disconnect on the same thread

2012-03-01 Thread Simon Wilkinson (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13220341#comment-13220341
 ] 

Simon Wilkinson commented on AVRO-1027:
---

Unfortunately I don't have any time to work on a test until Monday.

 NettyTransceiver will deadlock when attempting transceive/disconnect on the 
 same thread
 ---

 Key: AVRO-1027
 URL: https://issues.apache.org/jira/browse/AVRO-1027
 Project: Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.6.1
Reporter: Simon Wilkinson
Assignee: James Baldassari
 Fix For: 1.6.3

 Attachments: AVRO-1027-v2.patch, AVRO-1027.patch


 If an Exception is caught while trying to write to a Channel, Netty can 
 deliver the Exception to a ChannelUpstreamHandler on the same thread that 
 attempted to write to the Channel. If this occurs with the 
 NettyClientAvroHandler implementation of ChannelUpstreamHandler then the 
 thread will deadlock.
 Specifically, NettyClientAvroHandler overrides the 
 ChannelUpstreamHandler.exceptionCaught() method to perform a disconnect, 
 which requires the NettyTransceiver's write lock. However, in the above 
 situation, the thread will already have locked the NettyTransceiver's read 
 lock to write to the Channel. ReentrantReadWriteLock does not allow upgrading 
 from a read to a write lock, hence the thread deadlocks.
 Example stack trace (simplified):
 SessionManager-TimeoutPoller prio=10 tid=0x7b689c00 nid=0x375d waiting on 
 condition [0x7b0ad000..0x7b0ade70]
java.lang.Thread.State: WAITING (parking)
 at sun.misc.Unsafe.park(Native Method)
 - parking to wait for  0xf2a944d8 (a 
 java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
 at 
 java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807)
  [Acquire write lock] at 
  org.apache.avro.ipc.NettyTransceiver.disconnect(NettyTransceiver.java:285)
 at 
 org.apache.avro.ipc.NettyTransceiver.access$2(NettyTransceiver.java:281)
 at 
 org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.exceptionCaught(NettyTransceiver.java:499)
 at 
 org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
 at 
 org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.handleUpstream(NettyTransceiver.java:473)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783)
 at 
 org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:238)
 at 
 org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
 at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:432)
 at 
 org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:661)
 at 
 org.jboss.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:372)
 at 
 org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:117)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:771)
 at org.jboss.netty.channel.Channels.write(Channels.java:632)
 at 
 org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
 at org.jboss.netty.channel.Channels.write(Channels.java:611)
 at org.jboss.netty.channel.Channels.write(Channels.java:578)
 at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:251)
  [Acquire read lock] at 
  org.apache.avro.ipc.NettyTransceiver.writeDataPack(NettyTransceiver.java:413)
  [Acquire read lock] at 
  org.apache.avro.ipc.NettyTransceiver.transceive

[jira] [Commented] (AVRO-1027) NettyTransceiver will deadlock when attempting transceive/disconnect on the same thread

2012-03-01 Thread Doug Cutting (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13220363#comment-13220363
 ] 

Doug Cutting commented on AVRO-1027:


Should we:
- a. commit this without tests and roll a 1.6.3 RC today
- b. hold off on 1.6.3 until this has tests next week
- c. roll a 1.6.3 RC today without this

I don't have a strong opinion.

 NettyTransceiver will deadlock when attempting transceive/disconnect on the 
 same thread
 ---

 Key: AVRO-1027
 URL: https://issues.apache.org/jira/browse/AVRO-1027
 Project: Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.6.1
Reporter: Simon Wilkinson
Assignee: James Baldassari
 Fix For: 1.6.3

 Attachments: AVRO-1027-v2.patch, AVRO-1027.patch


 If an Exception is caught while trying to write to a Channel, Netty can 
 deliver the Exception to a ChannelUpstreamHandler on the same thread that 
 attempted to write to the Channel. If this occurs with the 
 NettyClientAvroHandler implementation of ChannelUpstreamHandler then the 
 thread will deadlock.
 Specifically, NettyClientAvroHandler overrides the 
 ChannelUpstreamHandler.exceptionCaught() method to perform a disconnect, 
 which requires the NettyTransceiver's write lock. However, in the above 
 situation, the thread will already have locked the NettyTransceiver's read 
 lock to write to the Channel. ReentrantReadWriteLock does not allow upgrading 
 from a read to a write lock, hence the thread deadlocks.
 Example stack trace (simplified):
 SessionManager-TimeoutPoller prio=10 tid=0x7b689c00 nid=0x375d waiting on 
 condition [0x7b0ad000..0x7b0ade70]
java.lang.Thread.State: WAITING (parking)
 at sun.misc.Unsafe.park(Native Method)
 - parking to wait for  0xf2a944d8 (a 
 java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
 at 
 java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807)
  [Acquire write lock] at 
  org.apache.avro.ipc.NettyTransceiver.disconnect(NettyTransceiver.java:285)
 at 
 org.apache.avro.ipc.NettyTransceiver.access$2(NettyTransceiver.java:281)
 at 
 org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.exceptionCaught(NettyTransceiver.java:499)
 at 
 org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
 at 
 org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.handleUpstream(NettyTransceiver.java:473)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783)
 at 
 org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:238)
 at 
 org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
 at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:432)
 at 
 org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:661)
 at 
 org.jboss.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:372)
 at 
 org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:117)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:771)
 at org.jboss.netty.channel.Channels.write(Channels.java:632)
 at 
 org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
 at org.jboss.netty.channel.Channels.write(Channels.java:611)
 at org.jboss.netty.channel.Channels.write(Channels.java:578)
 at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:251)
  [Acquire read lock] at 
  org.apache.avro.ipc.NettyTransceiver.writeDataPack

[jira] [Commented] (AVRO-1027) NettyTransceiver will deadlock when attempting transceive/disconnect on the same thread

2012-03-01 Thread James Baldassari (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13220399#comment-13220399
 ] 

James Baldassari commented on AVRO-1027:


Creating a test that reliably triggers these types of deadlocks can be tricky.  
I fixed a bug not too long ago that I could reproduce reliably with my own unit 
tests but couldn't seem to get it to happen in the Avro tests for whatever 
reason.  I think this is a fairly low-risk change, and it would be unfortunate 
if people hit this in production.  So I guess my vote would be a +0 on getting 
this in for 1.6.3 whether the RC gets rolled today or early next week.

I could build a snapshot off trunk + AVRO-1027 and run it through my own suite 
of Avro Netty unit tests.  That might at least give us some additional 
confidence that this won't introduce any regressions.

 NettyTransceiver will deadlock when attempting transceive/disconnect on the 
 same thread
 ---

 Key: AVRO-1027
 URL: https://issues.apache.org/jira/browse/AVRO-1027
 Project: Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.6.1
Reporter: Simon Wilkinson
Assignee: James Baldassari
 Fix For: 1.6.3

 Attachments: AVRO-1027-v2.patch, AVRO-1027.patch


 If an Exception is caught while trying to write to a Channel, Netty can 
 deliver the Exception to a ChannelUpstreamHandler on the same thread that 
 attempted to write to the Channel. If this occurs with the 
 NettyClientAvroHandler implementation of ChannelUpstreamHandler then the 
 thread will deadlock.
 Specifically, NettyClientAvroHandler overrides the 
 ChannelUpstreamHandler.exceptionCaught() method to perform a disconnect, 
 which requires the NettyTransceiver's write lock. However, in the above 
 situation, the thread will already have locked the NettyTransceiver's read 
 lock to write to the Channel. ReentrantReadWriteLock does not allow upgrading 
 from a read to a write lock, hence the thread deadlocks.
 Example stack trace (simplified):
 SessionManager-TimeoutPoller prio=10 tid=0x7b689c00 nid=0x375d waiting on 
 condition [0x7b0ad000..0x7b0ade70]
java.lang.Thread.State: WAITING (parking)
 at sun.misc.Unsafe.park(Native Method)
 - parking to wait for  0xf2a944d8 (a 
 java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
 at 
 java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807)
  [Acquire write lock] at 
  org.apache.avro.ipc.NettyTransceiver.disconnect(NettyTransceiver.java:285)
 at 
 org.apache.avro.ipc.NettyTransceiver.access$2(NettyTransceiver.java:281)
 at 
 org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.exceptionCaught(NettyTransceiver.java:499)
 at 
 org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
 at 
 org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.handleUpstream(NettyTransceiver.java:473)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783)
 at 
 org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:238)
 at 
 org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
 at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:432)
 at 
 org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:661)
 at 
 org.jboss.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:372)
 at 
 org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:117)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:771)
 at org.jboss.netty.channel.Channels.write(Channels.java:632)
 at 
 org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70

[jira] [Commented] (AVRO-1027) NettyTransceiver will deadlock when attempting transceive/disconnect on the same thread

2012-02-29 Thread Simon Wilkinson (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13219242#comment-13219242
 ] 

Simon Wilkinson commented on AVRO-1027:
---

I could trigger the bug as part of a larger test. I'll see if I can extract 
this into a specific unit test.

 NettyTransceiver will deadlock when attempting transceive/disconnect on the 
 same thread
 ---

 Key: AVRO-1027
 URL: https://issues.apache.org/jira/browse/AVRO-1027
 Project: Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.6.1
Reporter: Simon Wilkinson
Assignee: James Baldassari
 Fix For: 1.6.3

 Attachments: AVRO-1027-v2.patch, AVRO-1027.patch


 If an Exception is caught while trying to write to a Channel, Netty can 
 deliver the Exception to a ChannelUpstreamHandler on the same thread that 
 attempted to write to the Channel. If this occurs with the 
 NettyClientAvroHandler implementation of ChannelUpstreamHandler then the 
 thread will deadlock.
 Specifically, NettyClientAvroHandler overrides the 
 ChannelUpstreamHandler.exceptionCaught() method to perform a disconnect, 
 which requires the NettyTransceiver's write lock. However, in the above 
 situation, the thread will already have locked the NettyTransceiver's read 
 lock to write to the Channel. ReentrantReadWriteLock does not allow upgrading 
 from a read to a write lock, hence the thread deadlocks.
 Example stack trace (simplified):
 SessionManager-TimeoutPoller prio=10 tid=0x7b689c00 nid=0x375d waiting on 
 condition [0x7b0ad000..0x7b0ade70]
java.lang.Thread.State: WAITING (parking)
 at sun.misc.Unsafe.park(Native Method)
 - parking to wait for  0xf2a944d8 (a 
 java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
 at 
 java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807)
  [Acquire write lock] at 
  org.apache.avro.ipc.NettyTransceiver.disconnect(NettyTransceiver.java:285)
 at 
 org.apache.avro.ipc.NettyTransceiver.access$2(NettyTransceiver.java:281)
 at 
 org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.exceptionCaught(NettyTransceiver.java:499)
 at 
 org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
 at 
 org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.handleUpstream(NettyTransceiver.java:473)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783)
 at 
 org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:238)
 at 
 org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
 at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:432)
 at 
 org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:661)
 at 
 org.jboss.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:372)
 at 
 org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:117)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:771)
 at org.jboss.netty.channel.Channels.write(Channels.java:632)
 at 
 org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
 at org.jboss.netty.channel.Channels.write(Channels.java:611)
 at org.jboss.netty.channel.Channels.write(Channels.java:578)
 at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:251)
  [Acquire read lock] at 
  org.apache.avro.ipc.NettyTransceiver.writeDataPack(NettyTransceiver.java:413)
  [Acquire read lock

[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas

2012-02-29 Thread Raymie Stata (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13219815#comment-13219815
 ] 

Raymie Stata commented on AVRO-1006:


On the one hand, the Parsing Canonical Form as I've defined it has important 
uses and I think should be part of the Avro system.

On the other hand, it's a good point that there may be additional canonical 
forms that satisfy different needs in the future.  The class- and method-names 
of the previous patch implicitly assumed a single canonical form.  I've 
modified the patch with new names that will better support the addition of new 
canonical forms in the future.  (I think I also fixed the Forrest problems, 
although I haven't been able to verify.)

 Fingerprints for Avro Schemas
 -

 Key: AVRO-1006
 URL: https://issues.apache.org/jira/browse/AVRO-1006
 Project: Avro
  Issue Type: New Feature
  Components: java
Reporter: Raymie Stata
Assignee: Raymie Stata
  Labels: features
 Attachments: AVRO-1006-prelim.patch, AVRO-1006.patch, 
 AVRO-1006.patch, AVRO-1006.patch, schema-fingerprinting.html, 
 schema-fingerprinting.html, schema-fingerprinting.html


 Add function that returns a standardized, 64-bit fingerprint for schemas.  
 Fingerprints are designed such that the chances of collisions is very, very 
 low.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas

2012-02-29 Thread graham sanderson (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13219859#comment-13219859
 ] 

graham sanderson commented on AVRO-1006:


Raymie, thank you; I think we're in agreement. However the latest patch has 
class SchemaNormalization with static toParsingForm and static build methods

the other static methods reference these specific versions directly.

Perhaps it would be best (since the guts of the parsing form are in the current 
build method) to leave SchemaNormalization as a class of static methods, but 
pass a Normalizer instance to most of the static method... in this case the 
current build method would basically be the guts of the ParsingFormNormalizer 
class

Note the common Normalizer instances would likely be sensibly named singletons 
in the SchemaNormalizer class

 Fingerprints for Avro Schemas
 -

 Key: AVRO-1006
 URL: https://issues.apache.org/jira/browse/AVRO-1006
 Project: Avro
  Issue Type: New Feature
  Components: java
Reporter: Raymie Stata
Assignee: Raymie Stata
  Labels: features
 Attachments: AVRO-1006-prelim.patch, AVRO-1006.patch, 
 AVRO-1006.patch, AVRO-1006.patch, schema-fingerprinting.html, 
 schema-fingerprinting.html, schema-fingerprinting.html


 Add function that returns a standardized, 64-bit fingerprint for schemas.  
 Fingerprints are designed such that the chances of collisions is very, very 
 low.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-1027) NettyTransceiver will deadlock when attempting transceive/disconnect on the same thread

2012-02-28 Thread James Baldassari (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13218944#comment-13218944
 ] 

James Baldassari commented on AVRO-1027:


Simon, do you have a test for this?  If not, I'll try to make one.  Not sure 
when I'll have time to work on it, though.

 NettyTransceiver will deadlock when attempting transceive/disconnect on the 
 same thread
 ---

 Key: AVRO-1027
 URL: https://issues.apache.org/jira/browse/AVRO-1027
 Project: Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.6.1
Reporter: Simon Wilkinson
Assignee: James Baldassari
 Fix For: 1.6.3

 Attachments: AVRO-1027-v2.patch, AVRO-1027.patch


 If an Exception is caught while trying to write to a Channel, Netty can 
 deliver the Exception to a ChannelUpstreamHandler on the same thread that 
 attempted to write to the Channel. If this occurs with the 
 NettyClientAvroHandler implementation of ChannelUpstreamHandler then the 
 thread will deadlock.
 Specifically, NettyClientAvroHandler overrides the 
 ChannelUpstreamHandler.exceptionCaught() method to perform a disconnect, 
 which requires the NettyTransceiver's write lock. However, in the above 
 situation, the thread will already have locked the NettyTransceiver's read 
 lock to write to the Channel. ReentrantReadWriteLock does not allow upgrading 
 from a read to a write lock, hence the thread deadlocks.
 Example stack trace (simplified):
 SessionManager-TimeoutPoller prio=10 tid=0x7b689c00 nid=0x375d waiting on 
 condition [0x7b0ad000..0x7b0ade70]
java.lang.Thread.State: WAITING (parking)
 at sun.misc.Unsafe.park(Native Method)
 - parking to wait for  0xf2a944d8 (a 
 java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
 at 
 java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807)
  [Acquire write lock] at 
  org.apache.avro.ipc.NettyTransceiver.disconnect(NettyTransceiver.java:285)
 at 
 org.apache.avro.ipc.NettyTransceiver.access$2(NettyTransceiver.java:281)
 at 
 org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.exceptionCaught(NettyTransceiver.java:499)
 at 
 org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
 at 
 org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.handleUpstream(NettyTransceiver.java:473)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783)
 at 
 org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:238)
 at 
 org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
 at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:432)
 at 
 org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:661)
 at 
 org.jboss.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:372)
 at 
 org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:117)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:771)
 at org.jboss.netty.channel.Channels.write(Channels.java:632)
 at 
 org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
 at org.jboss.netty.channel.Channels.write(Channels.java:611)
 at org.jboss.netty.channel.Channels.write(Channels.java:578)
 at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:251)
  [Acquire read lock] at 
  org.apache.avro.ipc.NettyTransceiver.writeDataPack(NettyTransceiver.java:413)
  [Acquire read lock

[jira] [Commented] (AVRO-1027) NettyTransceiver will deadlock when attempting transceive/disconnect on the same thread

2012-02-26 Thread Scott Carey (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13216980#comment-13216980
 ] 

Scott Carey commented on AVRO-1027:
---

+1
Looks good.  Is there a test that could cover this?

 NettyTransceiver will deadlock when attempting transceive/disconnect on the 
 same thread
 ---

 Key: AVRO-1027
 URL: https://issues.apache.org/jira/browse/AVRO-1027
 Project: Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.6.1
Reporter: Simon Wilkinson
Assignee: James Baldassari
 Fix For: 1.6.3

 Attachments: AVRO-1027-v2.patch, AVRO-1027.patch


 If an Exception is caught while trying to write to a Channel, Netty can 
 deliver the Exception to a ChannelUpstreamHandler on the same thread that 
 attempted to write to the Channel. If this occurs with the 
 NettyClientAvroHandler implementation of ChannelUpstreamHandler then the 
 thread will deadlock.
 Specifically, NettyClientAvroHandler overrides the 
 ChannelUpstreamHandler.exceptionCaught() method to perform a disconnect, 
 which requires the NettyTransceiver's write lock. However, in the above 
 situation, the thread will already have locked the NettyTransceiver's read 
 lock to write to the Channel. ReentrantReadWriteLock does not allow upgrading 
 from a read to a write lock, hence the thread deadlocks.
 Example stack trace (simplified):
 SessionManager-TimeoutPoller prio=10 tid=0x7b689c00 nid=0x375d waiting on 
 condition [0x7b0ad000..0x7b0ade70]
java.lang.Thread.State: WAITING (parking)
 at sun.misc.Unsafe.park(Native Method)
 - parking to wait for  0xf2a944d8 (a 
 java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
 at 
 java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
 at 
 java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807)
  [Acquire write lock] at 
  org.apache.avro.ipc.NettyTransceiver.disconnect(NettyTransceiver.java:285)
 at 
 org.apache.avro.ipc.NettyTransceiver.access$2(NettyTransceiver.java:281)
 at 
 org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.exceptionCaught(NettyTransceiver.java:499)
 at 
 org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
 at 
 org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.handleUpstream(NettyTransceiver.java:473)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783)
 at 
 org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:238)
 at 
 org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
 at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:432)
 at 
 org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:661)
 at 
 org.jboss.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:372)
 at 
 org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:117)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:771)
 at org.jboss.netty.channel.Channels.write(Channels.java:632)
 at 
 org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
 at 
 org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
 at org.jboss.netty.channel.Channels.write(Channels.java:611)
 at org.jboss.netty.channel.Channels.write(Channels.java:578)
 at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:251)
  [Acquire read lock] at 
  org.apache.avro.ipc.NettyTransceiver.writeDataPack(NettyTransceiver.java:413)
  [Acquire read lock] at 
  org.apache.avro.ipc.NettyTransceiver.transceive(NettyTransceiver.java:394

[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas

2012-02-25 Thread graham sanderson (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13216629#comment-13216629
 ] 

graham sanderson commented on AVRO-1006:


A clarification, which addresses issues raised by Doug and Scott. The need I'm 
solving for is to capture that part of a writer's schema which a reader needs 
to read data. This is a relatively straight-forward notion of equivalence, 
and a very useful one. And the good news is that this notion of equivalence 
allows us to ignore many aspects of schemas (e.g., attributes, aliases, default 
values).

Perhaps this should be made clearer (when naming the class/method), I came 
across this feature because of a desire to hash/fingerprint avro schemas for 
messaging, and was seeing if there was already a util to do it. In my case I 
potentially might use custom properties on fields in the schema to indicate 
they are being transmitted using a certain named dictionary and thus in my case 
they affect the ability to interpret the message, so I'd rather stick with 
something that I can reliably use on the producer end to encode the entire 
state of the schema, rather than a particular well defined sub-set of the 
schema.

Note that (thanks to someone making Props a LinkedHashMap since the code base 
I'm using) and the particular implementation of Jackson, schema.toString() in 
the Java impl appears like it will be fine for my purposes, and if another 
language implementation happens to produce a different hash value I'm cool with 
that, as long as it is relatively stable; for example: 

SchemaInstance1 -toJson- string x
string x -fromJson- SchemInstance2 - toJson string y
string x and string y being equal seems a reasonable enough guarantee for me

 Fingerprints for Avro Schemas
 -

 Key: AVRO-1006
 URL: https://issues.apache.org/jira/browse/AVRO-1006
 Project: Avro
  Issue Type: New Feature
  Components: java
Reporter: Raymie Stata
Assignee: Raymie Stata
  Labels: features
 Attachments: AVRO-1006-prelim.patch, AVRO-1006.patch, 
 AVRO-1006.patch, schema-fingerprinting.html, schema-fingerprinting.html, 
 schema-fingerprinting.html


 Add function that returns a standardized, 64-bit fingerprint for schemas.  
 Fingerprints are designed such that the chances of collisions is very, very 
 low.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas

2012-02-25 Thread graham sanderson (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13216630#comment-13216630
 ] 

graham sanderson commented on AVRO-1006:


Note, more succinctly, given that I and other people in this issue have been 
discussing or inferring schema repositories, I think maybe it makes sense in 
general or at least as a separate issue to deal with a digest of the schema in 
its entirety, even if that results in a number of hash values mapping to 
equivalent schemas. Generally I think the use cases don't involve insane 
quantities of schemas in the first place (although I won't say that for 
certain, because the ability to define an anonymous schema on the fly for 
certain applications is rather appealing)

 Fingerprints for Avro Schemas
 -

 Key: AVRO-1006
 URL: https://issues.apache.org/jira/browse/AVRO-1006
 Project: Avro
  Issue Type: New Feature
  Components: java
Reporter: Raymie Stata
Assignee: Raymie Stata
  Labels: features
 Attachments: AVRO-1006-prelim.patch, AVRO-1006.patch, 
 AVRO-1006.patch, schema-fingerprinting.html, schema-fingerprinting.html, 
 schema-fingerprinting.html


 Add function that returns a standardized, 64-bit fingerprint for schemas.  
 Fingerprints are designed such that the chances of collisions is very, very 
 low.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas

2012-02-24 Thread Raymie Stata (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13215480#comment-13215480
 ] 

Raymie Stata commented on AVRO-1006:


 ['fp' or 'fingerprint']

I renamed classes to fingerprint, but left methods fp.  
SchemaFingerprint.fingerprint seems unnecessarily long...

 [put in o.a.avro]

Good idea -- I moved it.

 [missing javadoc]

Added

 [breaks Forrest]

I can't get Forrest running on my machine -- can you send me the errors?  (Or 
maybe just fix them?)

 Fingerprints for Avro Schemas
 -

 Key: AVRO-1006
 URL: https://issues.apache.org/jira/browse/AVRO-1006
 Project: Avro
  Issue Type: New Feature
  Components: java
Reporter: Raymie Stata
Assignee: Raymie Stata
  Labels: features
 Attachments: AVRO-1006-prelim.patch, AVRO-1006.patch, 
 AVRO-1006.patch, schema-fingerprinting.html, schema-fingerprinting.html, 
 schema-fingerprinting.html


 Add function that returns a standardized, 64-bit fingerprint for schemas.  
 Fingerprints are designed such that the chances of collisions is very, very 
 low.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-1036) IDL processing fails with multi-level nested imports

2012-02-23 Thread George Fletcher (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13214888#comment-13214888
 ] 

George Fletcher commented on AVRO-1036:
---

So the bug is that findFile() returns and InputStream and the Idl class 
constructor that takes an InputStream is created by the javacc processing of 
the Idl.jj file. This Idl constructor does not set the inputDir field of the 
Idl class and hence the Idl object created from the stream returned by 
findFile() has lost the sourceDirectory. This causes an import statement 
processed by this Idl object to fail, because the file can not be found as the 
sourceDirectory is lost.

I'm not very happy with my current solution as it seems a little hacky. 
Suggestions appreciated.

Current solution...

1. Add a setSrcDir() method to the Idl class (added to Idl.jj)
2. Modify the ImportIdl() method to first create the Idl instance from the 
InputStream returned by findFile() and then call the setSrcDir() setter 
chaining the current inputDir into the created Idl instance.

 IDL processing fails with multi-level nested imports
 

 Key: AVRO-1036
 URL: https://issues.apache.org/jira/browse/AVRO-1036
 Project: Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.6.2
Reporter: George Fletcher
 Fix For: 1.6.3


 The change to support finding IDL related files on the classpath in addition 
 to the maven-plugin defined directory caused the context of the 
 sourceDirectory to be lost when the InputStream return by findFile() is 
 used to create a new Idl instance.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-999) NPE in Java, RecordBuilderBase.defaultValue

2012-02-23 Thread Doug Cutting (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13214909#comment-13214909
 ] 

Doug Cutting commented on AVRO-999:
---

The test case added in this patch passes with the changes in AVRO-1007.  Also, 
the tests added in AVRO-1007 are substantially similar to the test added here.

 NPE in Java, RecordBuilderBase.defaultValue
 ---

 Key: AVRO-999
 URL: https://issues.apache.org/jira/browse/AVRO-999
 Project: Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.6.1
 Environment: Java
Reporter: Jay Rutten
Assignee: James Baldassari
 Attachments: AVRO-999.patch


 If you have a union with a default of null, the code in 
 RecordBuilderBase.defaultValue will cause an NPE in ConcurrentHashMap, since 
 it is trying to add a null to the map.
 Sample union:
 {code}
 record Sample {
 union{null, string} value = null;
 }
 {code}
 Code:
 {code}
 // If not cached, get the default Java value by encoding the default JSON
 // value and then decoding it:
 if (defaultValue == null) {
   ByteArrayOutputStream baos = new ByteArrayOutputStream();
   encoder = EncoderFactory.get().binaryEncoder(baos, encoder);
   ResolvingGrammarGenerator.encode(encoder, field.schema(), defaultJsonValue);
   encoder.flush();
   decoder = DecoderFactory.get().binaryDecoder(baos.toByteArray(), decoder);
   defaultValue = new GenericDatumReader(field.schema()).read(null, decoder);
   defaultSchemaValues.putIfAbsent(field.pos(), defaultValue); // -- NPE from 
 here
 }
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-999) NPE in Java, RecordBuilderBase.defaultValue

2012-02-23 Thread James Baldassari (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13214916#comment-13214916
 ] 

James Baldassari commented on AVRO-999:
---

OK, great.  Then it should be fine to resolve this one if the AVRO-1007 tests 
cover this case.

 NPE in Java, RecordBuilderBase.defaultValue
 ---

 Key: AVRO-999
 URL: https://issues.apache.org/jira/browse/AVRO-999
 Project: Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.6.1
 Environment: Java
Reporter: Jay Rutten
Assignee: James Baldassari
 Attachments: AVRO-999.patch


 If you have a union with a default of null, the code in 
 RecordBuilderBase.defaultValue will cause an NPE in ConcurrentHashMap, since 
 it is trying to add a null to the map.
 Sample union:
 {code}
 record Sample {
 union{null, string} value = null;
 }
 {code}
 Code:
 {code}
 // If not cached, get the default Java value by encoding the default JSON
 // value and then decoding it:
 if (defaultValue == null) {
   ByteArrayOutputStream baos = new ByteArrayOutputStream();
   encoder = EncoderFactory.get().binaryEncoder(baos, encoder);
   ResolvingGrammarGenerator.encode(encoder, field.schema(), defaultJsonValue);
   encoder.flush();
   decoder = DecoderFactory.get().binaryDecoder(baos.toByteArray(), decoder);
   defaultValue = new GenericDatumReader(field.schema()).read(null, decoder);
   defaultSchemaValues.putIfAbsent(field.pos(), defaultValue); // -- NPE from 
 here
 }
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas

2012-02-23 Thread Doug Cutting (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13214932#comment-13214932
 ] 

Doug Cutting commented on AVRO-1006:


This looks generally good.  A few nits:
 - In class and method names, should we abbreviate 'fp' or spell out 
'fingerprint'?  FP means floating point to my eye.
 - Might we instead put this in org.apache.avro.SchemaFingerprint, rather than 
in util?  Right now things in the util package depend only on the JDK, not on 
other parts of Avro.
 - Public methods and classes need javadoc comments.
 - The changes to the spec are not correctly processed by Forrest 0.8 for me.

 Fingerprints for Avro Schemas
 -

 Key: AVRO-1006
 URL: https://issues.apache.org/jira/browse/AVRO-1006
 Project: Avro
  Issue Type: New Feature
  Components: java
Reporter: Raymie Stata
Assignee: Raymie Stata
  Labels: features
 Attachments: AVRO-1006-prelim.patch, AVRO-1006.patch, 
 schema-fingerprinting.html, schema-fingerprinting.html, 
 schema-fingerprinting.html


 Add function that returns a standardized, 64-bit fingerprint for schemas.  
 Fingerprints are designed such that the chances of collisions is very, very 
 low.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-1036) IDL processing fails with multi-level nested imports

2012-02-23 Thread George Fletcher (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13215040#comment-13215040
 ] 

George Fletcher commented on AVRO-1036:
---

Decided a cleaner solution was to create a new constructor public Idl 
(InputStream, File, ClassLoader) that chains the source directory and classpath 
resource loader into the new Idl instance.

 IDL processing fails with multi-level nested imports
 

 Key: AVRO-1036
 URL: https://issues.apache.org/jira/browse/AVRO-1036
 Project: Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.6.2
Reporter: George Fletcher
 Fix For: 1.6.3


 The change to support finding IDL related files on the classpath in addition 
 to the maven-plugin defined directory caused the context of the 
 sourceDirectory to be lost when the InputStream return by findFile() is 
 used to create a new Idl instance.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-1036) IDL processing fails with multi-level nested imports

2012-02-23 Thread Doug Cutting (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13215125#comment-13215125
 ] 

Doug Cutting commented on AVRO-1036:


Looks like we patched this in parallel!  You added some tests, which is great!

My patch is a little different.  When you include a file in a different 
directory then any imports it contains should be relative to its directory, not 
the directory of the original file, no?  So this.inputDir is not the right 
value for the new inputDir, but rather it should come from the imported file.

 IDL processing fails with multi-level nested imports
 

 Key: AVRO-1036
 URL: https://issues.apache.org/jira/browse/AVRO-1036
 Project: Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.6.2
Reporter: George Fletcher
Assignee: Doug Cutting
 Fix For: 1.6.3

 Attachments: AVRO-1036.patch, jira-1036.patch


 The change to support finding IDL related files on the classpath in addition 
 to the maven-plugin defined directory caused the context of the 
 sourceDirectory to be lost when the InputStream return by findFile() is 
 used to create a new Idl instance.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-1037) Problems using Avro 1.6.2 with Hadoop (CDH3 or 1.0)

2012-02-23 Thread Scott Carey (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13215157#comment-13215157
 ] 

Scott Carey commented on AVRO-1037:
---

The stack trace I ran into was:

{noformat}
java.lang.NoSuchMethodError: org.codehaus.jackson.JsonNode.getValueAsDouble()D
at org.apache.avro.Schema$Field.defaultValueEquals(Schema.java:461)
at org.apache.avro.Schema$Field.equals(Schema.java:451)
at java.util.AbstractList.equals(AbstractList.java:507)
at org.apache.avro.Schema$RecordSchema.equals(Schema.java:647)
at java.util.HashMap.put(HashMap.java:376)
{noformat}

 Problems using Avro 1.6.2 with Hadoop (CDH3 or 1.0)
 ---

 Key: AVRO-1037
 URL: https://issues.apache.org/jira/browse/AVRO-1037
 Project: Avro
  Issue Type: Bug
  Components: java
Reporter: Scott Carey
Assignee: Scott Carey

 AVRO-972 uses a method on JsonNode that was introduced with Jackson 1.6.0.  
 Unfortunately, Hadoop in a distributed environment places jackson on the 
 mapreduce classpath prior to a user's Jar.
 Avro should function when this happens.
 There are two things we are using that are new in Jackson 1.6.0 -- 
 JsonNode.getValueAsDouble()  and MinimalPrettyPrinter
 MinimalPrettyPrinter is OK because the class is new, and if there is a 
 Jackson 1.5.x jar on a classpath before one that is = 1.6.0, it will find 
 the class in the second jar.  We need to avoid JsonNode.getValueAsDouble() 
 however.  There is a replacement that is equivalent and existed prior, 
 JsonNode.getDoubleValue().

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-1037) Problems using Avro 1.6.2 with Hadoop (CDH3 or 1.0)

2012-02-23 Thread Scott Carey (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13215173#comment-13215173
 ] 

Scott Carey commented on AVRO-1037:
---

If there are no objections, I will check this in tomorrow.  I would like to get 
a 1.6.3 with this in it out soon.

 Problems using Avro 1.6.2 with Hadoop (CDH3 or 1.0)
 ---

 Key: AVRO-1037
 URL: https://issues.apache.org/jira/browse/AVRO-1037
 Project: Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.6.2
Reporter: Scott Carey
Assignee: Scott Carey
 Fix For: 1.6.3

 Attachments: AVRO-1037.patch


 AVRO-972 uses a method on JsonNode that was introduced with Jackson 1.6.0.  
 Unfortunately, Hadoop in a distributed environment places jackson on the 
 mapreduce classpath prior to a user's Jar.
 Avro should function when this happens.
 There are two things we are using that are new in Jackson 1.6.0 -- 
 JsonNode.getValueAsDouble()  and MinimalPrettyPrinter
 MinimalPrettyPrinter is OK because the class is new, and if there is a 
 Jackson 1.5.x jar on a classpath before one that is = 1.6.0, it will find 
 the class in the second jar.  We need to avoid JsonNode.getValueAsDouble() 
 however.  There is a replacement that is equivalent and existed prior, 
 JsonNode.getDoubleValue().

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-1036) IDL processing fails with multi-level nested imports

2012-02-23 Thread George Fletcher (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13215214#comment-13215214
 ] 

George Fletcher commented on AVRO-1036:
---

Yes, that's a good point and I like your patch better. One other thought I had 
is that we could just add the sourceDirectory to the ClassLoader in the 
IDLProtocolMojo.java and then just use the ClassLoader mechanism across the 
board. 

 IDL processing fails with multi-level nested imports
 

 Key: AVRO-1036
 URL: https://issues.apache.org/jira/browse/AVRO-1036
 Project: Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.6.2
Reporter: George Fletcher
Assignee: Doug Cutting
 Fix For: 1.6.3

 Attachments: AVRO-1036.patch, jira-1036.patch


 The change to support finding IDL related files on the classpath in addition 
 to the maven-plugin defined directory caused the context of the 
 sourceDirectory to be lost when the InputStream return by findFile() is 
 used to create a new Idl instance.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-1036) IDL processing fails with multi-level nested imports

2012-02-23 Thread Scott Carey (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13215285#comment-13215285
 ] 

Scott Carey commented on AVRO-1036:
---

I do not see any tests in the last patch.

 IDL processing fails with multi-level nested imports
 

 Key: AVRO-1036
 URL: https://issues.apache.org/jira/browse/AVRO-1036
 Project: Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.6.2
Reporter: George Fletcher
Assignee: Doug Cutting
 Fix For: 1.6.3

 Attachments: AVRO-1036.patch, AVRO-1036.patch, jira-1036.patch


 The change to support finding IDL related files on the classpath in addition 
 to the maven-plugin defined directory caused the context of the 
 sourceDirectory to be lost when the InputStream return by findFile() is 
 used to create a new Idl instance.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-1029) I want to transfer files beetwin server and client by Avro-c.

2012-02-23 Thread menghaocheng (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13215334#comment-13215334
 ] 

menghaocheng commented on AVRO-1029:


Thanks Dounglas. 

I think you have got exactly what I meant. I know the step to do it.
1)got the binary encoding.
2)transmit them by TCP socket.
The 2nd step is easy. But there is a question to the 1st step. I didn't know 
how to get the binary encoding. The C API in the Documentation doesn't mention 
at all http://avro.apache.org/docs/1.5.4/api/c/index.html. So I have to read 
the rsc files. I find a function in the file src/avro/consumer.h:
avro_consume_binary(avro_reader_t reader,avro_consumer_t *consumer, void 
*ud);
I'm not sure if it is the right function that I need.

 I want to transfer files beetwin server and client by Avro-c.
 -

 Key: AVRO-1029
 URL: https://issues.apache.org/jira/browse/AVRO-1029
 Project: Avro
  Issue Type: Test
  Components: c
Affects Versions: 1.5.4
 Environment: avro in C, linux in vmware.
Reporter: menghaocheng
  Labels: test
 Fix For: 1.5.4

   Original Estimate: 168h
  Remaining Estimate: 168h

  I want to transfer files beetwin server and client by Avro-c. I know that 
 there must be two parts in this model, the server and the client. My question 
 is how to make the connection? Please forgive me as this maybe a simle 
 question, But I'm fresh in Avro, your simple worlds maybe helpful for me. If 
 only there is an example for it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-593) Avro mapreduce apis incompatible with hadoop 0.20.2

2012-02-22 Thread Doug Cutting (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13213866#comment-13213866
 ] 

Doug Cutting commented on AVRO-593:
---

 Ideally, anything in the .io, .util, and .file packages does not reference 
 the .mapred or .mapreduce packages [ ... ]

Much in these packages references AvroKey and AvroValue and/or AvroJob.  These 
uses aren't mapreduce-specific and could be refactored away, e.g., by moving 
AvroKey and AvroValue from o.a.a.mapred to o.a.a.hadoop.io, but that would be 
incompatible.

SortedKeyValueFile is the Avro equivalent of Hadoop's MapFile.  Arguably it 
should be moved into o.a.a.io.  It depends on AvroKeyValue, which might also be 
moved to the core.  AvroKeyValue is very similar in functionality to 
o.a.a.mapred.Pair.  Perhaps SortedKeyValueFile should be switched to use Pair 
and both moved to the core.

I have implemented a SequenceFile shim and it works.  There's now just a tiny 
class that needs to be in o.a.h.io, a base class that exposes two 
package-private nested classes from within SequenceFile.

I've re-arranged the classes per Scott's #4 variant but can revert that.  We 
need to decide how much refactoring we want to do here.

Finally, I note that io.SeekableHadoopInput replicates functionality that's 
already in mapred.FsInput, so we should replace the former with the latter in 
the new code.

 Avro mapreduce apis incompatible with hadoop 0.20.2
 ---

 Key: AVRO-593
 URL: https://issues.apache.org/jira/browse/AVRO-593
 Project: Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.3.2, 1.3.3
 Environment: Avro 1.3.3, Hadoop 0.20.2
Reporter: Steve Severance
Assignee: Garrett Wu
 Attachments: AVRO-593.patch, AVRO-593.patch


 The avro api's for hadoop use the hadoop mapreduce api that has been 
 deprecated. A new avro mapreduce api should be implemented for hadoop 0.20 
 and higher.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-1035) Add the possibility to append to existing avro files

2012-02-22 Thread Doug Cutting (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13213874#comment-13213874
 ] 

Doug Cutting commented on AVRO-1035:


Note that append is not reliable in current Hadoop releases.  Append support 
in Hadoop 1.0 just means that flush() works reliably, not that append actually 
works.  Append should be reliable in 0.23 releases although I doubt it's been 
well tested there yet.

 Add the possibility to append to existing avro files  
 --

 Key: AVRO-1035
 URL: https://issues.apache.org/jira/browse/AVRO-1035
 Project: Avro
  Issue Type: New Feature
Reporter: Vyacheslav Zholudev

 Currently it is not possible to append to avro files that were written and 
 closed. 
 Here is a Scott Carey's reply on the mailing list:
 {quote}
 It is not possible without modifying DataFileWriter. Please open a JIRA
 ticket.  
 It could not simply append to an OutputStream, since it must either:
 * Seek to the start to validate the schemas match and find the sync
 marker, or
 * Trust that the schemas match and find the sync marker from the last block
 DataFileWriter cannot refer to Hadoop classes such as FileSystem, but we
 could add something to the mapred module that takes a Path and FileSystem
 and returns
 something that implemements an interface that DataFileWriter can append
 to.  This would be something that is both a
 http://avro.apache.org/docs/1.6.2/api/java/org/apache/avro/file/SeekableInp
 ut.html
 and an OutputStream, or has both an InputStream from the start of the
 existing file and an OutputStream at the end.
 {quote}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-1023) Saved state should be restored in finally clause

2012-02-22 Thread Doug Cutting (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13214094#comment-13214094
 ] 

Doug Cutting commented on AVRO-1023:


The way this might be triggered is if a Schema.Parser is reused after a 
SchemaParseException is thrown.  Currently the default namespace is that of the 
preceding schema parsed.  If a SchemaParseException is thrown and the parser is 
reused then the default namespace could be that of a schema nested within the 
previous schema.

 Saved state should be restored in finally clause
 

 Key: AVRO-1023
 URL: https://issues.apache.org/jira/browse/AVRO-1023
 Project: Avro
  Issue Type: Bug
  Components: java
Reporter: Raymie Stata
Assignee: Raymie Stata
Priority: Minor
 Attachments: AVRO-1023.patch


 Schema.parse(JsonParse) and Schema.parse(JsonNode,Names) save global state in 
 a local variable; they should restore that state in a finally clause but they 
 don't.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-999) NPE in Java, RecordBuilderBase.defaultValue

2012-02-22 Thread Doug Cutting (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13214103#comment-13214103
 ] 

Doug Cutting commented on AVRO-999:
---

I think this was already fixed in AVRO-1007.  Can I close it as a duplicate?

 NPE in Java, RecordBuilderBase.defaultValue
 ---

 Key: AVRO-999
 URL: https://issues.apache.org/jira/browse/AVRO-999
 Project: Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.6.1
 Environment: Java
Reporter: Jay Rutten
Assignee: James Baldassari
 Attachments: AVRO-999.patch


 If you have a union with a default of null, the code in 
 RecordBuilderBase.defaultValue will cause an NPE in ConcurrentHashMap, since 
 it is trying to add a null to the map.
 Sample union:
 {code}
 record Sample {
 union{null, string} value = null;
 }
 {code}
 Code:
 {code}
 // If not cached, get the default Java value by encoding the default JSON
 // value and then decoding it:
 if (defaultValue == null) {
   ByteArrayOutputStream baos = new ByteArrayOutputStream();
   encoder = EncoderFactory.get().binaryEncoder(baos, encoder);
   ResolvingGrammarGenerator.encode(encoder, field.schema(), defaultJsonValue);
   encoder.flush();
   decoder = DecoderFactory.get().binaryDecoder(baos.toByteArray(), decoder);
   defaultValue = new GenericDatumReader(field.schema()).read(null, decoder);
   defaultSchemaValues.putIfAbsent(field.pos(), defaultValue); // -- NPE from 
 here
 }
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-999) NPE in Java, RecordBuilderBase.defaultValue

2012-02-22 Thread James Baldassari (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13214263#comment-13214263
 ] 

James Baldassari commented on AVRO-999:
---

I think AVRO-999 is slightly different from AVRO-1007 as AVRO-999 is 
specifically related to unions with null types.  I'm not positive, though.  
Either way I think we should commit the new test case this patch introduces to 
TestGenericRecordBuilder.  That should verify whether the changes in AVRO-1007 
actually fix this problem.

 NPE in Java, RecordBuilderBase.defaultValue
 ---

 Key: AVRO-999
 URL: https://issues.apache.org/jira/browse/AVRO-999
 Project: Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.6.1
 Environment: Java
Reporter: Jay Rutten
Assignee: James Baldassari
 Attachments: AVRO-999.patch


 If you have a union with a default of null, the code in 
 RecordBuilderBase.defaultValue will cause an NPE in ConcurrentHashMap, since 
 it is trying to add a null to the map.
 Sample union:
 {code}
 record Sample {
 union{null, string} value = null;
 }
 {code}
 Code:
 {code}
 // If not cached, get the default Java value by encoding the default JSON
 // value and then decoding it:
 if (defaultValue == null) {
   ByteArrayOutputStream baos = new ByteArrayOutputStream();
   encoder = EncoderFactory.get().binaryEncoder(baos, encoder);
   ResolvingGrammarGenerator.encode(encoder, field.schema(), defaultJsonValue);
   encoder.flush();
   decoder = DecoderFactory.get().binaryDecoder(baos.toByteArray(), decoder);
   defaultValue = new GenericDatumReader(field.schema()).read(null, decoder);
   defaultSchemaValues.putIfAbsent(field.pos(), defaultValue); // -- NPE from 
 here
 }
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-593) Avro mapreduce apis incompatible with hadoop 0.20.2

2012-02-21 Thread Doug Cutting (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13212819#comment-13212819
 ] 

Doug Cutting commented on AVRO-593:
---

I see a few choices:

1. org.apache.avro.{mapred,mapreduce,io,file,util}.  This is what the code on 
github does.  This would make the avro-mapred module contain things outside the 
org.apache.avro.mapred package, and splits Avro's io, file and util packages 
across multiple modules.

2. org.apache.avro.mapred.{mapreduce,io,file,util}.  This is what my patch 
does.  This is back-compatible and consistent with the module name, but places 
mapreduce under mapred, which is different than the Hadoop layout.

3. org.apache.avro.hadoop.{mapred,mapreduce,io,file,util}.  We'd rename the 
module to be avro-hadoop.  This would be incompatible but consistent with 
Hadoop.  For back-compatibility we might leave the mapred classes in their 
current package.

4. org.apache.avro.{mapred,mapreduce,mapred.io,mapred.file,mapred.util}.  This 
is back-compatible but includes a package that's not under the package of the 
module name.

Tom, are you advocating for (4)?  I'd be okay with that, I guess.

I'm also leaning towards moving AvroSequenceFile under org.apache.avro and 
adding just a shim base class into org.apache.hadoop.io that subclasses 
SequenceFile and makes public the bits we need.  That way if we get Hadoop to 
expose these bits the Avro API would not change.




 Avro mapreduce apis incompatible with hadoop 0.20.2
 ---

 Key: AVRO-593
 URL: https://issues.apache.org/jira/browse/AVRO-593
 Project: Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.3.2, 1.3.3
 Environment: Avro 1.3.3, Hadoop 0.20.2
Reporter: Steve Severance
Assignee: Garrett Wu
 Attachments: AVRO-593.patch, AVRO-593.patch


 The avro api's for hadoop use the hadoop mapreduce api that has been 
 deprecated. A new avro mapreduce api should be implemented for hadoop 0.20 
 and higher.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-593) Avro mapreduce apis incompatible with hadoop 0.20.2

2012-02-21 Thread Scott Carey (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13213012#comment-13213012
 ] 

Scott Carey commented on AVRO-593:
--

Re #1: its OK to have multiple packages in a single maven module, but not good 
to have a package split across modules as it causes problems for OSGi and in 
the future, Java 8 modules.

Re #2: This is OK, but a little confusing. Also, if we ever wanted to break 
apart the mapred module in to two or three (e.g. avro-hadoop, avro-mapred, 
avro-mapreduce with the common stuff in avro-hadoop and the two APIs in the 
others) it will be less consistent.

Re #3: This is fairly clean, but is incompatible.  

Re #4: This is decent, but I would propose: 
org.apache.avro.{hadoop,mapreduce,mapred,hadoop.io,hadoop.file,hadoop.util}.  
Then the current module would have o.a.a.{hadoop,mapreduce,mapred} and children 
packages.  A future split could divide on these cleanly.   One reason to split 
in the future is that some users may want hadoop stuff that is not related to 
mapreduce -- sequence files, avro data file access via FileSytem+Path, etc.  If 
we split the module, avoiding moving classes around is important.

Is it possible to move AvroSequenceFile under o.a.a ?  All classes in that 
package need to be in the base avro maven module, and cannot depend on any 
hadoop APIs.

 Avro mapreduce apis incompatible with hadoop 0.20.2
 ---

 Key: AVRO-593
 URL: https://issues.apache.org/jira/browse/AVRO-593
 Project: Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.3.2, 1.3.3
 Environment: Avro 1.3.3, Hadoop 0.20.2
Reporter: Steve Severance
Assignee: Garrett Wu
 Attachments: AVRO-593.patch, AVRO-593.patch


 The avro api's for hadoop use the hadoop mapreduce api that has been 
 deprecated. A new avro mapreduce api should be implemented for hadoop 0.20 
 and higher.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-593) Avro mapreduce apis incompatible with hadoop 0.20.2

2012-02-21 Thread Scott Carey (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13213018#comment-13213018
 ] 

Scott Carey commented on AVRO-593:
--

We also need to consider if we need to produce two otherwise identical modules 
in a build -- one 0.23.x + compatible and one for the 0.20 / 0.22 / 1.0 users.  
My understanding is that one needs to compile against 0.23.x to work properly 
there.  Organizing the modules so that it is possible to produce an Avro 
release that supports multiple Hadoop variants would be useful.

 Avro mapreduce apis incompatible with hadoop 0.20.2
 ---

 Key: AVRO-593
 URL: https://issues.apache.org/jira/browse/AVRO-593
 Project: Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.3.2, 1.3.3
 Environment: Avro 1.3.3, Hadoop 0.20.2
Reporter: Steve Severance
Assignee: Garrett Wu
 Attachments: AVRO-593.patch, AVRO-593.patch


 The avro api's for hadoop use the hadoop mapreduce api that has been 
 deprecated. A new avro mapreduce api should be implemented for hadoop 0.20 
 and higher.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-593) Avro mapreduce apis incompatible with hadoop 0.20.2

2012-02-21 Thread Doug Cutting (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13213071#comment-13213071
 ] 

Doug Cutting commented on AVRO-593:
---

 Is it possible to move AvroSequenceFile under o.a.a ?

I discussed that above.  We could move it, but we'd still need a shim in 
o.a.h.io, since the subclass accesses package-private bits.

 if we need to produce two otherwise identical modules in a build – one 0.23.x 
 + compatible and one for the 0.20 / 0.22 / 1.0 users

The nested Context classes in mapreduce's Mapper and Reducer went from abstract 
classes to interfaces (MAPREDUCE-954), requiring re-compilation of code that 
references these.  But the mapreduce support added here does not reference 
these.  So I think we're spared.

 Avro mapreduce apis incompatible with hadoop 0.20.2
 ---

 Key: AVRO-593
 URL: https://issues.apache.org/jira/browse/AVRO-593
 Project: Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.3.2, 1.3.3
 Environment: Avro 1.3.3, Hadoop 0.20.2
Reporter: Steve Severance
Assignee: Garrett Wu
 Attachments: AVRO-593.patch, AVRO-593.patch


 The avro api's for hadoop use the hadoop mapreduce api that has been 
 deprecated. A new avro mapreduce api should be implemented for hadoop 0.20 
 and higher.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-593) Avro mapreduce apis incompatible with hadoop 0.20.2

2012-02-21 Thread Tom White (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13213141#comment-13213141
 ] 

Tom White commented on AVRO-593:


+1 to Scott's #4 variant.

 Avro mapreduce apis incompatible with hadoop 0.20.2
 ---

 Key: AVRO-593
 URL: https://issues.apache.org/jira/browse/AVRO-593
 Project: Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.3.2, 1.3.3
 Environment: Avro 1.3.3, Hadoop 0.20.2
Reporter: Steve Severance
Assignee: Garrett Wu
 Attachments: AVRO-593.patch, AVRO-593.patch


 The avro api's for hadoop use the hadoop mapreduce api that has been 
 deprecated. A new avro mapreduce api should be implemented for hadoop 0.20 
 and higher.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-593) Avro mapreduce apis incompatible with hadoop 0.20.2

2012-02-21 Thread Scott Carey (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13213260#comment-13213260
 ] 

Scott Carey commented on AVRO-593:
--

{quote}I discussed that above. We could move it, but we'd still need a shim in 
o.a.h.io, since the subclass accesses package-private bits.{quote}

Let me clarify:  Is it possible for AvroSequenceFile to not reference anything 
in o.a.hadoop.** or o.a.a.{mapreduce, hadoop, mapred}.** ? 

It has:
{code}
import org.apache.avro.mapred.AvroKey;
import org.apache.avro.mapred.AvroValue;
{code}

which would indicate to me that it must be in o.a.a.mapred.

If it is in o.a.a it must not reference any classes that don't exist in the 
base avro module that encompases o.a.a (lang/java/avro)


Ideally, anything in the .io, .util, and .file packages does not reference the 
.mapred or .mapreduce packages, so that this can be packaged as a standalone 
hadoop dependency down the road.  I have not looked at all those yet to see 
what the package dependencies are.

 Avro mapreduce apis incompatible with hadoop 0.20.2
 ---

 Key: AVRO-593
 URL: https://issues.apache.org/jira/browse/AVRO-593
 Project: Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.3.2, 1.3.3
 Environment: Avro 1.3.3, Hadoop 0.20.2
Reporter: Steve Severance
Assignee: Garrett Wu
 Attachments: AVRO-593.patch, AVRO-593.patch


 The avro api's for hadoop use the hadoop mapreduce api that has been 
 deprecated. A new avro mapreduce api should be implemented for hadoop 0.20 
 and higher.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-593) Avro mapreduce apis incompatible with hadoop 0.20.2

2012-02-20 Thread Tom White (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13212361#comment-13212361
 ] 

Tom White commented on AVRO-593:


 I renamed all of the packages to reside under org.apache.avro.mapred. So that 
 package now has subpackages named io, file, util and mapreduce.

Keeping the package org.apache.avro.mapreduce would be more consistent with 
Hadoop, which has the mapred/mapreduce distinction.

 Avro mapreduce apis incompatible with hadoop 0.20.2
 ---

 Key: AVRO-593
 URL: https://issues.apache.org/jira/browse/AVRO-593
 Project: Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.3.2, 1.3.3
 Environment: Avro 1.3.3, Hadoop 0.20.2
Reporter: Steve Severance
Assignee: Garrett Wu
 Attachments: AVRO-593.patch, AVRO-593.patch


 The avro api's for hadoop use the hadoop mapreduce api that has been 
 deprecated. A new avro mapreduce api should be implemented for hadoop 0.20 
 and higher.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-1029) I want to transfer files beetwin server and client by Avro-c.

2012-02-17 Thread Douglas Creager (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13210257#comment-13210257
 ] 

Douglas Creager commented on AVRO-1029:
---

Hi there.  It sounds like you're trying to use the RPC mechanism defined in the 
Avro spec, is that right?  Unfortunately, the Avro C bindings don't support the 
RPC mechanism yet.  The C bindings will let you create instances of an Avro 
schema, and you can create the binary encoding of those instances.  But then 
you'd be responsible for transmitting the binary encoding using some other 
library, like ZeroMQ or raw TCP sockets.

I hope this helps, please let me know if you have any further questions.

 I want to transfer files beetwin server and client by Avro-c.
 -

 Key: AVRO-1029
 URL: https://issues.apache.org/jira/browse/AVRO-1029
 Project: Avro
  Issue Type: Test
  Components: c
Affects Versions: 1.5.4
 Environment: avro in C, linux in vmware.
Reporter: menghaocheng
  Labels: test
 Fix For: 1.5.4

   Original Estimate: 168h
  Remaining Estimate: 168h

  I want to transfer files beetwin server and client by Avro-c. I know that 
 there must be two parts in this model, the server and the client. My question 
 is how to make the connection? Please forgive me as this maybe a simle 
 question, But I'm fresh in Avro, your simple worlds maybe helpful for me. If 
 only there is an example for it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-672) Convert JSON Text Input to Avro Tool

2012-02-17 Thread Doug Cutting (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13210586#comment-13210586
 ] 

Doug Cutting commented on AVRO-672:
---

Leith, is the tool that Ron provided here the one you need?  If so, then we can 
probably resuscitate this patch and get it committed.  If not, is there a 
specific tool you need (e.g., CSV or TSV)?  Thanks!

 Convert JSON Text Input to Avro Tool
 

 Key: AVRO-672
 URL: https://issues.apache.org/jira/browse/AVRO-672
 Project: Avro
  Issue Type: New Feature
  Components: java
Reporter: Ron Bodkin
 Attachments: AVRO-672.patch, AVRO-672.patch


 The attached patch allows reading a JSON-formatted text file in, converting 
 to a conforming Avro text file, emitting one record per line, e.g., it can 
 read this input file:
 {intval:12}
 {intval:-73,strval:hello, there!!}
 with this schema:
 { type:record, name:TestRecord, fields: [ 
 {name:intval,type:int}, {name:strval,type:[string, null]}]}
 returning valid Avro. This is different than the DataFileWriteTool, which 
 would read in the following internal encoding:
 {intval:12,strval:null}
 {intval:-73,strval:{string:hello, there!!}}
 In general, the internal encodings used by Avro aren't natural when reading 
 in JSON text that appears in the wild. Likewise, this utility allows changing 
 invalid Avro identifier characters into an underscore, again to tolerate JSON 
 that wasn't designed to be readable by Avro.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-593) Avro mapreduce apis incompatible with hadoop 0.20.2

2012-02-17 Thread Doug Cutting (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13210813#comment-13210813
 ] 

Doug Cutting commented on AVRO-593:
---

Garrett, this code looks great!  Thanks for contributing it.

I renamed all of the packages to reside under org.apache.avro.mapred.  So that 
package now has subpackages named io, file, util and mapreduce.  That's 
consistent with other Avro modules, where classes are under 
org.apache.avro.module.

The only exception is org.apache.hadoop.io.AvroSequenceFile.  This is in a 
Hadoop package so that it can access some package-private parts of 
SequenceFile.  This is fragile, as SequenceFile could change these non-public 
APIs.  We should probably file an issue with Hadoop to make these items 
protected so that SequenceFile can be subclassed in a supported way.

I plan to improve the javadoc a bit (adding package.html files to new packages) 
and move versions for new dependencies from mapred/pom.xml into the parent pom. 
 Then I think this should be ready to commit.

 Avro mapreduce apis incompatible with hadoop 0.20.2
 ---

 Key: AVRO-593
 URL: https://issues.apache.org/jira/browse/AVRO-593
 Project: Avro
  Issue Type: Bug
  Components: java
Affects Versions: 1.3.2, 1.3.3
 Environment: Avro 1.3.3, Hadoop 0.20.2
Reporter: Steve Severance
Assignee: Garrett Wu
 Attachments: AVRO-593.patch, AVRO-593.patch


 The avro api's for hadoop use the hadoop mapreduce api that has been 
 deprecated. A new avro mapreduce api should be implemented for hadoop 0.20 
 and higher.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-1025) migrate website dist to svnpubsub

2012-02-16 Thread Doug Cutting (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13209735#comment-13209735
 ] 

Doug Cutting commented on AVRO-1025:


I committed the docs to subversion and asked Infrastructure to switch the 
website to automatically update from there in INFRA-4443.


 migrate website  dist to svnpubsub
 ---

 Key: AVRO-1025
 URL: https://issues.apache.org/jira/browse/AVRO-1025
 Project: Avro
  Issue Type: Improvement
Reporter: Doug Cutting
Assignee: Doug Cutting

 ASF infrastructure has requested that all projects migrate to svnpubsub for 
 their websites and release distributions.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-1026) Add namespace support to C++

2012-02-14 Thread Thiruvalluvan M. G. (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13207896#comment-13207896
 ] 

Thiruvalluvan M. G. commented on AVRO-1026:
---

The patch does not apply cleanly on the current trunk. I think this patch was 
generated before AVRO-956. We have made two important changes in the recent 
past to get rid of Flex/Bison dependency and to make it compile natively on 
Windows using Visual Studio.

I've not studied this patch deep. So my question may be naive.

Is it possible to make the name attribute include the namespace as well instead 
of adding a new attribute? If the namespace is not given, the name will fall 
back to the simple name as it is now. If namespace is explicitly given or 
implied, it will have a fully qualified name. That way backward compatibility 
with existing schemas and code will be maintained.

 Add namespace support to C++
 

 Key: AVRO-1026
 URL: https://issues.apache.org/jira/browse/AVRO-1026
 Project: Avro
  Issue Type: Improvement
  Components: c++
Affects Versions: 1.6.1
Reporter: Keh-Li Sheng
  Labels: patch
 Attachments: AVRO-1026.patch, AVRO-1026_json.patch


 This patch adds support for reading and resolving schemas that use namespaces 
 for named types. Looking for a code review and application of the patch to 
 codebase.
 Node is modified to take an additional NamespaceConcept attribute in the 
 template parameters. An additional stack has been added to the compiler 
 context to track when namespaces have been entered or left for resolving 
 further types/symbols. Symbolic resolution is now done against the fullname 
 of the type and not just by the name. Some string gymnastics were needed in 
 other areas of codegen in order to handle the new symbols.
 Added very trivial tests to AvrogencppTests for schema generation by 
 avrogencpp and added a test schema (tweet) that has namespaces, a record 
 definition inside a namespace, and then a later symbolic reference by name 
 within an outer namespace. Also patched to work with gen-cppcode.py output. 
 Note that NodeImpl::printBasicInfo intentionally does not output the 
 namespace since that caused downstream breakage of gen-cppcode.py expected 
 format.
 github project was forked at https://github.com/bifflabs/avro

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-1026) Add namespace support to C++

2012-02-14 Thread Keh-Li Sheng (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13207996#comment-13207996
 ] 

Keh-Li Sheng commented on AVRO-1026:


I forked from github and didn't update to the latest trunk before creating the 
patch. I'm resolving the merge now and will see if I can produce a clean patch.

I debated this a bit when implementing it and I chose in favor of keeping 
name/namespace/fullname as separate concepts since there were a lot of 
downstream effects to pushing namespaces into the name field - primarily in 
avrogencpp and gen-cppcode.py where name is used quite frequently as a field 
name or class/struct specifier (in which case the '.' in namespaces would be an 
invalid character). That meant additional logic would be required to strip the 
name of its fully qualified path, whereas it is more straightforward to know 
how to compose the full path when needed.

Additionally, if there was a future desire to nest declarations within C++ 
namespaces that tracked the schema namespace, having a separate namespace 
attribute would make that easier as well.

I'm not quite sure I understand what you mean by backward compatibility since 
this change does not *require* namespace attributes from existing schemas (all 
the existing schemas in jsonschemas seem to parse and validate and tests run 
the same as before), it merely knows what to do with them when it encounters 
them.

 Add namespace support to C++
 

 Key: AVRO-1026
 URL: https://issues.apache.org/jira/browse/AVRO-1026
 Project: Avro
  Issue Type: Improvement
  Components: c++
Affects Versions: 1.6.1
Reporter: Keh-Li Sheng
  Labels: patch
 Attachments: AVRO-1026.patch, AVRO-1026_json.patch


 This patch adds support for reading and resolving schemas that use namespaces 
 for named types. Looking for a code review and application of the patch to 
 codebase.
 Node is modified to take an additional NamespaceConcept attribute in the 
 template parameters. An additional stack has been added to the compiler 
 context to track when namespaces have been entered or left for resolving 
 further types/symbols. Symbolic resolution is now done against the fullname 
 of the type and not just by the name. Some string gymnastics were needed in 
 other areas of codegen in order to handle the new symbols.
 Added very trivial tests to AvrogencppTests for schema generation by 
 avrogencpp and added a test schema (tweet) that has namespaces, a record 
 definition inside a namespace, and then a later symbolic reference by name 
 within an outer namespace. Also patched to work with gen-cppcode.py output. 
 Note that NodeImpl::printBasicInfo intentionally does not output the 
 namespace since that caused downstream breakage of gen-cppcode.py expected 
 format.
 github project was forked at https://github.com/bifflabs/avro

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-1022) Error in validate name

2012-02-13 Thread Doug Cutting (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13207029#comment-13207029
 ] 

Doug Cutting commented on AVRO-1022:


 If names are restricted, then consuming schemas from other systems will be 
 difficult.

Good point.  The question is where the escaping burden lies: either with 
adapter layers (e.g., in Pig or Hive) or in the code generation layer.  I'd 
argue that code generation layer already has to handle reserved words so that 
adding character escaping is not a significant burden there.  It's also safer 
to not assume that other implementations have correctly escaped all names; to 
be tolerant.  Finally, escaping as late as possible maximizes legibility 
through the system.


 Error in validate name
 --

 Key: AVRO-1022
 URL: https://issues.apache.org/jira/browse/AVRO-1022
 Project: Avro
  Issue Type: Bug
  Components: java
Reporter: Raymie Stata
Priority: Minor
 Attachments: AVRO-1022.patch, AVRO-1022.patch


 Fix schema.validateName to allow only ASCII letters, not Unicode letters.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-1028) IPC transceiver doesn't gracefully handle server connection resets.

2012-02-13 Thread Bo Shi (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13207161#comment-13207161
 ] 

Bo Shi commented on AVRO-1028:
--

Also, note that this patch incorporates redirect functionality as originally 
requested in

https://issues.apache.org/jira/browse/AVRO-665

 IPC transceiver doesn't gracefully handle server connection resets.
 ---

 Key: AVRO-1028
 URL: https://issues.apache.org/jira/browse/AVRO-1028
 Project: Avro
  Issue Type: Improvement
  Components: python
Affects Versions: 1.6.2
Reporter: Bo Shi
Assignee: Bo Shi
 Fix For: 1.6.2

 Attachments: AVRO-1028.patch


 The current Python HTTPTransceiver class forces users to handle connection 
 resets.
 I've refactored the class using urllib3 and incorporated some features we get 
 for free from said library into the transceiver.  Added a test case for 
 test_ipc.py that uses the twisted server implementation.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-831) Enhance the Python `distutils` control file `setup.py` to build Avro/Python without a pre-processing step.

2012-02-13 Thread Bo Shi (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13207213#comment-13207213
 ] 

Bo Shi commented on AVRO-831:
-

Patch still seems to have some extra bits.

 Enhance the Python `distutils` control file `setup.py` to build Avro/Python 
 without a pre-processing step.
 --

 Key: AVRO-831
 URL: https://issues.apache.org/jira/browse/AVRO-831
 Project: Avro
  Issue Type: Improvement
  Components: python
 Environment: Operating environments supporting the Python programming 
 language.
Reporter: Joseph Koshy
Assignee: Joseph Koshy
  Labels: build, python
 Fix For: 1.7.0

 Attachments: setup.patch

   Original Estimate: 4h
  Remaining Estimate: 4h

 Avro's build process for its Python bindings currently needs a set of 
 pre-processing steps that need to be executed before the build proper is 
 started.  These pre-processing steps are executed by {{ant}}.
 The attached patch folds these pre-processing steps into the {{setup.py}} 
 control script that is input to Python's {{distutils}} build tool kit.  This 
 eliminates the dependency on {{ant}}.
 In the patch:
 * Source files that need keyword expansion are renamed to use a {{.in}} 
 extension.  The build process will generate the appropriately transformed 
 {{.py}} files from these {{.in}} files.
 * {{setup.py}} supports a new command {{make_interop_data}}, used to generate 
 test data for use with interoperability test code.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




[jira] [Commented] (AVRO-1022) Error in validate name

2012-02-13 Thread Scott Carey (Commented) (JIRA)

[ 
https://issues.apache.org/jira/browse/AVRO-1022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13207282#comment-13207282
 ] 

Scott Carey commented on AVRO-1022:
---

As late as possible also makes transitive equivalence easier.  If Hive and Pig 
both escaped the same string differently when writing into an Avro file, then 
that Avro file will not be shareable between them as easily.  This is why I 
feel that the Avro spec should define how names are escaped.  I think the only 
code point that truly  needs escaping is the namespace separator and an escape 
character.

A single source string recognized as equivalent in two different external 
systems, should become equivalent escaped strings if each of those systems uses 
different Avro language bindings to escape and store a schema.


 Error in validate name
 --

 Key: AVRO-1022
 URL: https://issues.apache.org/jira/browse/AVRO-1022
 Project: Avro
  Issue Type: Bug
  Components: java
Reporter: Raymie Stata
Priority: Minor
 Attachments: AVRO-1022.patch, AVRO-1022.patch


 Fix schema.validateName to allow only ASCII letters, not Unicode letters.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira




  1   2   3   4   5   >