[jira] [Commented] (AVRO-1068) Avro Java does not build on recent Eclipse versions with m2eclipse
[ https://issues.apache.org/jira/browse/AVRO-1068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13258286#comment-13258286 ] Scott Carey commented on AVRO-1068: --- +1 looks good. Avro Java does not build on recent Eclipse versions with m2eclipse -- Key: AVRO-1068 URL: https://issues.apache.org/jira/browse/AVRO-1068 Project: Avro Issue Type: Improvement Components: java Affects Versions: 1.7.0 Reporter: Thiruvalluvan M. G. Assignee: Thiruvalluvan M. G. Fix For: 1.7.0 Attachments: AVRO-1068.patch It is so because, in the recent versions of m2eclipse require manual mapping of non-standard maven lifecycle tasks. The forthcoming fix addresses that problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1066) ArrayIndexOutOfBoundsException in ParsingEncoder when trying to use a json encoder to serialize a deep object graph
[
https://issues.apache.org/jira/browse/AVRO-1066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13257601#comment-13257601
]
Thiruvalluvan M. G. commented on AVRO-1066:
---
Good catch. Daniel Lord's fix would work.
The problem has never been seen because we have not had nested array/maps with
10 levels or mores.
ArrayIndexOutOfBoundsException in ParsingEncoder when trying to use a json
encoder to serialize a deep object graph
---
Key: AVRO-1066
URL: https://issues.apache.org/jira/browse/AVRO-1066
Project: Avro
Issue Type: Bug
Affects Versions: 1.6.2
Reporter: Daniel Lord
I have an avro record that can have an array of children records of the same
type. If this structure gets to be too deep then I continually get an
ArrayOutOfBoundsException in ParsingEncoder when trying to use a json
encoder. This works fine when using a binary encoder. My schema looks
something like this:
{
name : MyRecord,
type : record,
fields : [
{
name : fooField,
type : int
},
{
name : childRecords,
type : [ null, { type : array, items : MyRecord } ]
}
]
}
The code I'm using to capture a JSON string for debugging looks like this:
ByteArrayOutputStream os = null;
try {
os = new ByteArrayOutputStream(2048);
final Encoder jsonEncoder =
EncoderFactory.get().jsonEncoder(MyRecord.SCHEMA$, os);
final DatumWriterMyRecord datumWriter = new
SpecificDatumWriterMyRecord(MyRecord.class);
datumWriter.write(record, jsonEncoder);
jsonEncoder.flush();
return new String(os.toByteArray(), Charset.defaultCharset());
} catch (IOException e) {
return null;
} finally {
if (os != null) {
try {
os.close();
} catch (IOException e) {
}
}
}
The error I get looks like this:
java.lang.ArrayIndexOutOfBoundsException: 10
at org.apache.avro.io.ParsingEncoder.push(ParsingEncoder.java:55)
at org.apache.avro.io.JsonEncoder.writeArrayStart(JsonEncoder.java:231)
at
org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:125)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:67)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:72)
at
org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:105)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
at
org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:105)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
at
org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:129)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:67)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:72)
at
org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:105)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
at
org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:129)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:67)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:72)
at
org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:105)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
at
org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:129)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:67)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:72)
at
org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:105)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
at
org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:129)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:67)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:72)
at
org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:105)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65
[jira] [Commented] (AVRO-1064) Encoder.writeString throws NullPointerException when the charSequence is null
[
https://issues.apache.org/jira/browse/AVRO-1064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13257609#comment-13257609
]
Thiruvalluvan M. G. commented on AVRO-1064:
---
The patch makes Encoder treat {{null}} for Avro {{String}} as empty. There is
no such requirement in Encoder's specification. This would make the the spec
somewhat non-uniform. For example, if we assign the interpretation for {{null}}
for Avro {{String}}, we should also add interpretation for {{null}} for Avro
{{bytes}}, {{fixed}} and so on.
I think ensuring the the bean doesn't have {{null}} member or annotating the
property {{@Nullable}} is a better way to handle this.
Encoder.writeString throws NullPointerException when the charSequence is null
-
Key: AVRO-1064
URL: https://issues.apache.org/jira/browse/AVRO-1064
Project: Avro
Issue Type: Bug
Components: java
Affects Versions: 1.6.3
Reporter: Kevin Zhao
Fix For: 1.7.0
Attachments: Encoder.java.patch
org.apache.avro.io.Encoder.writeString(CharSequence charSequence) throws
NullPointerException when dealing with a javabean which has a null value in
one of its fields, like the following one:
Person person = new Person();
person.setName(Kevin);
person.setId(null);
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-806) add a column-major codec for data files
[ https://issues.apache.org/jira/browse/AVRO-806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13257077#comment-13257077 ] Doug Cutting commented on AVRO-806: --- I've implemented a new column-file format at: https://github.com/cutting/trevni This supports writing Avro data. If folks find this useful then I intend to contribute it to Apache. add a column-major codec for data files --- Key: AVRO-806 URL: https://issues.apache.org/jira/browse/AVRO-806 Project: Avro Issue Type: New Feature Components: java, spec Reporter: Doug Cutting Assignee: Doug Cutting Fix For: 1.7.0 Attachments: AVRO-806-v2.patch, AVRO-806.patch, avro-file-columnar.pdf Define a codec that, when a data file's schema is a record schema, writes blocks within the file in column-major order. This would permit better compression and also permit efficient skipping of fields that are not of interest. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1066) ArrayIndexOutOfBoundsException in ParsingEncoder when trying to use a json encoder to serialize a deep object graph
[
https://issues.apache.org/jira/browse/AVRO-1066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13257162#comment-13257162
]
Daniel Lord commented on AVRO-1066:
---
There was a post on the mailing list about this a few weeks ago. I just tested
the suggested fix there and it works for me. Changing ParsingEncoder's push
method to this worked for me.
/** Push a new collection on to the stack. */
protected final void push() {
if (++pos = counts.length) {
counts = Arrays.copyOf(counts, pos + 10);
}
counts[pos] = 0;
}
ArrayIndexOutOfBoundsException in ParsingEncoder when trying to use a json
encoder to serialize a deep object graph
---
Key: AVRO-1066
URL: https://issues.apache.org/jira/browse/AVRO-1066
Project: Avro
Issue Type: Bug
Affects Versions: 1.6.2
Reporter: Daniel Lord
I have an avro record that can have an array of children records of the same
type. If this structure gets to be too deep then I continually get an
ArrayOutOfBoundsException in ParsingEncoder when trying to use a json
encoder. This works fine when using a binary encoder. My schema looks
something like this:
{
name : MyRecord,
type : record,
fields : [
{
name : fooField,
type : int
},
{
name : childRecords,
type : [ null, { type : array, items : MyRecord } ]
}
]
}
The code I'm using to capture a JSON string for debugging looks like this:
ByteArrayOutputStream os = null;
try {
os = new ByteArrayOutputStream(2048);
final Encoder jsonEncoder =
EncoderFactory.get().jsonEncoder(MyRecord.SCHEMA$, os);
final DatumWriterMyRecord datumWriter = new
SpecificDatumWriterMyRecord(MyRecord.class);
datumWriter.write(record, jsonEncoder);
jsonEncoder.flush();
return new String(os.toByteArray(), Charset.defaultCharset());
} catch (IOException e) {
return null;
} finally {
if (os != null) {
try {
os.close();
} catch (IOException e) {
}
}
}
The error I get looks like this:
java.lang.ArrayIndexOutOfBoundsException: 10
at org.apache.avro.io.ParsingEncoder.push(ParsingEncoder.java:55)
at org.apache.avro.io.JsonEncoder.writeArrayStart(JsonEncoder.java:231)
at
org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:125)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:67)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:72)
at
org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:105)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
at
org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:105)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
at
org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:129)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:67)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:72)
at
org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:105)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
at
org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:129)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:67)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:72)
at
org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:105)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
at
org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:129)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:67)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:72)
at
org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:105)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:65)
at
org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:129)
at
org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:67)
at
org.apache.avro.generic.GenericDatumWriter.write
[jira] [Commented] (AVRO-1062) DataFileWriter uses java.rmi.server.UID to generate unique id,which causes avro compilation problem on Android Delvik
[ https://issues.apache.org/jira/browse/AVRO-1062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13255419#comment-13255419 ] Kevin Zhao commented on AVRO-1062: -- use java.util.UUID as the alternative of java.rmi.server.UID to generate unique id DataFileWriter uses java.rmi.server.UID to generate unique id,which causes avro compilation problem on Android Delvik - Key: AVRO-1062 URL: https://issues.apache.org/jira/browse/AVRO-1062 Project: Avro Issue Type: Improvement Components: java Affects Versions: 1.6.3 Environment: Android 2.3.3-API level 10 Reporter: Kevin Zhao Fix For: 1.7.0 Attachments: AVRO-1062.patch Because Android Delvik does not have java.rmi.* packages and org.apache.avro.file.DataFileWriter has a reference of java.rmi.server.UID,avro fails in compilation process on Android. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1062) DataFileWriter uses java.rmi.server.UID to generate unique id,which causes avro compilation problem on Android Delvik
[ https://issues.apache.org/jira/browse/AVRO-1062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13255420#comment-13255420 ] Kevin Zhao commented on AVRO-1062: -- request for code reviewing~ :) DataFileWriter uses java.rmi.server.UID to generate unique id,which causes avro compilation problem on Android Delvik - Key: AVRO-1062 URL: https://issues.apache.org/jira/browse/AVRO-1062 Project: Avro Issue Type: Improvement Components: java Affects Versions: 1.6.3 Environment: Android 2.3.3-API level 10 Reporter: Kevin Zhao Fix For: 1.7.0 Attachments: AVRO-1062.patch Because Android Delvik does not have java.rmi.* packages and org.apache.avro.file.DataFileWriter has a reference of java.rmi.server.UID,avro fails in compilation process on Android. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1056) ReflectData thinks the instance of a java.util.Map as a record type
[ https://issues.apache.org/jira/browse/AVRO-1056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13255421#comment-13255421 ] Kevin Zhao commented on AVRO-1056: -- request for code reviewing~ ReflectData thinks the instance of a java.util.Map as a record type --- Key: AVRO-1056 URL: https://issues.apache.org/jira/browse/AVRO-1056 Project: Avro Issue Type: Bug Components: java Affects Versions: 1.6.3 Environment: MacOSX 10.7.2 Eclipse 3.7 JDK1.6 Reporter: Kevin Zhao Fix For: 1.6.3 Attachments: ReflectData.java.patch the avro's java api org.apache.avro.reflect.ReflectData.isRecord() should add the following code. if (datum instanceof Map) return false; -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1063) Ruby client should use multi_json rather than being locked down to yajl
[ https://issues.apache.org/jira/browse/AVRO-1063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13255516#comment-13255516 ] Paul Dlug commented on AVRO-1063: - I implemented this and sent a pull request via Github: https://github.com/apache/avro/pull/8 All tests pass after swapping yajl for multi_json. Ruby client should use multi_json rather than being locked down to yajl --- Key: AVRO-1063 URL: https://issues.apache.org/jira/browse/AVRO-1063 Project: Avro Issue Type: Improvement Components: ruby Reporter: Paul Dlug Priority: Minor The avro ruby client uses yajl for JSON serialization which is just one of many suitable JSON implementations for ruby. The multi_json gem provides a wrapper for JSON serialization selecting the fastest library available (Oj is now even faster than Yajl) and falling back to a pure ruby implementation bundled with multi_json. Requiring yajl also precludes the ruby gem from being used under jruby since it requires a C extension. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1062) DataFileWriter uses java.rmi.server.UID to generate unique id,which causes avro compilation problem on Android Delvik
[ https://issues.apache.org/jira/browse/AVRO-1062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13255802#comment-13255802 ] Doug Cutting commented on AVRO-1062: Looks good to me. We could perhaps improve sync marker generation, but this issue should make it no worse and permits Avro to run on Android. I'll commit this soon unless someone objects. DataFileWriter uses java.rmi.server.UID to generate unique id,which causes avro compilation problem on Android Delvik - Key: AVRO-1062 URL: https://issues.apache.org/jira/browse/AVRO-1062 Project: Avro Issue Type: Improvement Components: java Affects Versions: 1.6.3 Environment: Android 2.3.3-API level 10 Reporter: Kevin Zhao Labels: patch Fix For: 1.7.0 Attachments: AVRO-1062.patch Because Android Delvik does not have java.rmi.* packages and org.apache.avro.file.DataFileWriter has a reference of java.rmi.server.UID,avro fails in compilation process on Android. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1064) Encoder.writeString throws NullPointerException when the charSequence is null
[ https://issues.apache.org/jira/browse/AVRO-1064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13256188#comment-13256188 ] Kevin Zhao commented on AVRO-1064: -- request for code reviewing~ Encoder.writeString throws NullPointerException when the charSequence is null - Key: AVRO-1064 URL: https://issues.apache.org/jira/browse/AVRO-1064 Project: Avro Issue Type: Bug Components: java Affects Versions: 1.6.3 Reporter: Kevin Zhao Fix For: 1.7.0 Attachments: Encoder.java.patch org.apache.avro.io.Encoder.writeString(CharSequence charSequence) throws NullPointerException when dealing with a javabean which has a null value in one of its fields, like the following one: Person person = new Person(); person.setName(Kevin); person.setId(null); -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-565) Investigate Python encoding error
[ https://issues.apache.org/jira/browse/AVRO-565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13255064#comment-13255064 ] Miki Tebeka commented on AVRO-565: -- IMO it's not the library role. To quote [The Zen of Python|http://www.python.org/dev/peps/pep-0020/]: In the face of ambiguity, refuse the temptation to guess. You can do the same in your code (try to encode, if fails convert to utf8) Investigate Python encoding error - Key: AVRO-565 URL: https://issues.apache.org/jira/browse/AVRO-565 Project: Avro Issue Type: Bug Components: python Reporter: Jeff Hammerbacher Attachments: reproduce_avro_565.py Tyler B is seeing the following encoding error: http://avro.pastebin.com/b4HSYjCz. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1061) Add sync interval option to Avro commandline tools
[ https://issues.apache.org/jira/browse/AVRO-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13255174#comment-13255174 ] Doug Cutting commented on AVRO-1061: This looks good to me. We should probably add a test to TestDataFileTools that checks this. Should we also increase the default sync interval to 64k? Add sync interval option to Avro commandline tools -- Key: AVRO-1061 URL: https://issues.apache.org/jira/browse/AVRO-1061 Project: Avro Issue Type: Improvement Components: java Affects Versions: 1.7.0 Reporter: Ari Pollak Priority: Trivial Attachments: AVRO-1061.patch It would be nice to expose the sync interval to the avro commandline writer tools, since I've seen a 20%+ decrease in file size using deflate compression and a 64K+ sync interval instead of the default of 16K. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1057) Java builder API fails when default value does not match the first type in a union
[
https://issues.apache.org/jira/browse/AVRO-1057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13253666#comment-13253666
]
Doug Cutting commented on AVRO-1057:
I think this is correct. The specification says, Default values for union
fields correspond to the first schema in the union
(http://avro.apache.org/docs/current/spec.html). So the default value for
{boolean null} must be a boolean.
Java builder API fails when default value does not match the first type in a
union
--
Key: AVRO-1057
URL: https://issues.apache.org/jira/browse/AVRO-1057
Project: Avro
Issue Type: Bug
Components: java
Affects Versions: 1.6.3
Reporter: Christophe Taton
Priority: Minor
The following definition works fine with the builder:
record Rec {
union { boolean, null } field = false;
}
but this one fails:
record Rec {
union { boolean, null } field = null;
}
Rec.newBuilder().build() fails with this error:
org.apache.avro.AvroRuntimeException: org.apache.avro.AvroTypeException:
Non-boolean default for boolean: null
at Rec$Builder.build
Caused by: org.apache.avro.AvroTypeException: Non-boolean default for
boolean: null
at
org.apache.avro.io.parsing.ResolvingGrammarGenerator.encode(ResolvingGrammarGenerator.java:393)
at
org.apache.avro.io.parsing.ResolvingGrammarGenerator.encode(ResolvingGrammarGenerator.java:350)
at
org.apache.avro.data.RecordBuilderBase.defaultValue(RecordBuilderBase.java:178)
at Rec$Builder.build
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1057) Java builder API fails when default value does not match the first type in a union
[
https://issues.apache.org/jira/browse/AVRO-1057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13253675#comment-13253675
]
Christophe Taton commented on AVRO-1057:
Ah, you are right, I missed that bit.
I got confused as the IDL compiler and JSON schema currently allow bogus
declarations.
Could the IDL compiler and the schema parser check this more aggressively?
Java builder API fails when default value does not match the first type in a
union
--
Key: AVRO-1057
URL: https://issues.apache.org/jira/browse/AVRO-1057
Project: Avro
Issue Type: Bug
Components: java
Affects Versions: 1.6.3
Reporter: Christophe Taton
Priority: Minor
The following definition works fine with the builder:
record Rec {
union { boolean, null } field = false;
}
but this one fails:
record Rec {
union { boolean, null } field = null;
}
Rec.newBuilder().build() fails with this error:
org.apache.avro.AvroRuntimeException: org.apache.avro.AvroTypeException:
Non-boolean default for boolean: null
at Rec$Builder.build
Caused by: org.apache.avro.AvroTypeException: Non-boolean default for
boolean: null
at
org.apache.avro.io.parsing.ResolvingGrammarGenerator.encode(ResolvingGrammarGenerator.java:393)
at
org.apache.avro.io.parsing.ResolvingGrammarGenerator.encode(ResolvingGrammarGenerator.java:350)
at
org.apache.avro.data.RecordBuilderBase.defaultValue(RecordBuilderBase.java:178)
at Rec$Builder.build
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1057) Java builder API fails when default value does not match the first type in a union
[
https://issues.apache.org/jira/browse/AVRO-1057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13253692#comment-13253692
]
Doug Cutting commented on AVRO-1057:
Yes, this could be checked more aggressively. The existing logic for this is
in RecordBuilderBase#defaultValue(). It writes the default JSON parse tree,
then reads it into the appropriate Avro data structure (generic, specific,
reflect). The writing does the error checking, and that's in
ResolvingGrammarGenerator#encode. So the compiler and/or parser could call
that. We probably don't want to check it unconditionally in the schema parser,
as schema parsing is performance sensitive.
Java builder API fails when default value does not match the first type in a
union
--
Key: AVRO-1057
URL: https://issues.apache.org/jira/browse/AVRO-1057
Project: Avro
Issue Type: Bug
Components: java
Affects Versions: 1.6.3
Reporter: Christophe Taton
Priority: Minor
The following definition works fine with the builder:
record Rec {
union { boolean, null } field = false;
}
but this one fails:
record Rec {
union { boolean, null } field = null;
}
Rec.newBuilder().build() fails with this error:
org.apache.avro.AvroRuntimeException: org.apache.avro.AvroTypeException:
Non-boolean default for boolean: null
at Rec$Builder.build
Caused by: org.apache.avro.AvroTypeException: Non-boolean default for
boolean: null
at
org.apache.avro.io.parsing.ResolvingGrammarGenerator.encode(ResolvingGrammarGenerator.java:393)
at
org.apache.avro.io.parsing.ResolvingGrammarGenerator.encode(ResolvingGrammarGenerator.java:350)
at
org.apache.avro.data.RecordBuilderBase.defaultValue(RecordBuilderBase.java:178)
at Rec$Builder.build
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-593) Java: add support for Hadoop's new mapreduce APIs
[ https://issues.apache.org/jira/browse/AVRO-593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13253806#comment-13253806 ] Garrett Wu commented on AVRO-593: - Thank you, Doug, for doing the hard part -- integration! Java: add support for Hadoop's new mapreduce APIs - Key: AVRO-593 URL: https://issues.apache.org/jira/browse/AVRO-593 Project: Avro Issue Type: New Feature Components: java Affects Versions: 1.3.2, 1.3.3 Environment: Avro 1.3.3, Hadoop 0.20.2 Reporter: Steve Severance Assignee: Garrett Wu Fix For: 1.7.0 Attachments: AVRO-593-test.tgz, AVRO-593.patch, AVRO-593.patch, AVRO-593.patch, AVRO-593.patch Avro should work with Hadoop's newer org.apache.hadoop.mapreduce API, in addition to the older org.apache.hadoop.mapred API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-565) Investigate Python encoding error
[
https://issues.apache.org/jira/browse/AVRO-565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13253964#comment-13253964
]
Miki Tebeka commented on AVRO-565:
--
Russel, the problem is that you pass a *str* with unicode characters. Python
has no way of knowing the encoding and the default if 'ascii'. If you prepend a
'u' to make the body unicode:
{code}
email_hash = {'body': uVerit\xc3\xa1\r\nEstat\xc3\xadstica\r\n}
{code}
This will work since now it's a unicode string and it can be encoded.
As a side note - these kind of things were one of the reasons for Python 3. I
also recommend viewing [this
video|http://pyvideo.org/video/948/pragmatic-unicode-or-how-do-i-stop-the-pain]
which personally helped me.
I'll give you time to respond before closing this.
Investigate Python encoding error
-
Key: AVRO-565
URL: https://issues.apache.org/jira/browse/AVRO-565
Project: Avro
Issue Type: Bug
Components: python
Reporter: Jeff Hammerbacher
Attachments: reproduce_avro_565.py
Tyler B is seeing the following encoding error:
http://avro.pastebin.com/b4HSYjCz.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-565) Investigate Python encoding error
[ https://issues.apache.org/jira/browse/AVRO-565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13253967#comment-13253967 ] Russell Jurney commented on AVRO-565: - Pardon me, I mean it would be neat if Avro worked with ugly strings :) Investigate Python encoding error - Key: AVRO-565 URL: https://issues.apache.org/jira/browse/AVRO-565 Project: Avro Issue Type: Bug Components: python Reporter: Jeff Hammerbacher Attachments: reproduce_avro_565.py Tyler B is seeing the following encoding error: http://avro.pastebin.com/b4HSYjCz. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-565) Investigate Python encoding error
[ https://issues.apache.org/jira/browse/AVRO-565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13253966#comment-13253966 ] Russell Jurney commented on AVRO-565: - I create unicode objects when I know the codec, otherwise the decode crashes. codeif charset: subject = subject.decode(charset)/code It would be neat if Python worked with ugly strings. Investigate Python encoding error - Key: AVRO-565 URL: https://issues.apache.org/jira/browse/AVRO-565 Project: Avro Issue Type: Bug Components: python Reporter: Jeff Hammerbacher Attachments: reproduce_avro_565.py Tyler B is seeing the following encoding error: http://avro.pastebin.com/b4HSYjCz. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-593) Avro mapreduce apis incompatible with hadoop 0.20.2
[ https://issues.apache.org/jira/browse/AVRO-593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13252047#comment-13252047 ] Scott Carey commented on AVRO-593: -- No objections, but I have not had time for a deep review and won't for more than a week. I don't think we need to hold this up for my full review, I can always create another ticket for later changes. Avro mapreduce apis incompatible with hadoop 0.20.2 --- Key: AVRO-593 URL: https://issues.apache.org/jira/browse/AVRO-593 Project: Avro Issue Type: Bug Components: java Affects Versions: 1.3.2, 1.3.3 Environment: Avro 1.3.3, Hadoop 0.20.2 Reporter: Steve Severance Assignee: Garrett Wu Attachments: AVRO-593-test.tgz, AVRO-593.patch, AVRO-593.patch, AVRO-593.patch, AVRO-593.patch The avro api's for hadoop use the hadoop mapreduce api that has been deprecated. A new avro mapreduce api should be implemented for hadoop 0.20 and higher. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1055) Race condition in Java fingerprinting code
[ https://issues.apache.org/jira/browse/AVRO-1055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250094#comment-13250094 ] Doug Cutting commented on AVRO-1055: +1 Looks good to me. You might add a comment saying something like, Nested class used so that table is not built unless it's used. Race condition in Java fingerprinting code -- Key: AVRO-1055 URL: https://issues.apache.org/jira/browse/AVRO-1055 Project: Avro Issue Type: Bug Components: java Affects Versions: 1.7.0 Reporter: Thiruvalluvan M. G. Assignee: Thiruvalluvan M. G. Priority: Minor Attachments: AVRO-1055.patch There is a subtle race condition. If the fpTable64 is not yet initialized and two thread try to compute FP for two schemas (or the same schema) at the same time, one thread will start initializing the table while the other can start using the partially initialized table giving wrong result. The forthcoming patch fixes that issue -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1052) Support MultipleOutputFormat in Avro
[ https://issues.apache.org/jira/browse/AVRO-1052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247344#comment-13247344 ] Ashish Nagavaram commented on AVRO-1052: I tried to find one but was not able to. I will start working on a Avro Specific MultipleOutputs and see how it goes. Support MultipleOutputFormat in Avro Key: AVRO-1052 URL: https://issues.apache.org/jira/browse/AVRO-1052 Project: Avro Issue Type: New Feature Components: java Environment: Avro 1.6.1, Hadoop 0.20.205.0.3 Reporter: Ashish Nagavaram Labels: avro, newbie Original Estimate: 432h Remaining Estimate: 432h Api's for Avro to write records into multiple files similar to the MultipleTextOutputFormat. All the files have the same output schema -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-551) C: Build and pass tests on Win32
[ https://issues.apache.org/jira/browse/AVRO-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13245626#comment-13245626 ] Vivek Nadkarni commented on AVRO-551: - Hi - I am in the process of porting Avro-C, version 1.6.3, to Microsoft Visual C++ 2008. I would like to submit my changes for inclusion in the Avro-C project. In the porting process, I needed to include a few header files that are not present in the Microsoft environment. I have included the following files: 1. inttypes.h stdint.h Obtained from http://code.google.com/p/msinttypes/ This project uses the New BSD License: http://www.opensource.org/licenses/bsd-license.php 2. dirent.h Obtained from http://www.softagalleria.net/dirent.php http://www.softagalleria.net/download/dirent/dirent-1.11.zip The file copyright header matches the MIT-License. Can I just include these files in my patchset when I upload my patches? Is there anything special I need to do to include these files in the project? Thanks, Vivek C: Build and pass tests on Win32 Key: AVRO-551 URL: https://issues.apache.org/jira/browse/AVRO-551 Project: Avro Issue Type: Improvement Components: c Reporter: Bruce Mitchener Avro C does not currently build on Win32. We need to address that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-551) C: Build and pass tests on Win32
[ https://issues.apache.org/jira/browse/AVRO-551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13245856#comment-13245856 ] Doug Cutting commented on AVRO-551: --- It's fine to include MIT and BSD licensed code, but the licenses for these files should be appended to the end of Avro's top-level LICENSE.txt file. For more information see: http://apache.org/legal/resolved.html C: Build and pass tests on Win32 Key: AVRO-551 URL: https://issues.apache.org/jira/browse/AVRO-551 Project: Avro Issue Type: Improvement Components: c Reporter: Bruce Mitchener Attachments: AVRO-551.patch Avro C does not currently build on Win32. We need to address that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-565) Investigate Python encoding error
[
https://issues.apache.org/jira/browse/AVRO-565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239381#comment-13239381
]
Russell Jurney commented on AVRO-565:
-
Oh man, this has been KILLING me for a couple weeks.
What am I to do? What is the status of this? No matter what I do to the input
field, python avro cannot write the field.
Even though I do this: subject.decode(charset).encode('utf-8')
The write still dies. Oh man this has been KILLING me. Help :)
Investigate Python encoding error
-
Key: AVRO-565
URL: https://issues.apache.org/jira/browse/AVRO-565
Project: Avro
Issue Type: Bug
Components: python
Reporter: Jeff Hammerbacher
Tyler B is seeing the following encoding error:
http://avro.pastebin.com/b4HSYjCz.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-565) Investigate Python encoding error
[
https://issues.apache.org/jira/browse/AVRO-565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239401#comment-13239401
]
Russell Jurney commented on AVRO-565:
-
On further reflection...
I have changed my code to have all strings become Unicode objects, which Avro
can write: decode(mystring, mycharset)
Before I was using UTF-8 encoded strings:
mystring.decode(mycharset).encode('utf-8)
As outlined above, this was resulting in double UTF-age. Suggest that the
API/docs should only accept Unicode objects, not strings, since strings do not
work well.
Investigate Python encoding error
-
Key: AVRO-565
URL: https://issues.apache.org/jira/browse/AVRO-565
Project: Avro
Issue Type: Bug
Components: python
Reporter: Jeff Hammerbacher
Tyler B is seeing the following encoding error:
http://avro.pastebin.com/b4HSYjCz.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-565) Investigate Python encoding error
[ https://issues.apache.org/jira/browse/AVRO-565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13239444#comment-13239444 ] Miki Tebeka commented on AVRO-565: -- Russel, can you provide an example input that causes the problem? Even better will be an example script that reproduces the problem. Investigate Python encoding error - Key: AVRO-565 URL: https://issues.apache.org/jira/browse/AVRO-565 Project: Avro Issue Type: Bug Components: python Reporter: Jeff Hammerbacher Tyler B is seeing the following encoding error: http://avro.pastebin.com/b4HSYjCz. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1050) Avro PHP consumes too much memory due to code in io.php (AvroStringIO append_str)
[
https://issues.apache.org/jira/browse/AVRO-1050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13236657#comment-13236657
]
A B commented on AVRO-1050:
---
As I mentioned we have a fix in place as follows (patch will be forthcoming):
Original code (io.php, class AvroStringIO):
private function append_str($str)
{
$this-check_closed();
$ary = str_split($str);
$len = count($ary);
$this-buffer = array_merge($this-buffer, $ary);
$this-current_index += $len;
return $len;
}
Fixed code (io.php, class AvroStringIO):
private function append_str($str)
{
$this-check_closed();
$this-buffer .= $str;
$len = strlen($str);
$this-current_index += $len;
return $len;
}
Avro PHP consumes too much memory due to code in io.php (AvroStringIO
append_str)
-
Key: AVRO-1050
URL: https://issues.apache.org/jira/browse/AVRO-1050
Project: Avro
Issue Type: Bug
Components: php
Affects Versions: 1.6.2
Reporter: A B
Fix For: 1.7.0
While attempting to encode large messages, our team found that the PHP
run-time would exceed its memory limit and die. Profiling and use of
memory_get_usage showed us that we were spending a lot of time in append_str
and that the PHP process grew after every call to this method, resulting in
the leak-like behavior that caused the process to grow and then crash. We
rewrote append_str to use the string concatenation operator instead of the
somewhat elaborate scheme in place today (str_split, followed by the copying
of one byte at a time from the latter array to the final string) and the
memory usage issues disappeared with no loss of functionality. We would like
to provide the patch to fix this issue.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-823) Support Avro data files in C#
[ https://issues.apache.org/jira/browse/AVRO-823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13232072#comment-13232072 ] Edward Wohlman commented on AVRO-823: - Any progress on this? I was looking into using Avro as a new file format and I work primarily in C#. I tried adding File support by translating directly from the Java implementation, but I am not really familiar enough with the remainder of the library yet to get it to work. I may have another go, but I could probably use a few pointers. Support Avro data files in C# - Key: AVRO-823 URL: https://issues.apache.org/jira/browse/AVRO-823 Project: Avro Issue Type: New Feature Components: csharp Affects Versions: 1.5.1 Reporter: Patrick Angeles Currently, the C# implementation does not handle reading and writing of Avro data files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-823) Support Avro data files in C#
[ https://issues.apache.org/jira/browse/AVRO-823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13232142#comment-13232142 ] Thiruvalluvan M. G. commented on AVRO-823: -- If you can attach your work so far, someone may take up from there. I'm not sure if anyone would, but still one will be more motivated than if he has to start from scratch. Support Avro data files in C# - Key: AVRO-823 URL: https://issues.apache.org/jira/browse/AVRO-823 Project: Avro Issue Type: New Feature Components: csharp Affects Versions: 1.5.1 Reporter: Patrick Angeles Currently, the C# implementation does not handle reading and writing of Avro data files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1045) deepCopy of BYTES underflow exception
[ https://issues.apache.org/jira/browse/AVRO-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13230303#comment-13230303 ] Scott Carey commented on AVRO-1045: --- There are two choices that make sense to me: * Copy the whole buffer and set all positions in the destination to the same thing as the source (limit, pos, mark, capacity), being cognizant of arrayOffset in case the source buffer was a slice of a larger array. * Assume that the data of intereste is between pos and limit, copy that into a new byte buffer starting at index 0 with the new limit set to (limit - pos). In both cases, the original buffer needs to be returned to its original state. Avro isn't currently doing either. deepCopy of BYTES underflow exception - Key: AVRO-1045 URL: https://issues.apache.org/jira/browse/AVRO-1045 Project: Avro Issue Type: Bug Components: java Affects Versions: 1.6.2 Reporter: Jeremy Lewi Priority: Minor Fix For: 1.6.3 Attachments: AVRO-1045.patch In org.apache.avro.generic.GenericData.deepCopy - the code for copying a ByteBuffer is ByteBuffer byteBufferValue = (ByteBuffer) value; byte[] bytesCopy = new byte[byteBufferValue.capacity()]; byteBufferValue.rewind(); byteBufferValue.get(bytesCopy); byteBufferValue.rewind(); return ByteBuffer.wrap(bytesCopy); I think this is problematic because it will cause an UnderFlow exception to be thrown if the ByteBuffer limit is less than the capacity of the byte buffer. My use case is as follows. I have ByteBuffer's backed by large arrays so I can avoid resizing the array every time I write data. So limit capacity. When the data is written, or copied I think avro should respect this. When data is serialized, avro should automatically use the minimum number of bytes. When an object is copied, I think it makes sense to preserve the capacity of the underlying buffer as opposed to compacting it. So I think the code could be fixed by replacing get with byteBufferValue.get(bytesCopy, 0 , byteBufferValue.limit()); -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1045) deepCopy of BYTES underflow exception
[ https://issues.apache.org/jira/browse/AVRO-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13229630#comment-13229630 ] Doug Cutting commented on AVRO-1045: It's a little odd to require that deepCopy() preserve more than is checked by equals(). Some folks may might reasonably expect deepCopy() to compact large ByteBuffers. We could perhaps add a 'protected ByteBuffer GenericData#copyBytes(ByteBuffer)' method that could be overridden in a subclass? Would that work in your case? Am I being overly cautious? deepCopy of BYTES underflow exception - Key: AVRO-1045 URL: https://issues.apache.org/jira/browse/AVRO-1045 Project: Avro Issue Type: Bug Components: java Affects Versions: 1.6.2 Reporter: Jeremy Lewi Priority: Minor Fix For: 1.6.3 Attachments: AVRO-1045.patch In org.apache.avro.generic.GenericData.deepCopy - the code for copying a ByteBuffer is ByteBuffer byteBufferValue = (ByteBuffer) value; byte[] bytesCopy = new byte[byteBufferValue.capacity()]; byteBufferValue.rewind(); byteBufferValue.get(bytesCopy); byteBufferValue.rewind(); return ByteBuffer.wrap(bytesCopy); I think this is problematic because it will cause an UnderFlow exception to be thrown if the ByteBuffer limit is less than the capacity of the byte buffer. My use case is as follows. I have ByteBuffer's backed by large arrays so I can avoid resizing the array every time I write data. So limit capacity. When the data is written, or copied I think avro should respect this. When data is serialized, avro should automatically use the minimum number of bytes. When an object is copied, I think it makes sense to preserve the capacity of the underlying buffer as opposed to compacting it. So I think the code could be fixed by replacing get with byteBufferValue.get(bytesCopy, 0 , byteBufferValue.limit()); -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1045) deepCopy of BYTES underflow exception
[ https://issues.apache.org/jira/browse/AVRO-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13229718#comment-13229718 ] Jeremy Lewi commented on AVRO-1045: --- I think the issue is less my use case then maintaining compatibility with existing code. In my patch, I decided to preserve the capacity on copy rather than compact because this seemed slightly less likely to break existing code. The current implementation creates a new buffer of equal capacity, so in theory its not compacting the buffer. However, if the buffer isn't full, i.e. limit capacity, then the current code causes an exception. So based on the current implementation, its unclear whether compacting or not compacting is the expected behavior. With regards to my use case, compacting on copy is fine, so long as deep copy doesn't expect limit = capacity like it currently does. deepCopy of BYTES underflow exception - Key: AVRO-1045 URL: https://issues.apache.org/jira/browse/AVRO-1045 Project: Avro Issue Type: Bug Components: java Affects Versions: 1.6.2 Reporter: Jeremy Lewi Priority: Minor Fix For: 1.6.3 Attachments: AVRO-1045.patch In org.apache.avro.generic.GenericData.deepCopy - the code for copying a ByteBuffer is ByteBuffer byteBufferValue = (ByteBuffer) value; byte[] bytesCopy = new byte[byteBufferValue.capacity()]; byteBufferValue.rewind(); byteBufferValue.get(bytesCopy); byteBufferValue.rewind(); return ByteBuffer.wrap(bytesCopy); I think this is problematic because it will cause an UnderFlow exception to be thrown if the ByteBuffer limit is less than the capacity of the byte buffer. My use case is as follows. I have ByteBuffer's backed by large arrays so I can avoid resizing the array every time I write data. So limit capacity. When the data is written, or copied I think avro should respect this. When data is serialized, avro should automatically use the minimum number of bytes. When an object is copied, I think it makes sense to preserve the capacity of the underlying buffer as opposed to compacting it. So I think the code could be fixed by replacing get with byteBufferValue.get(bytesCopy, 0 , byteBufferValue.limit()); -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-301) Handle non-reserved properties appropriately in the Python implementation
[ https://issues.apache.org/jira/browse/AVRO-301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13229864#comment-13229864 ] Miki Tebeka commented on AVRO-301: -- Patch looks good, going to to commit tomorrow afternoon if there are no objections. Handle non-reserved properties appropriately in the Python implementation - Key: AVRO-301 URL: https://issues.apache.org/jira/browse/AVRO-301 Project: Avro Issue Type: New Feature Components: python Affects Versions: 1.6.2 Reporter: Jeff Hammerbacher Assignee: Marcio Silva Attachments: AVRO-301-patch-1.patch, AVRO-301-patch-2.patch, AVRO-301-patch-3.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1044) avro-maven-plugin requires dependency resolution which breaks multi-module projects
[ https://issues.apache.org/jira/browse/AVRO-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13228742#comment-13228742 ] Arvind Prabhakar commented on AVRO-1044: The tarball I attached does not reproduce this problem, it is complaining about something else. Hence I will remove it and work on getting a reproducible testcase. avro-maven-plugin requires dependency resolution which breaks multi-module projects --- Key: AVRO-1044 URL: https://issues.apache.org/jira/browse/AVRO-1044 Project: Avro Issue Type: Bug Components: java Affects Versions: 1.6.2 Reporter: Arvind Prabhakar Priority: Critical Fix For: 1.6.3 Use of avro-maven-plugin breaks multimodule projects since it forces the dependency resolution of all of the dependencies, some of which may be from within the reactor and not yet installed in the local cache. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-406) Support streaming RPC calls
[ https://issues.apache.org/jira/browse/AVRO-406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13226016#comment-13226016 ] André Cruz commented on AVRO-406: - Has streaming support been definitely abandoned? It would be very useful for Cassandra. Support streaming RPC calls --- Key: AVRO-406 URL: https://issues.apache.org/jira/browse/AVRO-406 Project: Avro Issue Type: New Feature Components: java, spec Reporter: Todd Lipcon Labels: rpc Avro nicely supports chunking of container types into multiple frames. We need to expose this to RPC layer to facilitate use cases like the Hadoop Datanode where a single RPC can yield far more data than should be buffered in memory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-987) Make Avro OSGi ready
[ https://issues.apache.org/jira/browse/AVRO-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13224088#comment-13224088 ] Ioannis Canellos commented on AVRO-987: --- I'll rework on the patch to align it to the current trunk. -- *Ioannis Canellos* * FuseSource http://fusesource.com ** Blog: http://iocanel.blogspot.com ** Twitter: iocanel * Make Avro OSGi ready Key: AVRO-987 URL: https://issues.apache.org/jira/browse/AVRO-987 Project: Avro Issue Type: New Feature Components: java Reporter: Ioannis Canellos Attachments: AVRO-987-patch-updated.txt, AVRO-987-patch.txt It would be really nice to be able to use Avro inside OSGi. To achieve this two things are required: i) Provide proper MANIFEST.MF. ii) Deal with potential class loading issues. Avro uses Class.forName a lot and that is not very OSGi friendly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-987) Make Avro OSGi ready
[ https://issues.apache.org/jira/browse/AVRO-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13223408#comment-13223408 ] Doug Cutting commented on AVRO-987: --- The patch looks reasonable to me, but fails to apply cleanly. Can someone please provide a version that applies to the current trunk? Thanks! Make Avro OSGi ready Key: AVRO-987 URL: https://issues.apache.org/jira/browse/AVRO-987 Project: Avro Issue Type: New Feature Components: java Reporter: Ioannis Canellos Attachments: AVRO-987-patch-updated.txt, AVRO-987-patch.txt It would be really nice to be able to use Avro inside OSGi. To achieve this two things are required: i) Provide proper MANIFEST.MF. ii) Deal with potential class loading issues. Avro uses Class.forName a lot and that is not very OSGi friendly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1022) Error in validate name
[ https://issues.apache.org/jira/browse/AVRO-1022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13223419#comment-13223419 ] Doug Cutting commented on AVRO-1022: Raymie, this is a good approach. The spec language that requires ASCII should be changed from MUST to SHOULD. One use case that Scott mentioned that your prose does not is transmitting schemas from other systems, e.g., Avro Schemas might often be generated automatically from Pig or SQL schemas. In these cases accepting liberally permits schemas to pass through Avro losslessly. Strict validation is really only useful when a developer is the schema author. In many (most?) cases Avro might be an underlying tool, used indirectly through an application, and in these cases strict validation is probably not useful. Error in validate name -- Key: AVRO-1022 URL: https://issues.apache.org/jira/browse/AVRO-1022 Project: Avro Issue Type: Bug Components: java Reporter: Raymie Stata Priority: Minor Attachments: AVRO-1022.patch, AVRO-1022.patch, unicode-recommendation.html Fix schema.validateName to allow only ASCII letters, not Unicode letters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13223428#comment-13223428 ] Doug Cutting commented on AVRO-1006: Patch looks good. +1 I'll commit this unless there are objections. Fingerprints for Avro Schemas - Key: AVRO-1006 URL: https://issues.apache.org/jira/browse/AVRO-1006 Project: Avro Issue Type: New Feature Components: java Reporter: Raymie Stata Assignee: Raymie Stata Labels: features Attachments: AVRO-1006-prelim.patch, AVRO-1006.patch, AVRO-1006.patch, AVRO-1006.patch, AVRO-1006.patch, AVRO-1006.patch, schema-fingerprinting.html, schema-fingerprinting.html, schema-fingerprinting.html Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-784) SpecificCompiler should generate accessors
[ https://issues.apache.org/jira/browse/AVRO-784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13222847#comment-13222847 ] Doug Cutting commented on AVRO-784: --- Scott, good point. Let's open a new issue to add unboxed accessors. SpecificCompiler should generate accessors -- Key: AVRO-784 URL: https://issues.apache.org/jira/browse/AVRO-784 Project: Avro Issue Type: Improvement Components: java Affects Versions: 1.5.0 Reporter: E. Sammer Labels: features Attachments: avro-784.diff, avro-784.diff Avro's Java SpecificCompiler should generate java bean style accessors. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-987) Make Avro OSGi ready
[ https://issues.apache.org/jira/browse/AVRO-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221862#comment-13221862 ] Claus Ibsen commented on AVRO-987: -- Any update on this? Make Avro OSGi ready Key: AVRO-987 URL: https://issues.apache.org/jira/browse/AVRO-987 Project: Avro Issue Type: New Feature Components: java Reporter: Ioannis Canellos Attachments: AVRO-987-patch-updated.txt, AVRO-987-patch.txt It would be really nice to be able to use Avro inside OSGi. To achieve this two things are required: i) Provide proper MANIFEST.MF. ii) Deal with potential class loading issues. Avro uses Class.forName a lot and that is not very OSGi friendly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-784) SpecificCompiler should generate accessors
[ https://issues.apache.org/jira/browse/AVRO-784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13222130#comment-13222130 ] Scott Carey commented on AVRO-784: -- I think there were some details that did not make it into AVRO-839. In particular, all accessors in AVRO-839 appear to be boxed, whether the value is nullable or not. I have not moved to using the API there because of the boxing overhead incurred. These issues will need to go into another ticket for later. SpecificCompiler should generate accessors -- Key: AVRO-784 URL: https://issues.apache.org/jira/browse/AVRO-784 Project: Avro Issue Type: Improvement Components: java Affects Versions: 1.5.0 Reporter: E. Sammer Labels: features Attachments: avro-784.diff, avro-784.diff Avro's Java SpecificCompiler should generate java bean style accessors. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1022) Error in validate name
[ https://issues.apache.org/jira/browse/AVRO-1022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221634#comment-13221634 ] Raymie Stata commented on AVRO-1022: Based on the dialog above, I propose that we turn this into a set of recommendations for Avro implementers and app developers. I've attached a draft of these recommendations, if folks like them, I can turn them into a patch for the spec. The basic idea is as follows. We cast this as an interoperability problem. On the one hand, the current spec for names facilitates language interoperability, and we shouldn't change it. On the other hand, we should apply the Robustness principle: accept liberally, generate strictly. (Looking closely at Scott's and Doug's comments, I think this is what they're really getting at.) In our context, Avro implementations accept schemas, Avro app developers generate them. Thus, we recommend that Avro implementations, by default, accept schemas liberally. This means accepting arbitrary Unicode strings for names (but with no normalization or other Unicode processing). At the same time, Avro developers for who want language interoperability are exhorted to generate strictly, and in particular, use only names that follow the strict definition in the Avro spec, because those names are easily handled by almost any language out there. To help developers generate strictly, we recommend that Avro implementations provide optional mechanisms to perform schema validation, preferably as early in the dev process as possible (eg, unit-test or even compile time). (For developers who don't care about language interop, they can ignore these mechanisms, and Avro implementations, being liberal by default, should happily consume their input.) The file I just posted says the above in a little more detail. Error in validate name -- Key: AVRO-1022 URL: https://issues.apache.org/jira/browse/AVRO-1022 Project: Avro Issue Type: Bug Components: java Reporter: Raymie Stata Priority: Minor Attachments: AVRO-1022.patch, AVRO-1022.patch, unicode-recommendation.html Fix schema.validateName to allow only ASCII letters, not Unicode letters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1027) NettyTransceiver will deadlock when attempting transceive/disconnect on the same thread
[ https://issues.apache.org/jira/browse/AVRO-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221055#comment-13221055 ] James Baldassari commented on AVRO-1027: All my tests are passing with trunk + AVRO-1027 NettyTransceiver will deadlock when attempting transceive/disconnect on the same thread --- Key: AVRO-1027 URL: https://issues.apache.org/jira/browse/AVRO-1027 Project: Avro Issue Type: Bug Components: java Affects Versions: 1.6.1 Reporter: Simon Wilkinson Assignee: James Baldassari Fix For: 1.6.3 Attachments: AVRO-1027-v2.patch, AVRO-1027.patch If an Exception is caught while trying to write to a Channel, Netty can deliver the Exception to a ChannelUpstreamHandler on the same thread that attempted to write to the Channel. If this occurs with the NettyClientAvroHandler implementation of ChannelUpstreamHandler then the thread will deadlock. Specifically, NettyClientAvroHandler overrides the ChannelUpstreamHandler.exceptionCaught() method to perform a disconnect, which requires the NettyTransceiver's write lock. However, in the above situation, the thread will already have locked the NettyTransceiver's read lock to write to the Channel. ReentrantReadWriteLock does not allow upgrading from a read to a write lock, hence the thread deadlocks. Example stack trace (simplified): SessionManager-TimeoutPoller prio=10 tid=0x7b689c00 nid=0x375d waiting on condition [0x7b0ad000..0x7b0ade70] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0xf2a944d8 (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114) at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807) [Acquire write lock] at org.apache.avro.ipc.NettyTransceiver.disconnect(NettyTransceiver.java:285) at org.apache.avro.ipc.NettyTransceiver.access$2(NettyTransceiver.java:281) at org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.exceptionCaught(NettyTransceiver.java:499) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122) at org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.handleUpstream(NettyTransceiver.java:473) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783) at org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:238) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:432) at org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:661) at org.jboss.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:372) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:117) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:771) at org.jboss.netty.channel.Channels.write(Channels.java:632) at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582) at org.jboss.netty.channel.Channels.write(Channels.java:611) at org.jboss.netty.channel.Channels.write(Channels.java:578) at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:251) [Acquire read lock] at org.apache.avro.ipc.NettyTransceiver.writeDataPack(NettyTransceiver.java:413) [Acquire read lock] at org.apache.avro.ipc.NettyTransceiver.transceive(NettyTransceiver.java:394
[jira] [Commented] (AVRO-1027) NettyTransceiver will deadlock when attempting transceive/disconnect on the same thread
[ https://issues.apache.org/jira/browse/AVRO-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221291#comment-13221291 ] Doug Cutting commented on AVRO-1027: James, thanks for running your tests. I guess I'll go ahead and commit this now and roll a release candidate today. NettyTransceiver will deadlock when attempting transceive/disconnect on the same thread --- Key: AVRO-1027 URL: https://issues.apache.org/jira/browse/AVRO-1027 Project: Avro Issue Type: Bug Components: java Affects Versions: 1.6.1 Reporter: Simon Wilkinson Assignee: James Baldassari Fix For: 1.6.3 Attachments: AVRO-1027-v2.patch, AVRO-1027.patch If an Exception is caught while trying to write to a Channel, Netty can deliver the Exception to a ChannelUpstreamHandler on the same thread that attempted to write to the Channel. If this occurs with the NettyClientAvroHandler implementation of ChannelUpstreamHandler then the thread will deadlock. Specifically, NettyClientAvroHandler overrides the ChannelUpstreamHandler.exceptionCaught() method to perform a disconnect, which requires the NettyTransceiver's write lock. However, in the above situation, the thread will already have locked the NettyTransceiver's read lock to write to the Channel. ReentrantReadWriteLock does not allow upgrading from a read to a write lock, hence the thread deadlocks. Example stack trace (simplified): SessionManager-TimeoutPoller prio=10 tid=0x7b689c00 nid=0x375d waiting on condition [0x7b0ad000..0x7b0ade70] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0xf2a944d8 (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114) at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807) [Acquire write lock] at org.apache.avro.ipc.NettyTransceiver.disconnect(NettyTransceiver.java:285) at org.apache.avro.ipc.NettyTransceiver.access$2(NettyTransceiver.java:281) at org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.exceptionCaught(NettyTransceiver.java:499) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122) at org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.handleUpstream(NettyTransceiver.java:473) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783) at org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:238) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:432) at org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:661) at org.jboss.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:372) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:117) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:771) at org.jboss.netty.channel.Channels.write(Channels.java:632) at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582) at org.jboss.netty.channel.Channels.write(Channels.java:611) at org.jboss.netty.channel.Channels.write(Channels.java:578) at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:251) [Acquire read lock] at org.apache.avro.ipc.NettyTransceiver.writeDataPack(NettyTransceiver.java:413) [Acquire read lock
[jira] [Commented] (AVRO-1027) NettyTransceiver will deadlock when attempting transceive/disconnect on the same thread
[ https://issues.apache.org/jira/browse/AVRO-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221305#comment-13221305 ] George Fletcher commented on AVRO-1027: --- A release candidate today would be fantastic! Thanks! NettyTransceiver will deadlock when attempting transceive/disconnect on the same thread --- Key: AVRO-1027 URL: https://issues.apache.org/jira/browse/AVRO-1027 Project: Avro Issue Type: Bug Components: java Affects Versions: 1.6.1 Reporter: Simon Wilkinson Assignee: James Baldassari Fix For: 1.6.3 Attachments: AVRO-1027-v2.patch, AVRO-1027.patch If an Exception is caught while trying to write to a Channel, Netty can deliver the Exception to a ChannelUpstreamHandler on the same thread that attempted to write to the Channel. If this occurs with the NettyClientAvroHandler implementation of ChannelUpstreamHandler then the thread will deadlock. Specifically, NettyClientAvroHandler overrides the ChannelUpstreamHandler.exceptionCaught() method to perform a disconnect, which requires the NettyTransceiver's write lock. However, in the above situation, the thread will already have locked the NettyTransceiver's read lock to write to the Channel. ReentrantReadWriteLock does not allow upgrading from a read to a write lock, hence the thread deadlocks. Example stack trace (simplified): SessionManager-TimeoutPoller prio=10 tid=0x7b689c00 nid=0x375d waiting on condition [0x7b0ad000..0x7b0ade70] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0xf2a944d8 (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114) at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807) [Acquire write lock] at org.apache.avro.ipc.NettyTransceiver.disconnect(NettyTransceiver.java:285) at org.apache.avro.ipc.NettyTransceiver.access$2(NettyTransceiver.java:281) at org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.exceptionCaught(NettyTransceiver.java:499) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122) at org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.handleUpstream(NettyTransceiver.java:473) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783) at org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:238) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:432) at org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:661) at org.jboss.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:372) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:117) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:771) at org.jboss.netty.channel.Channels.write(Channels.java:632) at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582) at org.jboss.netty.channel.Channels.write(Channels.java:611) at org.jboss.netty.channel.Channels.write(Channels.java:578) at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:251) [Acquire read lock] at org.apache.avro.ipc.NettyTransceiver.writeDataPack(NettyTransceiver.java:413) [Acquire read lock] at org.apache.avro.ipc.NettyTransceiver.transceive(NettyTransceiver.java:394
[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221342#comment-13221342 ] Doug Cutting commented on AVRO-1006: Raymie SchemaFingerprint.fingerprint seems unnecessarily long... Now this becomes SchemaNormalization.fp(SchemaNormalization.toParsingForm(schema)). The 'fp' might better be spelled out as 'fingerprint'. Also a utility method like SchemaNormalization.parsingFingerprint(schema) might be useful. Graham pass a Normalizer instance... With the latest API, someone can already call SchemaNormalization.fingerprint() with a differently normalized schema, so I don't see the need for this. As we add more normalizers to Avro we can add new methods, so I'm not (yet) seeing the advantage of adding a Normalization interface. Fingerprints for Avro Schemas - Key: AVRO-1006 URL: https://issues.apache.org/jira/browse/AVRO-1006 Project: Avro Issue Type: New Feature Components: java Reporter: Raymie Stata Assignee: Raymie Stata Labels: features Attachments: AVRO-1006-prelim.patch, AVRO-1006.patch, AVRO-1006.patch, AVRO-1006.patch, schema-fingerprinting.html, schema-fingerprinting.html, schema-fingerprinting.html Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1027) NettyTransceiver will deadlock when attempting transceive/disconnect on the same thread
[ https://issues.apache.org/jira/browse/AVRO-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221345#comment-13221345 ] James Baldassari commented on AVRO-1027: Doug, just a minor correction on the attribution for this patch. The original patch was contributed by Simon Wilkinson. I just updated it to apply cleanly with patch -p0, so I think Simon should probably be the contributor for this one. NettyTransceiver will deadlock when attempting transceive/disconnect on the same thread --- Key: AVRO-1027 URL: https://issues.apache.org/jira/browse/AVRO-1027 Project: Avro Issue Type: Bug Components: java Affects Versions: 1.6.1 Reporter: Simon Wilkinson Assignee: James Baldassari Fix For: 1.6.3 Attachments: AVRO-1027-v2.patch, AVRO-1027.patch If an Exception is caught while trying to write to a Channel, Netty can deliver the Exception to a ChannelUpstreamHandler on the same thread that attempted to write to the Channel. If this occurs with the NettyClientAvroHandler implementation of ChannelUpstreamHandler then the thread will deadlock. Specifically, NettyClientAvroHandler overrides the ChannelUpstreamHandler.exceptionCaught() method to perform a disconnect, which requires the NettyTransceiver's write lock. However, in the above situation, the thread will already have locked the NettyTransceiver's read lock to write to the Channel. ReentrantReadWriteLock does not allow upgrading from a read to a write lock, hence the thread deadlocks. Example stack trace (simplified): SessionManager-TimeoutPoller prio=10 tid=0x7b689c00 nid=0x375d waiting on condition [0x7b0ad000..0x7b0ade70] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0xf2a944d8 (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114) at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807) [Acquire write lock] at org.apache.avro.ipc.NettyTransceiver.disconnect(NettyTransceiver.java:285) at org.apache.avro.ipc.NettyTransceiver.access$2(NettyTransceiver.java:281) at org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.exceptionCaught(NettyTransceiver.java:499) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122) at org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.handleUpstream(NettyTransceiver.java:473) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783) at org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:238) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:432) at org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:661) at org.jboss.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:372) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:117) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:771) at org.jboss.netty.channel.Channels.write(Channels.java:632) at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582) at org.jboss.netty.channel.Channels.write(Channels.java:611) at org.jboss.netty.channel.Channels.write(Channels.java:578) at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:251) [Acquire read lock
[jira] [Commented] (AVRO-1027) NettyTransceiver will deadlock when attempting transceive/disconnect on the same thread
[ https://issues.apache.org/jira/browse/AVRO-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221380#comment-13221380 ] Simon Wilkinson commented on AVRO-1027: --- No problem. I've been thinking about how to construct the unit test for this. As James has said, it will be tricky. Specifically, to force the disconnect, the server has to be closed after a client thread has acquired a NettyTransceiver's read lock in transceive(), but before the request gets very far in the NettyTransceiver's Netty pipeline. There doesn't seem any way to make this happen consistently without adding some debug synchronization to NettyTransceiver. NettyTransceiver will deadlock when attempting transceive/disconnect on the same thread --- Key: AVRO-1027 URL: https://issues.apache.org/jira/browse/AVRO-1027 Project: Avro Issue Type: Bug Components: java Affects Versions: 1.6.1 Reporter: Simon Wilkinson Assignee: Simon Wilkinson Fix For: 1.6.3 Attachments: AVRO-1027-v2.patch, AVRO-1027.patch If an Exception is caught while trying to write to a Channel, Netty can deliver the Exception to a ChannelUpstreamHandler on the same thread that attempted to write to the Channel. If this occurs with the NettyClientAvroHandler implementation of ChannelUpstreamHandler then the thread will deadlock. Specifically, NettyClientAvroHandler overrides the ChannelUpstreamHandler.exceptionCaught() method to perform a disconnect, which requires the NettyTransceiver's write lock. However, in the above situation, the thread will already have locked the NettyTransceiver's read lock to write to the Channel. ReentrantReadWriteLock does not allow upgrading from a read to a write lock, hence the thread deadlocks. Example stack trace (simplified): SessionManager-TimeoutPoller prio=10 tid=0x7b689c00 nid=0x375d waiting on condition [0x7b0ad000..0x7b0ade70] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0xf2a944d8 (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114) at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807) [Acquire write lock] at org.apache.avro.ipc.NettyTransceiver.disconnect(NettyTransceiver.java:285) at org.apache.avro.ipc.NettyTransceiver.access$2(NettyTransceiver.java:281) at org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.exceptionCaught(NettyTransceiver.java:499) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122) at org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.handleUpstream(NettyTransceiver.java:473) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783) at org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:238) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:432) at org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:661) at org.jboss.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:372) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:117) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:771) at org.jboss.netty.channel.Channels.write(Channels.java:632) at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582
[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas
[
https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221386#comment-13221386
]
Doug Cutting commented on AVRO-1006:
A few minor nits:
- the 'href' in one of the documentation links is missing its 'h'
- the WHITESPACE comment should perhaps read, Eliminate all whitespace in
JSON outside of string literals.
- we might define a nested FingerprintAlgorithm Enum for the implemented
fingerprint algorithm names.
- SchemaNormalization should probably have a private constructor, e.g.,
'private SchemaNormalization() {}'
- the #fingerprint link in the class documentation is broken.
Otherwise I'm +1 and look forward to committing this early next week unless
there are objections.
Fingerprints for Avro Schemas
-
Key: AVRO-1006
URL: https://issues.apache.org/jira/browse/AVRO-1006
Project: Avro
Issue Type: New Feature
Components: java
Reporter: Raymie Stata
Assignee: Raymie Stata
Labels: features
Attachments: AVRO-1006-prelim.patch, AVRO-1006.patch,
AVRO-1006.patch, AVRO-1006.patch, AVRO-1006.patch,
schema-fingerprinting.html, schema-fingerprinting.html,
schema-fingerprinting.html
Add function that returns a standardized, 64-bit fingerprint for schemas.
Fingerprints are designed such that the chances of collisions is very, very
low.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221388#comment-13221388 ] Raymie Stata commented on AVRO-1006: The string passed to the fingerprint method is in turn passed through to MessageDigest.getInstance. Using an enum here would prevent folks from accessing digest algo's in the underlying library. I suggest leaving that as-is. Otherwise I'll fix the rest of these issues. Fingerprints for Avro Schemas - Key: AVRO-1006 URL: https://issues.apache.org/jira/browse/AVRO-1006 Project: Avro Issue Type: New Feature Components: java Reporter: Raymie Stata Assignee: Raymie Stata Labels: features Attachments: AVRO-1006-prelim.patch, AVRO-1006.patch, AVRO-1006.patch, AVRO-1006.patch, AVRO-1006.patch, schema-fingerprinting.html, schema-fingerprinting.html, schema-fingerprinting.html Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221403#comment-13221403 ] Raymie Stata commented on AVRO-1006: Yes, this is meant to be a cross-language feature, and the definition of parsing normal form is in the Avro spec as a result. To aid in assuring interoperability, this patch includes file-based test data that should be used across languages. Test inputs and outputs can be found in share/test/data/schema-tests.txt Fingerprints for Avro Schemas - Key: AVRO-1006 URL: https://issues.apache.org/jira/browse/AVRO-1006 Project: Avro Issue Type: New Feature Components: java Reporter: Raymie Stata Assignee: Raymie Stata Labels: features Attachments: AVRO-1006-prelim.patch, AVRO-1006.patch, AVRO-1006.patch, AVRO-1006.patch, AVRO-1006.patch, AVRO-1006.patch, schema-fingerprinting.html, schema-fingerprinting.html, schema-fingerprinting.html Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1027) NettyTransceiver will deadlock when attempting transceive/disconnect on the same thread
[
https://issues.apache.org/jira/browse/AVRO-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13221465#comment-13221465
]
Simon Wilkinson commented on AVRO-1027:
---
Correction, the server must close after the Netty Channel has been checked for
'readiness' in NettyTransceiver.writeDataPack(NettyDataPack dataPack), but
before the request gets very far in the NettyTransceiver's Netty pipeline.
I.e.:
private void writeDataPack(NettyDataPack dataPack) throws IOException {
Channel channel = getChannel(); // Channel appears ready
// Server closes
channel.write(dataPack); // Netty will notice the disconnection and handle on
the calling thread
}
NettyTransceiver will deadlock when attempting transceive/disconnect on the
same thread
---
Key: AVRO-1027
URL: https://issues.apache.org/jira/browse/AVRO-1027
Project: Avro
Issue Type: Bug
Components: java
Affects Versions: 1.6.1
Reporter: Simon Wilkinson
Assignee: Simon Wilkinson
Fix For: 1.6.3
Attachments: AVRO-1027-v2.patch, AVRO-1027.patch
If an Exception is caught while trying to write to a Channel, Netty can
deliver the Exception to a ChannelUpstreamHandler on the same thread that
attempted to write to the Channel. If this occurs with the
NettyClientAvroHandler implementation of ChannelUpstreamHandler then the
thread will deadlock.
Specifically, NettyClientAvroHandler overrides the
ChannelUpstreamHandler.exceptionCaught() method to perform a disconnect,
which requires the NettyTransceiver's write lock. However, in the above
situation, the thread will already have locked the NettyTransceiver's read
lock to write to the Channel. ReentrantReadWriteLock does not allow upgrading
from a read to a write lock, hence the thread deadlocks.
Example stack trace (simplified):
SessionManager-TimeoutPoller prio=10 tid=0x7b689c00 nid=0x375d waiting on
condition [0x7b0ad000..0x7b0ade70]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for 0xf2a944d8 (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114)
at
java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807)
[Acquire write lock] at
org.apache.avro.ipc.NettyTransceiver.disconnect(NettyTransceiver.java:285)
at
org.apache.avro.ipc.NettyTransceiver.access$2(NettyTransceiver.java:281)
at
org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.exceptionCaught(NettyTransceiver.java:499)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
at
org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.handleUpstream(NettyTransceiver.java:473)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783)
at
org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:238)
at
org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:432)
at
org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:661)
at
org.jboss.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:372)
at
org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:117)
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:771)
at org.jboss.netty.channel.Channels.write(Channels.java:632)
at
org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
at
org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream
[jira] [Commented] (AVRO-1027) NettyTransceiver will deadlock when attempting transceive/disconnect on the same thread
[ https://issues.apache.org/jira/browse/AVRO-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13220296#comment-13220296 ] Doug Cutting commented on AVRO-1027: Simon, do you expect to supply a test today? It would be good to include this in 1.6.3, but I don't think it's a showstopper, since it's not a regression, is it? NettyTransceiver will deadlock when attempting transceive/disconnect on the same thread --- Key: AVRO-1027 URL: https://issues.apache.org/jira/browse/AVRO-1027 Project: Avro Issue Type: Bug Components: java Affects Versions: 1.6.1 Reporter: Simon Wilkinson Assignee: James Baldassari Fix For: 1.6.3 Attachments: AVRO-1027-v2.patch, AVRO-1027.patch If an Exception is caught while trying to write to a Channel, Netty can deliver the Exception to a ChannelUpstreamHandler on the same thread that attempted to write to the Channel. If this occurs with the NettyClientAvroHandler implementation of ChannelUpstreamHandler then the thread will deadlock. Specifically, NettyClientAvroHandler overrides the ChannelUpstreamHandler.exceptionCaught() method to perform a disconnect, which requires the NettyTransceiver's write lock. However, in the above situation, the thread will already have locked the NettyTransceiver's read lock to write to the Channel. ReentrantReadWriteLock does not allow upgrading from a read to a write lock, hence the thread deadlocks. Example stack trace (simplified): SessionManager-TimeoutPoller prio=10 tid=0x7b689c00 nid=0x375d waiting on condition [0x7b0ad000..0x7b0ade70] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0xf2a944d8 (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114) at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807) [Acquire write lock] at org.apache.avro.ipc.NettyTransceiver.disconnect(NettyTransceiver.java:285) at org.apache.avro.ipc.NettyTransceiver.access$2(NettyTransceiver.java:281) at org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.exceptionCaught(NettyTransceiver.java:499) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122) at org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.handleUpstream(NettyTransceiver.java:473) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783) at org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:238) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:432) at org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:661) at org.jboss.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:372) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:117) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:771) at org.jboss.netty.channel.Channels.write(Channels.java:632) at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582) at org.jboss.netty.channel.Channels.write(Channels.java:611) at org.jboss.netty.channel.Channels.write(Channels.java:578) at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:251) [Acquire read lock] at org.apache.avro.ipc.NettyTransceiver.writeDataPack(NettyTransceiver.java:413
[jira] [Commented] (AVRO-1027) NettyTransceiver will deadlock when attempting transceive/disconnect on the same thread
[ https://issues.apache.org/jira/browse/AVRO-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13220341#comment-13220341 ] Simon Wilkinson commented on AVRO-1027: --- Unfortunately I don't have any time to work on a test until Monday. NettyTransceiver will deadlock when attempting transceive/disconnect on the same thread --- Key: AVRO-1027 URL: https://issues.apache.org/jira/browse/AVRO-1027 Project: Avro Issue Type: Bug Components: java Affects Versions: 1.6.1 Reporter: Simon Wilkinson Assignee: James Baldassari Fix For: 1.6.3 Attachments: AVRO-1027-v2.patch, AVRO-1027.patch If an Exception is caught while trying to write to a Channel, Netty can deliver the Exception to a ChannelUpstreamHandler on the same thread that attempted to write to the Channel. If this occurs with the NettyClientAvroHandler implementation of ChannelUpstreamHandler then the thread will deadlock. Specifically, NettyClientAvroHandler overrides the ChannelUpstreamHandler.exceptionCaught() method to perform a disconnect, which requires the NettyTransceiver's write lock. However, in the above situation, the thread will already have locked the NettyTransceiver's read lock to write to the Channel. ReentrantReadWriteLock does not allow upgrading from a read to a write lock, hence the thread deadlocks. Example stack trace (simplified): SessionManager-TimeoutPoller prio=10 tid=0x7b689c00 nid=0x375d waiting on condition [0x7b0ad000..0x7b0ade70] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0xf2a944d8 (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114) at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807) [Acquire write lock] at org.apache.avro.ipc.NettyTransceiver.disconnect(NettyTransceiver.java:285) at org.apache.avro.ipc.NettyTransceiver.access$2(NettyTransceiver.java:281) at org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.exceptionCaught(NettyTransceiver.java:499) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122) at org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.handleUpstream(NettyTransceiver.java:473) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783) at org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:238) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:432) at org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:661) at org.jboss.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:372) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:117) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:771) at org.jboss.netty.channel.Channels.write(Channels.java:632) at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582) at org.jboss.netty.channel.Channels.write(Channels.java:611) at org.jboss.netty.channel.Channels.write(Channels.java:578) at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:251) [Acquire read lock] at org.apache.avro.ipc.NettyTransceiver.writeDataPack(NettyTransceiver.java:413) [Acquire read lock] at org.apache.avro.ipc.NettyTransceiver.transceive
[jira] [Commented] (AVRO-1027) NettyTransceiver will deadlock when attempting transceive/disconnect on the same thread
[ https://issues.apache.org/jira/browse/AVRO-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13220363#comment-13220363 ] Doug Cutting commented on AVRO-1027: Should we: - a. commit this without tests and roll a 1.6.3 RC today - b. hold off on 1.6.3 until this has tests next week - c. roll a 1.6.3 RC today without this I don't have a strong opinion. NettyTransceiver will deadlock when attempting transceive/disconnect on the same thread --- Key: AVRO-1027 URL: https://issues.apache.org/jira/browse/AVRO-1027 Project: Avro Issue Type: Bug Components: java Affects Versions: 1.6.1 Reporter: Simon Wilkinson Assignee: James Baldassari Fix For: 1.6.3 Attachments: AVRO-1027-v2.patch, AVRO-1027.patch If an Exception is caught while trying to write to a Channel, Netty can deliver the Exception to a ChannelUpstreamHandler on the same thread that attempted to write to the Channel. If this occurs with the NettyClientAvroHandler implementation of ChannelUpstreamHandler then the thread will deadlock. Specifically, NettyClientAvroHandler overrides the ChannelUpstreamHandler.exceptionCaught() method to perform a disconnect, which requires the NettyTransceiver's write lock. However, in the above situation, the thread will already have locked the NettyTransceiver's read lock to write to the Channel. ReentrantReadWriteLock does not allow upgrading from a read to a write lock, hence the thread deadlocks. Example stack trace (simplified): SessionManager-TimeoutPoller prio=10 tid=0x7b689c00 nid=0x375d waiting on condition [0x7b0ad000..0x7b0ade70] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0xf2a944d8 (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114) at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807) [Acquire write lock] at org.apache.avro.ipc.NettyTransceiver.disconnect(NettyTransceiver.java:285) at org.apache.avro.ipc.NettyTransceiver.access$2(NettyTransceiver.java:281) at org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.exceptionCaught(NettyTransceiver.java:499) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122) at org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.handleUpstream(NettyTransceiver.java:473) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783) at org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:238) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:432) at org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:661) at org.jboss.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:372) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:117) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:771) at org.jboss.netty.channel.Channels.write(Channels.java:632) at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582) at org.jboss.netty.channel.Channels.write(Channels.java:611) at org.jboss.netty.channel.Channels.write(Channels.java:578) at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:251) [Acquire read lock] at org.apache.avro.ipc.NettyTransceiver.writeDataPack
[jira] [Commented] (AVRO-1027) NettyTransceiver will deadlock when attempting transceive/disconnect on the same thread
[ https://issues.apache.org/jira/browse/AVRO-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13220399#comment-13220399 ] James Baldassari commented on AVRO-1027: Creating a test that reliably triggers these types of deadlocks can be tricky. I fixed a bug not too long ago that I could reproduce reliably with my own unit tests but couldn't seem to get it to happen in the Avro tests for whatever reason. I think this is a fairly low-risk change, and it would be unfortunate if people hit this in production. So I guess my vote would be a +0 on getting this in for 1.6.3 whether the RC gets rolled today or early next week. I could build a snapshot off trunk + AVRO-1027 and run it through my own suite of Avro Netty unit tests. That might at least give us some additional confidence that this won't introduce any regressions. NettyTransceiver will deadlock when attempting transceive/disconnect on the same thread --- Key: AVRO-1027 URL: https://issues.apache.org/jira/browse/AVRO-1027 Project: Avro Issue Type: Bug Components: java Affects Versions: 1.6.1 Reporter: Simon Wilkinson Assignee: James Baldassari Fix For: 1.6.3 Attachments: AVRO-1027-v2.patch, AVRO-1027.patch If an Exception is caught while trying to write to a Channel, Netty can deliver the Exception to a ChannelUpstreamHandler on the same thread that attempted to write to the Channel. If this occurs with the NettyClientAvroHandler implementation of ChannelUpstreamHandler then the thread will deadlock. Specifically, NettyClientAvroHandler overrides the ChannelUpstreamHandler.exceptionCaught() method to perform a disconnect, which requires the NettyTransceiver's write lock. However, in the above situation, the thread will already have locked the NettyTransceiver's read lock to write to the Channel. ReentrantReadWriteLock does not allow upgrading from a read to a write lock, hence the thread deadlocks. Example stack trace (simplified): SessionManager-TimeoutPoller prio=10 tid=0x7b689c00 nid=0x375d waiting on condition [0x7b0ad000..0x7b0ade70] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0xf2a944d8 (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114) at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807) [Acquire write lock] at org.apache.avro.ipc.NettyTransceiver.disconnect(NettyTransceiver.java:285) at org.apache.avro.ipc.NettyTransceiver.access$2(NettyTransceiver.java:281) at org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.exceptionCaught(NettyTransceiver.java:499) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122) at org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.handleUpstream(NettyTransceiver.java:473) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783) at org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:238) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:432) at org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:661) at org.jboss.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:372) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:117) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:771) at org.jboss.netty.channel.Channels.write(Channels.java:632) at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70
[jira] [Commented] (AVRO-1027) NettyTransceiver will deadlock when attempting transceive/disconnect on the same thread
[ https://issues.apache.org/jira/browse/AVRO-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13219242#comment-13219242 ] Simon Wilkinson commented on AVRO-1027: --- I could trigger the bug as part of a larger test. I'll see if I can extract this into a specific unit test. NettyTransceiver will deadlock when attempting transceive/disconnect on the same thread --- Key: AVRO-1027 URL: https://issues.apache.org/jira/browse/AVRO-1027 Project: Avro Issue Type: Bug Components: java Affects Versions: 1.6.1 Reporter: Simon Wilkinson Assignee: James Baldassari Fix For: 1.6.3 Attachments: AVRO-1027-v2.patch, AVRO-1027.patch If an Exception is caught while trying to write to a Channel, Netty can deliver the Exception to a ChannelUpstreamHandler on the same thread that attempted to write to the Channel. If this occurs with the NettyClientAvroHandler implementation of ChannelUpstreamHandler then the thread will deadlock. Specifically, NettyClientAvroHandler overrides the ChannelUpstreamHandler.exceptionCaught() method to perform a disconnect, which requires the NettyTransceiver's write lock. However, in the above situation, the thread will already have locked the NettyTransceiver's read lock to write to the Channel. ReentrantReadWriteLock does not allow upgrading from a read to a write lock, hence the thread deadlocks. Example stack trace (simplified): SessionManager-TimeoutPoller prio=10 tid=0x7b689c00 nid=0x375d waiting on condition [0x7b0ad000..0x7b0ade70] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0xf2a944d8 (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114) at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807) [Acquire write lock] at org.apache.avro.ipc.NettyTransceiver.disconnect(NettyTransceiver.java:285) at org.apache.avro.ipc.NettyTransceiver.access$2(NettyTransceiver.java:281) at org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.exceptionCaught(NettyTransceiver.java:499) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122) at org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.handleUpstream(NettyTransceiver.java:473) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783) at org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:238) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:432) at org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:661) at org.jboss.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:372) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:117) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:771) at org.jboss.netty.channel.Channels.write(Channels.java:632) at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582) at org.jboss.netty.channel.Channels.write(Channels.java:611) at org.jboss.netty.channel.Channels.write(Channels.java:578) at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:251) [Acquire read lock] at org.apache.avro.ipc.NettyTransceiver.writeDataPack(NettyTransceiver.java:413) [Acquire read lock
[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13219815#comment-13219815 ] Raymie Stata commented on AVRO-1006: On the one hand, the Parsing Canonical Form as I've defined it has important uses and I think should be part of the Avro system. On the other hand, it's a good point that there may be additional canonical forms that satisfy different needs in the future. The class- and method-names of the previous patch implicitly assumed a single canonical form. I've modified the patch with new names that will better support the addition of new canonical forms in the future. (I think I also fixed the Forrest problems, although I haven't been able to verify.) Fingerprints for Avro Schemas - Key: AVRO-1006 URL: https://issues.apache.org/jira/browse/AVRO-1006 Project: Avro Issue Type: New Feature Components: java Reporter: Raymie Stata Assignee: Raymie Stata Labels: features Attachments: AVRO-1006-prelim.patch, AVRO-1006.patch, AVRO-1006.patch, AVRO-1006.patch, schema-fingerprinting.html, schema-fingerprinting.html, schema-fingerprinting.html Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13219859#comment-13219859 ] graham sanderson commented on AVRO-1006: Raymie, thank you; I think we're in agreement. However the latest patch has class SchemaNormalization with static toParsingForm and static build methods the other static methods reference these specific versions directly. Perhaps it would be best (since the guts of the parsing form are in the current build method) to leave SchemaNormalization as a class of static methods, but pass a Normalizer instance to most of the static method... in this case the current build method would basically be the guts of the ParsingFormNormalizer class Note the common Normalizer instances would likely be sensibly named singletons in the SchemaNormalizer class Fingerprints for Avro Schemas - Key: AVRO-1006 URL: https://issues.apache.org/jira/browse/AVRO-1006 Project: Avro Issue Type: New Feature Components: java Reporter: Raymie Stata Assignee: Raymie Stata Labels: features Attachments: AVRO-1006-prelim.patch, AVRO-1006.patch, AVRO-1006.patch, AVRO-1006.patch, schema-fingerprinting.html, schema-fingerprinting.html, schema-fingerprinting.html Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1027) NettyTransceiver will deadlock when attempting transceive/disconnect on the same thread
[ https://issues.apache.org/jira/browse/AVRO-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13218944#comment-13218944 ] James Baldassari commented on AVRO-1027: Simon, do you have a test for this? If not, I'll try to make one. Not sure when I'll have time to work on it, though. NettyTransceiver will deadlock when attempting transceive/disconnect on the same thread --- Key: AVRO-1027 URL: https://issues.apache.org/jira/browse/AVRO-1027 Project: Avro Issue Type: Bug Components: java Affects Versions: 1.6.1 Reporter: Simon Wilkinson Assignee: James Baldassari Fix For: 1.6.3 Attachments: AVRO-1027-v2.patch, AVRO-1027.patch If an Exception is caught while trying to write to a Channel, Netty can deliver the Exception to a ChannelUpstreamHandler on the same thread that attempted to write to the Channel. If this occurs with the NettyClientAvroHandler implementation of ChannelUpstreamHandler then the thread will deadlock. Specifically, NettyClientAvroHandler overrides the ChannelUpstreamHandler.exceptionCaught() method to perform a disconnect, which requires the NettyTransceiver's write lock. However, in the above situation, the thread will already have locked the NettyTransceiver's read lock to write to the Channel. ReentrantReadWriteLock does not allow upgrading from a read to a write lock, hence the thread deadlocks. Example stack trace (simplified): SessionManager-TimeoutPoller prio=10 tid=0x7b689c00 nid=0x375d waiting on condition [0x7b0ad000..0x7b0ade70] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0xf2a944d8 (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114) at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807) [Acquire write lock] at org.apache.avro.ipc.NettyTransceiver.disconnect(NettyTransceiver.java:285) at org.apache.avro.ipc.NettyTransceiver.access$2(NettyTransceiver.java:281) at org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.exceptionCaught(NettyTransceiver.java:499) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122) at org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.handleUpstream(NettyTransceiver.java:473) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783) at org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:238) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:432) at org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:661) at org.jboss.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:372) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:117) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:771) at org.jboss.netty.channel.Channels.write(Channels.java:632) at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582) at org.jboss.netty.channel.Channels.write(Channels.java:611) at org.jboss.netty.channel.Channels.write(Channels.java:578) at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:251) [Acquire read lock] at org.apache.avro.ipc.NettyTransceiver.writeDataPack(NettyTransceiver.java:413) [Acquire read lock
[jira] [Commented] (AVRO-1027) NettyTransceiver will deadlock when attempting transceive/disconnect on the same thread
[ https://issues.apache.org/jira/browse/AVRO-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13216980#comment-13216980 ] Scott Carey commented on AVRO-1027: --- +1 Looks good. Is there a test that could cover this? NettyTransceiver will deadlock when attempting transceive/disconnect on the same thread --- Key: AVRO-1027 URL: https://issues.apache.org/jira/browse/AVRO-1027 Project: Avro Issue Type: Bug Components: java Affects Versions: 1.6.1 Reporter: Simon Wilkinson Assignee: James Baldassari Fix For: 1.6.3 Attachments: AVRO-1027-v2.patch, AVRO-1027.patch If an Exception is caught while trying to write to a Channel, Netty can deliver the Exception to a ChannelUpstreamHandler on the same thread that attempted to write to the Channel. If this occurs with the NettyClientAvroHandler implementation of ChannelUpstreamHandler then the thread will deadlock. Specifically, NettyClientAvroHandler overrides the ChannelUpstreamHandler.exceptionCaught() method to perform a disconnect, which requires the NettyTransceiver's write lock. However, in the above situation, the thread will already have locked the NettyTransceiver's read lock to write to the Channel. ReentrantReadWriteLock does not allow upgrading from a read to a write lock, hence the thread deadlocks. Example stack trace (simplified): SessionManager-TimeoutPoller prio=10 tid=0x7b689c00 nid=0x375d waiting on condition [0x7b0ad000..0x7b0ade70] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0xf2a944d8 (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1114) at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:807) [Acquire write lock] at org.apache.avro.ipc.NettyTransceiver.disconnect(NettyTransceiver.java:285) at org.apache.avro.ipc.NettyTransceiver.access$2(NettyTransceiver.java:281) at org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.exceptionCaught(NettyTransceiver.java:499) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122) at org.apache.avro.ipc.NettyTransceiver$NettyClientAvroHandler.handleUpstream(NettyTransceiver.java:473) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:783) at org.jboss.netty.handler.codec.frame.FrameDecoder.exceptionCaught(FrameDecoder.java:238) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:122) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at org.jboss.netty.channel.Channels.fireExceptionCaught(Channels.java:432) at org.jboss.netty.channel.socket.nio.NioWorker.cleanUpWriteBuffer(NioWorker.java:661) at org.jboss.netty.channel.socket.nio.NioWorker.writeFromUserCode(NioWorker.java:372) at org.jboss.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:117) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:771) at org.jboss.netty.channel.Channels.write(Channels.java:632) at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582) at org.jboss.netty.channel.Channels.write(Channels.java:611) at org.jboss.netty.channel.Channels.write(Channels.java:578) at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:251) [Acquire read lock] at org.apache.avro.ipc.NettyTransceiver.writeDataPack(NettyTransceiver.java:413) [Acquire read lock] at org.apache.avro.ipc.NettyTransceiver.transceive(NettyTransceiver.java:394
[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13216629#comment-13216629 ] graham sanderson commented on AVRO-1006: A clarification, which addresses issues raised by Doug and Scott. The need I'm solving for is to capture that part of a writer's schema which a reader needs to read data. This is a relatively straight-forward notion of equivalence, and a very useful one. And the good news is that this notion of equivalence allows us to ignore many aspects of schemas (e.g., attributes, aliases, default values). Perhaps this should be made clearer (when naming the class/method), I came across this feature because of a desire to hash/fingerprint avro schemas for messaging, and was seeing if there was already a util to do it. In my case I potentially might use custom properties on fields in the schema to indicate they are being transmitted using a certain named dictionary and thus in my case they affect the ability to interpret the message, so I'd rather stick with something that I can reliably use on the producer end to encode the entire state of the schema, rather than a particular well defined sub-set of the schema. Note that (thanks to someone making Props a LinkedHashMap since the code base I'm using) and the particular implementation of Jackson, schema.toString() in the Java impl appears like it will be fine for my purposes, and if another language implementation happens to produce a different hash value I'm cool with that, as long as it is relatively stable; for example: SchemaInstance1 -toJson- string x string x -fromJson- SchemInstance2 - toJson string y string x and string y being equal seems a reasonable enough guarantee for me Fingerprints for Avro Schemas - Key: AVRO-1006 URL: https://issues.apache.org/jira/browse/AVRO-1006 Project: Avro Issue Type: New Feature Components: java Reporter: Raymie Stata Assignee: Raymie Stata Labels: features Attachments: AVRO-1006-prelim.patch, AVRO-1006.patch, AVRO-1006.patch, schema-fingerprinting.html, schema-fingerprinting.html, schema-fingerprinting.html Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13216630#comment-13216630 ] graham sanderson commented on AVRO-1006: Note, more succinctly, given that I and other people in this issue have been discussing or inferring schema repositories, I think maybe it makes sense in general or at least as a separate issue to deal with a digest of the schema in its entirety, even if that results in a number of hash values mapping to equivalent schemas. Generally I think the use cases don't involve insane quantities of schemas in the first place (although I won't say that for certain, because the ability to define an anonymous schema on the fly for certain applications is rather appealing) Fingerprints for Avro Schemas - Key: AVRO-1006 URL: https://issues.apache.org/jira/browse/AVRO-1006 Project: Avro Issue Type: New Feature Components: java Reporter: Raymie Stata Assignee: Raymie Stata Labels: features Attachments: AVRO-1006-prelim.patch, AVRO-1006.patch, AVRO-1006.patch, schema-fingerprinting.html, schema-fingerprinting.html, schema-fingerprinting.html Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13215480#comment-13215480 ] Raymie Stata commented on AVRO-1006: ['fp' or 'fingerprint'] I renamed classes to fingerprint, but left methods fp. SchemaFingerprint.fingerprint seems unnecessarily long... [put in o.a.avro] Good idea -- I moved it. [missing javadoc] Added [breaks Forrest] I can't get Forrest running on my machine -- can you send me the errors? (Or maybe just fix them?) Fingerprints for Avro Schemas - Key: AVRO-1006 URL: https://issues.apache.org/jira/browse/AVRO-1006 Project: Avro Issue Type: New Feature Components: java Reporter: Raymie Stata Assignee: Raymie Stata Labels: features Attachments: AVRO-1006-prelim.patch, AVRO-1006.patch, AVRO-1006.patch, schema-fingerprinting.html, schema-fingerprinting.html, schema-fingerprinting.html Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1036) IDL processing fails with multi-level nested imports
[ https://issues.apache.org/jira/browse/AVRO-1036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13214888#comment-13214888 ] George Fletcher commented on AVRO-1036: --- So the bug is that findFile() returns and InputStream and the Idl class constructor that takes an InputStream is created by the javacc processing of the Idl.jj file. This Idl constructor does not set the inputDir field of the Idl class and hence the Idl object created from the stream returned by findFile() has lost the sourceDirectory. This causes an import statement processed by this Idl object to fail, because the file can not be found as the sourceDirectory is lost. I'm not very happy with my current solution as it seems a little hacky. Suggestions appreciated. Current solution... 1. Add a setSrcDir() method to the Idl class (added to Idl.jj) 2. Modify the ImportIdl() method to first create the Idl instance from the InputStream returned by findFile() and then call the setSrcDir() setter chaining the current inputDir into the created Idl instance. IDL processing fails with multi-level nested imports Key: AVRO-1036 URL: https://issues.apache.org/jira/browse/AVRO-1036 Project: Avro Issue Type: Bug Components: java Affects Versions: 1.6.2 Reporter: George Fletcher Fix For: 1.6.3 The change to support finding IDL related files on the classpath in addition to the maven-plugin defined directory caused the context of the sourceDirectory to be lost when the InputStream return by findFile() is used to create a new Idl instance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-999) NPE in Java, RecordBuilderBase.defaultValue
[
https://issues.apache.org/jira/browse/AVRO-999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13214909#comment-13214909
]
Doug Cutting commented on AVRO-999:
---
The test case added in this patch passes with the changes in AVRO-1007. Also,
the tests added in AVRO-1007 are substantially similar to the test added here.
NPE in Java, RecordBuilderBase.defaultValue
---
Key: AVRO-999
URL: https://issues.apache.org/jira/browse/AVRO-999
Project: Avro
Issue Type: Bug
Components: java
Affects Versions: 1.6.1
Environment: Java
Reporter: Jay Rutten
Assignee: James Baldassari
Attachments: AVRO-999.patch
If you have a union with a default of null, the code in
RecordBuilderBase.defaultValue will cause an NPE in ConcurrentHashMap, since
it is trying to add a null to the map.
Sample union:
{code}
record Sample {
union{null, string} value = null;
}
{code}
Code:
{code}
// If not cached, get the default Java value by encoding the default JSON
// value and then decoding it:
if (defaultValue == null) {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
encoder = EncoderFactory.get().binaryEncoder(baos, encoder);
ResolvingGrammarGenerator.encode(encoder, field.schema(), defaultJsonValue);
encoder.flush();
decoder = DecoderFactory.get().binaryDecoder(baos.toByteArray(), decoder);
defaultValue = new GenericDatumReader(field.schema()).read(null, decoder);
defaultSchemaValues.putIfAbsent(field.pos(), defaultValue); // -- NPE from
here
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-999) NPE in Java, RecordBuilderBase.defaultValue
[
https://issues.apache.org/jira/browse/AVRO-999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13214916#comment-13214916
]
James Baldassari commented on AVRO-999:
---
OK, great. Then it should be fine to resolve this one if the AVRO-1007 tests
cover this case.
NPE in Java, RecordBuilderBase.defaultValue
---
Key: AVRO-999
URL: https://issues.apache.org/jira/browse/AVRO-999
Project: Avro
Issue Type: Bug
Components: java
Affects Versions: 1.6.1
Environment: Java
Reporter: Jay Rutten
Assignee: James Baldassari
Attachments: AVRO-999.patch
If you have a union with a default of null, the code in
RecordBuilderBase.defaultValue will cause an NPE in ConcurrentHashMap, since
it is trying to add a null to the map.
Sample union:
{code}
record Sample {
union{null, string} value = null;
}
{code}
Code:
{code}
// If not cached, get the default Java value by encoding the default JSON
// value and then decoding it:
if (defaultValue == null) {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
encoder = EncoderFactory.get().binaryEncoder(baos, encoder);
ResolvingGrammarGenerator.encode(encoder, field.schema(), defaultJsonValue);
encoder.flush();
decoder = DecoderFactory.get().binaryDecoder(baos.toByteArray(), decoder);
defaultValue = new GenericDatumReader(field.schema()).read(null, decoder);
defaultSchemaValues.putIfAbsent(field.pos(), defaultValue); // -- NPE from
here
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1006) Fingerprints for Avro Schemas
[ https://issues.apache.org/jira/browse/AVRO-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13214932#comment-13214932 ] Doug Cutting commented on AVRO-1006: This looks generally good. A few nits: - In class and method names, should we abbreviate 'fp' or spell out 'fingerprint'? FP means floating point to my eye. - Might we instead put this in org.apache.avro.SchemaFingerprint, rather than in util? Right now things in the util package depend only on the JDK, not on other parts of Avro. - Public methods and classes need javadoc comments. - The changes to the spec are not correctly processed by Forrest 0.8 for me. Fingerprints for Avro Schemas - Key: AVRO-1006 URL: https://issues.apache.org/jira/browse/AVRO-1006 Project: Avro Issue Type: New Feature Components: java Reporter: Raymie Stata Assignee: Raymie Stata Labels: features Attachments: AVRO-1006-prelim.patch, AVRO-1006.patch, schema-fingerprinting.html, schema-fingerprinting.html, schema-fingerprinting.html Add function that returns a standardized, 64-bit fingerprint for schemas. Fingerprints are designed such that the chances of collisions is very, very low. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1036) IDL processing fails with multi-level nested imports
[ https://issues.apache.org/jira/browse/AVRO-1036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13215040#comment-13215040 ] George Fletcher commented on AVRO-1036: --- Decided a cleaner solution was to create a new constructor public Idl (InputStream, File, ClassLoader) that chains the source directory and classpath resource loader into the new Idl instance. IDL processing fails with multi-level nested imports Key: AVRO-1036 URL: https://issues.apache.org/jira/browse/AVRO-1036 Project: Avro Issue Type: Bug Components: java Affects Versions: 1.6.2 Reporter: George Fletcher Fix For: 1.6.3 The change to support finding IDL related files on the classpath in addition to the maven-plugin defined directory caused the context of the sourceDirectory to be lost when the InputStream return by findFile() is used to create a new Idl instance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1036) IDL processing fails with multi-level nested imports
[ https://issues.apache.org/jira/browse/AVRO-1036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13215125#comment-13215125 ] Doug Cutting commented on AVRO-1036: Looks like we patched this in parallel! You added some tests, which is great! My patch is a little different. When you include a file in a different directory then any imports it contains should be relative to its directory, not the directory of the original file, no? So this.inputDir is not the right value for the new inputDir, but rather it should come from the imported file. IDL processing fails with multi-level nested imports Key: AVRO-1036 URL: https://issues.apache.org/jira/browse/AVRO-1036 Project: Avro Issue Type: Bug Components: java Affects Versions: 1.6.2 Reporter: George Fletcher Assignee: Doug Cutting Fix For: 1.6.3 Attachments: AVRO-1036.patch, jira-1036.patch The change to support finding IDL related files on the classpath in addition to the maven-plugin defined directory caused the context of the sourceDirectory to be lost when the InputStream return by findFile() is used to create a new Idl instance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1037) Problems using Avro 1.6.2 with Hadoop (CDH3 or 1.0)
[
https://issues.apache.org/jira/browse/AVRO-1037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13215157#comment-13215157
]
Scott Carey commented on AVRO-1037:
---
The stack trace I ran into was:
{noformat}
java.lang.NoSuchMethodError: org.codehaus.jackson.JsonNode.getValueAsDouble()D
at org.apache.avro.Schema$Field.defaultValueEquals(Schema.java:461)
at org.apache.avro.Schema$Field.equals(Schema.java:451)
at java.util.AbstractList.equals(AbstractList.java:507)
at org.apache.avro.Schema$RecordSchema.equals(Schema.java:647)
at java.util.HashMap.put(HashMap.java:376)
{noformat}
Problems using Avro 1.6.2 with Hadoop (CDH3 or 1.0)
---
Key: AVRO-1037
URL: https://issues.apache.org/jira/browse/AVRO-1037
Project: Avro
Issue Type: Bug
Components: java
Reporter: Scott Carey
Assignee: Scott Carey
AVRO-972 uses a method on JsonNode that was introduced with Jackson 1.6.0.
Unfortunately, Hadoop in a distributed environment places jackson on the
mapreduce classpath prior to a user's Jar.
Avro should function when this happens.
There are two things we are using that are new in Jackson 1.6.0 --
JsonNode.getValueAsDouble() and MinimalPrettyPrinter
MinimalPrettyPrinter is OK because the class is new, and if there is a
Jackson 1.5.x jar on a classpath before one that is = 1.6.0, it will find
the class in the second jar. We need to avoid JsonNode.getValueAsDouble()
however. There is a replacement that is equivalent and existed prior,
JsonNode.getDoubleValue().
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1037) Problems using Avro 1.6.2 with Hadoop (CDH3 or 1.0)
[ https://issues.apache.org/jira/browse/AVRO-1037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13215173#comment-13215173 ] Scott Carey commented on AVRO-1037: --- If there are no objections, I will check this in tomorrow. I would like to get a 1.6.3 with this in it out soon. Problems using Avro 1.6.2 with Hadoop (CDH3 or 1.0) --- Key: AVRO-1037 URL: https://issues.apache.org/jira/browse/AVRO-1037 Project: Avro Issue Type: Bug Components: java Affects Versions: 1.6.2 Reporter: Scott Carey Assignee: Scott Carey Fix For: 1.6.3 Attachments: AVRO-1037.patch AVRO-972 uses a method on JsonNode that was introduced with Jackson 1.6.0. Unfortunately, Hadoop in a distributed environment places jackson on the mapreduce classpath prior to a user's Jar. Avro should function when this happens. There are two things we are using that are new in Jackson 1.6.0 -- JsonNode.getValueAsDouble() and MinimalPrettyPrinter MinimalPrettyPrinter is OK because the class is new, and if there is a Jackson 1.5.x jar on a classpath before one that is = 1.6.0, it will find the class in the second jar. We need to avoid JsonNode.getValueAsDouble() however. There is a replacement that is equivalent and existed prior, JsonNode.getDoubleValue(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1036) IDL processing fails with multi-level nested imports
[ https://issues.apache.org/jira/browse/AVRO-1036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13215214#comment-13215214 ] George Fletcher commented on AVRO-1036: --- Yes, that's a good point and I like your patch better. One other thought I had is that we could just add the sourceDirectory to the ClassLoader in the IDLProtocolMojo.java and then just use the ClassLoader mechanism across the board. IDL processing fails with multi-level nested imports Key: AVRO-1036 URL: https://issues.apache.org/jira/browse/AVRO-1036 Project: Avro Issue Type: Bug Components: java Affects Versions: 1.6.2 Reporter: George Fletcher Assignee: Doug Cutting Fix For: 1.6.3 Attachments: AVRO-1036.patch, jira-1036.patch The change to support finding IDL related files on the classpath in addition to the maven-plugin defined directory caused the context of the sourceDirectory to be lost when the InputStream return by findFile() is used to create a new Idl instance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1036) IDL processing fails with multi-level nested imports
[ https://issues.apache.org/jira/browse/AVRO-1036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13215285#comment-13215285 ] Scott Carey commented on AVRO-1036: --- I do not see any tests in the last patch. IDL processing fails with multi-level nested imports Key: AVRO-1036 URL: https://issues.apache.org/jira/browse/AVRO-1036 Project: Avro Issue Type: Bug Components: java Affects Versions: 1.6.2 Reporter: George Fletcher Assignee: Doug Cutting Fix For: 1.6.3 Attachments: AVRO-1036.patch, AVRO-1036.patch, jira-1036.patch The change to support finding IDL related files on the classpath in addition to the maven-plugin defined directory caused the context of the sourceDirectory to be lost when the InputStream return by findFile() is used to create a new Idl instance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1029) I want to transfer files beetwin server and client by Avro-c.
[ https://issues.apache.org/jira/browse/AVRO-1029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13215334#comment-13215334 ] menghaocheng commented on AVRO-1029: Thanks Dounglas. I think you have got exactly what I meant. I know the step to do it. 1)got the binary encoding. 2)transmit them by TCP socket. The 2nd step is easy. But there is a question to the 1st step. I didn't know how to get the binary encoding. The C API in the Documentation doesn't mention at all http://avro.apache.org/docs/1.5.4/api/c/index.html. So I have to read the rsc files. I find a function in the file src/avro/consumer.h: avro_consume_binary(avro_reader_t reader,avro_consumer_t *consumer, void *ud); I'm not sure if it is the right function that I need. I want to transfer files beetwin server and client by Avro-c. - Key: AVRO-1029 URL: https://issues.apache.org/jira/browse/AVRO-1029 Project: Avro Issue Type: Test Components: c Affects Versions: 1.5.4 Environment: avro in C, linux in vmware. Reporter: menghaocheng Labels: test Fix For: 1.5.4 Original Estimate: 168h Remaining Estimate: 168h I want to transfer files beetwin server and client by Avro-c. I know that there must be two parts in this model, the server and the client. My question is how to make the connection? Please forgive me as this maybe a simle question, But I'm fresh in Avro, your simple worlds maybe helpful for me. If only there is an example for it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-593) Avro mapreduce apis incompatible with hadoop 0.20.2
[ https://issues.apache.org/jira/browse/AVRO-593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13213866#comment-13213866 ] Doug Cutting commented on AVRO-593: --- Ideally, anything in the .io, .util, and .file packages does not reference the .mapred or .mapreduce packages [ ... ] Much in these packages references AvroKey and AvroValue and/or AvroJob. These uses aren't mapreduce-specific and could be refactored away, e.g., by moving AvroKey and AvroValue from o.a.a.mapred to o.a.a.hadoop.io, but that would be incompatible. SortedKeyValueFile is the Avro equivalent of Hadoop's MapFile. Arguably it should be moved into o.a.a.io. It depends on AvroKeyValue, which might also be moved to the core. AvroKeyValue is very similar in functionality to o.a.a.mapred.Pair. Perhaps SortedKeyValueFile should be switched to use Pair and both moved to the core. I have implemented a SequenceFile shim and it works. There's now just a tiny class that needs to be in o.a.h.io, a base class that exposes two package-private nested classes from within SequenceFile. I've re-arranged the classes per Scott's #4 variant but can revert that. We need to decide how much refactoring we want to do here. Finally, I note that io.SeekableHadoopInput replicates functionality that's already in mapred.FsInput, so we should replace the former with the latter in the new code. Avro mapreduce apis incompatible with hadoop 0.20.2 --- Key: AVRO-593 URL: https://issues.apache.org/jira/browse/AVRO-593 Project: Avro Issue Type: Bug Components: java Affects Versions: 1.3.2, 1.3.3 Environment: Avro 1.3.3, Hadoop 0.20.2 Reporter: Steve Severance Assignee: Garrett Wu Attachments: AVRO-593.patch, AVRO-593.patch The avro api's for hadoop use the hadoop mapreduce api that has been deprecated. A new avro mapreduce api should be implemented for hadoop 0.20 and higher. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1035) Add the possibility to append to existing avro files
[
https://issues.apache.org/jira/browse/AVRO-1035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13213874#comment-13213874
]
Doug Cutting commented on AVRO-1035:
Note that append is not reliable in current Hadoop releases. Append support
in Hadoop 1.0 just means that flush() works reliably, not that append actually
works. Append should be reliable in 0.23 releases although I doubt it's been
well tested there yet.
Add the possibility to append to existing avro files
--
Key: AVRO-1035
URL: https://issues.apache.org/jira/browse/AVRO-1035
Project: Avro
Issue Type: New Feature
Reporter: Vyacheslav Zholudev
Currently it is not possible to append to avro files that were written and
closed.
Here is a Scott Carey's reply on the mailing list:
{quote}
It is not possible without modifying DataFileWriter. Please open a JIRA
ticket.
It could not simply append to an OutputStream, since it must either:
* Seek to the start to validate the schemas match and find the sync
marker, or
* Trust that the schemas match and find the sync marker from the last block
DataFileWriter cannot refer to Hadoop classes such as FileSystem, but we
could add something to the mapred module that takes a Path and FileSystem
and returns
something that implemements an interface that DataFileWriter can append
to. This would be something that is both a
http://avro.apache.org/docs/1.6.2/api/java/org/apache/avro/file/SeekableInp
ut.html
and an OutputStream, or has both an InputStream from the start of the
existing file and an OutputStream at the end.
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1023) Saved state should be restored in finally clause
[ https://issues.apache.org/jira/browse/AVRO-1023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13214094#comment-13214094 ] Doug Cutting commented on AVRO-1023: The way this might be triggered is if a Schema.Parser is reused after a SchemaParseException is thrown. Currently the default namespace is that of the preceding schema parsed. If a SchemaParseException is thrown and the parser is reused then the default namespace could be that of a schema nested within the previous schema. Saved state should be restored in finally clause Key: AVRO-1023 URL: https://issues.apache.org/jira/browse/AVRO-1023 Project: Avro Issue Type: Bug Components: java Reporter: Raymie Stata Assignee: Raymie Stata Priority: Minor Attachments: AVRO-1023.patch Schema.parse(JsonParse) and Schema.parse(JsonNode,Names) save global state in a local variable; they should restore that state in a finally clause but they don't. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-999) NPE in Java, RecordBuilderBase.defaultValue
[
https://issues.apache.org/jira/browse/AVRO-999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13214103#comment-13214103
]
Doug Cutting commented on AVRO-999:
---
I think this was already fixed in AVRO-1007. Can I close it as a duplicate?
NPE in Java, RecordBuilderBase.defaultValue
---
Key: AVRO-999
URL: https://issues.apache.org/jira/browse/AVRO-999
Project: Avro
Issue Type: Bug
Components: java
Affects Versions: 1.6.1
Environment: Java
Reporter: Jay Rutten
Assignee: James Baldassari
Attachments: AVRO-999.patch
If you have a union with a default of null, the code in
RecordBuilderBase.defaultValue will cause an NPE in ConcurrentHashMap, since
it is trying to add a null to the map.
Sample union:
{code}
record Sample {
union{null, string} value = null;
}
{code}
Code:
{code}
// If not cached, get the default Java value by encoding the default JSON
// value and then decoding it:
if (defaultValue == null) {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
encoder = EncoderFactory.get().binaryEncoder(baos, encoder);
ResolvingGrammarGenerator.encode(encoder, field.schema(), defaultJsonValue);
encoder.flush();
decoder = DecoderFactory.get().binaryDecoder(baos.toByteArray(), decoder);
defaultValue = new GenericDatumReader(field.schema()).read(null, decoder);
defaultSchemaValues.putIfAbsent(field.pos(), defaultValue); // -- NPE from
here
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-999) NPE in Java, RecordBuilderBase.defaultValue
[
https://issues.apache.org/jira/browse/AVRO-999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13214263#comment-13214263
]
James Baldassari commented on AVRO-999:
---
I think AVRO-999 is slightly different from AVRO-1007 as AVRO-999 is
specifically related to unions with null types. I'm not positive, though.
Either way I think we should commit the new test case this patch introduces to
TestGenericRecordBuilder. That should verify whether the changes in AVRO-1007
actually fix this problem.
NPE in Java, RecordBuilderBase.defaultValue
---
Key: AVRO-999
URL: https://issues.apache.org/jira/browse/AVRO-999
Project: Avro
Issue Type: Bug
Components: java
Affects Versions: 1.6.1
Environment: Java
Reporter: Jay Rutten
Assignee: James Baldassari
Attachments: AVRO-999.patch
If you have a union with a default of null, the code in
RecordBuilderBase.defaultValue will cause an NPE in ConcurrentHashMap, since
it is trying to add a null to the map.
Sample union:
{code}
record Sample {
union{null, string} value = null;
}
{code}
Code:
{code}
// If not cached, get the default Java value by encoding the default JSON
// value and then decoding it:
if (defaultValue == null) {
ByteArrayOutputStream baos = new ByteArrayOutputStream();
encoder = EncoderFactory.get().binaryEncoder(baos, encoder);
ResolvingGrammarGenerator.encode(encoder, field.schema(), defaultJsonValue);
encoder.flush();
decoder = DecoderFactory.get().binaryDecoder(baos.toByteArray(), decoder);
defaultValue = new GenericDatumReader(field.schema()).read(null, decoder);
defaultSchemaValues.putIfAbsent(field.pos(), defaultValue); // -- NPE from
here
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-593) Avro mapreduce apis incompatible with hadoop 0.20.2
[
https://issues.apache.org/jira/browse/AVRO-593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13212819#comment-13212819
]
Doug Cutting commented on AVRO-593:
---
I see a few choices:
1. org.apache.avro.{mapred,mapreduce,io,file,util}. This is what the code on
github does. This would make the avro-mapred module contain things outside the
org.apache.avro.mapred package, and splits Avro's io, file and util packages
across multiple modules.
2. org.apache.avro.mapred.{mapreduce,io,file,util}. This is what my patch
does. This is back-compatible and consistent with the module name, but places
mapreduce under mapred, which is different than the Hadoop layout.
3. org.apache.avro.hadoop.{mapred,mapreduce,io,file,util}. We'd rename the
module to be avro-hadoop. This would be incompatible but consistent with
Hadoop. For back-compatibility we might leave the mapred classes in their
current package.
4. org.apache.avro.{mapred,mapreduce,mapred.io,mapred.file,mapred.util}. This
is back-compatible but includes a package that's not under the package of the
module name.
Tom, are you advocating for (4)? I'd be okay with that, I guess.
I'm also leaning towards moving AvroSequenceFile under org.apache.avro and
adding just a shim base class into org.apache.hadoop.io that subclasses
SequenceFile and makes public the bits we need. That way if we get Hadoop to
expose these bits the Avro API would not change.
Avro mapreduce apis incompatible with hadoop 0.20.2
---
Key: AVRO-593
URL: https://issues.apache.org/jira/browse/AVRO-593
Project: Avro
Issue Type: Bug
Components: java
Affects Versions: 1.3.2, 1.3.3
Environment: Avro 1.3.3, Hadoop 0.20.2
Reporter: Steve Severance
Assignee: Garrett Wu
Attachments: AVRO-593.patch, AVRO-593.patch
The avro api's for hadoop use the hadoop mapreduce api that has been
deprecated. A new avro mapreduce api should be implemented for hadoop 0.20
and higher.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-593) Avro mapreduce apis incompatible with hadoop 0.20.2
[
https://issues.apache.org/jira/browse/AVRO-593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13213012#comment-13213012
]
Scott Carey commented on AVRO-593:
--
Re #1: its OK to have multiple packages in a single maven module, but not good
to have a package split across modules as it causes problems for OSGi and in
the future, Java 8 modules.
Re #2: This is OK, but a little confusing. Also, if we ever wanted to break
apart the mapred module in to two or three (e.g. avro-hadoop, avro-mapred,
avro-mapreduce with the common stuff in avro-hadoop and the two APIs in the
others) it will be less consistent.
Re #3: This is fairly clean, but is incompatible.
Re #4: This is decent, but I would propose:
org.apache.avro.{hadoop,mapreduce,mapred,hadoop.io,hadoop.file,hadoop.util}.
Then the current module would have o.a.a.{hadoop,mapreduce,mapred} and children
packages. A future split could divide on these cleanly. One reason to split
in the future is that some users may want hadoop stuff that is not related to
mapreduce -- sequence files, avro data file access via FileSytem+Path, etc. If
we split the module, avoiding moving classes around is important.
Is it possible to move AvroSequenceFile under o.a.a ? All classes in that
package need to be in the base avro maven module, and cannot depend on any
hadoop APIs.
Avro mapreduce apis incompatible with hadoop 0.20.2
---
Key: AVRO-593
URL: https://issues.apache.org/jira/browse/AVRO-593
Project: Avro
Issue Type: Bug
Components: java
Affects Versions: 1.3.2, 1.3.3
Environment: Avro 1.3.3, Hadoop 0.20.2
Reporter: Steve Severance
Assignee: Garrett Wu
Attachments: AVRO-593.patch, AVRO-593.patch
The avro api's for hadoop use the hadoop mapreduce api that has been
deprecated. A new avro mapreduce api should be implemented for hadoop 0.20
and higher.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-593) Avro mapreduce apis incompatible with hadoop 0.20.2
[ https://issues.apache.org/jira/browse/AVRO-593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13213018#comment-13213018 ] Scott Carey commented on AVRO-593: -- We also need to consider if we need to produce two otherwise identical modules in a build -- one 0.23.x + compatible and one for the 0.20 / 0.22 / 1.0 users. My understanding is that one needs to compile against 0.23.x to work properly there. Organizing the modules so that it is possible to produce an Avro release that supports multiple Hadoop variants would be useful. Avro mapreduce apis incompatible with hadoop 0.20.2 --- Key: AVRO-593 URL: https://issues.apache.org/jira/browse/AVRO-593 Project: Avro Issue Type: Bug Components: java Affects Versions: 1.3.2, 1.3.3 Environment: Avro 1.3.3, Hadoop 0.20.2 Reporter: Steve Severance Assignee: Garrett Wu Attachments: AVRO-593.patch, AVRO-593.patch The avro api's for hadoop use the hadoop mapreduce api that has been deprecated. A new avro mapreduce api should be implemented for hadoop 0.20 and higher. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-593) Avro mapreduce apis incompatible with hadoop 0.20.2
[ https://issues.apache.org/jira/browse/AVRO-593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13213071#comment-13213071 ] Doug Cutting commented on AVRO-593: --- Is it possible to move AvroSequenceFile under o.a.a ? I discussed that above. We could move it, but we'd still need a shim in o.a.h.io, since the subclass accesses package-private bits. if we need to produce two otherwise identical modules in a build – one 0.23.x + compatible and one for the 0.20 / 0.22 / 1.0 users The nested Context classes in mapreduce's Mapper and Reducer went from abstract classes to interfaces (MAPREDUCE-954), requiring re-compilation of code that references these. But the mapreduce support added here does not reference these. So I think we're spared. Avro mapreduce apis incompatible with hadoop 0.20.2 --- Key: AVRO-593 URL: https://issues.apache.org/jira/browse/AVRO-593 Project: Avro Issue Type: Bug Components: java Affects Versions: 1.3.2, 1.3.3 Environment: Avro 1.3.3, Hadoop 0.20.2 Reporter: Steve Severance Assignee: Garrett Wu Attachments: AVRO-593.patch, AVRO-593.patch The avro api's for hadoop use the hadoop mapreduce api that has been deprecated. A new avro mapreduce api should be implemented for hadoop 0.20 and higher. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-593) Avro mapreduce apis incompatible with hadoop 0.20.2
[ https://issues.apache.org/jira/browse/AVRO-593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13213141#comment-13213141 ] Tom White commented on AVRO-593: +1 to Scott's #4 variant. Avro mapreduce apis incompatible with hadoop 0.20.2 --- Key: AVRO-593 URL: https://issues.apache.org/jira/browse/AVRO-593 Project: Avro Issue Type: Bug Components: java Affects Versions: 1.3.2, 1.3.3 Environment: Avro 1.3.3, Hadoop 0.20.2 Reporter: Steve Severance Assignee: Garrett Wu Attachments: AVRO-593.patch, AVRO-593.patch The avro api's for hadoop use the hadoop mapreduce api that has been deprecated. A new avro mapreduce api should be implemented for hadoop 0.20 and higher. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-593) Avro mapreduce apis incompatible with hadoop 0.20.2
[
https://issues.apache.org/jira/browse/AVRO-593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13213260#comment-13213260
]
Scott Carey commented on AVRO-593:
--
{quote}I discussed that above. We could move it, but we'd still need a shim in
o.a.h.io, since the subclass accesses package-private bits.{quote}
Let me clarify: Is it possible for AvroSequenceFile to not reference anything
in o.a.hadoop.** or o.a.a.{mapreduce, hadoop, mapred}.** ?
It has:
{code}
import org.apache.avro.mapred.AvroKey;
import org.apache.avro.mapred.AvroValue;
{code}
which would indicate to me that it must be in o.a.a.mapred.
If it is in o.a.a it must not reference any classes that don't exist in the
base avro module that encompases o.a.a (lang/java/avro)
Ideally, anything in the .io, .util, and .file packages does not reference the
.mapred or .mapreduce packages, so that this can be packaged as a standalone
hadoop dependency down the road. I have not looked at all those yet to see
what the package dependencies are.
Avro mapreduce apis incompatible with hadoop 0.20.2
---
Key: AVRO-593
URL: https://issues.apache.org/jira/browse/AVRO-593
Project: Avro
Issue Type: Bug
Components: java
Affects Versions: 1.3.2, 1.3.3
Environment: Avro 1.3.3, Hadoop 0.20.2
Reporter: Steve Severance
Assignee: Garrett Wu
Attachments: AVRO-593.patch, AVRO-593.patch
The avro api's for hadoop use the hadoop mapreduce api that has been
deprecated. A new avro mapreduce api should be implemented for hadoop 0.20
and higher.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-593) Avro mapreduce apis incompatible with hadoop 0.20.2
[ https://issues.apache.org/jira/browse/AVRO-593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13212361#comment-13212361 ] Tom White commented on AVRO-593: I renamed all of the packages to reside under org.apache.avro.mapred. So that package now has subpackages named io, file, util and mapreduce. Keeping the package org.apache.avro.mapreduce would be more consistent with Hadoop, which has the mapred/mapreduce distinction. Avro mapreduce apis incompatible with hadoop 0.20.2 --- Key: AVRO-593 URL: https://issues.apache.org/jira/browse/AVRO-593 Project: Avro Issue Type: Bug Components: java Affects Versions: 1.3.2, 1.3.3 Environment: Avro 1.3.3, Hadoop 0.20.2 Reporter: Steve Severance Assignee: Garrett Wu Attachments: AVRO-593.patch, AVRO-593.patch The avro api's for hadoop use the hadoop mapreduce api that has been deprecated. A new avro mapreduce api should be implemented for hadoop 0.20 and higher. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1029) I want to transfer files beetwin server and client by Avro-c.
[ https://issues.apache.org/jira/browse/AVRO-1029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13210257#comment-13210257 ] Douglas Creager commented on AVRO-1029: --- Hi there. It sounds like you're trying to use the RPC mechanism defined in the Avro spec, is that right? Unfortunately, the Avro C bindings don't support the RPC mechanism yet. The C bindings will let you create instances of an Avro schema, and you can create the binary encoding of those instances. But then you'd be responsible for transmitting the binary encoding using some other library, like ZeroMQ or raw TCP sockets. I hope this helps, please let me know if you have any further questions. I want to transfer files beetwin server and client by Avro-c. - Key: AVRO-1029 URL: https://issues.apache.org/jira/browse/AVRO-1029 Project: Avro Issue Type: Test Components: c Affects Versions: 1.5.4 Environment: avro in C, linux in vmware. Reporter: menghaocheng Labels: test Fix For: 1.5.4 Original Estimate: 168h Remaining Estimate: 168h I want to transfer files beetwin server and client by Avro-c. I know that there must be two parts in this model, the server and the client. My question is how to make the connection? Please forgive me as this maybe a simle question, But I'm fresh in Avro, your simple worlds maybe helpful for me. If only there is an example for it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-672) Convert JSON Text Input to Avro Tool
[
https://issues.apache.org/jira/browse/AVRO-672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13210586#comment-13210586
]
Doug Cutting commented on AVRO-672:
---
Leith, is the tool that Ron provided here the one you need? If so, then we can
probably resuscitate this patch and get it committed. If not, is there a
specific tool you need (e.g., CSV or TSV)? Thanks!
Convert JSON Text Input to Avro Tool
Key: AVRO-672
URL: https://issues.apache.org/jira/browse/AVRO-672
Project: Avro
Issue Type: New Feature
Components: java
Reporter: Ron Bodkin
Attachments: AVRO-672.patch, AVRO-672.patch
The attached patch allows reading a JSON-formatted text file in, converting
to a conforming Avro text file, emitting one record per line, e.g., it can
read this input file:
{intval:12}
{intval:-73,strval:hello, there!!}
with this schema:
{ type:record, name:TestRecord, fields: [
{name:intval,type:int}, {name:strval,type:[string, null]}]}
returning valid Avro. This is different than the DataFileWriteTool, which
would read in the following internal encoding:
{intval:12,strval:null}
{intval:-73,strval:{string:hello, there!!}}
In general, the internal encodings used by Avro aren't natural when reading
in JSON text that appears in the wild. Likewise, this utility allows changing
invalid Avro identifier characters into an underscore, again to tolerate JSON
that wasn't designed to be readable by Avro.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-593) Avro mapreduce apis incompatible with hadoop 0.20.2
[ https://issues.apache.org/jira/browse/AVRO-593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13210813#comment-13210813 ] Doug Cutting commented on AVRO-593: --- Garrett, this code looks great! Thanks for contributing it. I renamed all of the packages to reside under org.apache.avro.mapred. So that package now has subpackages named io, file, util and mapreduce. That's consistent with other Avro modules, where classes are under org.apache.avro.module. The only exception is org.apache.hadoop.io.AvroSequenceFile. This is in a Hadoop package so that it can access some package-private parts of SequenceFile. This is fragile, as SequenceFile could change these non-public APIs. We should probably file an issue with Hadoop to make these items protected so that SequenceFile can be subclassed in a supported way. I plan to improve the javadoc a bit (adding package.html files to new packages) and move versions for new dependencies from mapred/pom.xml into the parent pom. Then I think this should be ready to commit. Avro mapreduce apis incompatible with hadoop 0.20.2 --- Key: AVRO-593 URL: https://issues.apache.org/jira/browse/AVRO-593 Project: Avro Issue Type: Bug Components: java Affects Versions: 1.3.2, 1.3.3 Environment: Avro 1.3.3, Hadoop 0.20.2 Reporter: Steve Severance Assignee: Garrett Wu Attachments: AVRO-593.patch, AVRO-593.patch The avro api's for hadoop use the hadoop mapreduce api that has been deprecated. A new avro mapreduce api should be implemented for hadoop 0.20 and higher. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1025) migrate website dist to svnpubsub
[ https://issues.apache.org/jira/browse/AVRO-1025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13209735#comment-13209735 ] Doug Cutting commented on AVRO-1025: I committed the docs to subversion and asked Infrastructure to switch the website to automatically update from there in INFRA-4443. migrate website dist to svnpubsub --- Key: AVRO-1025 URL: https://issues.apache.org/jira/browse/AVRO-1025 Project: Avro Issue Type: Improvement Reporter: Doug Cutting Assignee: Doug Cutting ASF infrastructure has requested that all projects migrate to svnpubsub for their websites and release distributions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1026) Add namespace support to C++
[ https://issues.apache.org/jira/browse/AVRO-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13207896#comment-13207896 ] Thiruvalluvan M. G. commented on AVRO-1026: --- The patch does not apply cleanly on the current trunk. I think this patch was generated before AVRO-956. We have made two important changes in the recent past to get rid of Flex/Bison dependency and to make it compile natively on Windows using Visual Studio. I've not studied this patch deep. So my question may be naive. Is it possible to make the name attribute include the namespace as well instead of adding a new attribute? If the namespace is not given, the name will fall back to the simple name as it is now. If namespace is explicitly given or implied, it will have a fully qualified name. That way backward compatibility with existing schemas and code will be maintained. Add namespace support to C++ Key: AVRO-1026 URL: https://issues.apache.org/jira/browse/AVRO-1026 Project: Avro Issue Type: Improvement Components: c++ Affects Versions: 1.6.1 Reporter: Keh-Li Sheng Labels: patch Attachments: AVRO-1026.patch, AVRO-1026_json.patch This patch adds support for reading and resolving schemas that use namespaces for named types. Looking for a code review and application of the patch to codebase. Node is modified to take an additional NamespaceConcept attribute in the template parameters. An additional stack has been added to the compiler context to track when namespaces have been entered or left for resolving further types/symbols. Symbolic resolution is now done against the fullname of the type and not just by the name. Some string gymnastics were needed in other areas of codegen in order to handle the new symbols. Added very trivial tests to AvrogencppTests for schema generation by avrogencpp and added a test schema (tweet) that has namespaces, a record definition inside a namespace, and then a later symbolic reference by name within an outer namespace. Also patched to work with gen-cppcode.py output. Note that NodeImpl::printBasicInfo intentionally does not output the namespace since that caused downstream breakage of gen-cppcode.py expected format. github project was forked at https://github.com/bifflabs/avro -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1026) Add namespace support to C++
[ https://issues.apache.org/jira/browse/AVRO-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13207996#comment-13207996 ] Keh-Li Sheng commented on AVRO-1026: I forked from github and didn't update to the latest trunk before creating the patch. I'm resolving the merge now and will see if I can produce a clean patch. I debated this a bit when implementing it and I chose in favor of keeping name/namespace/fullname as separate concepts since there were a lot of downstream effects to pushing namespaces into the name field - primarily in avrogencpp and gen-cppcode.py where name is used quite frequently as a field name or class/struct specifier (in which case the '.' in namespaces would be an invalid character). That meant additional logic would be required to strip the name of its fully qualified path, whereas it is more straightforward to know how to compose the full path when needed. Additionally, if there was a future desire to nest declarations within C++ namespaces that tracked the schema namespace, having a separate namespace attribute would make that easier as well. I'm not quite sure I understand what you mean by backward compatibility since this change does not *require* namespace attributes from existing schemas (all the existing schemas in jsonschemas seem to parse and validate and tests run the same as before), it merely knows what to do with them when it encounters them. Add namespace support to C++ Key: AVRO-1026 URL: https://issues.apache.org/jira/browse/AVRO-1026 Project: Avro Issue Type: Improvement Components: c++ Affects Versions: 1.6.1 Reporter: Keh-Li Sheng Labels: patch Attachments: AVRO-1026.patch, AVRO-1026_json.patch This patch adds support for reading and resolving schemas that use namespaces for named types. Looking for a code review and application of the patch to codebase. Node is modified to take an additional NamespaceConcept attribute in the template parameters. An additional stack has been added to the compiler context to track when namespaces have been entered or left for resolving further types/symbols. Symbolic resolution is now done against the fullname of the type and not just by the name. Some string gymnastics were needed in other areas of codegen in order to handle the new symbols. Added very trivial tests to AvrogencppTests for schema generation by avrogencpp and added a test schema (tweet) that has namespaces, a record definition inside a namespace, and then a later symbolic reference by name within an outer namespace. Also patched to work with gen-cppcode.py output. Note that NodeImpl::printBasicInfo intentionally does not output the namespace since that caused downstream breakage of gen-cppcode.py expected format. github project was forked at https://github.com/bifflabs/avro -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1022) Error in validate name
[ https://issues.apache.org/jira/browse/AVRO-1022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13207029#comment-13207029 ] Doug Cutting commented on AVRO-1022: If names are restricted, then consuming schemas from other systems will be difficult. Good point. The question is where the escaping burden lies: either with adapter layers (e.g., in Pig or Hive) or in the code generation layer. I'd argue that code generation layer already has to handle reserved words so that adding character escaping is not a significant burden there. It's also safer to not assume that other implementations have correctly escaped all names; to be tolerant. Finally, escaping as late as possible maximizes legibility through the system. Error in validate name -- Key: AVRO-1022 URL: https://issues.apache.org/jira/browse/AVRO-1022 Project: Avro Issue Type: Bug Components: java Reporter: Raymie Stata Priority: Minor Attachments: AVRO-1022.patch, AVRO-1022.patch Fix schema.validateName to allow only ASCII letters, not Unicode letters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1028) IPC transceiver doesn't gracefully handle server connection resets.
[ https://issues.apache.org/jira/browse/AVRO-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13207161#comment-13207161 ] Bo Shi commented on AVRO-1028: -- Also, note that this patch incorporates redirect functionality as originally requested in https://issues.apache.org/jira/browse/AVRO-665 IPC transceiver doesn't gracefully handle server connection resets. --- Key: AVRO-1028 URL: https://issues.apache.org/jira/browse/AVRO-1028 Project: Avro Issue Type: Improvement Components: python Affects Versions: 1.6.2 Reporter: Bo Shi Assignee: Bo Shi Fix For: 1.6.2 Attachments: AVRO-1028.patch The current Python HTTPTransceiver class forces users to handle connection resets. I've refactored the class using urllib3 and incorporated some features we get for free from said library into the transceiver. Added a test case for test_ipc.py that uses the twisted server implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-831) Enhance the Python `distutils` control file `setup.py` to build Avro/Python without a pre-processing step.
[
https://issues.apache.org/jira/browse/AVRO-831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13207213#comment-13207213
]
Bo Shi commented on AVRO-831:
-
Patch still seems to have some extra bits.
Enhance the Python `distutils` control file `setup.py` to build Avro/Python
without a pre-processing step.
--
Key: AVRO-831
URL: https://issues.apache.org/jira/browse/AVRO-831
Project: Avro
Issue Type: Improvement
Components: python
Environment: Operating environments supporting the Python programming
language.
Reporter: Joseph Koshy
Assignee: Joseph Koshy
Labels: build, python
Fix For: 1.7.0
Attachments: setup.patch
Original Estimate: 4h
Remaining Estimate: 4h
Avro's build process for its Python bindings currently needs a set of
pre-processing steps that need to be executed before the build proper is
started. These pre-processing steps are executed by {{ant}}.
The attached patch folds these pre-processing steps into the {{setup.py}}
control script that is input to Python's {{distutils}} build tool kit. This
eliminates the dependency on {{ant}}.
In the patch:
* Source files that need keyword expansion are renamed to use a {{.in}}
extension. The build process will generate the appropriately transformed
{{.py}} files from these {{.in}} files.
* {{setup.py}} supports a new command {{make_interop_data}}, used to generate
test data for use with interoperability test code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1022) Error in validate name
[ https://issues.apache.org/jira/browse/AVRO-1022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13207282#comment-13207282 ] Scott Carey commented on AVRO-1022: --- As late as possible also makes transitive equivalence easier. If Hive and Pig both escaped the same string differently when writing into an Avro file, then that Avro file will not be shareable between them as easily. This is why I feel that the Avro spec should define how names are escaped. I think the only code point that truly needs escaping is the namespace separator and an escape character. A single source string recognized as equivalent in two different external systems, should become equivalent escaped strings if each of those systems uses different Avro language bindings to escape and store a schema. Error in validate name -- Key: AVRO-1022 URL: https://issues.apache.org/jira/browse/AVRO-1022 Project: Avro Issue Type: Bug Components: java Reporter: Raymie Stata Priority: Minor Attachments: AVRO-1022.patch, AVRO-1022.patch Fix schema.validateName to allow only ASCII letters, not Unicode letters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
