[jira] [Resolved] (AVRO-1588) ReflectData.AllowNull incorrectly handles primitive types.
[
https://issues.apache.org/jira/browse/AVRO-1588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joey Echeverria resolved AVRO-1588.
---
Resolution: Duplicate
ReflectData.AllowNull incorrectly handles primitive types.
--
Key: AVRO-1588
URL: https://issues.apache.org/jira/browse/AVRO-1588
Project: Avro
Issue Type: Bug
Components: java
Affects Versions: 1.7.7
Reporter: Joey Echeverria
When doing the following:
{code:java}
private static class PoJo {
private long id;
private String name;
}
ReflectData.AllowNull.get().getSchema(PoJo.class);
{code}
I'd expect a schema like this:
{code:javascript}
{
type : record,
name : PoJo,
namespace : MyObject$,
fields : [ {
name : id,
type : long,
}, {
name : name,
type : [ null, string ],
default : null
} ]
}
{code}
But instead I get:
{code:javascript}
{
type : record,
name : PoJo,
namespace : MyObject$,
fields : [ {
name : id,
type : [ null, long ],
default : null
}, {
name : name,
type : [ null, string ],
default : null
} ]
}
{code}
This will later result in an NullPointerException when you try to deserialize
bytes where id was null when it tries to convert the Long to a long.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (AVRO-1588) ReflectData.AllowNull incorrectly handles primitive types.
Joey Echeverria created AVRO-1588:
-
Summary: ReflectData.AllowNull incorrectly handles primitive types.
Key: AVRO-1588
URL: https://issues.apache.org/jira/browse/AVRO-1588
Project: Avro
Issue Type: Bug
Affects Versions: 1.7.7
Reporter: Joey Echeverria
When doing the following:
{code:java}
private static class PoJo {
private long id;
private String name;
}
ReflectData.AllowNull.get().getSchema(PoJo.class);
{code}
I'd expect a schema like this:
{code:javascript}
{
type : record,
name : PoJo,
namespace : MyObject$,
fields : [ {
name : id,
type : long,
}, {
name : name,
type : [ null, string ],
default : null
} ]
}
{code}
But instead I get:
{code:javascript}
{
type : record,
name : PoJo,
namespace : MyObject$,
fields : [ {
name : id,
type : [ null, long ],
default : null
}, {
name : name,
type : [ null, string ],
default : null
} ]
}
{code}
This will later result in an NullPointerException when you try to deserialize
bytes where id was null when it tries to convert the Long to a long.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AVRO-1588) ReflectData.AllowNull incorrectly handles primitive types.
[
https://issues.apache.org/jira/browse/AVRO-1588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14138117#comment-14138117
]
Joey Echeverria commented on AVRO-1588:
---
I think it should special case primitive fields to not be nullable. I view
ReflectData.AllowNull to be a short cut to adding {{@Nullable}} to the fields
of my PoJo, or for cases where I don't control the source of the PoJo. But I
wouldn't expect it to produce a schema that isn't actually compatible with my
class. In particular, if I'm using ReflectData.AllowNull to extract a schema
and then use it to read data, I'd like to be able to check schema compatibility
in advance and get an error if the writer schema allows nulls for one of the
fields that is a primitive in my class. As it stands today, I won't get an
error until I actually try to read the data and the error I'll get is a
NullPointerException.
ReflectData.AllowNull incorrectly handles primitive types.
--
Key: AVRO-1588
URL: https://issues.apache.org/jira/browse/AVRO-1588
Project: Avro
Issue Type: Bug
Components: java
Affects Versions: 1.7.7
Reporter: Joey Echeverria
When doing the following:
{code:java}
private static class PoJo {
private long id;
private String name;
}
ReflectData.AllowNull.get().getSchema(PoJo.class);
{code}
I'd expect a schema like this:
{code:javascript}
{
type : record,
name : PoJo,
namespace : MyObject$,
fields : [ {
name : id,
type : long,
}, {
name : name,
type : [ null, string ],
default : null
} ]
}
{code}
But instead I get:
{code:javascript}
{
type : record,
name : PoJo,
namespace : MyObject$,
fields : [ {
name : id,
type : [ null, long ],
default : null
}, {
name : name,
type : [ null, string ],
default : null
} ]
}
{code}
This will later result in an NullPointerException when you try to deserialize
bytes where id was null when it tries to convert the Long to a long.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (AVRO-1243) Support all compression codecs
[ https://issues.apache.org/jira/browse/AVRO-1243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573706#comment-13573706 ] Joey Echeverria commented on AVRO-1243: --- q. You might add a HadoopCodec to the avro-mapred module that implements an Avro codec in terms of a Hadoop CompressionCodec.. I like the idea of a Hadoop Codec, but should it be only available as part of avro-mapred or should it be it's own module? As I understand it, with the exception of null and deflate, compression codecs are an optional part of the Avro spec so there should already be ways of handling trying to load a codec that doesn't exist. Support all compression codecs -- Key: AVRO-1243 URL: https://issues.apache.org/jira/browse/AVRO-1243 Project: Avro Issue Type: Improvement Components: java Affects Versions: 1.7.3 Reporter: Ted Malaska Priority: Minor I may be reading this wrong but at this time org.apache.avro.file.CodecFactory only supports null, deflate, and snappy compression codecs. I would like to change the fromString method to use Class.forName(codec).newInstance(); after the codec was not found in the REGISTERED map but before the AvroRuntimeException is thrown. Here are some of my supporting thoughts 1. This should not interduce much slowness because it will only be called initialize. 2. This will allow for support for GZip, BZip2, and LZO with out adding more dependances to the maven pom file. 3. This will allow for a future Jiri I would like to do that would allow AvroOutputFormat to be able to use the following configs: mapred.output.compress and mapred.output.compression.codec -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1242) AvroTrevniOutputFormat Metadata filter subString bug
[ https://issues.apache.org/jira/browse/AVRO-1242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13572788#comment-13572788 ] Joey Echeverria commented on AVRO-1242: --- Thanks for the patch Ted! If you think your patch is ready to be reviewed, please follow the steps outlined here: https://cwiki.apache.org/confluence/display/AVRO/How+To+Contribute and another contributor or a committer will come along and take a look. -Joey AvroTrevniOutputFormat Metadata filter subString bug Key: AVRO-1242 URL: https://issues.apache.org/jira/browse/AVRO-1242 Project: Avro Issue Type: Bug Affects Versions: 1.7.4 Reporter: Ted Malaska Attachments: AVRO-1242.patch In AvroTrevniOutputFormat there seems to be a cut and post error in the metadata filtering. The result is not trevni.meta. metadatas will successfully get to the writer. if (e.getKey().startsWith(META_PREFIX)) meta.put(e.getKey().substring(AvroJob.TEXT_PREFIX.length()), e.getValue().getBytes(MetaData.UTF8)); META_PREFIX = trevni.meta. TEXT_PREFIX = avro.meta.text. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1112) Java: Snappy codec not supported by Avro mapreduce API
[ https://issues.apache.org/jira/browse/AVRO-1112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13295764#comment-13295764 ] Joey Echeverria commented on AVRO-1112: --- I'm +1 on the patch, hopefully a committer can review and push it in. Nice job Matt! Java: Snappy codec not supported by Avro mapreduce API -- Key: AVRO-1112 URL: https://issues.apache.org/jira/browse/AVRO-1112 Project: Avro Issue Type: Bug Components: java Affects Versions: 1.7.0, 1.7.1 Reporter: matt mead Priority: Minor Labels: mapreduce, snappy Fix For: 1.7.1 Attachments: AVRO-1112.patch, AVRO-1112_201206150842.patch, AVRO-1112_201206150912.patch Original Estimate: 2h Remaining Estimate: 2h The implementation of the mapreduce API support does not allow for the user to specify the snappy codec for output compression. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
