[jira] [Commented] (AVRO-975) Support RPC in C#
[ https://issues.apache.org/jira/browse/AVRO-975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13699893#comment-13699893 ] Bruce Collie commented on AVRO-975: --- I have just discovered a type mapping issue when using arrays or maps as ipc parameters, I will try to submit a patch before the weekend. Support RPC in C# - Key: AVRO-975 URL: https://issues.apache.org/jira/browse/AVRO-975 Project: Avro Issue Type: New Feature Components: csharp Affects Versions: 1.6.1 Reporter: Jeff Hammerbacher Attachments: 975.patch, Avro-975-00.patch, Avro-975-complete2.patch, Avro-975-complete3.patch, Avro975-complete.patch, buildtask.patch, Castle.Core.dll, errors-and-remote-protocols.975.diff, java-compat.diff, propagate-exception.diff, timeout-unhandled-exception.diff -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1341) Allow controlling avro via java annotations when using reflection.
[
https://issues.apache.org/jira/browse/AVRO-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vincenz Priesnitz updated AVRO-1341:
Release Note:
Added the java annotations @AvroName, @AvroIgnore, @AvroMeta, @AvroAlias and
@AvroEncode to control the behavior of avro when using reflection.
was:
Added the java annotations @AvroName, @AvroIgnore, @AvroMeta and @AvroEncode to
control the behavior of avro when using reflection.
Allow controlling avro via java annotations when using reflection.
---
Key: AVRO-1341
URL: https://issues.apache.org/jira/browse/AVRO-1341
Project: Avro
Issue Type: New Feature
Components: java
Reporter: Vincenz Priesnitz
Assignee: Vincenz Priesnitz
Attachments: AVRO-1341.patch, AVRO-1341.patch, AVRO-1341.patch
It would be great if one could control avro with java annotations. As of now,
it is already possible to mark fields as Nullable or classes being encoded as
a String. I propose a bigger set of annotations to control the behavior of
avro on fields and classes. Such annotations have proven useful with jacksons
json serialization and morphias mongoDB serialization.
I propose the following additional annotations:
@AvroName(alternativeName)
@AvroIgnore
@AvroMeta(key=K, value=V)
@AvroEncode(using=CustomEncoding.class)
Java fields with the @AvroName(alternativeName) annotation will be renamed
in the induced schema. When reading an avro file via reflection, the
reflection reader will look for fields in the schema with alternativeName.
For example:
{code}
@AvroName(foo)
int bar;
{code}
is serialized as
{code}
{ name : foo, type : int }
{code}
Fields with the @AvroIgnore annotation will be treated as if they had a
transient modifier, i.e. they will not be written to or read from avro files.
The @AvroMeta(key=K, value=V) annotation allows you to store an arbitrary
key : value pair at every node in the schema.
{code}
@AvroMeta(key=fieldKey, value=fieldValue)
int foo;
{code}
will create the following schema
{code}
{name : foo, type : int, fieldKey : fieldValue }
{code}
Fields can be custom encoded with the AvroEncode(using=CustomEncoding.class)
annotation. This annotation is a generalization of the @Stringable
annotation. The @Stringable annotation is limited to classes with string
argument constructors. Some classes can be similarly reduced to a smaller
class or even a single primitive, but dont fit the requirements for
@Stringable. A prominent example is java.util.Date, which instances can
essentially be described with a single long. Such classes can now be encoded
with a CustomEncoding, which reads and writes directly from the
encoder/decoder.
One simply extends the abstract CustomEncodings class by implementing a
schema, a read method and a write method. A java field can then be annotated
like this:
{code}
@AvroEncode(using=DateAslongEncoding.class)
Date date;
{code}
The custom encoding implementation would look like
{code}
public class DateAsLongEncoding extends CustomEncodingDate {
{
schema = Schema.create(Schema.Type.LONG);
schema.addProp(CustomEncoding, DateAsLongEncoding);
}
@Override
public void write(Object datum, Encoder out) throws IOException {
out.writeLong(((Date)datum).getTime());
}
@Override
public Date read(Object reuse, Decoder in) throws IOException {
if (reuse != null) {
((Date)reuse).setTime(in.readLong());
return (Date)reuse;
}
else return new Date(in.readLong());
}
}
{code}
I implemented said annotations and a custom encoding for java.util.Date as a
proof of concept and also extended the @Stringable annotations to fields.
This issue is a followup of AVRO-1328 and AVRO-1330.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1341) Allow controlling avro via java annotations when using reflection.
[
https://issues.apache.org/jira/browse/AVRO-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vincenz Priesnitz updated AVRO-1341:
Attachment: AVRO-1341.patch
Allow controlling avro via java annotations when using reflection.
---
Key: AVRO-1341
URL: https://issues.apache.org/jira/browse/AVRO-1341
Project: Avro
Issue Type: New Feature
Components: java
Reporter: Vincenz Priesnitz
Assignee: Vincenz Priesnitz
Attachments: AVRO-1341.patch, AVRO-1341.patch, AVRO-1341.patch
It would be great if one could control avro with java annotations. As of now,
it is already possible to mark fields as Nullable or classes being encoded as
a String. I propose a bigger set of annotations to control the behavior of
avro on fields and classes. Such annotations have proven useful with jacksons
json serialization and morphias mongoDB serialization.
I propose the following additional annotations:
@AvroName(alternativeName)
@AvroIgnore
@AvroMeta(key=K, value=V)
@AvroEncode(using=CustomEncoding.class)
Java fields with the @AvroName(alternativeName) annotation will be renamed
in the induced schema. When reading an avro file via reflection, the
reflection reader will look for fields in the schema with alternativeName.
For example:
{code}
@AvroName(foo)
int bar;
{code}
is serialized as
{code}
{ name : foo, type : int }
{code}
Fields with the @AvroIgnore annotation will be treated as if they had a
transient modifier, i.e. they will not be written to or read from avro files.
The @AvroMeta(key=K, value=V) annotation allows you to store an arbitrary
key : value pair at every node in the schema.
{code}
@AvroMeta(key=fieldKey, value=fieldValue)
int foo;
{code}
will create the following schema
{code}
{name : foo, type : int, fieldKey : fieldValue }
{code}
Fields can be custom encoded with the AvroEncode(using=CustomEncoding.class)
annotation. This annotation is a generalization of the @Stringable
annotation. The @Stringable annotation is limited to classes with string
argument constructors. Some classes can be similarly reduced to a smaller
class or even a single primitive, but dont fit the requirements for
@Stringable. A prominent example is java.util.Date, which instances can
essentially be described with a single long. Such classes can now be encoded
with a CustomEncoding, which reads and writes directly from the
encoder/decoder.
One simply extends the abstract CustomEncodings class by implementing a
schema, a read method and a write method. A java field can then be annotated
like this:
{code}
@AvroEncode(using=DateAslongEncoding.class)
Date date;
{code}
The custom encoding implementation would look like
{code}
public class DateAsLongEncoding extends CustomEncodingDate {
{
schema = Schema.create(Schema.Type.LONG);
schema.addProp(CustomEncoding, DateAsLongEncoding);
}
@Override
public void write(Object datum, Encoder out) throws IOException {
out.writeLong(((Date)datum).getTime());
}
@Override
public Date read(Object reuse, Decoder in) throws IOException {
if (reuse != null) {
((Date)reuse).setTime(in.readLong());
return (Date)reuse;
}
else return new Date(in.readLong());
}
}
{code}
I implemented said annotations and a custom encoding for java.util.Date as a
proof of concept and also extended the @Stringable annotations to fields.
This issue is a followup of AVRO-1328 and AVRO-1330.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AVRO-1341) Allow controlling avro via java annotations when using reflection.
[
https://issues.apache.org/jira/browse/AVRO-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13700188#comment-13700188
]
Vincenz Priesnitz commented on AVRO-1341:
-
Attached is a new patch with more JavaDocs, including warnings for using custom
encodings.
I also moved the @AvroAlias annotation from issue AVRO-1347 here, but excluded
the controversial writer aliases and added unit tests. It is now also possible
to add an alias without a namespace.
This patch still contains the @AvroIgnore annotation, which I would really like
to see committed.
Allow controlling avro via java annotations when using reflection.
---
Key: AVRO-1341
URL: https://issues.apache.org/jira/browse/AVRO-1341
Project: Avro
Issue Type: New Feature
Components: java
Reporter: Vincenz Priesnitz
Assignee: Vincenz Priesnitz
Attachments: AVRO-1341.patch, AVRO-1341.patch, AVRO-1341.patch
It would be great if one could control avro with java annotations. As of now,
it is already possible to mark fields as Nullable or classes being encoded as
a String. I propose a bigger set of annotations to control the behavior of
avro on fields and classes. Such annotations have proven useful with jacksons
json serialization and morphias mongoDB serialization.
I propose the following additional annotations:
@AvroName(alternativeName)
@AvroIgnore
@AvroMeta(key=K, value=V)
@AvroEncode(using=CustomEncoding.class)
Java fields with the @AvroName(alternativeName) annotation will be renamed
in the induced schema. When reading an avro file via reflection, the
reflection reader will look for fields in the schema with alternativeName.
For example:
{code}
@AvroName(foo)
int bar;
{code}
is serialized as
{code}
{ name : foo, type : int }
{code}
Fields with the @AvroIgnore annotation will be treated as if they had a
transient modifier, i.e. they will not be written to or read from avro files.
The @AvroMeta(key=K, value=V) annotation allows you to store an arbitrary
key : value pair at every node in the schema.
{code}
@AvroMeta(key=fieldKey, value=fieldValue)
int foo;
{code}
will create the following schema
{code}
{name : foo, type : int, fieldKey : fieldValue }
{code}
Fields can be custom encoded with the AvroEncode(using=CustomEncoding.class)
annotation. This annotation is a generalization of the @Stringable
annotation. The @Stringable annotation is limited to classes with string
argument constructors. Some classes can be similarly reduced to a smaller
class or even a single primitive, but dont fit the requirements for
@Stringable. A prominent example is java.util.Date, which instances can
essentially be described with a single long. Such classes can now be encoded
with a CustomEncoding, which reads and writes directly from the
encoder/decoder.
One simply extends the abstract CustomEncodings class by implementing a
schema, a read method and a write method. A java field can then be annotated
like this:
{code}
@AvroEncode(using=DateAslongEncoding.class)
Date date;
{code}
The custom encoding implementation would look like
{code}
public class DateAsLongEncoding extends CustomEncodingDate {
{
schema = Schema.create(Schema.Type.LONG);
schema.addProp(CustomEncoding, DateAsLongEncoding);
}
@Override
public void write(Object datum, Encoder out) throws IOException {
out.writeLong(((Date)datum).getTime());
}
@Override
public Date read(Object reuse, Decoder in) throws IOException {
if (reuse != null) {
((Date)reuse).setTime(in.readLong());
return (Date)reuse;
}
else return new Date(in.readLong());
}
}
{code}
I implemented said annotations and a custom encoding for java.util.Date as a
proof of concept and also extended the @Stringable annotations to fields.
This issue is a followup of AVRO-1328 and AVRO-1330.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AVRO-1341) Allow controlling avro via java annotations when using reflection.
[
https://issues.apache.org/jira/browse/AVRO-1341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vincenz Priesnitz updated AVRO-1341:
Description:
It would be great if one could control avro with java annotations. As of now,
it is already possible to mark fields as Nullable or classes being encoded as a
String. I propose a bigger set of annotations to control the behavior of avro
on fields and classes. Such annotations have proven useful with jacksons json
serialization and morphias mongoDB serialization.
I propose the following additional annotations:
@AvroName(alternativeName)
@AvroAlias(alias=alias, space=space)
@AvroIgnore
@AvroMeta(key=K, value=V)
@AvroEncode(using=CustomEncoding.class)
Java fields with the @AvroName(alternativeName) annotation will be renamed in
the induced schema. When reading an avro file via reflection, the reflection
reader will look for fields in the schema with alternativeName.
For example:
{code}
@AvroName(foo)
int bar;
{code}
is serialized as
{code}
{ name : foo, type : int }
{code}
The @AvroAlias annotation will add a new alias to the induced schema of a
record, enum or field. The space parameter is optional and defaults to the
namespace of the named schema the alias is added to.
Fields with the @AvroIgnore annotation will be treated as if they had a
transient modifier, i.e. they will not be written to or read from avro files.
The @AvroMeta(key=K, value=V) annotation allows you to store an arbitrary
key : value pair at every node in the schema.
{code}
@AvroMeta(key=fieldKey, value=fieldValue)
int foo;
{code}
will create the following schema
{code}
{name : foo, type : int, fieldKey : fieldValue }
{code}
Fields can be custom encoded with the AvroEncode(using=CustomEncoding.class)
annotation. This annotation is a generalization of the @Stringable annotation.
The @Stringable annotation is limited to classes with string argument
constructors. Some classes can be similarly reduced to a smaller class or even
a single primitive, but dont fit the requirements for @Stringable. A prominent
example is java.util.Date, which instances can essentially be described with a
single long. Such classes can now be encoded with a CustomEncoding, which reads
and writes directly from the encoder/decoder.
One simply extends the abstract CustomEncodings class by implementing a schema,
a read method and a write method. A java field can then be annotated like this:
{code}
@AvroEncode(using=DateAslongEncoding.class)
Date date;
{code}
The custom encoding implementation would look like
{code}
public class DateAsLongEncoding extends CustomEncodingDate {
{
schema = Schema.create(Schema.Type.LONG);
schema.addProp(CustomEncoding, DateAsLongEncoding);
}
@Override
public void write(Object datum, Encoder out) throws IOException {
out.writeLong(((Date)datum).getTime());
}
@Override
public Date read(Object reuse, Decoder in) throws IOException {
if (reuse != null) {
((Date)reuse).setTime(in.readLong());
return (Date)reuse;
}
else return new Date(in.readLong());
}
}
{code}
I implemented said annotations and a custom encoding for java.util.Date as a
proof of concept and also extended the @Stringable annotations to fields.
This issue is a followup of AVRO-1328 and AVRO-1330.
was:
It would be great if one could control avro with java annotations. As of now,
it is already possible to mark fields as Nullable or classes being encoded as a
String. I propose a bigger set of annotations to control the behavior of avro
on fields and classes. Such annotations have proven useful with jacksons json
serialization and morphias mongoDB serialization.
I propose the following additional annotations:
@AvroName(alternativeName)
@AvroIgnore
@AvroMeta(key=K, value=V)
@AvroEncode(using=CustomEncoding.class)
Java fields with the @AvroName(alternativeName) annotation will be renamed in
the induced schema. When reading an avro file via reflection, the reflection
reader will look for fields in the schema with alternativeName.
For example:
{code}
@AvroName(foo)
int bar;
{code}
is serialized as
{code}
{ name : foo, type : int }
{code}
Fields with the @AvroIgnore annotation will be treated as if they had a
transient modifier, i.e. they will not be written to or read from avro files.
The @AvroMeta(key=K, value=V) annotation allows you to store an arbitrary
key : value pair at every node in the schema.
{code}
@AvroMeta(key=fieldKey, value=fieldValue)
int foo;
{code}
will create the following schema
{code}
{name : foo, type : int, fieldKey : fieldValue }
{code}
Fields can be custom encoded with the AvroEncode(using=CustomEncoding.class)
annotation. This annotation is a generalization of the @Stringable annotation.
The @Stringable annotation is limited to classes with string argument
constructors. Some classes can be
[jira] [Commented] (AVRO-1347) Improve name and alias matching for named schemas
[
https://issues.apache.org/jira/browse/AVRO-1347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13700214#comment-13700214
]
Vincenz Priesnitz commented on AVRO-1347:
-
Thanks a lot for the indepth review.
I moved the annotation to AVRO-1341. I think the annotation is still handy even
without writer schema aliases.
There are scenarios where it is not possible to add aliases to the readers
schema when using reflection, for example if i have no control over the class
or when reading an avro file with a specificreader, it is not possible to add
aliases to the reader.
The change is backwards compatible, but it enhances flexibility when using
reflection.
Improve name and alias matching for named schemas
-
Key: AVRO-1347
URL: https://issues.apache.org/jira/browse/AVRO-1347
Project: Avro
Issue Type: Improvement
Components: java
Reporter: Vincenz Priesnitz
Attachments: AVRO-1347.patch, AVRO-1347.patch
When reading an avro file with a named schema, the aliases of the writers
schema are not taken into account; only the aliases of the readers are
matched against the writers name. Even if the writers aliases match the
readers name, the schemas will not be matched.
For example, the following two enum schemas will not be matched, even though
they share a common alias.
{code}
{
type : enum,
name : foo,
alias : CommonAlias,
symbols : [LEFT, RIGHT]
}
{code}
{code}
{
type : enum,
name : bar,
alias : CommonAlias,
symbols : [LEFT, RIGHT]
}
{code}
In most cases, the DatumReader resolves records of different names or
namespaces by matching their fields.
Unfortunately, there are some cases, where this sort of matching is not
happening, but just the names are compared:
* Other named nodes, like enums, fixed or fieldschemas are not matched this
way.
* A record inside a union is also only matched by the full name.
The latter one is especially tricky, since two recordschemas that match
structurally but differ in name or space, are interexchangable until they are
put into an union, at which point an exception is thrown.
I propose that two named schemas are matched, when they share a common name
or alias.
I implemented said changes and added a java annotation @AvroAlias(alias,
space) that allows one to add an alias to a record, enum or field.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
