date:20180222

[jira] [Commented] (AVRO-2035) enable validation of default values in schemas by default

2018-02-22 Thread Zoltan Farkas (JIRA)


[ 
https://issues.apache.org/jira/browse/AVRO-2035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373777#comment-16373777
 ] 

Zoltan Farkas commented on AVRO-2035:
-

[~cutting] looks good. thanks!  

> enable validation of default values in schemas by default
> -
>
> Key: AVRO-2035
> URL: https://issues.apache.org/jira/browse/AVRO-2035
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.1
>Reporter: radai rosenblatt
>Assignee: Doug Cutting
>Priority: Major
> Fix For: 1.9.0
>
> Attachments: AVRO-2035.patch
>
>
> suppose i have the following schema evolution:
> {code}
> {
>   "name": "Bob",
>   "type": "record",
>   "fields": [
> {"name": "f1", "type": "int"}
>   ]
> }
> {code}
> and then:
> {code}
> {
>   "name": "Bob",
>   "type": "record",
>   "fields": [
> {"name": "f1", "type": "int"},
> {"name": "f2", "type": "boolean", "default": "true"}
>   ]
> }
> {code}
> the default value for "f2" is specified as the _STRING_ "true" (and not the 
> literal boolean true). 
> if this default value is ever accessed (when reading a gen1-serialized object 
> as a gen2) we get this:
> {code}
> org.apache.avro.AvroTypeException: Non-boolean default for boolean: "true"
>   at 
> org.apache.avro.io.parsing.ResolvingGrammarGenerator.encode(ResolvingGrammarGenerator.java:408)
>   at 
> org.apache.avro.io.parsing.ResolvingGrammarGenerator.getBinary(ResolvingGrammarGenerator.java:307)
>   at 
> org.apache.avro.io.parsing.ResolvingGrammarGenerator.resolveRecords(ResolvingGrammarGenerator.java:285)
>   at 
> org.apache.avro.io.parsing.ResolvingGrammarGenerator.generate(ResolvingGrammarGenerator.java:118)
>   at 
> org.apache.avro.io.parsing.ResolvingGrammarGenerator.generate(ResolvingGrammarGenerator.java:50)
>   at org.apache.avro.io.ResolvingDecoder.resolve(ResolvingDecoder.java:85)
>   at org.apache.avro.io.ResolvingDecoder.(ResolvingDecoder.java:49)
>   at 
> org.apache.avro.io.DecoderFactory.resolvingDecoder(DecoderFactory.java:307)
>   at 
> org.apache.avro.generic.GenericDatumReader.getResolver(GenericDatumReader.java:127)
>   at 
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)
> {code}
> yet Schema.parse() passes for this



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2132) Avro IDL: Support dot ('.') character in property annotation names

2018-02-22 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/AVRO-2132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373722#comment-16373722
 ] 

ASF GitHub Bot commented on AVRO-2132:
--

kdrakon closed pull request #287: AVRO-2132 added missing defaults for new 
fields
URL: https://github.com/apache/avro/pull/287
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/lang/java/compiler/src/test/idl/input/simple.avdl 
b/lang/java/compiler/src/test/idl/input/simple.avdl
index 1bfcd7b27..d41d7116f 100644
--- a/lang/java/compiler/src/test/idl/input/simple.avdl
+++ b/lang/java/compiler/src/test/idl/input/simple.avdl
@@ -50,8 +50,8 @@ protocol Simple {
 date d = 0;
 time_ms t = 0;
 
-@foo.bar("bar.foo") long l;
-union {null, @foo.foo.bar(42) @foo.foo.foo("3foo") string} 
nested_properties;
+@foo.bar("bar.foo") long l = 0;
+union {null, @foo.foo.bar(42) @foo.foo.foo("3foo") string} 
nested_properties = null;
   }
 
   error TestError {
diff --git a/lang/java/compiler/src/test/idl/output/simple.avpr 
b/lang/java/compiler/src/test/idl/output/simple.avpr
index f40030790..e315d7851 100644
--- a/lang/java/compiler/src/test/idl/output/simple.avpr
+++ b/lang/java/compiler/src/test/idl/output/simple.avpr
@@ -56,10 +56,12 @@
   "default": 0
 } , {
   "name": "l",
-  "type": {"type": "long", "foo.bar": "bar.foo"}
+  "type": {"type": "long", "foo.bar": "bar.foo"},
+  "default": 0
 } , {
   "name": "nested_properties",
-  "type": [ "null" , {"type":"string", "foo.foo.bar": 42, "foo.foo.foo": 
"3foo"} ]
+  "type": [ "null" , {"type":"string", "foo.foo.bar": 42, "foo.foo.foo": 
"3foo"} ],
+  "default": null
 }],
 "my-property" : {
   "key" : 3


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Avro IDL: Support dot ('.') character in property annotation names
> --
>
> Key: AVRO-2132
> URL: https://issues.apache.org/jira/browse/AVRO-2132
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Affects Versions: 1.8.2
>Reporter: Sean Policarpio
>Assignee: Doug Cutting
>Priority: Major
> Fix For: 1.9.0
>
>
> Unless there is a strong reason why names like {{@foo.bar}} can't be used as 
> property annotations in IDL, I propose an enhancement to the IDL parser to 
> allow it.
> The major drive for this change comes from Kafka Connect; for a certain 
> fields – namely timestamps – additional metadata must be present in the 
> schema when certain consumers read the data (e.g. [the JDBC 
> connector|https://github.com/confluentinc/kafka-connect-jdbc]). What I hoped 
> when using IDL was to write the following for a record field:
> {code:java}
> union {null, @connect.version(1) 
> @connect.name("org.apache.kafka.connect.data.Timestamp") long} 
> queryTime;{code}
> so that the following would be available in the schemata:
> {code:java}
> {
>   "name": "queryTime",
>   "type": [
> "null",
> {
>   "type": "long",
>   "connect.version": 1,
>   "connect.name": "org.apache.kafka.connect.data.Timestamp"
> }
>   ],
>   "default": null
> }{code}
> Unfortunately, both {{connect.version}} and {{connect.name}} are unacceptable 
> by the parser.
> The change for this is quite minimal as it can be based on AVRO-1267.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2132) Avro IDL: Support dot ('.') character in property annotation names

2018-02-22 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/AVRO-2132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373721#comment-16373721
 ] 

ASF GitHub Bot commented on AVRO-2132:
--

kdrakon commented on issue #287: AVRO-2132 added missing defaults for new fields
URL: https://github.com/apache/avro/pull/287#issuecomment-367866284
 
 
   no problem @cutting 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Avro IDL: Support dot ('.') character in property annotation names
> --
>
> Key: AVRO-2132
> URL: https://issues.apache.org/jira/browse/AVRO-2132
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Affects Versions: 1.8.2
>Reporter: Sean Policarpio
>Assignee: Doug Cutting
>Priority: Major
> Fix For: 1.9.0
>
>
> Unless there is a strong reason why names like {{@foo.bar}} can't be used as 
> property annotations in IDL, I propose an enhancement to the IDL parser to 
> allow it.
> The major drive for this change comes from Kafka Connect; for a certain 
> fields – namely timestamps – additional metadata must be present in the 
> schema when certain consumers read the data (e.g. [the JDBC 
> connector|https://github.com/confluentinc/kafka-connect-jdbc]). What I hoped 
> when using IDL was to write the following for a record field:
> {code:java}
> union {null, @connect.version(1) 
> @connect.name("org.apache.kafka.connect.data.Timestamp") long} 
> queryTime;{code}
> so that the following would be available in the schemata:
> {code:java}
> {
>   "name": "queryTime",
>   "type": [
> "null",
> {
>   "type": "long",
>   "connect.version": 1,
>   "connect.name": "org.apache.kafka.connect.data.Timestamp"
> }
>   ],
>   "default": null
> }{code}
> Unfortunately, both {{connect.version}} and {{connect.name}} are unacceptable 
> by the parser.
> The change for this is quite minimal as it can be based on AVRO-1267.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2132) Avro IDL: Support dot ('.') character in property annotation names

2018-02-22 Thread Doug Cutting (JIRA)


[ 
https://issues.apache.org/jira/browse/AVRO-2132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373696#comment-16373696
 ] 

Doug Cutting commented on AVRO-2132:


It turns out that the problem was that defaults were only checked when parsing 
schemas from JSON, not when creating them from constructors.  Fixing that 
caused a few other things to fail.  I've submitted the fix in AVRO-2035.

> Avro IDL: Support dot ('.') character in property annotation names
> --
>
> Key: AVRO-2132
> URL: https://issues.apache.org/jira/browse/AVRO-2132
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Affects Versions: 1.8.2
>Reporter: Sean Policarpio
>Assignee: Doug Cutting
>Priority: Major
> Fix For: 1.9.0
>
>
> Unless there is a strong reason why names like {{@foo.bar}} can't be used as 
> property annotations in IDL, I propose an enhancement to the IDL parser to 
> allow it.
> The major drive for this change comes from Kafka Connect; for a certain 
> fields – namely timestamps – additional metadata must be present in the 
> schema when certain consumers read the data (e.g. [the JDBC 
> connector|https://github.com/confluentinc/kafka-connect-jdbc]). What I hoped 
> when using IDL was to write the following for a record field:
> {code:java}
> union {null, @connect.version(1) 
> @connect.name("org.apache.kafka.connect.data.Timestamp") long} 
> queryTime;{code}
> so that the following would be available in the schemata:
> {code:java}
> {
>   "name": "queryTime",
>   "type": [
> "null",
> {
>   "type": "long",
>   "connect.version": 1,
>   "connect.name": "org.apache.kafka.connect.data.Timestamp"
> }
>   ],
>   "default": null
> }{code}
> Unfortunately, both {{connect.version}} and {{connect.name}} are unacceptable 
> by the parser.
> The change for this is quite minimal as it can be based on AVRO-1267.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (AVRO-2035) enable validation of default values in schemas by default

2018-02-22 Thread Doug Cutting (JIRA)


 [ 
https://issues.apache.org/jira/browse/AVRO-2035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Cutting updated AVRO-2035:
---
Status: Patch Available  (was: Reopened)

> enable validation of default values in schemas by default
> -
>
> Key: AVRO-2035
> URL: https://issues.apache.org/jira/browse/AVRO-2035
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.1
>Reporter: radai rosenblatt
>Assignee: Doug Cutting
>Priority: Major
> Fix For: 1.9.0
>
> Attachments: AVRO-2035.patch
>
>
> suppose i have the following schema evolution:
> {code}
> {
>   "name": "Bob",
>   "type": "record",
>   "fields": [
> {"name": "f1", "type": "int"}
>   ]
> }
> {code}
> and then:
> {code}
> {
>   "name": "Bob",
>   "type": "record",
>   "fields": [
> {"name": "f1", "type": "int"},
> {"name": "f2", "type": "boolean", "default": "true"}
>   ]
> }
> {code}
> the default value for "f2" is specified as the _STRING_ "true" (and not the 
> literal boolean true). 
> if this default value is ever accessed (when reading a gen1-serialized object 
> as a gen2) we get this:
> {code}
> org.apache.avro.AvroTypeException: Non-boolean default for boolean: "true"
>   at 
> org.apache.avro.io.parsing.ResolvingGrammarGenerator.encode(ResolvingGrammarGenerator.java:408)
>   at 
> org.apache.avro.io.parsing.ResolvingGrammarGenerator.getBinary(ResolvingGrammarGenerator.java:307)
>   at 
> org.apache.avro.io.parsing.ResolvingGrammarGenerator.resolveRecords(ResolvingGrammarGenerator.java:285)
>   at 
> org.apache.avro.io.parsing.ResolvingGrammarGenerator.generate(ResolvingGrammarGenerator.java:118)
>   at 
> org.apache.avro.io.parsing.ResolvingGrammarGenerator.generate(ResolvingGrammarGenerator.java:50)
>   at org.apache.avro.io.ResolvingDecoder.resolve(ResolvingDecoder.java:85)
>   at org.apache.avro.io.ResolvingDecoder.(ResolvingDecoder.java:49)
>   at 
> org.apache.avro.io.DecoderFactory.resolvingDecoder(DecoderFactory.java:307)
>   at 
> org.apache.avro.generic.GenericDatumReader.getResolver(GenericDatumReader.java:127)
>   at 
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)
> {code}
> yet Schema.parse() passes for this



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2035) enable validation of default values in schemas by default

2018-02-22 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/AVRO-2035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373686#comment-16373686
 ] 

ASF GitHub Bot commented on AVRO-2035:
--

cutting opened a new pull request #288: AVRO-2035. Java: enable default 
validation for schemas created by constructors.
URL: https://github.com/apache/avro/pull/288
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> enable validation of default values in schemas by default
> -
>
> Key: AVRO-2035
> URL: https://issues.apache.org/jira/browse/AVRO-2035
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.1
>Reporter: radai rosenblatt
>Assignee: Doug Cutting
>Priority: Major
> Fix For: 1.9.0
>
> Attachments: AVRO-2035.patch
>
>
> suppose i have the following schema evolution:
> {code}
> {
>   "name": "Bob",
>   "type": "record",
>   "fields": [
> {"name": "f1", "type": "int"}
>   ]
> }
> {code}
> and then:
> {code}
> {
>   "name": "Bob",
>   "type": "record",
>   "fields": [
> {"name": "f1", "type": "int"},
> {"name": "f2", "type": "boolean", "default": "true"}
>   ]
> }
> {code}
> the default value for "f2" is specified as the _STRING_ "true" (and not the 
> literal boolean true). 
> if this default value is ever accessed (when reading a gen1-serialized object 
> as a gen2) we get this:
> {code}
> org.apache.avro.AvroTypeException: Non-boolean default for boolean: "true"
>   at 
> org.apache.avro.io.parsing.ResolvingGrammarGenerator.encode(ResolvingGrammarGenerator.java:408)
>   at 
> org.apache.avro.io.parsing.ResolvingGrammarGenerator.getBinary(ResolvingGrammarGenerator.java:307)
>   at 
> org.apache.avro.io.parsing.ResolvingGrammarGenerator.resolveRecords(ResolvingGrammarGenerator.java:285)
>   at 
> org.apache.avro.io.parsing.ResolvingGrammarGenerator.generate(ResolvingGrammarGenerator.java:118)
>   at 
> org.apache.avro.io.parsing.ResolvingGrammarGenerator.generate(ResolvingGrammarGenerator.java:50)
>   at org.apache.avro.io.ResolvingDecoder.resolve(ResolvingDecoder.java:85)
>   at org.apache.avro.io.ResolvingDecoder.(ResolvingDecoder.java:49)
>   at 
> org.apache.avro.io.DecoderFactory.resolvingDecoder(DecoderFactory.java:307)
>   at 
> org.apache.avro.generic.GenericDatumReader.getResolver(GenericDatumReader.java:127)
>   at 
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)
> {code}
> yet Schema.parse() passes for this



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Reopened] (AVRO-2035) enable validation of default values in schemas by default

2018-02-22 Thread Doug Cutting (JIRA)


 [ 
https://issues.apache.org/jira/browse/AVRO-2035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Cutting reopened AVRO-2035:


This patch enabled validation for parsed schemas, but not for those created 
using schema constructors.  These are still unvalidated.

> enable validation of default values in schemas by default
> -
>
> Key: AVRO-2035
> URL: https://issues.apache.org/jira/browse/AVRO-2035
> Project: Avro
>  Issue Type: Bug
>  Components: java
>Affects Versions: 1.8.1
>Reporter: radai rosenblatt
>Assignee: Doug Cutting
>Priority: Major
> Fix For: 1.9.0
>
> Attachments: AVRO-2035.patch
>
>
> suppose i have the following schema evolution:
> {code}
> {
>   "name": "Bob",
>   "type": "record",
>   "fields": [
> {"name": "f1", "type": "int"}
>   ]
> }
> {code}
> and then:
> {code}
> {
>   "name": "Bob",
>   "type": "record",
>   "fields": [
> {"name": "f1", "type": "int"},
> {"name": "f2", "type": "boolean", "default": "true"}
>   ]
> }
> {code}
> the default value for "f2" is specified as the _STRING_ "true" (and not the 
> literal boolean true). 
> if this default value is ever accessed (when reading a gen1-serialized object 
> as a gen2) we get this:
> {code}
> org.apache.avro.AvroTypeException: Non-boolean default for boolean: "true"
>   at 
> org.apache.avro.io.parsing.ResolvingGrammarGenerator.encode(ResolvingGrammarGenerator.java:408)
>   at 
> org.apache.avro.io.parsing.ResolvingGrammarGenerator.getBinary(ResolvingGrammarGenerator.java:307)
>   at 
> org.apache.avro.io.parsing.ResolvingGrammarGenerator.resolveRecords(ResolvingGrammarGenerator.java:285)
>   at 
> org.apache.avro.io.parsing.ResolvingGrammarGenerator.generate(ResolvingGrammarGenerator.java:118)
>   at 
> org.apache.avro.io.parsing.ResolvingGrammarGenerator.generate(ResolvingGrammarGenerator.java:50)
>   at org.apache.avro.io.ResolvingDecoder.resolve(ResolvingDecoder.java:85)
>   at org.apache.avro.io.ResolvingDecoder.(ResolvingDecoder.java:49)
>   at 
> org.apache.avro.io.DecoderFactory.resolvingDecoder(DecoderFactory.java:307)
>   at 
> org.apache.avro.generic.GenericDatumReader.getResolver(GenericDatumReader.java:127)
>   at 
> org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:142)
> {code}
> yet Schema.parse() passes for this



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2132) Avro IDL: Support dot ('.') character in property annotation names

2018-02-22 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/AVRO-2132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373580#comment-16373580
 ] 

ASF GitHub Bot commented on AVRO-2132:
--

kdrakon opened a new pull request #287: AVRO-2132 added missing defaults for 
new fields
URL: https://github.com/apache/avro/pull/287
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Avro IDL: Support dot ('.') character in property annotation names
> --
>
> Key: AVRO-2132
> URL: https://issues.apache.org/jira/browse/AVRO-2132
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Affects Versions: 1.8.2
>Reporter: Sean Policarpio
>Assignee: Doug Cutting
>Priority: Major
> Fix For: 1.9.0
>
>
> Unless there is a strong reason why names like {{@foo.bar}} can't be used as 
> property annotations in IDL, I propose an enhancement to the IDL parser to 
> allow it.
> The major drive for this change comes from Kafka Connect; for a certain 
> fields – namely timestamps – additional metadata must be present in the 
> schema when certain consumers read the data (e.g. [the JDBC 
> connector|https://github.com/confluentinc/kafka-connect-jdbc]). What I hoped 
> when using IDL was to write the following for a record field:
> {code:java}
> union {null, @connect.version(1) 
> @connect.name("org.apache.kafka.connect.data.Timestamp") long} 
> queryTime;{code}
> so that the following would be available in the schemata:
> {code:java}
> {
>   "name": "queryTime",
>   "type": [
> "null",
> {
>   "type": "long",
>   "connect.version": 1,
>   "connect.name": "org.apache.kafka.connect.data.Timestamp"
> }
>   ],
>   "default": null
> }{code}
> Unfortunately, both {{connect.version}} and {{connect.name}} are unacceptable 
> by the parser.
> The change for this is quite minimal as it can be based on AVRO-1267.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (AVRO-2132) Avro IDL: Support dot ('.') character in property annotation names

2018-02-22 Thread Sean Policarpio (JIRA)


[ 
https://issues.apache.org/jira/browse/AVRO-2132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373572#comment-16373572
 ] 

Sean Policarpio edited comment on AVRO-2132 at 2/22/18 10:24 PM:
-

[~zolyfarkas] [~cutting]
If I'm understanding this correctly, 1.9.0 will enforce default values? If so, 
I can append defaults for both my new fields {{l}} and {{nested_properties}} 
({{0}} and {{null}} respectively). However, shouldn't the enum field {{kind}} 
also specified a default?


was (Author: kdrakon):
[~zolyfarkas] [~cutting]
If I'm understanding this correctly, 1.9.0 will enforce default values? If so, 
I can append defaults for both my new fields `l` and `nested_properties` (`0` 
and `null` respectively). However, shouldn't the enum field `kind` also 
specified a default?

> Avro IDL: Support dot ('.') character in property annotation names
> --
>
> Key: AVRO-2132
> URL: https://issues.apache.org/jira/browse/AVRO-2132
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Affects Versions: 1.8.2
>Reporter: Sean Policarpio
>Assignee: Doug Cutting
>Priority: Major
> Fix For: 1.9.0
>
>
> Unless there is a strong reason why names like {{@foo.bar}} can't be used as 
> property annotations in IDL, I propose an enhancement to the IDL parser to 
> allow it.
> The major drive for this change comes from Kafka Connect; for a certain 
> fields – namely timestamps – additional metadata must be present in the 
> schema when certain consumers read the data (e.g. [the JDBC 
> connector|https://github.com/confluentinc/kafka-connect-jdbc]). What I hoped 
> when using IDL was to write the following for a record field:
> {code:java}
> union {null, @connect.version(1) 
> @connect.name("org.apache.kafka.connect.data.Timestamp") long} 
> queryTime;{code}
> so that the following would be available in the schemata:
> {code:java}
> {
>   "name": "queryTime",
>   "type": [
> "null",
> {
>   "type": "long",
>   "connect.version": 1,
>   "connect.name": "org.apache.kafka.connect.data.Timestamp"
> }
>   ],
>   "default": null
> }{code}
> Unfortunately, both {{connect.version}} and {{connect.name}} are unacceptable 
> by the parser.
> The change for this is quite minimal as it can be based on AVRO-1267.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2132) Avro IDL: Support dot ('.') character in property annotation names

2018-02-22 Thread Sean Policarpio (JIRA)


[ 
https://issues.apache.org/jira/browse/AVRO-2132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373572#comment-16373572
 ] 

Sean Policarpio commented on AVRO-2132:
---

[~zolyfarkas] [~cutting]
If I'm understanding this correctly, 1.9.0 will enforce default values? If so, 
I can append defaults for both my new fields `l` and `nested_properties` (`0` 
and `null` respectively). However, shouldn't the enum field `kind` also 
specified a default?

> Avro IDL: Support dot ('.') character in property annotation names
> --
>
> Key: AVRO-2132
> URL: https://issues.apache.org/jira/browse/AVRO-2132
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Affects Versions: 1.8.2
>Reporter: Sean Policarpio
>Assignee: Doug Cutting
>Priority: Major
> Fix For: 1.9.0
>
>
> Unless there is a strong reason why names like {{@foo.bar}} can't be used as 
> property annotations in IDL, I propose an enhancement to the IDL parser to 
> allow it.
> The major drive for this change comes from Kafka Connect; for a certain 
> fields – namely timestamps – additional metadata must be present in the 
> schema when certain consumers read the data (e.g. [the JDBC 
> connector|https://github.com/confluentinc/kafka-connect-jdbc]). What I hoped 
> when using IDL was to write the following for a record field:
> {code:java}
> union {null, @connect.version(1) 
> @connect.name("org.apache.kafka.connect.data.Timestamp") long} 
> queryTime;{code}
> so that the following would be available in the schemata:
> {code:java}
> {
>   "name": "queryTime",
>   "type": [
> "null",
> {
>   "type": "long",
>   "connect.version": 1,
>   "connect.name": "org.apache.kafka.connect.data.Timestamp"
> }
>   ],
>   "default": null
> }{code}
> Unfortunately, both {{connect.version}} and {{connect.name}} are unacceptable 
> by the parser.
> The change for this is quite minimal as it can be based on AVRO-1267.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2132) Avro IDL: Support dot ('.') character in property annotation names

2018-02-22 Thread Doug Cutting (JIRA)


[ 
https://issues.apache.org/jira/browse/AVRO-2132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373515#comment-16373515
 ] 

Doug Cutting commented on AVRO-2132:


[~zolyfarkas], Yes, you are right.  However I'm not seeing any tests fail, are 
you?  There are multiple versions of simple.avdl in the sources, and I think 
the one altered here is compiled to JSON but never subsequently parsed as a 
protocol.

Thanks for catching this.

> Avro IDL: Support dot ('.') character in property annotation names
> --
>
> Key: AVRO-2132
> URL: https://issues.apache.org/jira/browse/AVRO-2132
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Affects Versions: 1.8.2
>Reporter: Sean Policarpio
>Assignee: Doug Cutting
>Priority: Major
> Fix For: 1.9.0
>
>
> Unless there is a strong reason why names like {{@foo.bar}} can't be used as 
> property annotations in IDL, I propose an enhancement to the IDL parser to 
> allow it.
> The major drive for this change comes from Kafka Connect; for a certain 
> fields – namely timestamps – additional metadata must be present in the 
> schema when certain consumers read the data (e.g. [the JDBC 
> connector|https://github.com/confluentinc/kafka-connect-jdbc]). What I hoped 
> when using IDL was to write the following for a record field:
> {code:java}
> union {null, @connect.version(1) 
> @connect.name("org.apache.kafka.connect.data.Timestamp") long} 
> queryTime;{code}
> so that the following would be available in the schemata:
> {code:java}
> {
>   "name": "queryTime",
>   "type": [
> "null",
> {
>   "type": "long",
>   "connect.version": 1,
>   "connect.name": "org.apache.kafka.connect.data.Timestamp"
> }
>   ],
>   "default": null
> }{code}
> Unfortunately, both {{connect.version}} and {{connect.name}} are unacceptable 
> by the parser.
> The change for this is quite minimal as it can be based on AVRO-1267.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2132) Avro IDL: Support dot ('.') character in property annotation names

2018-02-22 Thread Zoltan Farkas (JIRA)


[ 
https://issues.apache.org/jira/browse/AVRO-2132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373418#comment-16373418
 ] 

Zoltan Farkas commented on AVRO-2132:
-

[~cutting] [~kdrakon]

I just looked at this commit and tried to apply it to my fork.. and since my 
fork does not allow invalid default values defined in schemas/idl, the changes 
made here break the unit tests in the fork.

here is where the issue is:

the new declared fields in simple.avdl do not have default values:

{code}
record TestRecord {
...
@foo.bar("bar.foo") long l;
union {null, @foo.foo.bar(42) @foo.foo.foo("3foo") string} 
nested_properties;
{code}

and as such the method declaration bellow is invalid:

  TestRecord echo(TestRecord `record` = {"name":"bar","kind":"BAR"});

and to be correct since the new added fields do not have defaults it should be 
something like:

  TestRecord echo(TestRecord `record` = 
{"name":"bar","kind":"BAR","l":0,"nested_properties":null});

I thought https://issues.apache.org/jira/browse/AVRO-2035 is resolved?

worthwhile to look into this... cheers.

> Avro IDL: Support dot ('.') character in property annotation names
> --
>
> Key: AVRO-2132
> URL: https://issues.apache.org/jira/browse/AVRO-2132
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Affects Versions: 1.8.2
>Reporter: Sean Policarpio
>Assignee: Doug Cutting
>Priority: Major
> Fix For: 1.9.0
>
>
> Unless there is a strong reason why names like {{@foo.bar}} can't be used as 
> property annotations in IDL, I propose an enhancement to the IDL parser to 
> allow it.
> The major drive for this change comes from Kafka Connect; for a certain 
> fields – namely timestamps – additional metadata must be present in the 
> schema when certain consumers read the data (e.g. [the JDBC 
> connector|https://github.com/confluentinc/kafka-connect-jdbc]). What I hoped 
> when using IDL was to write the following for a record field:
> {code:java}
> union {null, @connect.version(1) 
> @connect.name("org.apache.kafka.connect.data.Timestamp") long} 
> queryTime;{code}
> so that the following would be available in the schemata:
> {code:java}
> {
>   "name": "queryTime",
>   "type": [
> "null",
> {
>   "type": "long",
>   "connect.version": 1,
>   "connect.name": "org.apache.kafka.connect.data.Timestamp"
> }
>   ],
>   "default": null
> }{code}
> Unfortunately, both {{connect.version}} and {{connect.name}} are unacceptable 
> by the parser.
> The change for this is quite minimal as it can be based on AVRO-1267.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2132) Avro IDL: Support dot ('.') character in property annotation names

2018-02-22 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/AVRO-2132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373316#comment-16373316
 ] 

ASF GitHub Bot commented on AVRO-2132:
--

kdrakon commented on issue #279: AVRO-2132: allow the dot character to be used 
in the name of an IDL property annotation
URL: https://github.com/apache/avro/pull/279#issuecomment-367795242
 
 
   Cheers @cutting 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Avro IDL: Support dot ('.') character in property annotation names
> --
>
> Key: AVRO-2132
> URL: https://issues.apache.org/jira/browse/AVRO-2132
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Affects Versions: 1.8.2
>Reporter: Sean Policarpio
>Assignee: Doug Cutting
>Priority: Major
> Fix For: 1.9.0
>
>
> Unless there is a strong reason why names like {{@foo.bar}} can't be used as 
> property annotations in IDL, I propose an enhancement to the IDL parser to 
> allow it.
> The major drive for this change comes from Kafka Connect; for a certain 
> fields – namely timestamps – additional metadata must be present in the 
> schema when certain consumers read the data (e.g. [the JDBC 
> connector|https://github.com/confluentinc/kafka-connect-jdbc]). What I hoped 
> when using IDL was to write the following for a record field:
> {code:java}
> union {null, @connect.version(1) 
> @connect.name("org.apache.kafka.connect.data.Timestamp") long} 
> queryTime;{code}
> so that the following would be available in the schemata:
> {code:java}
> {
>   "name": "queryTime",
>   "type": [
> "null",
> {
>   "type": "long",
>   "connect.version": 1,
>   "connect.name": "org.apache.kafka.connect.data.Timestamp"
> }
>   ],
>   "default": null
> }{code}
> Unfortunately, both {{connect.version}} and {{connect.name}} are unacceptable 
> by the parser.
> The change for this is quite minimal as it can be based on AVRO-1267.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2132) Avro IDL: Support dot ('.') character in property annotation names

2018-02-22 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/AVRO-2132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373291#comment-16373291
 ] 

ASF GitHub Bot commented on AVRO-2132:
--

cutting closed pull request #279: AVRO-2132: allow the dot character to be used 
in the name of an IDL property annotation
URL: https://github.com/apache/avro/pull/279
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/lang/java/compiler/src/main/javacc/org/apache/avro/compiler/idl/idl.jj 
b/lang/java/compiler/src/main/javacc/org/apache/avro/compiler/idl/idl.jj
index 421321019..7de123e20 100644
--- a/lang/java/compiler/src/main/javacc/org/apache/avro/compiler/idl/idl.jj
+++ b/lang/java/compiler/src/main/javacc/org/apache/avro/compiler/idl/idl.jj
@@ -1528,6 +1528,8 @@ String PropertyName():
 {
   t ={ name.append(t.image); }
   ( t ={ name.append(t.image); }
+t =  { name.append(t.image); } |
+t = { name.append(t.image); }
 t =  { name.append(t.image); }
   ) *
   { return name.toString(); }
diff --git a/lang/java/compiler/src/test/idl/input/simple.avdl 
b/lang/java/compiler/src/test/idl/input/simple.avdl
index 37aed9e11..1bfcd7b27 100644
--- a/lang/java/compiler/src/test/idl/input/simple.avdl
+++ b/lang/java/compiler/src/test/idl/input/simple.avdl
@@ -49,6 +49,9 @@ protocol Simple {
 float average = -Infinity;
 date d = 0;
 time_ms t = 0;
+
+@foo.bar("bar.foo") long l;
+union {null, @foo.foo.bar(42) @foo.foo.foo("3foo") string} 
nested_properties;
   }
 
   error TestError {
diff --git a/lang/java/compiler/src/test/idl/output/simple.avpr 
b/lang/java/compiler/src/test/idl/output/simple.avpr
index c3e18e237..f40030790 100644
--- a/lang/java/compiler/src/test/idl/output/simple.avpr
+++ b/lang/java/compiler/src/test/idl/output/simple.avpr
@@ -54,7 +54,13 @@
   "name": "t",
   "type": {"type": "int", "logicalType": "time-millis"},
   "default": 0
-} ],
+} , {
+  "name": "l",
+  "type": {"type": "long", "foo.bar": "bar.foo"}
+} , {
+  "name": "nested_properties",
+  "type": [ "null" , {"type":"string", "foo.foo.bar": 42, "foo.foo.foo": 
"3foo"} ]
+}],
 "my-property" : {
   "key" : 3
 }


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Avro IDL: Support dot ('.') character in property annotation names
> --
>
> Key: AVRO-2132
> URL: https://issues.apache.org/jira/browse/AVRO-2132
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Affects Versions: 1.8.2
>Reporter: Sean Policarpio
>Assignee: Doug Cutting
>Priority: Major
> Fix For: 1.9.0
>
>
> Unless there is a strong reason why names like {{@foo.bar}} can't be used as 
> property annotations in IDL, I propose an enhancement to the IDL parser to 
> allow it.
> The major drive for this change comes from Kafka Connect; for a certain 
> fields – namely timestamps – additional metadata must be present in the 
> schema when certain consumers read the data (e.g. [the JDBC 
> connector|https://github.com/confluentinc/kafka-connect-jdbc]). What I hoped 
> when using IDL was to write the following for a record field:
> {code:java}
> union {null, @connect.version(1) 
> @connect.name("org.apache.kafka.connect.data.Timestamp") long} 
> queryTime;{code}
> so that the following would be available in the schemata:
> {code:java}
> {
>   "name": "queryTime",
>   "type": [
> "null",
> {
>   "type": "long",
>   "connect.version": 1,
>   "connect.name": "org.apache.kafka.connect.data.Timestamp"
> }
>   ],
>   "default": null
> }{code}
> Unfortunately, both {{connect.version}} and {{connect.name}} are unacceptable 
> by the parser.
> The change for this is quite minimal as it can be based on AVRO-1267.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (AVRO-2132) Avro IDL: Support dot ('.') character in property annotation names

2018-02-22 Thread Doug Cutting (JIRA)


 [ 
https://issues.apache.org/jira/browse/AVRO-2132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Doug Cutting resolved AVRO-2132.

   Resolution: Fixed
 Assignee: Doug Cutting
Fix Version/s: 1.9.0

I committed this.

> Avro IDL: Support dot ('.') character in property annotation names
> --
>
> Key: AVRO-2132
> URL: https://issues.apache.org/jira/browse/AVRO-2132
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Affects Versions: 1.8.2
>Reporter: Sean Policarpio
>Assignee: Doug Cutting
>Priority: Major
> Fix For: 1.9.0
>
>
> Unless there is a strong reason why names like {{@foo.bar}} can't be used as 
> property annotations in IDL, I propose an enhancement to the IDL parser to 
> allow it.
> The major drive for this change comes from Kafka Connect; for a certain 
> fields – namely timestamps – additional metadata must be present in the 
> schema when certain consumers read the data (e.g. [the JDBC 
> connector|https://github.com/confluentinc/kafka-connect-jdbc]). What I hoped 
> when using IDL was to write the following for a record field:
> {code:java}
> union {null, @connect.version(1) 
> @connect.name("org.apache.kafka.connect.data.Timestamp") long} 
> queryTime;{code}
> so that the following would be available in the schemata:
> {code:java}
> {
>   "name": "queryTime",
>   "type": [
> "null",
> {
>   "type": "long",
>   "connect.version": 1,
>   "connect.name": "org.apache.kafka.connect.data.Timestamp"
> }
>   ],
>   "default": null
> }{code}
> Unfortunately, both {{connect.version}} and {{connect.name}} are unacceptable 
> by the parser.
> The change for this is quite minimal as it can be based on AVRO-1267.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2150) Improved idl syntax support for "marker properties"

2018-02-22 Thread Zoltan Farkas (JIRA)


[ 
https://issues.apache.org/jira/browse/AVRO-2150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373288#comment-16373288
 ] 

Zoltan Farkas commented on AVRO-2150:
-

Implementation of this is trivial, see: 

https://github.com/zolyfarkas/avro/commit/6659315f5f78feac37f5501bf0057d3f2bc817d0#diff-136e900cc327974ae416a44248e47d0a

for a potential implementation.

> Improved idl syntax support for "marker properties"
> ---
>
> Key: AVRO-2150
> URL: https://issues.apache.org/jira/browse/AVRO-2150
> Project: Avro
>  Issue Type: Improvement
>Reporter: Zoltan Farkas
>Priority: Minor
>
> It would be nice to allow in IDL "marker properties" like:
> {code}
> @MarkerProperty
> record TestRecord {
> 
> }
> {code}
> this would be only a simpler syntax for:
> {code}
> @MarkerProperty("")
> record TestRecord {
> 
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2132) Avro IDL: Support dot ('.') character in property annotation names

2018-02-22 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/AVRO-2132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373285#comment-16373285
 ] 

ASF subversion and git services commented on AVRO-2132:
---

Commit 9c200c993147dd3ac0b5f6ed94b558ad863d7c18 in avro's branch 
refs/heads/master from [~cutting]
[ https://gitbox.apache.org/repos/asf?p=avro.git;h=9c200c9 ]

AVRO-2132: Allow dots in IDL property names. (kdrakon via cutting)


> Avro IDL: Support dot ('.') character in property annotation names
> --
>
> Key: AVRO-2132
> URL: https://issues.apache.org/jira/browse/AVRO-2132
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Affects Versions: 1.8.2
>Reporter: Sean Policarpio
>Priority: Major
>
> Unless there is a strong reason why names like {{@foo.bar}} can't be used as 
> property annotations in IDL, I propose an enhancement to the IDL parser to 
> allow it.
> The major drive for this change comes from Kafka Connect; for a certain 
> fields – namely timestamps – additional metadata must be present in the 
> schema when certain consumers read the data (e.g. [the JDBC 
> connector|https://github.com/confluentinc/kafka-connect-jdbc]). What I hoped 
> when using IDL was to write the following for a record field:
> {code:java}
> union {null, @connect.version(1) 
> @connect.name("org.apache.kafka.connect.data.Timestamp") long} 
> queryTime;{code}
> so that the following would be available in the schemata:
> {code:java}
> {
>   "name": "queryTime",
>   "type": [
> "null",
> {
>   "type": "long",
>   "connect.version": 1,
>   "connect.name": "org.apache.kafka.connect.data.Timestamp"
> }
>   ],
>   "default": null
> }{code}
> Unfortunately, both {{connect.version}} and {{connect.name}} are unacceptable 
> by the parser.
> The change for this is quite minimal as it can be based on AVRO-1267.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (AVRO-2150) Improved idl syntax support for "marker properties"

2018-02-22 Thread Zoltan Farkas (JIRA)

Zoltan Farkas created AVRO-2150:
---

 Summary: Improved idl syntax support for "marker properties"
 Key: AVRO-2150
 URL: https://issues.apache.org/jira/browse/AVRO-2150
 Project: Avro
  Issue Type: Improvement
Reporter: Zoltan Farkas


It would be nice to allow in IDL "marker properties" like:

{code}
@MarkerProperty
record TestRecord {

}
{code}

this would be only a simpler syntax for:

{code}
@MarkerProperty("")
record TestRecord {

}
{code}





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (AVRO-2149) Using GenericDatumWriter for writing GenericRecord with schema from the specified generated type

2018-02-22 Thread Doug Cutting (JIRA)


[ 
https://issues.apache.org/jira/browse/AVRO-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373035#comment-16373035
 ] 

Doug Cutting commented on AVRO-2149:


By "generated type" you mean an enum class definition generated by Avro's 
specific compiler?  If so, then you should use SpecificDatumReader instead.  
GenericDatumReader works for the generic data model.  SpecificDatumReader works 
for classes generated by the specific compiler (and generic data, because it 
inherits from GenericDatumReader).  ReflectDatumReader works for most POJO 
classes (plus specific & generic).

> Using GenericDatumWriter for writing GenericRecord with schema from the 
> specified generated type
> 
>
> Key: AVRO-2149
> URL: https://issues.apache.org/jira/browse/AVRO-2149
> Project: Avro
>  Issue Type: Improvement
>  Components: java
>Affects Versions: 1.8.2
>Reporter: Wojciech Sroka
>Priority: Major
>
> Currently using GenericDatumWriter for writing GenericRecord with schema from 
> the specified generated type cause exception when one of fields is enum type:
> {code:java}
> Exception in thread "main" 
> org.apache.avro.file.DataFileWriter$AppendWriteException: 
> org.apache.avro.AvroTypeException: Not an enum: YELLOW
> at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:308)
> at 
> pl.ingbank.kafka.AvroErrorTester.runGenericWriterReaderTest(AvroErrorTester.java:65)
> at pl.ingbank.kafka.AvroErrorTester.main(AvroErrorTester.java:27)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
> Caused by: org.apache.avro.AvroTypeException: Not an enum: YELLOW
> at 
> org.apache.avro.generic.GenericDatumWriter.writeEnum(GenericDatumWriter.java:195)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:128)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:75)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:177)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:167)
> at 
> org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:118)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:75)
> at 
> org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:62)
> at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:302)
> ... 7 more{code}
> It is caused by method protected void writeEnum(Schema schema, Object datum, 
> Encoder out) in class org.apache.avro.generic.GenericDatumWriter:
> {code:java}
>   protected void writeEnum(Schema schema, Object datum, Encoder out) throws 
> IOException {
> if (!data.isEnum(datum)) {
>   throw new AvroTypeException("Not an enum: " + datum);
> }
> out.writeEnum(schema.getEnumOrdinal(datum.toString()));
>   }{code}
> Fix for this is to add additional check does datum is enum type:
> {code:java}
> protected void writeEnum(Schema schema, Object datum, Encoder out) throws 
> IOException {
>   if(datum.getClass().isEnum() && !data.isEnum(datum)){
>throw new AvroTypeException("Not an enum: " + datum);
>   }
>   out.writeEnum(schema.getEnumOrdinal(datum.toString()));
> }
> {code}
> This fix alows to serialize and deserialize generic record with schema from 
> the specified generated type.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Avro serialization in streaming context

2018-02-22 Thread Doug Cutting

These sound to me like they'd be useful additions.

Please file an issue in Jira and then issue a Github pull request against
it.

https://cwiki.apache.org/confluence/display/AVRO/How+To+Contribute

Thanks!

Doug

On Wed, Feb 21, 2018 at 10:18 PM, Commeau, Gabriel <
gabriel.comm...@gmail.com> wrote:

> Hi all,
>
> I’ve been working for several years now on a streaming data platform, and
> we’ve been using Avro to serialize the messages that flow through the
> distributed queue (Kafka/Kinesis). Because the message payload contains
> just one record or a small batch of Avro records, the serialization
> mechanisms are slightly different than the typical file-based ones. I wrote
> a few utility classes that facilitate the serialization and deserialization
> of records in the data streaming context, and I’m poking the community to
> see if there’s an appetite for me contributing it back to the main Avro
> project – as opposed to creating a small independent library.
>
> The utility allows to convert a record with a single method call:
>
> - from a GenericRecord (and therefore SpecificRecord as well) to binary or
> json
>
> - from binary or json to a GenericRecord
>
> - from binary or json to a SpecificRecord
>
> There’s also a few additional utility methods to generically get an
> attribute or its schema based on a path, provided as a string array.
>
> So first, I haven’t seen a utility like that, but please correct me if I
> missed it. Then do you think it’ll be a contribution that you’d welcome?
>
> Thanks!
>
>
>
> Gabriel
>
>

[jira] [Commented] (AVRO-2035) enable validation of default values in schemas by default

[jira] [Commented] (AVRO-2132) Avro IDL: Support dot ('.') character in property annotation names

[jira] [Commented] (AVRO-2132) Avro IDL: Support dot ('.') character in property annotation names

[jira] [Commented] (AVRO-2132) Avro IDL: Support dot ('.') character in property annotation names

[jira] [Updated] (AVRO-2035) enable validation of default values in schemas by default

[jira] [Commented] (AVRO-2035) enable validation of default values in schemas by default

[jira] [Reopened] (AVRO-2035) enable validation of default values in schemas by default

[jira] [Commented] (AVRO-2132) Avro IDL: Support dot ('.') character in property annotation names

[jira] [Comment Edited] (AVRO-2132) Avro IDL: Support dot ('.') character in property annotation names

[jira] [Commented] (AVRO-2132) Avro IDL: Support dot ('.') character in property annotation names

[jira] [Commented] (AVRO-2132) Avro IDL: Support dot ('.') character in property annotation names

[jira] [Commented] (AVRO-2132) Avro IDL: Support dot ('.') character in property annotation names

[jira] [Commented] (AVRO-2132) Avro IDL: Support dot ('.') character in property annotation names

[jira] [Commented] (AVRO-2132) Avro IDL: Support dot ('.') character in property annotation names

[jira] [Resolved] (AVRO-2132) Avro IDL: Support dot ('.') character in property annotation names

[jira] [Commented] (AVRO-2150) Improved idl syntax support for "marker properties"

[jira] [Commented] (AVRO-2132) Avro IDL: Support dot ('.') character in property annotation names

[jira] [Created] (AVRO-2150) Improved idl syntax support for "marker properties"

[jira] [Commented] (AVRO-2149) Using GenericDatumWriter for writing GenericRecord with schema from the specified generated type

Re: Avro serialization in streaming context

20 matches

Site Navigation

Mail list logo

Footer information