[jira] [Commented] (AVRO-1810) GenericDatumWriter broken with Enum
[ https://issues.apache.org/jira/browse/AVRO-1810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17058917#comment-17058917 ] Daniel Blazevski commented on AVRO-1810: Would it be possible to have a backport release of this in 1.8x? I would rather not have to upgrade to 1.9x just for this since I may have to pull in jars compiled w/ 1.8 in the project. > GenericDatumWriter broken with Enum > --- > > Key: AVRO-1810 > URL: https://issues.apache.org/jira/browse/AVRO-1810 > Project: Apache Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.0 >Reporter: Ryon Day >Assignee: Fokko Driesprong >Priority: Blocker > Fix For: 1.9.0 > > > {panel:title=Description|titleBGColor=#3FA|bgColor=#DDD} > Using the GenericDatumWriter with either Generic OR SpecificRecord will break > if an Enum is present. > {panel} > {panel:title=Steps To Reproduce|titleBGColor=#8DB|bgColor=#DDD} > I have been tracking Avro decoding oddities for a while. > The tests for this issue can be found > [here|https://github.com/ryonday/avroDecodingHelp/blob/master/src/test/java/com/ryonday/test/Avro180EnumFail.java] > {panel} > {panel:title=Notes|titleBGColor=#3AF|bgColor=#DDD} > Due to the debacle that is the Avro "UTF8" object, we have been avoiding it > by using the following scheme: > * Write incoming records to a byte array using the GenericDatumWriter > * Read back the byte array to our compiled Java domain objects using a > SpecificDatumWriter > This worked great with Avro 1.7.7, and this is a binary-incompatable breaking > change with 1.8.0. > This would appear to be caused by an addition in the > {{GenericDatumWriter:163-164}}: > {code} > if (!data.isEnum(datum)) > throw new AvroTypeException("Not an enum: "+datum); > {code} > {panel} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (AVRO-1810) GenericDatumWriter broken with Enum
[ https://issues.apache.org/jira/browse/AVRO-1810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16783909#comment-16783909 ] Hudson commented on AVRO-1810: -- FAILURE: Integrated in Jenkins build AvroJava #612 (See [https://builds.apache.org/job/AvroJava/612/]) AVRO-1810: Fix GenericDatumWriter w/ enums (#462) (fokko: [https://github.com/apache/avro/commit/4a3609cd854e9ef5bdda5b43e188a3309a93aef9]) * (edit) lang/java/tools/src/test/compiler/output-string/Position.java * (edit) lang/java/tools/src/test/compiler/output/Position.java * (add) lang/java/avro/src/test/java/org/apache/avro/generic/TestGenericConcreteEnum.java * (edit) lang/java/compiler/src/main/velocity/org/apache/avro/compiler/specific/templates/java/classic/enum.vm * (edit) lang/java/avro/src/main/java/org/apache/avro/generic/GenericEnumSymbol.java * (edit) lang/java/avro/src/test/java/org/apache/avro/TypeEnum.java * (edit) lang/java/avro/src/main/java/org/apache/avro/generic/GenericData.java * (edit) lang/java/tools/src/test/compiler/output-string/avro/examples/baseball/Position.java > GenericDatumWriter broken with Enum > --- > > Key: AVRO-1810 > URL: https://issues.apache.org/jira/browse/AVRO-1810 > Project: Apache Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.0 >Reporter: Ryon Day >Assignee: Fokko Driesprong >Priority: Blocker > Fix For: 1.9.0, 1.8.4 > > > {panel:title=Description|titleBGColor=#3FA|bgColor=#DDD} > Using the GenericDatumWriter with either Generic OR SpecificRecord will break > if an Enum is present. > {panel} > {panel:title=Steps To Reproduce|titleBGColor=#8DB|bgColor=#DDD} > I have been tracking Avro decoding oddities for a while. > The tests for this issue can be found > [here|https://github.com/ryonday/avroDecodingHelp/blob/master/src/test/java/com/ryonday/test/Avro180EnumFail.java] > {panel} > {panel:title=Notes|titleBGColor=#3AF|bgColor=#DDD} > Due to the debacle that is the Avro "UTF8" object, we have been avoiding it > by using the following scheme: > * Write incoming records to a byte array using the GenericDatumWriter > * Read back the byte array to our compiled Java domain objects using a > SpecificDatumWriter > This worked great with Avro 1.7.7, and this is a binary-incompatable breaking > change with 1.8.0. > This would appear to be caused by an addition in the > {{GenericDatumWriter:163-164}}: > {code} > if (!data.isEnum(datum)) > throw new AvroTypeException("Not an enum: "+datum); > {code} > {panel} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-1810) GenericDatumWriter broken with Enum
[ https://issues.apache.org/jira/browse/AVRO-1810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16783853#comment-16783853 ] ASF subversion and git services commented on AVRO-1810: --- Commit 4a3609cd854e9ef5bdda5b43e188a3309a93aef9 in avro's branch refs/heads/master from ivangreene [ https://gitbox.apache.org/repos/asf?p=avro.git;h=4a3609c ] AVRO-1810: Fix GenericDatumWriter w/ enums (#462) > GenericDatumWriter broken with Enum > --- > > Key: AVRO-1810 > URL: https://issues.apache.org/jira/browse/AVRO-1810 > Project: Apache Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.0 >Reporter: Ryon Day >Priority: Blocker > Fix For: 1.9.0, 1.8.4 > > > {panel:title=Description|titleBGColor=#3FA|bgColor=#DDD} > Using the GenericDatumWriter with either Generic OR SpecificRecord will break > if an Enum is present. > {panel} > {panel:title=Steps To Reproduce|titleBGColor=#8DB|bgColor=#DDD} > I have been tracking Avro decoding oddities for a while. > The tests for this issue can be found > [here|https://github.com/ryonday/avroDecodingHelp/blob/master/src/test/java/com/ryonday/test/Avro180EnumFail.java] > {panel} > {panel:title=Notes|titleBGColor=#3AF|bgColor=#DDD} > Due to the debacle that is the Avro "UTF8" object, we have been avoiding it > by using the following scheme: > * Write incoming records to a byte array using the GenericDatumWriter > * Read back the byte array to our compiled Java domain objects using a > SpecificDatumWriter > This worked great with Avro 1.7.7, and this is a binary-incompatable breaking > change with 1.8.0. > This would appear to be caused by an addition in the > {{GenericDatumWriter:163-164}}: > {code} > if (!data.isEnum(datum)) > throw new AvroTypeException("Not an enum: "+datum); > {code} > {panel} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-1810) GenericDatumWriter broken with Enum
[ https://issues.apache.org/jira/browse/AVRO-1810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463921#comment-16463921 ] Adam Bellemare commented on AVRO-1810: -- [~zolyfarkas] - How has the use of your branch been in the past months? I am interested in having this ticket revived as this is something that I will also be needing to do. Is there anything I could do to help? > GenericDatumWriter broken with Enum > --- > > Key: AVRO-1810 > URL: https://issues.apache.org/jira/browse/AVRO-1810 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.0 >Reporter: Ryon Day >Priority: Blocker > Fix For: 1.9.0, 1.8.4 > > > {panel:title=Description|titleBGColor=#3FA|bgColor=#DDD} > Using the GenericDatumWriter with either Generic OR SpecificRecord will break > if an Enum is present. > {panel} > {panel:title=Steps To Reproduce|titleBGColor=#8DB|bgColor=#DDD} > I have been tracking Avro decoding oddities for a while. > The tests for this issue can be found > [here|https://github.com/ryonday/avroDecodingHelp/blob/master/src/test/java/com/ryonday/test/Avro180EnumFail.java] > {panel} > {panel:title=Notes|titleBGColor=#3AF|bgColor=#DDD} > Due to the debacle that is the Avro "UTF8" object, we have been avoiding it > by using the following scheme: > * Write incoming records to a byte array using the GenericDatumWriter > * Read back the byte array to our compiled Java domain objects using a > SpecificDatumWriter > This worked great with Avro 1.7.7, and this is a binary-incompatable breaking > change with 1.8.0. > This would appear to be caused by an addition in the > {{GenericDatumWriter:163-164}}: > {code} > if (!data.isEnum(datum)) > throw new AvroTypeException("Not an enum: "+datum); > {code} > {panel} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-1810) GenericDatumWriter broken with Enum
[ https://issues.apache.org/jira/browse/AVRO-1810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166847#comment-16166847 ] Bridger Howell commented on AVRO-1810: -- No particular use case, I was just looking at the way that {{GenericData.EnumSymbol}} supports being compared with any other {{GenericEnumSymbol}} and wondering if there was a good reason for that. Considering that {{equals}} is final for enums, it seems harder to keep that property (although {{compareTo}} should be fine because the type signatures for {{java.lang.Enum}}'s {{compareTo}} only matches the enum type, not {{GenericEnumSymbol}}?). At least It would be nice to avoid weird cases where the following test fails: {noformat} /* assuming SpecificEnum is generated from enumSchema */ final SpecificEnum specificSymbol = SpecificEnum.FOO; final GenericData.EnumSymbol genericSymbol = new GenericData.EnumSymbol(enumSchema, "FOO"); assertTrue(genericSymbol.equals(specificSymbol) == specificSymbol.equals(genericSymbol)); {noformat} > GenericDatumWriter broken with Enum > --- > > Key: AVRO-1810 > URL: https://issues.apache.org/jira/browse/AVRO-1810 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.0 >Reporter: Ryon Day >Priority: Blocker > Fix For: 1.9.0, 1.8.4 > > > {panel:title=Description|titleBGColor=#3FA|bgColor=#DDD} > Using the GenericDatumWriter with either Generic OR SpecificRecord will break > if an Enum is present. > {panel} > {panel:title=Steps To Reproduce|titleBGColor=#8DB|bgColor=#DDD} > I have been tracking Avro decoding oddities for a while. > The tests for this issue can be found > [here|https://github.com/ryonday/avroDecodingHelp/blob/master/src/test/java/com/ryonday/test/Avro180EnumFail.java] > {panel} > {panel:title=Notes|titleBGColor=#3AF|bgColor=#DDD} > Due to the debacle that is the Avro "UTF8" object, we have been avoiding it > by using the following scheme: > * Write incoming records to a byte array using the GenericDatumWriter > * Read back the byte array to our compiled Java domain objects using a > SpecificDatumWriter > This worked great with Avro 1.7.7, and this is a binary-incompatable breaking > change with 1.8.0. > This would appear to be caused by an addition in the > {{GenericDatumWriter:163-164}}: > {code} > if (!data.isEnum(datum)) > throw new AvroTypeException("Not an enum: "+datum); > {code} > {panel} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-1810) GenericDatumWriter broken with Enum
[ https://issues.apache.org/jira/browse/AVRO-1810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166749#comment-16166749 ] Zoltan Farkas commented on AVRO-1810: - [~howellbridger] java.lang.Enum equals, hashCode, compareTo are final and cannot be overloaded... So if one would need to compare generated enums with generic enums a custom comparator would be the way... what is the use case you are thinking of? > GenericDatumWriter broken with Enum > --- > > Key: AVRO-1810 > URL: https://issues.apache.org/jira/browse/AVRO-1810 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.0 >Reporter: Ryon Day >Priority: Blocker > Fix For: 1.9.0, 1.8.4 > > > {panel:title=Description|titleBGColor=#3FA|bgColor=#DDD} > Using the GenericDatumWriter with either Generic OR SpecificRecord will break > if an Enum is present. > {panel} > {panel:title=Steps To Reproduce|titleBGColor=#8DB|bgColor=#DDD} > I have been tracking Avro decoding oddities for a while. > The tests for this issue can be found > [here|https://github.com/ryonday/avroDecodingHelp/blob/master/src/test/java/com/ryonday/test/Avro180EnumFail.java] > {panel} > {panel:title=Notes|titleBGColor=#3AF|bgColor=#DDD} > Due to the debacle that is the Avro "UTF8" object, we have been avoiding it > by using the following scheme: > * Write incoming records to a byte array using the GenericDatumWriter > * Read back the byte array to our compiled Java domain objects using a > SpecificDatumWriter > This worked great with Avro 1.7.7, and this is a binary-incompatable breaking > change with 1.8.0. > This would appear to be caused by an addition in the > {{GenericDatumWriter:163-164}}: > {code} > if (!data.isEnum(datum)) > throw new AvroTypeException("Not an enum: "+datum); > {code} > {panel} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-1810) GenericDatumWriter broken with Enum
[ https://issues.apache.org/jira/browse/AVRO-1810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166383#comment-16166383 ] Bridger Howell commented on AVRO-1810: -- [~zolyfarkas] Would it be bad to interoperate {{GenericData.EnumSymbol}} instances with generated classes in terms of {{equals}} and {{compareTo}}? > GenericDatumWriter broken with Enum > --- > > Key: AVRO-1810 > URL: https://issues.apache.org/jira/browse/AVRO-1810 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.0 >Reporter: Ryon Day >Priority: Blocker > Fix For: 1.9.0, 1.8.4 > > > {panel:title=Description|titleBGColor=#3FA|bgColor=#DDD} > Using the GenericDatumWriter with either Generic OR SpecificRecord will break > if an Enum is present. > {panel} > {panel:title=Steps To Reproduce|titleBGColor=#8DB|bgColor=#DDD} > I have been tracking Avro decoding oddities for a while. > The tests for this issue can be found > [here|https://github.com/ryonday/avroDecodingHelp/blob/master/src/test/java/com/ryonday/test/Avro180EnumFail.java] > {panel} > {panel:title=Notes|titleBGColor=#3AF|bgColor=#DDD} > Due to the debacle that is the Avro "UTF8" object, we have been avoiding it > by using the following scheme: > * Write incoming records to a byte array using the GenericDatumWriter > * Read back the byte array to our compiled Java domain objects using a > SpecificDatumWriter > This worked great with Avro 1.7.7, and this is a binary-incompatable breaking > change with 1.8.0. > This would appear to be caused by an addition in the > {{GenericDatumWriter:163-164}}: > {code} > if (!data.isEnum(datum)) > throw new AvroTypeException("Not an enum: "+datum); > {code} > {panel} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-1810) GenericDatumWriter broken with Enum
[ https://issues.apache.org/jira/browse/AVRO-1810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166377#comment-16166377 ] Zoltan Farkas commented on AVRO-1810: - The way I resolved this in my for was to make the Generated enums implement org.apache.avro.generic.GenericEnumSymbol: https://github.com/zolyfarkas/avro/blob/trunk/lang/java/compiler/src/main/velocity/org/apache/avro/compiler/specific/templates/java/classic/enum.vm#L29 Also changed GenericEnumSymbol from: {code} /** An enum symbol. */ public interface GenericEnumSymbol extends GenericContainer, Comparable { /** Return the symbol. */ String toString(); } {code} to: {code} /** An enum symbol. */ public interface GenericEnumSymbol extends GenericContainer, Comparable { /** Return the symbol. */ String toString(); } {code} I can prepare a PR if this approach is OK with everyone. > GenericDatumWriter broken with Enum > --- > > Key: AVRO-1810 > URL: https://issues.apache.org/jira/browse/AVRO-1810 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.0 >Reporter: Ryon Day >Priority: Blocker > Fix For: 1.9.0, 1.8.4 > > > {panel:title=Description|titleBGColor=#3FA|bgColor=#DDD} > Using the GenericDatumWriter with either Generic OR SpecificRecord will break > if an Enum is present. > {panel} > {panel:title=Steps To Reproduce|titleBGColor=#8DB|bgColor=#DDD} > I have been tracking Avro decoding oddities for a while. > The tests for this issue can be found > [here|https://github.com/ryonday/avroDecodingHelp/blob/master/src/test/java/com/ryonday/test/Avro180EnumFail.java] > {panel} > {panel:title=Notes|titleBGColor=#3AF|bgColor=#DDD} > Due to the debacle that is the Avro "UTF8" object, we have been avoiding it > by using the following scheme: > * Write incoming records to a byte array using the GenericDatumWriter > * Read back the byte array to our compiled Java domain objects using a > SpecificDatumWriter > This worked great with Avro 1.7.7, and this is a binary-incompatable breaking > change with 1.8.0. > This would appear to be caused by an addition in the > {{GenericDatumWriter:163-164}}: > {code} > if (!data.isEnum(datum)) > throw new AvroTypeException("Not an enum: "+datum); > {code} > {panel} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-1810) GenericDatumWriter broken with Enum
[ https://issues.apache.org/jira/browse/AVRO-1810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16166299#comment-16166299 ] Sean Busbey commented on AVRO-1810: --- Personally I'd love to see this get fixed. I'm happy to review, but almost certainly don't have the time to post a patch up. If anyone wants to try to work through things please either ping here or ping me off-list and I'll help get you up and running. > GenericDatumWriter broken with Enum > --- > > Key: AVRO-1810 > URL: https://issues.apache.org/jira/browse/AVRO-1810 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.0 >Reporter: Ryon Day >Priority: Blocker > > {panel:title=Description|titleBGColor=#3FA|bgColor=#DDD} > Using the GenericDatumWriter with either Generic OR SpecificRecord will break > if an Enum is present. > {panel} > {panel:title=Steps To Reproduce|titleBGColor=#8DB|bgColor=#DDD} > I have been tracking Avro decoding oddities for a while. > The tests for this issue can be found > [here|https://github.com/ryonday/avroDecodingHelp/blob/master/src/test/java/com/ryonday/test/Avro180EnumFail.java] > {panel} > {panel:title=Notes|titleBGColor=#3AF|bgColor=#DDD} > Due to the debacle that is the Avro "UTF8" object, we have been avoiding it > by using the following scheme: > * Write incoming records to a byte array using the GenericDatumWriter > * Read back the byte array to our compiled Java domain objects using a > SpecificDatumWriter > This worked great with Avro 1.7.7, and this is a binary-incompatable breaking > change with 1.8.0. > This would appear to be caused by an addition in the > {{GenericDatumWriter:163-164}}: > {code} > if (!data.isEnum(datum)) > throw new AvroTypeException("Not an enum: "+datum); > {code} > {panel} -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (AVRO-1810) GenericDatumWriter broken with Enum
[ https://issues.apache.org/jira/browse/AVRO-1810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15768250#comment-15768250 ] Caleb Cushing commented on AVRO-1810: - Grabbed Spring Platform Athens SR1 which specifies 1.8.1 of Avro, and hibernate search threw this error, downgrading to 1.7.7 works. > GenericDatumWriter broken with Enum > --- > > Key: AVRO-1810 > URL: https://issues.apache.org/jira/browse/AVRO-1810 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.0 >Reporter: Ryon Day >Priority: Blocker > > {panel:title=Description|titleBGColor=#3FA|bgColor=#DDD} > Using the GenericDatumWriter with either Generic OR SpecificRecord will break > if an Enum is present. > {panel} > {panel:title=Steps To Reproduce|titleBGColor=#8DB|bgColor=#DDD} > I have been tracking Avro decoding oddities for a while. > The tests for this issue can be found > [here|https://github.com/ryonday/avroDecodingHelp/blob/master/src/test/java/com/ryonday/test/Avro180EnumFail.java] > {panel} > {panel:title=Notes|titleBGColor=#3AF|bgColor=#DDD} > Due to the debacle that is the Avro "UTF8" object, we have been avoiding it > by using the following scheme: > * Write incoming records to a byte array using the GenericDatumWriter > * Read back the byte array to our compiled Java domain objects using a > SpecificDatumWriter > This worked great with Avro 1.7.7, and this is a binary-incompatable breaking > change with 1.8.0. > This would appear to be caused by an addition in the > {{GenericDatumWriter:163-164}}: > {code} > if (!data.isEnum(datum)) > throw new AvroTypeException("Not an enum: "+datum); > {code} > {panel} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AVRO-1810) GenericDatumWriter broken with Enum
[ https://issues.apache.org/jira/browse/AVRO-1810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15654926#comment-15654926 ] Steve Jerman commented on AVRO-1810: I'm seeing this as well basically SpecificRecords generated by maven plugin in 1.8.1 don't satisfy contract for GenericRecord wrt enums. Rolling back to 1.7.7 works. > GenericDatumWriter broken with Enum > --- > > Key: AVRO-1810 > URL: https://issues.apache.org/jira/browse/AVRO-1810 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.0 >Reporter: Ryon Day >Priority: Blocker > > {panel:title=Description|titleBGColor=#3FA|bgColor=#DDD} > Using the GenericDatumWriter with either Generic OR SpecificRecord will break > if an Enum is present. > {panel} > {panel:title=Steps To Reproduce|titleBGColor=#8DB|bgColor=#DDD} > I have been tracking Avro decoding oddities for a while. > The tests for this issue can be found > [here|https://github.com/ryonday/avroDecodingHelp/blob/master/src/test/java/com/ryonday/test/Avro180EnumFail.java] > {panel} > {panel:title=Notes|titleBGColor=#3AF|bgColor=#DDD} > Due to the debacle that is the Avro "UTF8" object, we have been avoiding it > by using the following scheme: > * Write incoming records to a byte array using the GenericDatumWriter > * Read back the byte array to our compiled Java domain objects using a > SpecificDatumWriter > This worked great with Avro 1.7.7, and this is a binary-incompatable breaking > change with 1.8.0. > This would appear to be caused by an addition in the > {{GenericDatumWriter:163-164}}: > {code} > if (!data.isEnum(datum)) > throw new AvroTypeException("Not an enum: "+datum); > {code} > {panel} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AVRO-1810) GenericDatumWriter broken with Enum
[ https://issues.apache.org/jira/browse/AVRO-1810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573691#comment-15573691 ] Zoltan Farkas commented on AVRO-1810: - I hit this issue when serializing a SpecificRecord (generated from idl) with a GenericDatumWriter. Everything works fine when serializing withe SpecificDatumWriter, but since all generated avro records implement GenericRecord I don't see why they should not be serializable with a GenericDatumWriter... I see 2 ways to fix this: 1) make GenericDatumWriter handle java enums. 2) make the generated enums (like Reuben suggested) implement GenericEnumSymbol. I used approach 1 to fix my fork. I am not sure the separation between GenericRecord and SpecificRecords reader/writers is ideal... For example I wrote some utilities to on the fly generate Generic Data like: https://github.com/zolyfarkas/spf4j/blob/master/spf4j-avro/src/main/java/org/spf4j/avro/GenericRecordBuilder.java how to use: https://github.com/zolyfarkas/spf4j/blob/master/spf4j-avro/src/test/java/org/spf4j/avro/GenericRecordBuilderTest.java This is still beta quality, but the results are slightly more efficient (10%) GenericRecord implementations. (see JMH benchmark: https://github.com/zolyfarkas/spf4j/blob/master/spf4j-benchmarks/src/test/java/org/spf4j/avro/GenericRecordBenchmark.java) > GenericDatumWriter broken with Enum > --- > > Key: AVRO-1810 > URL: https://issues.apache.org/jira/browse/AVRO-1810 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.0 >Reporter: Ryon Day >Priority: Blocker > > {panel:title=Description|titleBGColor=#3FA|bgColor=#DDD} > Using the GenericDatumWriter with either Generic OR SpecificRecord will break > if an Enum is present. > {panel} > {panel:title=Steps To Reproduce|titleBGColor=#8DB|bgColor=#DDD} > I have been tracking Avro decoding oddities for a while. > The tests for this issue can be found > [here|https://github.com/ryonday/avroDecodingHelp/blob/master/src/test/java/com/ryonday/test/Avro180EnumFail.java] > {panel} > {panel:title=Notes|titleBGColor=#3AF|bgColor=#DDD} > Due to the debacle that is the Avro "UTF8" object, we have been avoiding it > by using the following scheme: > * Write incoming records to a byte array using the GenericDatumWriter > * Read back the byte array to our compiled Java domain objects using a > SpecificDatumWriter > This worked great with Avro 1.7.7, and this is a binary-incompatable breaking > change with 1.8.0. > This would appear to be caused by an addition in the > {{GenericDatumWriter:163-164}}: > {code} > if (!data.isEnum(datum)) > throw new AvroTypeException("Not an enum: "+datum); > {code} > {panel} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AVRO-1810) GenericDatumWriter broken with Enum
[ https://issues.apache.org/jira/browse/AVRO-1810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504036#comment-15504036 ] Sean Busbey commented on AVRO-1810: --- That does seem broken. Could you share a code snippet that results in the stacktrace? > GenericDatumWriter broken with Enum > --- > > Key: AVRO-1810 > URL: https://issues.apache.org/jira/browse/AVRO-1810 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.0 >Reporter: Ryon Day >Priority: Blocker > > {panel:title=Description|titleBGColor=#3FA|bgColor=#DDD} > Using the GenericDatumWriter with either Generic OR SpecificRecord will break > if an Enum is present. > {panel} > {panel:title=Steps To Reproduce|titleBGColor=#8DB|bgColor=#DDD} > I have been tracking Avro decoding oddities for a while. > The tests for this issue can be found > [here|https://github.com/ryonday/avroDecodingHelp/blob/master/src/test/java/com/ryonday/test/Avro180EnumFail.java] > {panel} > {panel:title=Notes|titleBGColor=#3AF|bgColor=#DDD} > Due to the debacle that is the Avro "UTF8" object, we have been avoiding it > by using the following scheme: > * Write incoming records to a byte array using the GenericDatumWriter > * Read back the byte array to our compiled Java domain objects using a > SpecificDatumWriter > This worked great with Avro 1.7.7, and this is a binary-incompatable breaking > change with 1.8.0. > This would appear to be caused by an addition in the > {{GenericDatumWriter:163-164}}: > {code} > if (!data.isEnum(datum)) > throw new AvroTypeException("Not an enum: "+datum); > {code} > {panel} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AVRO-1810) GenericDatumWriter broken with Enum
[ https://issues.apache.org/jira/browse/AVRO-1810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504015#comment-15504015 ] Reuben Kuhnert commented on AVRO-1810: -- Sup Sean, getting burned by this: {code} org.apache.avro.AvroTypeException: Not an enum: UNIT at org.apache.avro.generic.GenericDatumWriter.writeEnum(GenericDatumWriter.java:164) at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:106) at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73) at org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:153) at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143) at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105) at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73) at org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:153) at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:143) at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:105) at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73) at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:60) at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:302) {code} Looks like enums have to subclass [GenericEnumSymbol|https://github.com/apache/avro/blob/master/lang/java/avro/src/main/java/org/apache/avro/generic/GenericData.java#L800]. Fair enough. But, it looks like the class auto-generation tools that we're using are not putting that into the schema. We're using: {code} org.apache.avro avro-maven-plugin ${avro.version} generate-sources schema protocol idl-protocol PRIVATE {code} But it generates: {code} @org.apache.avro.specific.AvroGenerated public enum EntityType { // LOOK MA, NO INTERFACE LISTING, UNIT ; public static final org.apache.avro.Schema SCHEMA$ = new org.apache.avro.Schema.Parser().parse("{\"type\":\"enum\",\"name\":\"EntityType\",\"namespace\":\"com.homeaway.commons.logging.events.lm\",\"symbols\":[\"LISTING\",\"UNIT\"]}"); public static org.apache.avro.Schema getClassSchema() { return SCHEMA$; } } {code} For now, rolling back to {{1.7.7}}, but that does seem brokenish. > GenericDatumWriter broken with Enum > --- > > Key: AVRO-1810 > URL: https://issues.apache.org/jira/browse/AVRO-1810 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.0 >Reporter: Ryon Day >Priority: Blocker > > {panel:title=Description|titleBGColor=#3FA|bgColor=#DDD} > Using the GenericDatumWriter with either Generic OR SpecificRecord will break > if an Enum is present. > {panel} > {panel:title=Steps To Reproduce|titleBGColor=#8DB|bgColor=#DDD} > I have been tracking Avro decoding oddities for a while. > The tests for this issue can be found > [here|https://github.com/ryonday/avroDecodingHelp/blob/master/src/test/java/com/ryonday/test/Avro180EnumFail.java] > {panel} > {panel:title=Notes|titleBGColor=#3AF|bgColor=#DDD} > Due to the debacle that is the Avro "UTF8" object, we have been avoiding it > by using the following scheme: > * Write incoming records to a byte array using the GenericDatumWriter > * Read back the byte array to our compiled Java domain objects using a > SpecificDatumWriter > This worked great with Avro 1.7.7, and this is a binary-incompatable breaking > change with 1.8.0. > This would appear to be caused by an addition in the > {{GenericDatumWriter:163-164}}: > {code} > if (!data.isEnum(datum)) > throw new AvroTypeException("Not an enum: "+datum); > {code} > {panel} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AVRO-1810) GenericDatumWriter broken with Enum
[ https://issues.apache.org/jira/browse/AVRO-1810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15190164#comment-15190164 ] Sean Busbey commented on AVRO-1810: --- This looks like an impedance mismatch between the Generic and Specific APIs. The Specific API generates bare java enums for avro enum types, while the Generic API now requires GenericEnumSymbol. (technically the switch to GenericEnumSymbol happened in 1.4.0, but enforcement has been inconsistent.) Short term, the {{genericDatumWriter_failsForGenericRecord_populatedWithRawEnum}} test can be fixed similar to the string one by wrapping things in GenericData.EnumSymbol. It looks like the root cause for your helper utility is that SpecificRecordBase claims to implement GenericRecord, but then returns the raw java enums for the two avro enum fields rather than the correct o.a.avro.generic type. > GenericDatumWriter broken with Enum > --- > > Key: AVRO-1810 > URL: https://issues.apache.org/jira/browse/AVRO-1810 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.0 >Reporter: Ryon Day >Priority: Blocker > > {panel:title=Description|titleBGColor=#3FA|bgColor=#DDD} > Using the GenericDatumWriter with either Generic OR SpecificRecord will break > if an Enum is present. > {panel} > {panel:title=Steps To Reproduce|titleBGColor=#8DB|bgColor=#DDD} > I have been tracking Avro decoding oddities for a while. > The tests for this issue can be found > [here|https://github.com/ryonday/avroDecodingHelp/blob/master/src/test/java/com/ryonday/test/Avro180EnumFail.java] > {panel} > {panel:title=Notes|titleBGColor=#3AF|bgColor=#DDD} > Due to the debacle that is the Avro "UTF8" object, we have been avoiding it > by using the following scheme: > * Write incoming records to a byte array using the GenericDatumWriter > * Read back the byte array to our compiled Java domain objects using a > SpecificDatumWriter > This worked great with Avro 1.7.7, and this is a binary-incompatable breaking > change with 1.8.0. > This would appear to be caused by an addition in the > {{GenericDatumWriter:163-164}}: > {code} > if (!data.isEnum(datum)) > throw new AvroTypeException("Not an enum: "+datum); > {code} > {panel} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (AVRO-1810) GenericDatumWriter broken with Enum
[ https://issues.apache.org/jira/browse/AVRO-1810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15189944#comment-15189944 ] Sean Busbey commented on AVRO-1810: --- {quote} Due to the debacle that is the Avro "UTF8" object, we have been avoiding it by using the following scheme: {quote} I am curious to hear what the debacle is, perhaps you could send a description to user@avro or dev@avro? {quote} This worked great with Avro 1.7.7, and this is a binary-incompatable breaking change with 1.8.0. This would appear to be caused by an addition in the GenericDatumWriter:163-164: {code} if (!data.isEnum(datum)) throw new AvroTypeException("Not an enum: "+datum); {code} {quote} This was the breaking change documented in AVRO-997. The example in {{genericDatumWriter_andSpecificDatumWriter_failForGenericRecord_populatedWithTextualEnum}} is using Strings for enum types. Those should be GenericEnumSymbol, as documented in the [javadocs for type mapping|http://avro.apache.org/docs/1.8.0/api/java/org/apache/avro/generic/package-summary.html] and the release notes for AVRO-997. The other two examples are based on what look like generated specific classes from [the schemas in your test project|https://github.com/ryonday/avroDecodingHelp/tree/master/src/main/avro], is that right? If I build the test project at the top level is it straight forward to examine hte generated classes? > GenericDatumWriter broken with Enum > --- > > Key: AVRO-1810 > URL: https://issues.apache.org/jira/browse/AVRO-1810 > Project: Avro > Issue Type: Bug > Components: java >Affects Versions: 1.8.0 >Reporter: Ryon Day >Priority: Blocker > > {panel:title=Description|titleBGColor=#3FA|bgColor=#DDD} > Using the GenericDatumWriter with either Generic OR SpecificRecord will break > if an Enum is present. > {panel} > {panel:title=Steps To Reproduce|titleBGColor=#8DB|bgColor=#DDD} > I have been tracking Avro decoding oddities for a while. > The tests for this issue can be found > [here|https://github.com/ryonday/avroDecodingHelp/blob/master/src/test/java/com/ryonday/test/Avro180EnumFail.java] > {panel} > {panel:title=Notes|titleBGColor=#3AF|bgColor=#DDD} > Due to the debacle that is the Avro "UTF8" object, we have been avoiding it > by using the following scheme: > * Write incoming records to a byte array using the GenericDatumWriter > * Read back the byte array to our compiled Java domain objects using a > SpecificDatumWriter > This worked great with Avro 1.7.7, and this is a binary-incompatable breaking > change with 1.8.0. > This would appear to be caused by an addition in the > {{GenericDatumWriter:163-164}}: > {code} > if (!data.isEnum(datum)) > throw new AvroTypeException("Not an enum: "+datum); > {code} > {panel} -- This message was sent by Atlassian JIRA (v6.3.4#6332)