[jira] [Commented] (AVRO-2130) [RUBY] QUESTION: Is Avro::SchemaValidator class accessible in version 1.8.2?
[ https://issues.apache.org/jira/browse/AVRO-2130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16327647#comment-16327647 ] Tim Perkins commented on AVRO-2130: --- [~shabab] you might find this gem of interest: [https://github.com/salsify/avro-patches] It collects several of the Ruby changes that have not yet made it into an official release. > [RUBY] QUESTION: Is Avro::SchemaValidator class accessible in version 1.8.2? > > > Key: AVRO-2130 > URL: https://issues.apache.org/jira/browse/AVRO-2130 > Project: Avro > Issue Type: Task > Components: ruby >Affects Versions: 1.8.2 >Reporter: Shabab Ayub >Priority: Major > Fix For: 1.8.2 > > Attachments: image-2018-01-15-22-25-27-964.png > > Original Estimate: 24h > Remaining Estimate: 24h > > I can't seem to access `Avro::SchemaValidator.validate!()` in the current > version 1.8.2. I was wondering if this class is being made public in a future > version or am I missing something in how I use it? It seems that it is not > one of the accessible classes from the Avro module, if this is the case, I > was wondering what the reason for this would be? > I figured doing > ``` > require 'avro' > Avro::SchemaValidator.validate!(schema, data) > ``` > Would be enough. > I was interested in the ValidtionError class to return specific error > messages. > file: > https://github.com/apache/avro/blob/master/lang/ruby/lib/avro/schema_validator.rb > > Thank you! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-1340) use default to allow old readers to specify default enum value when encountering new enum symbols
[ https://issues.apache.org/jira/browse/AVRO-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16327640#comment-16327640 ] Doug Cutting commented on AVRO-1340: [~zolyfarkas], couldn't one specify (type1, type2, unknown) with default of unknown regardless of whether the default is in the field or in the enum definition itself? Also, enums already support the default field value, in the case that the writer didn't have the enum field at all. The proposal is simply to extend it to the case where the writer's symbol didn't match any in the reader's enum. Do these cases need to be distinguished? The advantage I see to adding the default to the enum's schema definition is that it better localizes things: the default lives with the enum rather than with every use of it. Perhaps that's your point. In this case, I prefer it be called "default" rather than "fallback" since it's conceptually similar to a field's default (both are used when the writer's missing something from the reader's schema). Lastly, if we define it in the enum, then we need to decide how it interacts with the field's default value. My intuition is that the field's default should override the enum's default. > use default to allow old readers to specify default enum value when > encountering new enum symbols > - > > Key: AVRO-1340 > URL: https://issues.apache.org/jira/browse/AVRO-1340 > Project: Avro > Issue Type: Improvement > Components: spec > Environment: N/A >Reporter: Jim Donofrio >Priority: Minor > > The schema resolution page says: > > if both are enums: > > if the writer's symbol is not present in the reader's enum, then an > error is signalled. > This makes it difficult to use enum's because you can never add a enum value > and keep old reader's compatible. Why not use the default option to refer to > one of enum values so that when a old reader encounters a enum ordinal it > does not recognize, it can default to the optional schema provided one. If > the old schema does not provide a default then the older reader can continue > to fail as it does today. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-1340) use default to allow old readers to specify default enum value when encountering new enum symbols
[
https://issues.apache.org/jira/browse/AVRO-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16327575#comment-16327575
]
Zoltan Farkas commented on AVRO-1340:
-
[~cutting] I believe the best place to define a fallback symbol is at type
level (enum) ("fallbackSymbol": "SOME SYMBOL")
field default value in my opinion is conceptually something different that a
"enum unknown value"... and I believe it is error prone to assume field default
is the right value when a unknown symbol is received...
here is some example to try to explain what I mean:
Lets have for example a record: Transaction with a enum field transactionType
(type1, type2) with default value type1...
if somebody extends this enum later with type3 it will be problematic since
older versions of Transaction will treat this as type1 which will probably be
really bad...
To not have the above we can do:
Transaction with a enum field transactionType (type1, type2, unknown) with
default value unknown...
This will make extending the enum safer as long as developers use the right
default value every time they use this enum which tells me that the right
place this needs to be defined is at the type level...
> use default to allow old readers to specify default enum value when
> encountering new enum symbols
> -
>
> Key: AVRO-1340
> URL: https://issues.apache.org/jira/browse/AVRO-1340
> Project: Avro
> Issue Type: Improvement
> Components: spec
> Environment: N/A
>Reporter: Jim Donofrio
>Priority: Minor
>
> The schema resolution page says:
> > if both are enums:
> > if the writer's symbol is not present in the reader's enum, then an
> error is signalled.
> This makes it difficult to use enum's because you can never add a enum value
> and keep old reader's compatible. Why not use the default option to refer to
> one of enum values so that when a old reader encounters a enum ordinal it
> does not recognize, it can default to the optional schema provided one. If
> the old schema does not provide a default then the older reader can continue
> to fail as it does today.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-2130) [RUBY] QUESTION: Is Avro::SchemaValidator class accessible in version 1.8.2?
[ https://issues.apache.org/jira/browse/AVRO-2130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shabab Ayub resolved AVRO-2130. --- Resolution: Fixed Release Note: The SchemaValidator class is not in the 1.8.2 release. > [RUBY] QUESTION: Is Avro::SchemaValidator class accessible in version 1.8.2? > > > Key: AVRO-2130 > URL: https://issues.apache.org/jira/browse/AVRO-2130 > Project: Avro > Issue Type: Task > Components: ruby >Affects Versions: 1.8.2 >Reporter: Shabab Ayub >Priority: Major > Fix For: 1.8.2 > > Attachments: image-2018-01-15-22-25-27-964.png > > Original Estimate: 24h > Remaining Estimate: 24h > > I can't seem to access `Avro::SchemaValidator.validate!()` in the current > version 1.8.2. I was wondering if this class is being made public in a future > version or am I missing something in how I use it? It seems that it is not > one of the accessible classes from the Avro module, if this is the case, I > was wondering what the reason for this would be? > I figured doing > ``` > require 'avro' > Avro::SchemaValidator.validate!(schema, data) > ``` > Would be enough. > I was interested in the ValidtionError class to return specific error > messages. > file: > https://github.com/apache/avro/blob/master/lang/ruby/lib/avro/schema_validator.rb > > Thank you! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-1340) use default to allow old readers to specify default enum value when encountering new enum symbols
[ https://issues.apache.org/jira/browse/AVRO-1340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16327480#comment-16327480 ] Doug Cutting commented on AVRO-1340: So it seems there's an emerging consensus to add a default value for enums but not aliases. Conveniently, there's also a patch by [~kse...@gmail.com] that implements this. Rather than extend enum syntax, the patch uses the default from the field whose value is the enum, extending Avro's existing default mechanism to work for enums. Is that implementation acceptable? If not, what why, and what would be better? > use default to allow old readers to specify default enum value when > encountering new enum symbols > - > > Key: AVRO-1340 > URL: https://issues.apache.org/jira/browse/AVRO-1340 > Project: Avro > Issue Type: Improvement > Components: spec > Environment: N/A >Reporter: Jim Donofrio >Priority: Minor > > The schema resolution page says: > > if both are enums: > > if the writer's symbol is not present in the reader's enum, then an > error is signalled. > This makes it difficult to use enum's because you can never add a enum value > and keep old reader's compatible. Why not use the default option to refer to > one of enum values so that when a old reader encounters a enum ordinal it > does not recognize, it can default to the optional schema provided one. If > the old schema does not provide a default then the older reader can continue > to fail as it does today. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2122) Cannot validate schemas with recursive definitions
[
https://issues.apache.org/jira/browse/AVRO-2122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16327149#comment-16327149
]
Bart commented on AVRO-2122:
[~nkollar] I've processed your review comments. I did not update CHANGES.txt
(on purpose, I saw you add entries yourself for pull requests the you merged).
> Cannot validate schemas with recursive definitions
> --
>
> Key: AVRO-2122
> URL: https://issues.apache.org/jira/browse/AVRO-2122
> Project: Avro
> Issue Type: Bug
> Components: java
>Affects Versions: 1.8.2
>Reporter: Bart
>Priority: Major
>
> Validating a schema with a recursive definition will lead to a stack
> overflow. When using the following schema definition:
> {noformat}
> @namespace("avro")
> protocol Unused {
> record Node {
> union { null, Node } value = null;
> }
> }
> {noformat}
> {code:java}
> final SchemaValidator backwardValidator = new SchemaValidatorBuilder()
> .canReadStrategy().validateLatest();
> backwardValidator.validate(Node.SCHEMA$, Arrays.asList(Node.SCHEMA$));
> {code}
> It results in a stack trace:
> {noformat}
> java.lang.StackOverflowError
> at org.apache.avro.io.parsing.Symbol.hasErrors(Symbol.java:406)
> at org.apache.avro.io.parsing.Symbol.hasErrors(Symbol.java:392)
> at org.apache.avro.io.parsing.Symbol.hasErrors(Symbol.java:383)
> at org.apache.avro.io.parsing.Symbol.hasErrors(Symbol.java:406)
> at org.apache.avro.io.parsing.Symbol.hasErrors(Symbol.java:392)
> at org.apache.avro.io.parsing.Symbol.hasErrors(Symbol.java:406)
> at org.apache.avro.io.parsing.Symbol.hasErrors(Symbol.java:374)
> at org.apache.avro.io.parsing.Symbol.hasErrors(Symbol.java:406)
> ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-2125) AvroRuntimeException: Not a union
[
https://issues.apache.org/jira/browse/AVRO-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16327094#comment-16327094
]
Nandor Kollar commented on AVRO-2125:
-
Closing this since it doesn't seems to be an issue, Andreas feel free to reopen
if my suggestion didn't solve the problem.
> AvroRuntimeException: Not a union
> -
>
> Key: AVRO-2125
> URL: https://issues.apache.org/jira/browse/AVRO-2125
> Project: Avro
> Issue Type: Bug
>Reporter: Andreas Christodoulou
>Priority: Major
> Attachments: all-types-schema.avsc, data.json
>
>
> I have an Avro schema which I am using as part of our Apache Beam pipeline
> tests, and using the avro-tools-1.8.2 tool I have generated random data from
> the Avro schema. In the tests I am using this data with the schema, but I get
> the following Java exception:
> {code:java}
> org.apache.avro.AvroRuntimeException: Not a union:
> {"type":"record","name":"sampleArrayRecord","namespace":"gamesys.sample","fields":[{"name":"sampleArrayRecordString","type":"string"}]}
> {code}
> I have attached the schema and the data I've used, and the relevant
> stacktrace is given below:
> {code:java}
> Caused by: com.fasterxml.jackson.databind.JsonMappingException: Not a union:
> {"type":"record","name":"sampleArrayRecord","namespace":"gamesys.sample","fields":[{"name":"sampleArrayRecordString","type":"string"}]}
> (through reference chain:
> com.google.api.services.bigquery.model.TableRow["sampleArray"]->org.apache.avro.generic.GenericData$Array[0]->org.apache.avro.generic.GenericData$Record["schema"]->org.apache.avro.Schema$RecordSchema["types"])
> at
> com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:388)
> at
> com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:348)
> at
> com.fasterxml.jackson.databind.ser.std.StdSerializer.wrapAndThrow(StdSerializer.java:343)
> at
> com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:697)
> at
> com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
> at
> com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:704)
> at
> com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:689)
> at
> com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
> at
> com.fasterxml.jackson.databind.ser.std.CollectionSerializer.serializeContents(CollectionSerializer.java:149)
> at
> com.fasterxml.jackson.databind.ser.std.CollectionSerializer.serialize(CollectionSerializer.java:112)
> at
> com.fasterxml.jackson.databind.ser.std.CollectionSerializer.serialize(CollectionSerializer.java:25)
> at
> com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:633)
> at
> com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:536)
> at
> com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:30)
> at
> com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:292)
> at
> com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3697)
> at
> com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:3073)
> at
> org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder.encode(TableRowJsonCoder.java:49)
> at
> org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder.encode(TableRowJsonCoder.java:34)
> at
> org.apache.beam.sdk.util.CoderUtils.encodeToSafeStream(CoderUtils.java:93)
> ... 22 more
> Caused by: org.apache.avro.AvroRuntimeException: Not a union:
> {"type":"record","name":"sampleArrayRecord","namespace":"gamesys.sample","fields":[{"name":"sampleArrayRecordString","type":"string"}]}
> at org.apache.avro.Schema.getTypes(Schema.java:299)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:664)
> at
> com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:689)
> ... 38 more
> {code}
> I am using the org.apache.avro:avro:1.8.1 library in the project.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (AVRO-2125) AvroRuntimeException: Not a union
[
https://issues.apache.org/jira/browse/AVRO-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandor Kollar resolved AVRO-2125.
-
Resolution: Not A Bug
> AvroRuntimeException: Not a union
> -
>
> Key: AVRO-2125
> URL: https://issues.apache.org/jira/browse/AVRO-2125
> Project: Avro
> Issue Type: Bug
>Reporter: Andreas Christodoulou
>Priority: Major
> Attachments: all-types-schema.avsc, data.json
>
>
> I have an Avro schema which I am using as part of our Apache Beam pipeline
> tests, and using the avro-tools-1.8.2 tool I have generated random data from
> the Avro schema. In the tests I am using this data with the schema, but I get
> the following Java exception:
> {code:java}
> org.apache.avro.AvroRuntimeException: Not a union:
> {"type":"record","name":"sampleArrayRecord","namespace":"gamesys.sample","fields":[{"name":"sampleArrayRecordString","type":"string"}]}
> {code}
> I have attached the schema and the data I've used, and the relevant
> stacktrace is given below:
> {code:java}
> Caused by: com.fasterxml.jackson.databind.JsonMappingException: Not a union:
> {"type":"record","name":"sampleArrayRecord","namespace":"gamesys.sample","fields":[{"name":"sampleArrayRecordString","type":"string"}]}
> (through reference chain:
> com.google.api.services.bigquery.model.TableRow["sampleArray"]->org.apache.avro.generic.GenericData$Array[0]->org.apache.avro.generic.GenericData$Record["schema"]->org.apache.avro.Schema$RecordSchema["types"])
> at
> com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:388)
> at
> com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:348)
> at
> com.fasterxml.jackson.databind.ser.std.StdSerializer.wrapAndThrow(StdSerializer.java:343)
> at
> com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:697)
> at
> com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
> at
> com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:704)
> at
> com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:689)
> at
> com.fasterxml.jackson.databind.ser.BeanSerializer.serialize(BeanSerializer.java:155)
> at
> com.fasterxml.jackson.databind.ser.std.CollectionSerializer.serializeContents(CollectionSerializer.java:149)
> at
> com.fasterxml.jackson.databind.ser.std.CollectionSerializer.serialize(CollectionSerializer.java:112)
> at
> com.fasterxml.jackson.databind.ser.std.CollectionSerializer.serialize(CollectionSerializer.java:25)
> at
> com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:633)
> at
> com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:536)
> at
> com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:30)
> at
> com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:292)
> at
> com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3697)
> at
> com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:3073)
> at
> org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder.encode(TableRowJsonCoder.java:49)
> at
> org.apache.beam.sdk.io.gcp.bigquery.TableRowJsonCoder.encode(TableRowJsonCoder.java:34)
> at
> org.apache.beam.sdk.util.CoderUtils.encodeToSafeStream(CoderUtils.java:93)
> ... 22 more
> Caused by: org.apache.avro.AvroRuntimeException: Not a union:
> {"type":"record","name":"sampleArrayRecord","namespace":"gamesys.sample","fields":[{"name":"sampleArrayRecordString","type":"string"}]}
> at org.apache.avro.Schema.getTypes(Schema.java:299)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> com.fasterxml.jackson.databind.ser.BeanPropertyWriter.serializeAsField(BeanPropertyWriter.java:664)
> at
> com.fasterxml.jackson.databind.ser.std.BeanSerializerBase.serializeFields(BeanSerializerBase.java:689)
> ... 38 more
> {code}
> I am using the org.apache.avro:avro:1.8.1 library in the project.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
