[jira] [Commented] (AVRO-2199) Validate that field defaults have the correct type
[ https://issues.apache.org/jira/browse/AVRO-2199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806214#comment-16806214 ] Hudson commented on AVRO-2199: -- FAILURE: Integrated in Jenkins build AvroJava #632 (See [https://builds.apache.org/job/AvroJava/632/]) AVRO-2199: Validate that field defaults have the correct type (#497) (fokko: [https://github.com/apache/avro/commit/392c761952ec7f679b29ae28129403ea8cf307ea]) * (edit) lang/ruby/test/test_schema.rb * (edit) lang/ruby/test/test_datafile.rb * (edit) lang/ruby/.gitignore * (edit) lang/ruby/lib/avro.rb * (edit) lang/ruby/lib/avro/schema.rb > Validate that field defaults have the correct type > --- > > Key: AVRO-2199 > URL: https://issues.apache.org/jira/browse/AVRO-2199 > Project: Apache Avro > Issue Type: Improvement > Components: ruby >Affects Versions: 1.8.2 >Reporter: Daniel Orner >Assignee: Tim Perkins >Priority: Minor > Fix For: 1.9.0 > > Attachments: AVRO-2199.patch > > > Currently, on the master branch, when a schema is parsed, it is possible to > define a field with a type and a default of a totally different type. E.g. if > the field has type "string", the default can be set to "null". > I'd like to open a PR which will fix this by running the default through the > SchemaValidator whenever a new Field is created. See > [https://github.com/salsify/avro-patches/pull/16] > cc: [~tjwp] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2199) Validate that field defaults have the correct type
[ https://issues.apache.org/jira/browse/AVRO-2199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806193#comment-16806193 ] ASF subversion and git services commented on AVRO-2199: --- Commit 392c761952ec7f679b29ae28129403ea8cf307ea in avro's branch refs/heads/master from Tim Perkins [ https://gitbox.apache.org/repos/asf?p=avro.git;h=392c761 ] AVRO-2199: Validate that field defaults have the correct type (#497) * AVRO-2199: Validate that field defaults have the correct type * AVRO-2199: Code review comments * AVRO-2199: Make it configurable whether field default validation is enforced for Ruby > Validate that field defaults have the correct type > --- > > Key: AVRO-2199 > URL: https://issues.apache.org/jira/browse/AVRO-2199 > Project: Apache Avro > Issue Type: Improvement > Components: ruby >Affects Versions: 1.8.2 >Reporter: Daniel Orner >Priority: Minor > Fix For: 1.9.0 > > Attachments: AVRO-2199.patch > > > Currently, on the master branch, when a schema is parsed, it is possible to > define a field with a type and a default of a totally different type. E.g. if > the field has type "string", the default can be set to "null". > I'd like to open a PR which will fix this by running the default through the > SchemaValidator whenever a new Field is created. See > [https://github.com/salsify/avro-patches/pull/16] > cc: [~tjwp] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2199) Validate that field defaults have the correct type
[ https://issues.apache.org/jira/browse/AVRO-2199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806192#comment-16806192 ] ASF subversion and git services commented on AVRO-2199: --- Commit 392c761952ec7f679b29ae28129403ea8cf307ea in avro's branch refs/heads/master from Tim Perkins [ https://gitbox.apache.org/repos/asf?p=avro.git;h=392c761 ] AVRO-2199: Validate that field defaults have the correct type (#497) * AVRO-2199: Validate that field defaults have the correct type * AVRO-2199: Code review comments * AVRO-2199: Make it configurable whether field default validation is enforced for Ruby > Validate that field defaults have the correct type > --- > > Key: AVRO-2199 > URL: https://issues.apache.org/jira/browse/AVRO-2199 > Project: Apache Avro > Issue Type: Improvement > Components: ruby >Affects Versions: 1.8.2 >Reporter: Daniel Orner >Priority: Minor > Fix For: 1.9.0 > > Attachments: AVRO-2199.patch > > > Currently, on the master branch, when a schema is parsed, it is possible to > define a field with a type and a default of a totally different type. E.g. if > the field has type "string", the default can be set to "null". > I'd like to open a PR which will fix this by running the default through the > SchemaValidator whenever a new Field is created. See > [https://github.com/salsify/avro-patches/pull/16] > cc: [~tjwp] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2199) Validate that field defaults have the correct type
[ https://issues.apache.org/jira/browse/AVRO-2199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16806195#comment-16806195 ] ASF subversion and git services commented on AVRO-2199: --- Commit 392c761952ec7f679b29ae28129403ea8cf307ea in avro's branch refs/heads/master from Tim Perkins [ https://gitbox.apache.org/repos/asf?p=avro.git;h=392c761 ] AVRO-2199: Validate that field defaults have the correct type (#497) * AVRO-2199: Validate that field defaults have the correct type * AVRO-2199: Code review comments * AVRO-2199: Make it configurable whether field default validation is enforced for Ruby > Validate that field defaults have the correct type > --- > > Key: AVRO-2199 > URL: https://issues.apache.org/jira/browse/AVRO-2199 > Project: Apache Avro > Issue Type: Improvement > Components: ruby >Affects Versions: 1.8.2 >Reporter: Daniel Orner >Priority: Minor > Fix For: 1.9.0 > > Attachments: AVRO-2199.patch > > > Currently, on the master branch, when a schema is parsed, it is possible to > define a field with a type and a default of a totally different type. E.g. if > the field has type "string", the default can be set to "null". > I'd like to open a PR which will fix this by running the default through the > SchemaValidator whenever a new Field is created. See > [https://github.com/salsify/avro-patches/pull/16] > cc: [~tjwp] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2199) Validate that field defaults have the correct type
[ https://issues.apache.org/jira/browse/AVRO-2199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646752#comment-16646752 ] ASF GitHub Bot commented on AVRO-2199: -- dorner commented on issue #320: AVRO-2199: Validate that field defaults have the correct type URL: https://github.com/apache/avro/pull/320#issuecomment-429034100 @theturtle32 I definitely feel your pain. But the fact is that the current implementation is incorrect - the fact that you have bad data out there is a result of the incorrect implementation. In my case we had other Avro consumers not written in Ruby whose parsers were stricter that were crashing because we were sending bad data, so IMO it's more important to fix the behavior in Ruby which doesn't match the Avro specification. We should definitely make this a breaking change (1.9.0 instead of 1.8.3) so people can be made aware of this. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Validate that field defaults have the correct type > --- > > Key: AVRO-2199 > URL: https://issues.apache.org/jira/browse/AVRO-2199 > Project: Avro > Issue Type: Improvement > Components: ruby >Affects Versions: 1.8.2 >Reporter: Daniel Orner >Priority: Minor > > Currently, on the master branch, when a schema is parsed, it is possible to > define a field with a type and a default of a totally different type. E.g. if > the field has type "string", the default can be set to "null". > I'd like to open a PR which will fix this by running the default through the > SchemaValidator whenever a new Field is created. See > [https://github.com/salsify/avro-patches/pull/16] > cc: [~tjwp] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2199) Validate that field defaults have the correct type
[ https://issues.apache.org/jira/browse/AVRO-2199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16645606#comment-16645606 ] Daniel Orner commented on AVRO-2199: [~theturtle32] I definitely feel your pain. :( But the fact is that the current implementation is incorrect - the fact that you have bad data out there is a result of the incorrect implementation. In my case we had other Avro consumers not written in Ruby whose parsers were stricter that were crashing because we were sending bad data, so IMO it's more important to fix the behavior in Ruby which doesn't match the Avro specification. > Validate that field defaults have the correct type > --- > > Key: AVRO-2199 > URL: https://issues.apache.org/jira/browse/AVRO-2199 > Project: Avro > Issue Type: Improvement > Components: ruby >Affects Versions: 1.8.2 >Reporter: Daniel Orner >Priority: Minor > > Currently, on the master branch, when a schema is parsed, it is possible to > define a field with a type and a default of a totally different type. E.g. if > the field has type "string", the default can be set to "null". > I'd like to open a PR which will fix this by running the default through the > SchemaValidator whenever a new Field is created. See > [https://github.com/salsify/avro-patches/pull/16] > cc: [~tjwp] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2199) Validate that field defaults have the correct type
[ https://issues.apache.org/jira/browse/AVRO-2199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16645527#comment-16645527 ] ASF GitHub Bot commented on AVRO-2199: -- theturtle32 commented on issue #320: AVRO-2199: Validate that field defaults have the correct type URL: https://github.com/apache/avro/pull/320#issuecomment-428729499 This has an unfortunate side effect: it breaks applications from reading existing data that's been encoded with older versions of schemas that had incompatible default values specified. The library won't even finish parsing the old schemas, so you can't decode existing data. The Avro specification says that when you have a union type, any default value must be of the first type specified in the union. We have millions of records of data encoded using a schema where the default value was of the second or third type in the union instead of the first type. This always worked just fine before but with this more stringent validation, our application suddenly fails to decode our existing data because it raises an error while parsing the writer's schema. I'm going to try to write a monkey-patch to relax this validation temporarily, long enough to read all existing data, migrate it to a newer schema, and re-save it. But that's a lot of time and effort to compensate for a sudden change in schema validation. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Validate that field defaults have the correct type > --- > > Key: AVRO-2199 > URL: https://issues.apache.org/jira/browse/AVRO-2199 > Project: Avro > Issue Type: Improvement > Components: ruby >Affects Versions: 1.8.2 >Reporter: Daniel Orner >Priority: Minor > > Currently, on the master branch, when a schema is parsed, it is possible to > define a field with a type and a default of a totally different type. E.g. if > the field has type "string", the default can be set to "null". > I'd like to open a PR which will fix this by running the default through the > SchemaValidator whenever a new Field is created. See > [https://github.com/salsify/avro-patches/pull/16] > cc: [~tjwp] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2199) Validate that field defaults have the correct type
[ https://issues.apache.org/jira/browse/AVRO-2199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16645523#comment-16645523 ] Brian McKelvey commented on AVRO-2199: -- This has an unfortunate side effect: it breaks applications from reading existing data that's been encoded with older versions of schemas that had incorrect default values specified. The library won't even finish parsing the old schemas, so you can't decode existing data. The Avro specification says that when you have a union type, any default value must be of the first type specified in the union. We have millions of records of data encoded using a schema where the default value was of the second or third type in the union. This always worked before, but with this more stringent validation, our application suddenly fails to decode our existing data because it raises an error while parsing the old historical schema. I'm going to try to write a monkey-patch to relax this validation temporarily, long enough to read all existing data, migrate it to a newer schema, and re-save it. But that's a lot of time and effort to compensate for a sudden change in schema validation. > Validate that field defaults have the correct type > --- > > Key: AVRO-2199 > URL: https://issues.apache.org/jira/browse/AVRO-2199 > Project: Avro > Issue Type: Improvement > Components: ruby >Affects Versions: 1.8.2 >Reporter: Daniel Orner >Priority: Minor > > Currently, on the master branch, when a schema is parsed, it is possible to > define a field with a type and a default of a totally different type. E.g. if > the field has type "string", the default can be set to "null". > I'd like to open a PR which will fix this by running the default through the > SchemaValidator whenever a new Field is created. See > [https://github.com/salsify/avro-patches/pull/16] > cc: [~tjwp] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2199) Validate that field defaults have the correct type
[
https://issues.apache.org/jira/browse/AVRO-2199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16547900#comment-16547900
]
ASF GitHub Bot commented on AVRO-2199:
--
dorner commented on issue #320: AVRO-2199: Validate that field defaults have
the correct type
URL: https://github.com/apache/avro/pull/320#issuecomment-405951524
@tjwp added another commit. Funny thing - once this was in, a different test
failed precisely because it was using the wrong type ("false" instead of
`false`)! :)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Validate that field defaults have the correct type
> ---
>
> Key: AVRO-2199
> URL: https://issues.apache.org/jira/browse/AVRO-2199
> Project: Avro
> Issue Type: Improvement
> Components: ruby
>Affects Versions: 1.8.2
>Reporter: Daniel Orner
>Priority: Minor
>
> Currently, on the master branch, when a schema is parsed, it is possible to
> define a field with a type and a default of a totally different type. E.g. if
> the field has type "string", the default can be set to "null".
> I'd like to open a PR which will fix this by running the default through the
> SchemaValidator whenever a new Field is created. See
> [https://github.com/salsify/avro-patches/pull/16]
> cc: [~tjwp]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (AVRO-2199) Validate that field defaults have the correct type
[ https://issues.apache.org/jira/browse/AVRO-2199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16547253#comment-16547253 ] ASF GitHub Bot commented on AVRO-2199: -- tjwp commented on a change in pull request #320: AVRO-2199: Validate that field defaults have the correct type URL: https://github.com/apache/avro/pull/320#discussion_r203221105 ## File path: lang/ruby/test/test_default_validation.rb ## @@ -0,0 +1,127 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +require 'test_help' + +class TestDefaultValidation < Test::Unit::TestCase Review comment: I separate these tests out for the avro-patches gem, but it may make sense to move them into `test_schema.rb` here. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Validate that field defaults have the correct type > --- > > Key: AVRO-2199 > URL: https://issues.apache.org/jira/browse/AVRO-2199 > Project: Avro > Issue Type: Improvement > Components: ruby >Affects Versions: 1.8.2 >Reporter: Daniel Orner >Priority: Minor > > Currently, on the master branch, when a schema is parsed, it is possible to > define a field with a type and a default of a totally different type. E.g. if > the field has type "string", the default can be set to "null". > I'd like to open a PR which will fix this by running the default through the > SchemaValidator whenever a new Field is created. See > [https://github.com/salsify/avro-patches/pull/16] > cc: [~tjwp] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2199) Validate that field defaults have the correct type
[ https://issues.apache.org/jira/browse/AVRO-2199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16547252#comment-16547252 ] ASF GitHub Bot commented on AVRO-2199: -- tjwp commented on a change in pull request #320: AVRO-2199: Validate that field defaults have the correct type URL: https://github.com/apache/avro/pull/320#discussion_r203220999 ## File path: lang/ruby/lib/avro/schema.rb ## @@ -373,6 +374,18 @@ def to_avro(names=Set.new) avro['doc'] = doc if doc end end + + def validate_default! Review comment: Should this method be made private? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Validate that field defaults have the correct type > --- > > Key: AVRO-2199 > URL: https://issues.apache.org/jira/browse/AVRO-2199 > Project: Avro > Issue Type: Improvement > Components: ruby >Affects Versions: 1.8.2 >Reporter: Daniel Orner >Priority: Minor > > Currently, on the master branch, when a schema is parsed, it is possible to > define a field with a type and a default of a totally different type. E.g. if > the field has type "string", the default can be set to "null". > I'd like to open a PR which will fix this by running the default through the > SchemaValidator whenever a new Field is created. See > [https://github.com/salsify/avro-patches/pull/16] > cc: [~tjwp] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AVRO-2199) Validate that field defaults have the correct type
[ https://issues.apache.org/jira/browse/AVRO-2199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16546795#comment-16546795 ] ASF GitHub Bot commented on AVRO-2199: -- dorner opened a new pull request #320: AVRO-2199: Validate that field defaults have the correct type URL: https://github.com/apache/avro/pull/320 See https://issues.apache.org/jira/browse/AVRO-2199 Currently, on the master branch, when a schema is parsed, it is possible to define a field with a type and a default of a totally different type. E.g. if the field has type "string", the default can be set to "null". This PR fixes this by running the default through the SchemaValidator whenever a new Field is created. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Validate that field defaults have the correct type > --- > > Key: AVRO-2199 > URL: https://issues.apache.org/jira/browse/AVRO-2199 > Project: Avro > Issue Type: Improvement > Components: ruby >Affects Versions: 1.8.2 >Reporter: Daniel Orner >Priority: Minor > > Currently, on the master branch, when a schema is parsed, it is possible to > define a field with a type and a default of a totally different type. E.g. if > the field has type "string", the default can be set to "null". > I'd like to open a PR which will fix this by running the default through the > SchemaValidator whenever a new Field is created. See > [https://github.com/salsify/avro-patches/pull/16] > cc: [~tjwp] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
