Actually, having re-read the "Schema resolution" section, I think you're right. If "Schema resolution" is the term used when discussing how to resolve differences between reader and writer schemas, then that sentence under the Decimal heading is currently not implemented / misleading / wrong.
Probably best that this is corrected as part of the PR to change the behaviour. It's probably worth adding something specific to the "Schema Resolution" rules section to cover Decimal, as well as changing the "Decimal" section. Unfortunately, at the moment I don't have time to write a PR. Will try to find the time soon if no one else gets a chance. Andy On Fri, 27 Jul 2018 at 15:18, Nandor Kollar <[email protected]> wrote: > Hi Andy, > > I think the intent of that sentence was to define a schema evolution rule, > because it starts with "purposes of schema resolution", though it is not in > the proper 'Schema resolution' part of Avro spec. If so, then in my opinion > either implementation should enforce this rule (AVRO-2078), or we should > delete this sentence from the spec (because it is misleading) and clarify > that Avro doesn't enforce any evolution rule for logical types, it is up to > the libraries that use Avro to handle reader/writer logical type changes. > Thoughts? > > Nandor > > On Fri, Jul 27, 2018 at 3:25 PM, Andy Coates <[email protected]> wrote: > > > Hi [~nkollar], > > > > I'd say that's a rule for decimal comparison, not evolution. > > > > On Fri, 20 Jul 2018 at 13:10, Nandor Kollar (JIRA) <[email protected]> > > wrote: > > > > > > > > [ > > > https://issues.apache.org/jira/browse/AVRO-2164?page= > > com.atlassian.jira.plugin.system.issuetabpanels:comment- > > tabpanel&focusedCommentId=16550713#comment-16550713 > > > ] > > > > > > Nandor Kollar commented on AVRO-2164: > > > ------------------------------------- > > > > > > [~BigAndy] I wanted to refer to this in the Decimal section of the > spec: > > > "For the purposes of schema resolution, two schemas that > > > are decimal logical types _match_ if their scales and precisions > match." > > > This sounds like a rule for schema evolution, and scale and precision > > > should match (the current doesn't check for this). > > > > > > > Make Decimal a first class type. > > > > -------------------------------- > > > > > > > > Key: AVRO-2164 > > > > URL: https://issues.apache.org/jira/browse/AVRO-2164 > > > > Project: Avro > > > > Issue Type: Improvement > > > > Components: logical types > > > > Affects Versions: 1.8.2 > > > > Reporter: Andy Coates > > > > Priority: Major > > > > > > > > I'd be interested to hear the communities thoughts on making decimal > a > > > first class type. > > > > The current logical type encodes a decimal into a _bytes_ or _fixed_. > > > This encoding does not include any information about the scale, i.e. > this > > > encoding is lossy. > > > > There are open issues around the compatibility / evolvability of > > schemas > > > containing decimal logical types, (e.g. AVRO-2078 & AVRO-1721), that > mean > > > reading data that was previously written with a different scale will > > result > > > in data corruption. > > > > If these issues were fixed, with suitable compatibility checks put in > > > place, this would then make it impossible to evolve an Avro schema > where > > > the scale needs to be changed. This inability to evolve the scale is > very > > > restrictive, and can result in high overhead for organizations that > > _need_ > > > to change the scale, i.e. they may potentially need to copy their > entire > > > data set, deserializing with the old scale and re-serializing with the > > new. > > > > If _decimal_ were promoted to a first class type, this would allow > the > > > scale to be captured in the serialized form, allow for schema evolution > > > support. > > > > > > > > > > > > -- > > > This message was sent by Atlassian JIRA > > > (v7.6.3#76005) > > > > > >
