Andy Coates commented on AVRO-2164:

[~rdblue], if we go with always deserializing using the write schema's scale 
then we'll avoid data corruption. How would it work if read/write schema's had 
mix of logical and non-logical types?

||Write Schema||Read Schema||Action||
|Logical|Logical with same scale|Read with specified scale|
|Logical|Logical with different scale|Read with scale specified in write schema|
|Logical|Not logical|Read a bytes|
|Not logical|Logical|Read as decimal using read schema scale???|



> Make Decimal a first class type.
> --------------------------------
>                 Key: AVRO-2164
>                 URL: https://issues.apache.org/jira/browse/AVRO-2164
>             Project: Avro
>          Issue Type: Improvement
>          Components: logical types
>    Affects Versions: 1.8.2
>            Reporter: Andy Coates
>            Priority: Major
> I'd be interested to hear the communities thoughts on making decimal a first 
> class type. 
> The current logical type encodes a decimal into a _bytes_ or _fixed_. This 
> encoding does not include any information about the scale, i.e. this encoding 
> is lossy. 
> There are open issues around the compatibility / evolvability of schemas 
> containing decimal logical types, (e.g. AVRO-2078 & AVRO-1721), that mean 
> reading data that was previously written with a different scale will result 
> in data corruption.
> If these issues were fixed, with suitable compatibility checks put in place, 
> this would then make it impossible to evolve an Avro schema where the scale 
> needs to be changed. This inability to evolve the scale is very restrictive, 
> and can result in high overhead for organizations that _need_ to change the 
> scale, i.e. they may potentially need to copy their entire data set, 
> deserializing with the old scale and re-serializing with the new.
> If _decimal_ were promoted to a first class type, this would allow the scale 
> to be captured in the serialized form, allow for schema evolution support.

This message was sent by Atlassian JIRA

Reply via email to