[
https://issues.apache.org/jira/browse/AVRO-1402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13962954#comment-13962954
]
Tom White commented on AVRO-1402:
---------------------------------
There has been some further (offline) discussion about whether it would be
possible to store the scale in the Avro schema, and not in the data for
efficiency reasons. Something like:
{code}
{
"type":"record”,
"name":”org.apache.avro.FixedDecimal”,
"fields”: [{
"name":"value”,
"type":”bytes"
}],
"scale":"2”,
"precision":”4"
}
{code}
In the implementation committed here the name does not uniquely determine the
RecordMapping so a FixedDecimal(4, 2) has a different RecordMapping to a
FixedDecimal(3, 0). GenericData has a map of name to RecordMappings, so
org.apache.avro.FixedDecimal would map to either FixedDecimalRecordMapping(4,
2) or FixedDecimalRecordMapping(3, 0), but not both.
We could solve this problem by having a stateless FixedDecimalRecordMapping and
having the read and write methods pass through the record schema to get the
scale. However, consider the case where there are multiple decimals (with
different scales) in a single schema. Since you can’t redefine a type multiple
times (http://avro.apache.org/docs/1.7.6/spec.html#Names), the first one serves
as the definition, and later ones are just references:
{code}
{"type":"record","name":"rec","fields":[
{"name":"dec1","type":{"type":"record","name":"org.apache.avro.FixedDecimal","fields":[{"name":"value","type":"bytes"}],"scale":"2","precision":"4"}},
{"name":"dec2","type":"org.apache.avro.FixedDecimal","precision":"3","scale":"0"}
]}
{code}
When GenericDatumReader/Writer is processing dec2, the value of scale seen is
2, not 0, since the read/write method sees the record schema, not the
field-level schema. I can’t see a simple way around this.
Note that in the Decimal schema committed in this JIRA we allow maxPrecision
and maxScale values to be specified as JSON properties that are not interpreted
by Avro. E.g.
{code}
{"type":"record","name":"rec","fields":[
{"name":"dec1","type":{"type":"record","name":”org.apache.avro.Decimal","fields":[{"name":"scale","type":"int"},{"name":"value","type":"bytes"}],"maxPrecision":"4","maxScale":"2"}},
{"name":"dec2","type":"org.apache.avro.Decimal","maxPrecision":"3","maxScale":"0"}
]}
{code}
As it stands an application using this extra metadata would have to be careful
to read the JSON properties either from the field (if they are present there)
or the org.apache.avro.Decimal record type. This might be something we improve
- e.g. by only having the metadata as a field-level properties, not as a part
of the record definition. That would work for Hive.
> Support for DECIMAL type
> ------------------------
>
> Key: AVRO-1402
> URL: https://issues.apache.org/jira/browse/AVRO-1402
> Project: Avro
> Issue Type: New Feature
> Affects Versions: 1.7.5
> Reporter: Mariano Dominguez
> Assignee: Tom White
> Priority: Minor
> Labels: Hive
> Fix For: 1.7.7
>
> Attachments: AVRO-1402.patch, AVRO-1402.patch, AVRO-1402.patch,
> AVRO-1402.patch, UnixEpochRecordMapping.patch
>
>
> Currently, Avro does not seem to support a DECIMAL type or equivalent.
> http://avro.apache.org/docs/1.7.5/spec.html#schema_primitive
> Adding DECIMAL support would be particularly interesting when converting
> types from Avro to Hive, since DECIMAL is already a supported data type in
> Hive (0.11.0).
--
This message was sent by Atlassian JIRA
(v6.2#6252)
