Qian Xu created SQOOP-1964:
------------------------------
Summary: Sqoop2: Encode the decimal type accordingly to AVRO-1402
Key: SQOOP-1964
URL: https://issues.apache.org/jira/browse/SQOOP-1964
Project: Sqoop
Issue Type: Improvement
Reporter: Qian Xu
Decimal has been added to Avro relatively recently via
[AVRO-1402|https://issues.apache.org/jira/browse/AVRO-1402]. It was added only
as a so called “logical type” that is annotating already existing physical
type. The user (in this case us, Sqoop developers) have to do the proper
encoding to final bytes - e.g. convert the logical decimal value and store as
described in the spec via “Fixed” Avro type.
Currently converting decimal to Avro String is to be consistent with Sqoop 1
that [does the
same|https://github.com/apache/sqoop/blob/trunk/src/java/org/apache/sqoop/manager/ConnManager.java#L201].
Sqoop 1 uses String because at the time we added Avro support into Sqoop 1
(several years back), there was no better way how to store decimal in Avro. At
that time, there was no support for decimal in the platform at all. As the
platform evolved and added a proper decimal type.
We should encode it in Sqoop 2 as such. E.g. in this particular case, I believe
that it’s better to encode the decimal type accordingly to AVRO-1402 that rest
of the platform (Hive, Impala, …) can easily process rather then keep backward
compatibility/consistency with Sqoop 1.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
