[
https://issues.apache.org/jira/browse/SQOOP-1964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Qian Xu updated SQOOP-1964:
---------------------------
Description:
Decimal has been added to Avro relatively recently via AVRO-1402. It was added
only as a so called “logical type” that is annotating already existing physical
type. The user (in this case us, Sqoop developers) have to do the proper
encoding to final bytes - e.g. convert the logical decimal value and store as
described in the spec via “Fixed” Avro type.
Currently converting decimal to Avro String is to be consistent with Sqoop 1
that [does the
same|https://github.com/apache/sqoop/blob/trunk/src/java/org/apache/sqoop/manager/ConnManager.java#L201].
Sqoop 1 uses String because at the time we added Avro support into Sqoop 1
(several years back), there was no better way how to store decimal in Avro. At
that time, there was no support for decimal in the platform at all. As the
platform evolved and added a proper decimal type.
We should encode it in Sqoop 2 as such. E.g. in this particular case, I believe
that it’s better to encode the decimal type accordingly to AVRO-1402 that rest
of the platform (Hive, Impala, …) can easily process rather then keep backward
compatibility/consistency with Sqoop 1.
was:
Decimal has been added to Avro relatively recently via
[AVRO-1402|https://issues.apache.org/jira/browse/AVRO-1402]. It was added only
as a so called “logical type” that is annotating already existing physical
type. The user (in this case us, Sqoop developers) have to do the proper
encoding to final bytes - e.g. convert the logical decimal value and store as
described in the spec via “Fixed” Avro type.
Currently converting decimal to Avro String is to be consistent with Sqoop 1
that [does the
same|https://github.com/apache/sqoop/blob/trunk/src/java/org/apache/sqoop/manager/ConnManager.java#L201].
Sqoop 1 uses String because at the time we added Avro support into Sqoop 1
(several years back), there was no better way how to store decimal in Avro. At
that time, there was no support for decimal in the platform at all. As the
platform evolved and added a proper decimal type.
We should encode it in Sqoop 2 as such. E.g. in this particular case, I believe
that it’s better to encode the decimal type accordingly to AVRO-1402 that rest
of the platform (Hive, Impala, …) can easily process rather then keep backward
compatibility/consistency with Sqoop 1.
> Sqoop2: Encode the decimal type accordingly to AVRO-1402
> ---------------------------------------------------------
>
> Key: SQOOP-1964
> URL: https://issues.apache.org/jira/browse/SQOOP-1964
> Project: Sqoop
> Issue Type: Improvement
> Reporter: Qian Xu
>
> Decimal has been added to Avro relatively recently via AVRO-1402. It was
> added only as a so called “logical type” that is annotating already existing
> physical type. The user (in this case us, Sqoop developers) have to do the
> proper encoding to final bytes - e.g. convert the logical decimal value and
> store as described in the spec via “Fixed” Avro type.
> Currently converting decimal to Avro String is to be consistent with Sqoop 1
> that [does the
> same|https://github.com/apache/sqoop/blob/trunk/src/java/org/apache/sqoop/manager/ConnManager.java#L201].
> Sqoop 1 uses String because at the time we added Avro support into Sqoop 1
> (several years back), there was no better way how to store decimal in Avro.
> At that time, there was no support for decimal in the platform at all. As the
> platform evolved and added a proper decimal type.
> We should encode it in Sqoop 2 as such. E.g. in this particular case, I
> believe that it’s better to encode the decimal type accordingly to AVRO-1402
> that rest of the platform (Hive, Impala, …) can easily process rather then
> keep backward compatibility/consistency with Sqoop 1.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
