[
https://issues.apache.org/jira/browse/SPARK-20427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15978451#comment-15978451
]
Ostap Gonchar commented on SPARK-20427:
---------------------------------------
Hi Sean,
What we've spotted is that by default Oracle's NUMBER type is translated as
Decimal(38,10). And if value has no zeroes - Spark still adds those.
Therefore imagine you have number which is 30 digits long, e.g. =
12312321321321312312312312123. When Spark SQL (in Java) reads such value from
Oracle's NUMBER column - it perceives it as
12312321321321312312312312123,0000000000 (10 zeroes). Then somehow precision is
counted as 30 + 10 = 40, which is higher than specified in Decimal(38,10). Even
though zeros shouldn't be counted as precision (see
http://stackoverflow.com/questions/35435691/bigdecimal-precision-and-scale).
That is an issue. And the only solution we have found so far is to change
Column's type to INTEGER (so that it interprets it as Decimal(38.0)). But
unfortunately, we do not have a control over all of tables.
Hope it clarifies
> Issue with Spark interpreting Oracle datatype NUMBER
> ----------------------------------------------------
>
> Key: SPARK-20427
> URL: https://issues.apache.org/jira/browse/SPARK-20427
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.1.0
> Reporter: Alexander Andrushenko
>
> In Oracle exists data type NUMBER. When defining a filed in a table of type
> NUMBER the field has two components, precision and scale.
> For example, NUMBER(p,s) has precision p and scale s.
> Precision can range from 1 to 38.
> Scale can range from -84 to 127.
> When reading such a filed Spark can create numbers with precision exceeding
> 38. In our case it has created fields with precision 44,
> calculated as sum of the precision (in our case 34 digits) and the scale (10):
> "...java.lang.IllegalArgumentException: requirement failed: Decimal precision
> 44 exceeds max precision 38...".
> The result was, that a data frame was read from a table on one schema but
> could not be inserted in the identical table on other schema.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
