[
https://issues.apache.org/jira/browse/SPARK-30100?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17052050#comment-17052050
]
Joby Joje commented on SPARK-30100:
-----------------------------------
[~hyukjin.kwon] is there any workaround for this precision data loss ?
> Decimal Precision Inferred from JDBC via Spark
> ----------------------------------------------
>
> Key: SPARK-30100
> URL: https://issues.apache.org/jira/browse/SPARK-30100
> Project: Spark
> Issue Type: Bug
> Components: PySpark, SQL
> Affects Versions: 2.4.0
> Reporter: Joby Joje
> Priority: Major
>
> When trying to load data from JDBC(Oracle) into Spark, there seems to be
> precision loss in the decimal field, as per my understanding Spark supports
> *DECIMAL(38,18)*. The field from the Oracle is DECIMAL(38,14), whereas Spark
> rounds off the last four digits making it a precision of DECIMAL(38,10). This
> is happening to few fields in the dataframe where the column is fetched using
> a CASE statement whereas in the same query another field populates the right
> schema.
> Tried to pass the
> {code:java}
> spark.sql.decimalOperations.allowPrecisionLoss=false{code}
> conf in the Spark-submit though didn't get the desired results.
> {code:java}
> jdbcDF = spark.read \
> .format("jdbc") \
> .option("url", "ORACLE") \
> .option("dbtable", "QUERY") \
> .option("user", "USERNAME") \
> .option("password", "PASSWORD") \
> .load(){code}
> So considering that the Spark infers the schema from a sample records, how
> does this work here? Does it use the results of the query i.e (SELECT * FROM
> TABLE_NAME JOIN ...) or does it take a different route to guess the schema
> for itself? Can someone throw some light on this and advise how to achieve
> the right decimal precision on this regards without manipulating the query as
> doing a CAST on the query does solve the issue, but would prefer to get some
> alternatives.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
