[
https://issues.apache.org/jira/browse/SPARK-28152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shiv Prashant Sood updated SPARK-28152:
---------------------------------------
Description:
ShortType and FloatTypes are not correctly mapped to right JDBC types when
using JDBC connector. This results in tables and spark data frame being created
with unintended types. The issue was observed when validating against SQLServer.
Some example issue
* Write from df with column type results in a SQL table of with column type as
INTEGER as opposed to SMALLINT. Thus a larger table that expected.
* read results in a dataframe with type INTEGER as opposed to ShortType
FloatTypes have a issue with read path. In the write path Spark data type
'FloatType' is correctly mapped to JDBC equivalent data type 'Real'. But in the
read path when JDBC data types need to be converted to Catalyst data types (
getCatalystType) 'Real' gets incorrectly gets mapped to 'DoubleType' rather
than 'FloatType'.
was:
ShortType and FloatTypes are not correctly mapped to right JDBC types when
using JDBC connector. This results in tables and spark data frame being created
with unintended types.
Some example issue
* Write from df with column type results in a SQL table of with column type as
INTEGER as opposed to SMALLINT. Thus a larger table that expected.
* read results in a dataframe with type INTEGER as opposed to ShortType
FloatTypes have a issue with read path. In the write path Spark data type
'FloatType' is correctly mapped to JDBC equivalent data type 'Real'. But in the
read path when JDBC data types need to be converted to Catalyst data types (
getCatalystType) 'Real' gets incorrectly gets mapped to 'DoubleType' rather
than 'FloatType'.
> [JDBC Connector] ShortType and FloatTypes are not mapped correctly
> ------------------------------------------------------------------
>
> Key: SPARK-28152
> URL: https://issues.apache.org/jira/browse/SPARK-28152
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.0.0, 2.4.3
> Reporter: Shiv Prashant Sood
> Priority: Minor
>
> ShortType and FloatTypes are not correctly mapped to right JDBC types when
> using JDBC connector. This results in tables and spark data frame being
> created with unintended types. The issue was observed when validating against
> SQLServer.
> Some example issue
> * Write from df with column type results in a SQL table of with column type
> as INTEGER as opposed to SMALLINT. Thus a larger table that expected.
> * read results in a dataframe with type INTEGER as opposed to ShortType
> FloatTypes have a issue with read path. In the write path Spark data type
> 'FloatType' is correctly mapped to JDBC equivalent data type 'Real'. But in
> the read path when JDBC data types need to be converted to Catalyst data
> types ( getCatalystType) 'Real' gets incorrectly gets mapped to 'DoubleType'
> rather than 'FloatType'.
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
