eugene created SPARK-38846:
------------------------------
Summary: Teradata's Number is either converted to its floor value
or ceiling value despite its fractional part.
Key: SPARK-38846
URL: https://issues.apache.org/jira/browse/SPARK-38846
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.3.0
Environment: Spark2.3.0 on Yarn
Teradata 16.20.32.59
Reporter: eugene
I'm trying to load data from Teradata, the code using is:
sparkSession.read
.format("jdbc")
.options(
Map(
"url" -> "jdbc:teradata://hostname, user=$username,
password=$password",
"MAYBENULL" -> "ON",
"SIP_SUPPORT" -> "ON",
"driver" -> "com.teradata.jdbc.TeraDriver",
"dbtable" -> $table_name
)
)
.load()
However, some data lost its fractional part after loading. To be more concise,
the column in Teradata is in the type of [Number][1] and after loading, the
data type in Spark is `DecimalType(38,0)`, the scale value is 0 which means no
digits after decimal point.
Data in Teradata is something like,
id column1 column2
1 50.23 100.23
2 25.8 20.669
3 30.2 19.23
The `dataframe` of Spark is like,
id column1 column2
1 50 100
2 26 21
3 30 19
The meta data of the table in Teradata is like:
CREATE SET TABLE table_name (id BIGINT, column1 NUMBER, column2 NUMBER)
PRIMARY INDEX (id);
The Spark version is 2.3.0 and Teradata is 16.20.32.59.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
