eugene created SPARK-38846: ------------------------------ Summary: Teradata's Number is either converted to its floor value or ceiling value despite its fractional part. Key: SPARK-38846 URL: https://issues.apache.org/jira/browse/SPARK-38846 Project: Spark Issue Type: Bug Components: SQL Affects Versions: 2.3.0 Environment: Spark2.3.0 on Yarn
Teradata 16.20.32.59 Reporter: eugene I'm trying to load data from Teradata, the code using is: sparkSession.read .format("jdbc") .options( Map( "url" -> "jdbc:teradata://hostname, user=$username, password=$password", "MAYBENULL" -> "ON", "SIP_SUPPORT" -> "ON", "driver" -> "com.teradata.jdbc.TeraDriver", "dbtable" -> $table_name ) ) .load() However, some data lost its fractional part after loading. To be more concise, the column in Teradata is in the type of [Number][1] and after loading, the data type in Spark is `DecimalType(38,0)`, the scale value is 0 which means no digits after decimal point. Data in Teradata is something like, id column1 column2 1 50.23 100.23 2 25.8 20.669 3 30.2 19.23 The `dataframe` of Spark is like, id column1 column2 1 50 100 2 26 21 3 30 19 The meta data of the table in Teradata is like: CREATE SET TABLE table_name (id BIGINT, column1 NUMBER, column2 NUMBER) PRIMARY INDEX (id); The Spark version is 2.3.0 and Teradata is 16.20.32.59. -- This message was sent by Atlassian Jira (v8.20.1#820001) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org