[ https://issues.apache.org/jira/browse/SPARK-28151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shiv Prashant Sood updated SPARK-28151: --------------------------------------- Summary: ByteType, ShortType and FloatTypes are not correctly mapped for read/write of SQLServer tables (was: ByteType is not correctly mapped for read/write of SQLServer tables) > ByteType, ShortType and FloatTypes are not correctly mapped for read/write of > SQLServer tables > ---------------------------------------------------------------------------------------------- > > Key: SPARK-28151 > URL: https://issues.apache.org/jira/browse/SPARK-28151 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 3.0.0, 2.4.3 > Reporter: Shiv Prashant Sood > Priority: Minor > > Writing dataframe with column type BYTETYPE fails when using JDBC connector > for SQL Server. Append and Read of tables also fail. The problem is due > 1. (Write path) Incorrect mapping of BYTETYPE in getCommonJDBCType() in > jdbcutils.scala where BYTETYPE gets mapped to BYTE text. It should be mapped > to TINYINT > {color:#cc7832}case {color}ByteType => > Option(JdbcType({color:#6a8759}"BYTE"{color}{color:#cc7832}, > {color}java.sql.Types.{color:#9876aa}TINYINT{color})) > In getCatalystType() ( JDBC to Catalyst type mapping) TINYINT is mapped to > INTEGER, while it should be mapped to BYTETYPE. Mapping to integer is ok from > the point of view of upcasting, but will lead to 4 byte allocation rather > than 1 byte for BYTETYPE. > 2. (read path) Read path ends up calling makeGetter(dt: DataType, metadata: > Metadata). The function sets the value in RDD row. The value is set per the > data type. Here there is no mapping for BYTETYPE and thus results will result > in an error when getCatalystType() is fixed. > Note : These issues were found when reading/writing with SQLServer. Will be > submitting a PR soon to fix these mappings in MSSQLServerDialect. > Error seen when writing table > (JDBC Write failed,com.microsoft.sqlserver.jdbc.SQLServerException: Column, > parameter, or variable #2: *Cannot find data type BYTE*.) > com.microsoft.sqlserver.jdbc.SQLServerException: Column, parameter, or > variable #2: Cannot find data type BYTE. > com.microsoft.sqlserver.jdbc.SQLServerException.makeFromDatabaseError(SQLServerException.java:254) > com.microsoft.sqlserver.jdbc.SQLServerStatement.getNextResult(SQLServerStatement.java:1608) > com.microsoft.sqlserver.jdbc.SQLServerStatement.doExecuteStatement(SQLServerStatement.java:859) > .. > > > > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org