Suresh Thalamati created SPARK-10170:
----------------------------------------
Summary: Writing from data frame into db2 database using jdbc data
source api fails with error for string, and boolean column types.
Key: SPARK-10170
URL: https://issues.apache.org/jira/browse/SPARK-10170
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 1.4.1, 1.5.0
Reporter: Suresh Thalamati
Repro :
-- start spark shell with classpath set to the db2 jdbc driver.
SPARK_CLASSPATH=~/myjars/db2jcc.jar ./spark-shell
// set connetion properties
val properties = new java.util.Properties()
properties.setProperty("user" , "user")
properties.setProperty("password" , "password")
// load the driver.
Class.forName("com.ibm.db2.jcc.DB2Driver").newInstance
// create data frame with a String type
val empdf = sc.parallelize( Array((1,"John"), (2,"Mike"))).toDF("id", "name" )
// write the data frame. this will fail with error.
empdf.write.jdbc("jdbc:db2://bdvs150.svl.ibm.com:60000/SAMPLE:retrieveMessagesFromServerOnGetMessage=true;",
"emp_data", properties)
Error :
com.ibm.db2.jcc.am.SqlSyntaxErrorException: TEXT
at com.ibm.db2.jcc.am.fd.a(fd.java:679)
at com.ibm.db2.jcc.am.fd.a(fd.java:60)
..
// create data frame with String , and Boolean types
val empdf = sc.parallelize( Array((1,"true".toBoolean ), (2, "false".toBoolean
))).toDF("id", "isManager")
// write the data frame. this will fail with error.
empdf.write.jdbc("jdbc:db2://<server>:<port>
/SAMPLE:retrieveMessagesFromServerOnGetMessage=true;", "emp_data", properties)
Error :
com.ibm.db2.jcc.am.SqlSyntaxErrorException: TEXT
at com.ibm.db2.jcc.am.fd.a(fd.java:679)
at com.ibm.db2.jcc.am.fd.a(fd.java:60)
Write is failing because by default JDBC data source implementation generating
table schema with unsupported data types TEXT for String, and BIT1(1) for
Boolean. I think String type should get mapped to CLOB/VARCHAR, and boolean
type should be mapped to CHAR(1) for DB2 database.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
