[
https://issues.apache.org/jira/browse/SPARK-11953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siva Gudavalli updated SPARK-11953:
-----------------------------------
Description:
In Spark 1.3.1 we have 2 methods i.e.. CreateJdbcTable and InsertIntoJdbc.
They are replaced with write.jdbc() in Spark 1.4.1
When we specify SaveMode.Append we are letting application know that there is a
table in the database which means "tableExists = true". And we do not need to
perform "JdbcUtils.tableExists(conn, table)".
Please let me know if you think differently.
Regards
Shiv
def jdbc(url: String, table: String, connectionProperties: Properties): Unit = {
val conn = JdbcUtils.createConnection(url, connectionProperties)
try {
var tableExists = JdbcUtils.tableExists(conn, table)
if (mode == SaveMode.Ignore && tableExists)
{ return }
if (mode == SaveMode.ErrorIfExists && tableExists)
{ sys.error(s"Table $table already exists.") }
if (mode == SaveMode.Overwrite && tableExists)
{ JdbcUtils.dropTable(conn, table) tableExists = false }
// Create the table if the table didn't exist.
if (!tableExists)
{ val schema = JDBCWriteDetails.schemaString(df, url) val sql = s"CREATE TABLE
$table ($schema)" conn.prepareStatement(sql).executeUpdate() }
} finally
{ conn.close() }
JDBCWriteDetails.saveTable(df, url, table, connectionProperties)
}
was:
I am running spark-submit in window 8.1 with spark standalone cluster (01
worker and 01 master), the job throw Exception in DataFrameWrite.jdbc(..)
scala function.
We found that the following test:
var tableExists = JdbcUtils.tableExists(conn, table)
always return false event if we already created a table.
That drive the function to do creating table from specified DataFrame and the
SQL Syntax error for creating table, we locate the SQL execution statement
hereafter:
if (!tableExists) {
val schema = JDBCWriteDetails.schemaString(df, url)
val sql = s"CREATE TABLE $table ($schema)"
conn.prepareStatement(sql).executeUpdate() // This execution cause sql
syntax error
}
This happened with spark-1.4.1 and Spark-1.5.1 (our dev environment).
Please help!
> CLONE - Sparksql-1.4.1 DataFrameWrite.jdbc() SaveMode.Append Bug
> ----------------------------------------------------------------
>
> Key: SPARK-11953
> URL: https://issues.apache.org/jira/browse/SPARK-11953
> Project: Spark
> Issue Type: Bug
> Components: Java API, Spark Submit, SQL
> Affects Versions: 1.4.1, 1.5.1
> Environment: Spark stand alone cluster
> Reporter: Siva Gudavalli
>
> In Spark 1.3.1 we have 2 methods i.e.. CreateJdbcTable and InsertIntoJdbc.
> They are replaced with write.jdbc() in Spark 1.4.1
> When we specify SaveMode.Append we are letting application know that there is
> a table in the database which means "tableExists = true". And we do not need
> to perform "JdbcUtils.tableExists(conn, table)".
> Please let me know if you think differently.
> Regards
> Shiv
> def jdbc(url: String, table: String, connectionProperties: Properties): Unit
> = {
> val conn = JdbcUtils.createConnection(url, connectionProperties)
> try {
> var tableExists = JdbcUtils.tableExists(conn, table)
> if (mode == SaveMode.Ignore && tableExists)
> { return }
> if (mode == SaveMode.ErrorIfExists && tableExists)
> { sys.error(s"Table $table already exists.") }
> if (mode == SaveMode.Overwrite && tableExists)
> { JdbcUtils.dropTable(conn, table) tableExists = false }
> // Create the table if the table didn't exist.
> if (!tableExists)
> { val schema = JDBCWriteDetails.schemaString(df, url) val sql = s"CREATE
> TABLE $table ($schema)" conn.prepareStatement(sql).executeUpdate() }
> } finally
> { conn.close() }
> JDBCWriteDetails.saveTable(df, url, table, connectionProperties)
> }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
