[
https://issues.apache.org/jira/browse/SPARK-11953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15024991#comment-15024991
]
Huaxin Gao commented on SPARK-11953:
------------------------------------
My understanding is that SaveMode.Append doesn't mean that table exists for
sure.
When SaveMode is Append, if table exists, append to the existing table. If
table doesn't exist, create the table and append to the newly created table. So
the code looks right to me.
> CLONE - Sparksql-1.4.1 DataFrameWrite.jdbc() SaveMode.Append Bug
> ----------------------------------------------------------------
>
> Key: SPARK-11953
> URL: https://issues.apache.org/jira/browse/SPARK-11953
> Project: Spark
> Issue Type: Bug
> Components: Java API, Spark Submit, SQL
> Affects Versions: 1.4.1, 1.5.1
> Environment: Spark stand alone cluster
> Reporter: Siva Gudavalli
>
> In Spark 1.3.1 we have 2 methods i.e.. CreateJdbcTable and InsertIntoJdbc.
> They are replaced with write.jdbc() in Spark 1.4.1
> When we specify SaveMode.Append we are letting application know that there is
> a table in the database which means "tableExists = true". And we do not need
> to perform "JdbcUtils.tableExists(conn, table)".
> Please let me know if you think differently.
> Regards
> Shiv
> def jdbc(url: String, table: String, connectionProperties: Properties): Unit
> = {
> val conn = JdbcUtils.createConnection(url, connectionProperties)
> try {
> var tableExists = JdbcUtils.tableExists(conn, table)
> if (mode == SaveMode.Ignore && tableExists)
> { return }
> if (mode == SaveMode.ErrorIfExists && tableExists)
> { sys.error(s"Table $table already exists.") }
> if (mode == SaveMode.Overwrite && tableExists)
> { JdbcUtils.dropTable(conn, table) tableExists = false }
> // Create the table if the table didn't exist.
> if (!tableExists)
> { val schema = JDBCWriteDetails.schemaString(df, url) val sql = s"CREATE
> TABLE $table ($schema)" conn.prepareStatement(sql).executeUpdate() }
> } finally
> { conn.close() }
> JDBCWriteDetails.saveTable(df, url, table, connectionProperties)
> }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
