[
https://issues.apache.org/jira/browse/SPARK-11953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15025538#comment-15025538
]
Siva Gudavalli edited comment on SPARK-11953 at 11/25/15 9:46 AM:
------------------------------------------------------------------
I Agree. It depends on how we define SaveMode.Append.
We are implementing Spark Streaming in our Speed Layer. For now we are using
Oracle as DataStore.
Issue1# I don't have a luxury to allow Spark Job to create the table as only
DBA can perform DDL. And I believe this will be the case with most production
deployments.
Issue2# tableExists check on Oracle is always failing doesn't mean to CREATE
TABLE. We should only create table if db returns status saying table doesnt
exist in the database
I hope it makes sense. Let me know your Ideas.
was (Author: sgudavalli):
I Agree. It depends on how we define SaveMode.Append.
I believe we should not allow CREATE Table in Append Mode
> Acknowledge Append Mode in DataFrameWriter
> ------------------------------------------
>
> Key: SPARK-11953
> URL: https://issues.apache.org/jira/browse/SPARK-11953
> Project: Spark
> Issue Type: Improvement
> Components: Java API, Spark Submit, SQL
> Affects Versions: 1.4.1, 1.5.1
> Environment: Spark stand alone cluster
> Reporter: Siva Gudavalli
> Priority: Blocker
>
> DataFramewriter is not acknowledging Append Mode.
> In case of Append Mode => It is verfiying if the table exists and creating a
> new table if it is not there and then Inserting data.
> If the table exists check is failed for whatsover reason, it still assumes
> table doesnt exists and creates a new table.
> As per my understanding, Append should perform "no DDL". It is okay to throw
> error saying "table doesnt exist for Inserts'. But creating a Table should
> not be allowed in Append Mode.
> *I believe we should do two things here*
> perform DDL => if and only if table exists check results "Not exists"
> do not perform DDL => if SaveMode is Append
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
