[jira] [Commented] (SPARK-23402) Dataset write method not working as expected for postgresql database
[
https://issues.apache.org/jira/browse/SPARK-23402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16365549#comment-16365549
]
Sean Owen commented on SPARK-23402:
---
This is all just the master branch of github.com/apache/spark and yes you need
to build it yourself.
> Dataset write method not working as expected for postgresql database
>
>
> Key: SPARK-23402
> URL: https://issues.apache.org/jira/browse/SPARK-23402
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, SQL
>Affects Versions: 2.2.1
> Environment: PostgreSQL: 9.5.8 (10 + Also same issue)
> OS: Cent OS 7 & Windows 7,8
> JDBC: 9.4-1201-jdbc41
>
> Spark: I executed in both 2.1.0 and 2.2.1
> Mode: Standalone
> OS: Windows 7
>Reporter: Pallapothu Jyothi Swaroop
>Priority: Major
> Attachments: Emsku[1].jpg
>
>
> I am using spark dataset write to insert data on postgresql existing table.
> For this I am using write method mode as append mode. While using i am
> getting exception like table already exists. But, I gave option as append
> mode.
> It's strange. When i change options to sqlserver/oracle append mode is
> working as expected.
>
> *Database Properties:*
> {{destinationProps.put("driver", "org.postgresql.Driver");
> destinationProps.put("url", "jdbc:postgresql://127.0.0.1:30001/dbmig");
> destinationProps.put("user", "dbmig");}}
> {{destinationProps.put("password", "dbmig");}}
>
> *Dataset Write Code:*
> {{valueAnalysisDataset.write().mode(SaveMode.Append).jdbc(destinationDbMap.get("url"),
> "dqvalue", destinationdbProperties);}}
>
>
> {{Exception in thread "main" org.postgresql.util.PSQLException: ERROR:
> relation "dqvalue" already exists at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2412)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2125)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:297)
> at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:428) at
> org.postgresql.jdbc.PgStatement.execute(PgStatement.java:354) at
> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:301) at
> org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:287) at
> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:264) at
> org.postgresql.jdbc.PgStatement.executeUpdate(PgStatement.java:244) at
> org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.createTable(JdbcUtils.scala:806)
> at
> org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:95)
> at
> org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:469)
> at
> org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:50)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:138)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
> at
> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:135) at
> org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:116) at
> org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:92)
> at
> org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:92)
> at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:609)
> at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:233) at
> org.apache.spark.sql.DataFrameWriter.jdbc(DataFrameWriter.scala:460) at
> com.ads.dqam.action.impl.PostgresValueAnalysis.persistValueAnalysis(PostgresValueAnalysis.java:25)
> at
> com.ads.dqam.action.AbstractValueAnalysis.persistAnalysis(AbstractValueAnalysis.java:81)
> at com.ads.dqam.Analysis.doAnalysis(Analysis.java:32) at
> com.ads.dqam.Client.main(Client.java:71)}}
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23402) Dataset write method not working as expected for postgresql database
[
https://issues.apache.org/jira/browse/SPARK-23402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364088#comment-16364088
]
Pallapothu Jyothi Swaroop commented on SPARK-23402:
---
Ok I will try with master.
I need to build in my local?
Or
Where I can get master branch maven dependencies?
> Dataset write method not working as expected for postgresql database
>
>
> Key: SPARK-23402
> URL: https://issues.apache.org/jira/browse/SPARK-23402
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, SQL
>Affects Versions: 2.2.1
> Environment: PostgreSQL: 9.5.8 (10 + Also same issue)
> OS: Cent OS 7 & Windows 7,8
> JDBC: 9.4-1201-jdbc41
>
> Spark: I executed in both 2.1.0 and 2.2.1
> Mode: Standalone
> OS: Windows 7
>Reporter: Pallapothu Jyothi Swaroop
>Priority: Major
> Attachments: Emsku[1].jpg
>
>
> I am using spark dataset write to insert data on postgresql existing table.
> For this I am using write method mode as append mode. While using i am
> getting exception like table already exists. But, I gave option as append
> mode.
> It's strange. When i change options to sqlserver/oracle append mode is
> working as expected.
>
> *Database Properties:*
> {{destinationProps.put("driver", "org.postgresql.Driver");
> destinationProps.put("url", "jdbc:postgresql://127.0.0.1:30001/dbmig");
> destinationProps.put("user", "dbmig");}}
> {{destinationProps.put("password", "dbmig");}}
>
> *Dataset Write Code:*
> {{valueAnalysisDataset.write().mode(SaveMode.Append).jdbc(destinationDbMap.get("url"),
> "dqvalue", destinationdbProperties);}}
>
>
> {{Exception in thread "main" org.postgresql.util.PSQLException: ERROR:
> relation "dqvalue" already exists at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2412)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2125)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:297)
> at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:428) at
> org.postgresql.jdbc.PgStatement.execute(PgStatement.java:354) at
> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:301) at
> org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:287) at
> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:264) at
> org.postgresql.jdbc.PgStatement.executeUpdate(PgStatement.java:244) at
> org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.createTable(JdbcUtils.scala:806)
> at
> org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:95)
> at
> org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:469)
> at
> org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:50)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:138)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
> at
> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:135) at
> org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:116) at
> org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:92)
> at
> org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:92)
> at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:609)
> at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:233) at
> org.apache.spark.sql.DataFrameWriter.jdbc(DataFrameWriter.scala:460) at
> com.ads.dqam.action.impl.PostgresValueAnalysis.persistValueAnalysis(PostgresValueAnalysis.java:25)
> at
> com.ads.dqam.action.AbstractValueAnalysis.persistAnalysis(AbstractValueAnalysis.java:81)
> at com.ads.dqam.Analysis.doAnalysis(Analysis.java:32) at
> com.ads.dqam.Client.main(Client.java:71)}}
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23402) Dataset write method not working as expected for postgresql database
[
https://issues.apache.org/jira/browse/SPARK-23402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16364053#comment-16364053
]
Sean Owen commented on SPARK-23402:
---
[~swaroopp] you would need to try with master. It's possible this was fixed
separately somehow.
> Dataset write method not working as expected for postgresql database
>
>
> Key: SPARK-23402
> URL: https://issues.apache.org/jira/browse/SPARK-23402
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, SQL
>Affects Versions: 2.2.1
> Environment: PostgreSQL: 9.5.8 (10 + Also same issue)
> OS: Cent OS 7 & Windows 7,8
> JDBC: 9.4-1201-jdbc41
>
> Spark: I executed in both 2.1.0 and 2.2.1
> Mode: Standalone
> OS: Windows 7
>Reporter: Pallapothu Jyothi Swaroop
>Priority: Major
> Attachments: Emsku[1].jpg
>
>
> I am using spark dataset write to insert data on postgresql existing table.
> For this I am using write method mode as append mode. While using i am
> getting exception like table already exists. But, I gave option as append
> mode.
> It's strange. When i change options to sqlserver/oracle append mode is
> working as expected.
>
> *Database Properties:*
> {{destinationProps.put("driver", "org.postgresql.Driver");
> destinationProps.put("url", "jdbc:postgresql://127.0.0.1:30001/dbmig");
> destinationProps.put("user", "dbmig");}}
> {{destinationProps.put("password", "dbmig");}}
>
> *Dataset Write Code:*
> {{valueAnalysisDataset.write().mode(SaveMode.Append).jdbc(destinationDbMap.get("url"),
> "dqvalue", destinationdbProperties);}}
>
>
> {{Exception in thread "main" org.postgresql.util.PSQLException: ERROR:
> relation "dqvalue" already exists at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2412)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2125)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:297)
> at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:428) at
> org.postgresql.jdbc.PgStatement.execute(PgStatement.java:354) at
> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:301) at
> org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:287) at
> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:264) at
> org.postgresql.jdbc.PgStatement.executeUpdate(PgStatement.java:244) at
> org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.createTable(JdbcUtils.scala:806)
> at
> org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:95)
> at
> org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:469)
> at
> org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:50)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:138)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
> at
> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:135) at
> org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:116) at
> org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:92)
> at
> org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:92)
> at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:609)
> at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:233) at
> org.apache.spark.sql.DataFrameWriter.jdbc(DataFrameWriter.scala:460) at
> com.ads.dqam.action.impl.PostgresValueAnalysis.persistValueAnalysis(PostgresValueAnalysis.java:25)
> at
> com.ads.dqam.action.AbstractValueAnalysis.persistAnalysis(AbstractValueAnalysis.java:81)
> at com.ads.dqam.Analysis.doAnalysis(Analysis.java:32) at
> com.ads.dqam.Client.main(Client.java:71)}}
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23402) Dataset write method not working as expected for postgresql database
[
https://issues.apache.org/jira/browse/SPARK-23402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363749#comment-16363749
]
Pallapothu Jyothi Swaroop commented on SPARK-23402:
---
I am using 2.2.1 version in my project. So please check with 2.2.1
> Dataset write method not working as expected for postgresql database
>
>
> Key: SPARK-23402
> URL: https://issues.apache.org/jira/browse/SPARK-23402
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, SQL
>Affects Versions: 2.2.1
> Environment: PostgreSQL: 9.5.8 (10 + Also same issue)
> OS: Cent OS 7 & Windows 7,8
> JDBC: 9.4-1201-jdbc41
>
> Spark: I executed in both 2.1.0 and 2.2.1
> Mode: Standalone
> OS: Windows 7
>Reporter: Pallapothu Jyothi Swaroop
>Priority: Major
> Attachments: Emsku[1].jpg
>
>
> I am using spark dataset write to insert data on postgresql existing table.
> For this I am using write method mode as append mode. While using i am
> getting exception like table already exists. But, I gave option as append
> mode.
> It's strange. When i change options to sqlserver/oracle append mode is
> working as expected.
>
> *Database Properties:*
> {{destinationProps.put("driver", "org.postgresql.Driver");
> destinationProps.put("url", "jdbc:postgresql://127.0.0.1:30001/dbmig");
> destinationProps.put("user", "dbmig");}}
> {{destinationProps.put("password", "dbmig");}}
>
> *Dataset Write Code:*
> {{valueAnalysisDataset.write().mode(SaveMode.Append).jdbc(destinationDbMap.get("url"),
> "dqvalue", destinationdbProperties);}}
>
>
> {{Exception in thread "main" org.postgresql.util.PSQLException: ERROR:
> relation "dqvalue" already exists at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2412)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2125)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:297)
> at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:428) at
> org.postgresql.jdbc.PgStatement.execute(PgStatement.java:354) at
> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:301) at
> org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:287) at
> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:264) at
> org.postgresql.jdbc.PgStatement.executeUpdate(PgStatement.java:244) at
> org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.createTable(JdbcUtils.scala:806)
> at
> org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:95)
> at
> org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:469)
> at
> org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:50)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:138)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
> at
> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:135) at
> org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:116) at
> org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:92)
> at
> org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:92)
> at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:609)
> at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:233) at
> org.apache.spark.sql.DataFrameWriter.jdbc(DataFrameWriter.scala:460) at
> com.ads.dqam.action.impl.PostgresValueAnalysis.persistValueAnalysis(PostgresValueAnalysis.java:25)
> at
> com.ads.dqam.action.AbstractValueAnalysis.persistAnalysis(AbstractValueAnalysis.java:81)
> at com.ads.dqam.Analysis.doAnalysis(Analysis.java:32) at
> com.ads.dqam.Client.main(Client.java:71)}}
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23402) Dataset write method not working as expected for postgresql database
[
https://issues.apache.org/jira/browse/SPARK-23402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363741#comment-16363741
]
Marco Gaido commented on SPARK-23402:
-
Yes the table existed. please try with the current master. I used the current
master for my try.
> Dataset write method not working as expected for postgresql database
>
>
> Key: SPARK-23402
> URL: https://issues.apache.org/jira/browse/SPARK-23402
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, SQL
>Affects Versions: 2.2.1
> Environment: PostgreSQL: 9.5.8 (10 + Also same issue)
> OS: Cent OS 7 & Windows 7,8
> JDBC: 9.4-1201-jdbc41
>
> Spark: I executed in both 2.1.0 and 2.2.1
> Mode: Standalone
> OS: Windows 7
>Reporter: Pallapothu Jyothi Swaroop
>Priority: Major
> Attachments: Emsku[1].jpg
>
>
> I am using spark dataset write to insert data on postgresql existing table.
> For this I am using write method mode as append mode. While using i am
> getting exception like table already exists. But, I gave option as append
> mode.
> It's strange. When i change options to sqlserver/oracle append mode is
> working as expected.
>
> *Database Properties:*
> {{destinationProps.put("driver", "org.postgresql.Driver");
> destinationProps.put("url", "jdbc:postgresql://127.0.0.1:30001/dbmig");
> destinationProps.put("user", "dbmig");}}
> {{destinationProps.put("password", "dbmig");}}
>
> *Dataset Write Code:*
> {{valueAnalysisDataset.write().mode(SaveMode.Append).jdbc(destinationDbMap.get("url"),
> "dqvalue", destinationdbProperties);}}
>
>
> {{Exception in thread "main" org.postgresql.util.PSQLException: ERROR:
> relation "dqvalue" already exists at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2412)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2125)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:297)
> at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:428) at
> org.postgresql.jdbc.PgStatement.execute(PgStatement.java:354) at
> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:301) at
> org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:287) at
> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:264) at
> org.postgresql.jdbc.PgStatement.executeUpdate(PgStatement.java:244) at
> org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.createTable(JdbcUtils.scala:806)
> at
> org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:95)
> at
> org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:469)
> at
> org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:50)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:138)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
> at
> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:135) at
> org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:116) at
> org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:92)
> at
> org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:92)
> at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:609)
> at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:233) at
> org.apache.spark.sql.DataFrameWriter.jdbc(DataFrameWriter.scala:460) at
> com.ads.dqam.action.impl.PostgresValueAnalysis.persistValueAnalysis(PostgresValueAnalysis.java:25)
> at
> com.ads.dqam.action.AbstractValueAnalysis.persistAnalysis(AbstractValueAnalysis.java:81)
> at com.ads.dqam.Analysis.doAnalysis(Analysis.java:32) at
> com.ads.dqam.Client.main(Client.java:71)}}
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23402) Dataset write method not working as expected for postgresql database
[
https://issues.apache.org/jira/browse/SPARK-23402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363726#comment-16363726
]
Pallapothu Jyothi Swaroop commented on SPARK-23402:
---
[~mgaido] please confirm, is table already existed in database? I am getting
issue only for tables that are already existed in schema.
I tried with Postgres 10, driver 42.2.1 in windows 8. No success.
> Dataset write method not working as expected for postgresql database
>
>
> Key: SPARK-23402
> URL: https://issues.apache.org/jira/browse/SPARK-23402
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, SQL
>Affects Versions: 2.2.1
> Environment: PostgreSQL: 9.5.8 (10 + Also same issue)
> OS: Cent OS 7 & Windows 7,8
> JDBC: 9.4-1201-jdbc41
>
> Spark: I executed in both 2.1.0 and 2.2.1
> Mode: Standalone
> OS: Windows 7
>Reporter: Pallapothu Jyothi Swaroop
>Priority: Major
> Attachments: Emsku[1].jpg
>
>
> I am using spark dataset write to insert data on postgresql existing table.
> For this I am using write method mode as append mode. While using i am
> getting exception like table already exists. But, I gave option as append
> mode.
> It's strange. When i change options to sqlserver/oracle append mode is
> working as expected.
>
> *Database Properties:*
> {{destinationProps.put("driver", "org.postgresql.Driver");
> destinationProps.put("url", "jdbc:postgresql://127.0.0.1:30001/dbmig");
> destinationProps.put("user", "dbmig");}}
> {{destinationProps.put("password", "dbmig");}}
>
> *Dataset Write Code:*
> {{valueAnalysisDataset.write().mode(SaveMode.Append).jdbc(destinationDbMap.get("url"),
> "dqvalue", destinationdbProperties);}}
>
>
> {{Exception in thread "main" org.postgresql.util.PSQLException: ERROR:
> relation "dqvalue" already exists at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2412)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2125)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:297)
> at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:428) at
> org.postgresql.jdbc.PgStatement.execute(PgStatement.java:354) at
> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:301) at
> org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:287) at
> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:264) at
> org.postgresql.jdbc.PgStatement.executeUpdate(PgStatement.java:244) at
> org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.createTable(JdbcUtils.scala:806)
> at
> org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:95)
> at
> org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:469)
> at
> org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:50)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:138)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
> at
> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:135) at
> org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:116) at
> org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:92)
> at
> org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:92)
> at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:609)
> at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:233) at
> org.apache.spark.sql.DataFrameWriter.jdbc(DataFrameWriter.scala:460) at
> com.ads.dqam.action.impl.PostgresValueAnalysis.persistValueAnalysis(PostgresValueAnalysis.java:25)
> at
> com.ads.dqam.action.AbstractValueAnalysis.persistAnalysis(AbstractValueAnalysis.java:81)
> at com.ads.dqam.Analysis.doAnalysis(Analysis.java:32) at
> com.ads.dqam.Client.main(Client.java:71)}}
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional
[jira] [Commented] (SPARK-23402) Dataset write method not working as expected for postgresql database
[
https://issues.apache.org/jira/browse/SPARK-23402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363657#comment-16363657
]
Marco Gaido commented on SPARK-23402:
-
I tried with Postgres 10, driver 42.2.1 and I was unable to reproduce. It may
be a windows specific issue or a driver/server incompatibility.
> Dataset write method not working as expected for postgresql database
>
>
> Key: SPARK-23402
> URL: https://issues.apache.org/jira/browse/SPARK-23402
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, SQL
>Affects Versions: 2.2.1
> Environment: PostgreSQL: 9.5.8 (10 + Also same issue)
> OS: Cent OS 7 & Windows 7,8
> JDBC: 9.4-1201-jdbc41
>
> Spark: I executed in both 2.1.0 and 2.2.1
> Mode: Standalone
> OS: Windows 7
>Reporter: Pallapothu Jyothi Swaroop
>Priority: Major
> Attachments: Emsku[1].jpg
>
>
> I am using spark dataset write to insert data on postgresql existing table.
> For this I am using write method mode as append mode. While using i am
> getting exception like table already exists. But, I gave option as append
> mode.
> It's strange. When i change options to sqlserver/oracle append mode is
> working as expected.
>
> *Database Properties:*
> {{destinationProps.put("driver", "org.postgresql.Driver");
> destinationProps.put("url", "jdbc:postgresql://127.0.0.1:30001/dbmig");
> destinationProps.put("user", "dbmig");}}
> {{destinationProps.put("password", "dbmig");}}
>
> *Dataset Write Code:*
> {{valueAnalysisDataset.write().mode(SaveMode.Append).jdbc(destinationDbMap.get("url"),
> "dqvalue", destinationdbProperties);}}
>
>
> {{Exception in thread "main" org.postgresql.util.PSQLException: ERROR:
> relation "dqvalue" already exists at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2412)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2125)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:297)
> at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:428) at
> org.postgresql.jdbc.PgStatement.execute(PgStatement.java:354) at
> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:301) at
> org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:287) at
> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:264) at
> org.postgresql.jdbc.PgStatement.executeUpdate(PgStatement.java:244) at
> org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.createTable(JdbcUtils.scala:806)
> at
> org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:95)
> at
> org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:469)
> at
> org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:50)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:138)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
> at
> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:135) at
> org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:116) at
> org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:92)
> at
> org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:92)
> at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:609)
> at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:233) at
> org.apache.spark.sql.DataFrameWriter.jdbc(DataFrameWriter.scala:460) at
> com.ads.dqam.action.impl.PostgresValueAnalysis.persistValueAnalysis(PostgresValueAnalysis.java:25)
> at
> com.ads.dqam.action.AbstractValueAnalysis.persistAnalysis(AbstractValueAnalysis.java:81)
> at com.ads.dqam.Analysis.doAnalysis(Analysis.java:32) at
> com.ads.dqam.Client.main(Client.java:71)}}
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23402) Dataset write method not working as expected for postgresql database
[
https://issues.apache.org/jira/browse/SPARK-23402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363584#comment-16363584
]
kevin yu commented on SPARK-23402:
--
Thanks, I will install the 9.5.8, and try again.
Sent from my iPhone
> Dataset write method not working as expected for postgresql database
>
>
> Key: SPARK-23402
> URL: https://issues.apache.org/jira/browse/SPARK-23402
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, SQL
>Affects Versions: 2.2.1
> Environment: PostgreSQL: 9.5.8 (10 + Also same issue)
> OS: Cent OS 7 & Windows 7,8
> JDBC: 9.4-1201-jdbc41
>
> Spark: I executed in both 2.1.0 and 2.2.1
> Mode: Standalone
> OS: Windows 7
>Reporter: Pallapothu Jyothi Swaroop
>Priority: Major
> Attachments: Emsku[1].jpg
>
>
> I am using spark dataset write to insert data on postgresql existing table.
> For this I am using write method mode as append mode. While using i am
> getting exception like table already exists. But, I gave option as append
> mode.
> It's strange. When i change options to sqlserver/oracle append mode is
> working as expected.
>
> *Database Properties:*
> {{destinationProps.put("driver", "org.postgresql.Driver");
> destinationProps.put("url", "jdbc:postgresql://127.0.0.1:30001/dbmig");
> destinationProps.put("user", "dbmig");}}
> {{destinationProps.put("password", "dbmig");}}
>
> *Dataset Write Code:*
> {{valueAnalysisDataset.write().mode(SaveMode.Append).jdbc(destinationDbMap.get("url"),
> "dqvalue", destinationdbProperties);}}
>
>
> {{Exception in thread "main" org.postgresql.util.PSQLException: ERROR:
> relation "dqvalue" already exists at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2412)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2125)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:297)
> at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:428) at
> org.postgresql.jdbc.PgStatement.execute(PgStatement.java:354) at
> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:301) at
> org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:287) at
> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:264) at
> org.postgresql.jdbc.PgStatement.executeUpdate(PgStatement.java:244) at
> org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.createTable(JdbcUtils.scala:806)
> at
> org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:95)
> at
> org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:469)
> at
> org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:50)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:138)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
> at
> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:135) at
> org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:116) at
> org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:92)
> at
> org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:92)
> at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:609)
> at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:233) at
> org.apache.spark.sql.DataFrameWriter.jdbc(DataFrameWriter.scala:460) at
> com.ads.dqam.action.impl.PostgresValueAnalysis.persistValueAnalysis(PostgresValueAnalysis.java:25)
> at
> com.ads.dqam.action.AbstractValueAnalysis.persistAnalysis(AbstractValueAnalysis.java:81)
> at com.ads.dqam.Analysis.doAnalysis(Analysis.java:32) at
> com.ads.dqam.Client.main(Client.java:71)}}
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23402) Dataset write method not working as expected for postgresql database
[
https://issues.apache.org/jira/browse/SPARK-23402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363555#comment-16363555
]
Pallapothu Jyothi Swaroop commented on SPARK-23402:
---
Thanks for checking again. I tested with 9.5.4 Append mode is working with out
exception.
I analyzed some thing that may use full for you.
Can you check above scala file.
https://github.com/apache/spark/blob/v2.2.1/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcRelationProvider.scala
Below is the statement for checking table exists or not. In this statement it
is failing.
val tableExists = JdbcUtils.tableExists(conn, options)
But i am not sure. Why it is failing. I executed sql for table exists command
taken from the postgres dialect. It is executed successfully in database.
> Dataset write method not working as expected for postgresql database
>
>
> Key: SPARK-23402
> URL: https://issues.apache.org/jira/browse/SPARK-23402
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, SQL
>Affects Versions: 2.2.1
> Environment: PostgreSQL: 9.5.8 (10 + Also same issue)
> OS: Cent OS 7 & Windows 7,8
> JDBC: 9.4-1201-jdbc41
>
> Spark: I executed in both 2.1.0 and 2.2.1
> Mode: Standalone
> OS: Windows 7
>Reporter: Pallapothu Jyothi Swaroop
>Priority: Major
> Attachments: Emsku[1].jpg
>
>
> I am using spark dataset write to insert data on postgresql existing table.
> For this I am using write method mode as append mode. While using i am
> getting exception like table already exists. But, I gave option as append
> mode.
> It's strange. When i change options to sqlserver/oracle append mode is
> working as expected.
>
> *Database Properties:*
> {{destinationProps.put("driver", "org.postgresql.Driver");
> destinationProps.put("url", "jdbc:postgresql://127.0.0.1:30001/dbmig");
> destinationProps.put("user", "dbmig");}}
> {{destinationProps.put("password", "dbmig");}}
>
> *Dataset Write Code:*
> {{valueAnalysisDataset.write().mode(SaveMode.Append).jdbc(destinationDbMap.get("url"),
> "dqvalue", destinationdbProperties);}}
>
>
> {{Exception in thread "main" org.postgresql.util.PSQLException: ERROR:
> relation "dqvalue" already exists at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2412)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2125)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:297)
> at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:428) at
> org.postgresql.jdbc.PgStatement.execute(PgStatement.java:354) at
> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:301) at
> org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:287) at
> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:264) at
> org.postgresql.jdbc.PgStatement.executeUpdate(PgStatement.java:244) at
> org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.createTable(JdbcUtils.scala:806)
> at
> org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:95)
> at
> org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:469)
> at
> org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:50)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:138)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
> at
> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:135) at
> org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:116) at
> org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:92)
> at
> org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:92)
> at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:609)
> at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:233) at
> org.apache.spark.sql.DataFrameWriter.jdbc(DataFrameWriter.scala:460) at
> com.ads.dqam.action.impl.PostgresValueAnalysis.persistValueAnalysis(PostgresValueAnalysis.java:25)
> at
>
[jira] [Commented] (SPARK-23402) Dataset write method not working as expected for postgresql database
[
https://issues.apache.org/jira/browse/SPARK-23402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363539#comment-16363539
]
kevin yu commented on SPARK-23402:
--
Yes, I create empty table (emptytable) in database (mydb) in the postgresql,
then I run the above statement from spark-shell, it works fine. The only
difference I see is that my postgres is at 9.5.6, yours is at 9.5.8 +.
> Dataset write method not working as expected for postgresql database
>
>
> Key: SPARK-23402
> URL: https://issues.apache.org/jira/browse/SPARK-23402
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, SQL
>Affects Versions: 2.2.1
> Environment: PostgreSQL: 9.5.8 (10 + Also same issue)
> OS: Cent OS 7 & Windows 7,8
> JDBC: 9.4-1201-jdbc41
>
> Spark: I executed in both 2.1.0 and 2.2.1
> Mode: Standalone
> OS: Windows 7
>Reporter: Pallapothu Jyothi Swaroop
>Priority: Major
> Attachments: Emsku[1].jpg
>
>
> I am using spark dataset write to insert data on postgresql existing table.
> For this I am using write method mode as append mode. While using i am
> getting exception like table already exists. But, I gave option as append
> mode.
> It's strange. When i change options to sqlserver/oracle append mode is
> working as expected.
>
> *Database Properties:*
> {{destinationProps.put("driver", "org.postgresql.Driver");
> destinationProps.put("url", "jdbc:postgresql://127.0.0.1:30001/dbmig");
> destinationProps.put("user", "dbmig");}}
> {{destinationProps.put("password", "dbmig");}}
>
> *Dataset Write Code:*
> {{valueAnalysisDataset.write().mode(SaveMode.Append).jdbc(destinationDbMap.get("url"),
> "dqvalue", destinationdbProperties);}}
>
>
> {{Exception in thread "main" org.postgresql.util.PSQLException: ERROR:
> relation "dqvalue" already exists at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2412)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2125)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:297)
> at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:428) at
> org.postgresql.jdbc.PgStatement.execute(PgStatement.java:354) at
> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:301) at
> org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:287) at
> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:264) at
> org.postgresql.jdbc.PgStatement.executeUpdate(PgStatement.java:244) at
> org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.createTable(JdbcUtils.scala:806)
> at
> org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:95)
> at
> org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:469)
> at
> org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:50)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:138)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
> at
> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:135) at
> org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:116) at
> org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:92)
> at
> org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:92)
> at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:609)
> at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:233) at
> org.apache.spark.sql.DataFrameWriter.jdbc(DataFrameWriter.scala:460) at
> com.ads.dqam.action.impl.PostgresValueAnalysis.persistValueAnalysis(PostgresValueAnalysis.java:25)
> at
> com.ads.dqam.action.AbstractValueAnalysis.persistAnalysis(AbstractValueAnalysis.java:81)
> at com.ads.dqam.Analysis.doAnalysis(Analysis.java:32) at
> com.ads.dqam.Client.main(Client.java:71)}}
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail:
[jira] [Commented] (SPARK-23402) Dataset write method not working as expected for postgresql database
[
https://issues.apache.org/jira/browse/SPARK-23402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363512#comment-16363512
]
Pallapothu Jyothi Swaroop commented on SPARK-23402:
---
[~kevinyu98] Did you create table before execute above instructions? It will
throw exception only when table already exists in database. Please run above
statements again you will get exception and let me know issue replicated or not.
> Dataset write method not working as expected for postgresql database
>
>
> Key: SPARK-23402
> URL: https://issues.apache.org/jira/browse/SPARK-23402
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, SQL
>Affects Versions: 2.2.1
> Environment: PostgreSQL: 9.5.8 (10 + Also same issue)
> OS: Cent OS 7 & Windows 7,8
> JDBC: 9.4-1201-jdbc41
>
> Spark: I executed in both 2.1.0 and 2.2.1
> Mode: Standalone
> OS: Windows 7
>Reporter: Pallapothu Jyothi Swaroop
>Priority: Major
> Attachments: Emsku[1].jpg
>
>
> I am using spark dataset write to insert data on postgresql existing table.
> For this I am using write method mode as append mode. While using i am
> getting exception like table already exists. But, I gave option as append
> mode.
> It's strange. When i change options to sqlserver/oracle append mode is
> working as expected.
>
> *Database Properties:*
> {{destinationProps.put("driver", "org.postgresql.Driver");
> destinationProps.put("url", "jdbc:postgresql://127.0.0.1:30001/dbmig");
> destinationProps.put("user", "dbmig");}}
> {{destinationProps.put("password", "dbmig");}}
>
> *Dataset Write Code:*
> {{valueAnalysisDataset.write().mode(SaveMode.Append).jdbc(destinationDbMap.get("url"),
> "dqvalue", destinationdbProperties);}}
>
>
> {{Exception in thread "main" org.postgresql.util.PSQLException: ERROR:
> relation "dqvalue" already exists at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2412)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2125)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:297)
> at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:428) at
> org.postgresql.jdbc.PgStatement.execute(PgStatement.java:354) at
> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:301) at
> org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:287) at
> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:264) at
> org.postgresql.jdbc.PgStatement.executeUpdate(PgStatement.java:244) at
> org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.createTable(JdbcUtils.scala:806)
> at
> org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:95)
> at
> org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:469)
> at
> org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:50)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:138)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
> at
> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:135) at
> org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:116) at
> org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:92)
> at
> org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:92)
> at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:609)
> at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:233) at
> org.apache.spark.sql.DataFrameWriter.jdbc(DataFrameWriter.scala:460) at
> com.ads.dqam.action.impl.PostgresValueAnalysis.persistValueAnalysis(PostgresValueAnalysis.java:25)
> at
> com.ads.dqam.action.AbstractValueAnalysis.persistAnalysis(AbstractValueAnalysis.java:81)
> at com.ads.dqam.Analysis.doAnalysis(Analysis.java:32) at
> com.ads.dqam.Client.main(Client.java:71)}}
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail:
[jira] [Commented] (SPARK-23402) Dataset write method not working as expected for postgresql database
[
https://issues.apache.org/jira/browse/SPARK-23402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16363110#comment-16363110
]
kevin yu commented on SPARK-23402:
--
I just tried with
PostgreSQL 9.5.6 on x86_64-pc-linux-gnu
with jdbc driver:
postgresql-9.4.1210
I can't reproduce the error.
scala> val df1 = Seq((1)).toDF("c1")
df1: org.apache.spark.sql.DataFrame = [c1: int]
scala>
df1.write.mode(SaveMode.Append).jdbc("jdbc:postgresql://9.30.167.220:5432/mydb",
"emptytable", destProperties)
scala> val df3 = spark.read.jdbc("jdbc:postgresql://9.30.167.220:5432/mydb",
"emptytable", destProperties).show
+---+
| c1|
+---+
| 1|
+---+
df3: Unit = ()
> Dataset write method not working as expected for postgresql database
>
>
> Key: SPARK-23402
> URL: https://issues.apache.org/jira/browse/SPARK-23402
> Project: Spark
> Issue Type: Bug
> Components: Spark Core, SQL
>Affects Versions: 2.2.1
> Environment: PostgreSQL: 9.5.8 (10 + Also same issue)
> OS: Cent OS 7 & Windows 7,8
> JDBC: 9.4-1201-jdbc41
>
> Spark: I executed in both 2.1.0 and 2.2.1
> Mode: Standalone
> OS: Windows 7
>Reporter: Pallapothu Jyothi Swaroop
>Priority: Major
> Attachments: Emsku[1].jpg
>
>
> I am using spark dataset write to insert data on postgresql existing table.
> For this I am using write method mode as append mode. While using i am
> getting exception like table already exists. But, I gave option as append
> mode.
> It's strange. When i change options to sqlserver/oracle append mode is
> working as expected.
>
> *Database Properties:*
> {{destinationProps.put("driver", "org.postgresql.Driver");
> destinationProps.put("url", "jdbc:postgresql://127.0.0.1:30001/dbmig");
> destinationProps.put("user", "dbmig");}}
> {{destinationProps.put("password", "dbmig");}}
>
> *Dataset Write Code:*
> {{valueAnalysisDataset.write().mode(SaveMode.Append).jdbc(destinationDbMap.get("url"),
> "dqvalue", destinationdbProperties);}}
>
>
> {{Exception in thread "main" org.postgresql.util.PSQLException: ERROR:
> relation "dqvalue" already exists at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2412)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2125)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:297)
> at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:428) at
> org.postgresql.jdbc.PgStatement.execute(PgStatement.java:354) at
> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:301) at
> org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:287) at
> org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:264) at
> org.postgresql.jdbc.PgStatement.executeUpdate(PgStatement.java:244) at
> org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.createTable(JdbcUtils.scala:806)
> at
> org.apache.spark.sql.execution.datasources.jdbc.JdbcRelationProvider.createRelation(JdbcRelationProvider.scala:95)
> at
> org.apache.spark.sql.execution.datasources.DataSource.write(DataSource.scala:469)
> at
> org.apache.spark.sql.execution.datasources.SaveIntoDataSourceCommand.run(SaveIntoDataSourceCommand.scala:50)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56)
> at
> org.apache.spark.sql.execution.command.ExecutedCommandExec.doExecute(commands.scala:74)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:117)
> at
> org.apache.spark.sql.execution.SparkPlan$$anonfun$executeQuery$1.apply(SparkPlan.scala:138)
> at
> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
> at
> org.apache.spark.sql.execution.SparkPlan.executeQuery(SparkPlan.scala:135) at
> org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:116) at
> org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:92)
> at
> org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:92)
> at org.apache.spark.sql.DataFrameWriter.runCommand(DataFrameWriter.scala:609)
> at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:233) at
> org.apache.spark.sql.DataFrameWriter.jdbc(DataFrameWriter.scala:460) at
> com.ads.dqam.action.impl.PostgresValueAnalysis.persistValueAnalysis(PostgresValueAnalysis.java:25)
>
