[jira] [Commented] (SPARK-25789) Support for Dataset of Avro

[Dheeren Beborrtha (Jira)]

[ 
https://issues.apache.org/jira/browse/SPARK-25789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17817184#comment-17817184
 ] 

Dheeren Beborrtha commented on SPARK-25789:
---

Any chance that the PR can be merged ?

> Support for Dataset of Avro
> ---
>
> Key: SPARK-25789
> URL: https://issues.apache.org/jira/browse/SPARK-25789
> Project: Spark
>  Issue Type: New Feature
>  Components: SQL
>Affects Versions: 3.0.0
>Reporter: Aleksander Eskilson
>Priority: Major
>
> Support for Dataset of Avro records in an API that would allow the user to 
> provide a class to an {{Encoder}} for Avro, analogous to the {{Bean}} 
> encoder. This functionality was previously to be provided by SPARK-22739 and 
> [Spark-Avro #169|https://github.com/databricks/spark-avro/issues/169]. Avro 
> functionality was folded into Spark-proper by SPARK-24768, eliminating the 
> need to maintain a separate library for Avro in Spark. Resolution of this 
> issue would:
>  * Add necessary {{Expression}} elements to Spark
>  * Add an {{AvroEncoder}} for Datasets of Avro records to Spark



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-21740) DataFrame.write does not work with Phoenix JDBC Driver

[Dheeren Beborrtha (JIRA)]

[ 
https://issues.apache.org/jira/browse/SPARK-21740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16375134#comment-16375134
 ] 

Dheeren Beborrtha commented on SPARK-21740:
---

What workaround did you use? DId you modify Spark-core or modify phoenix jdbc 
client?

> DataFrame.write does not work with Phoenix JDBC Driver
> --
>
> Key: SPARK-21740
> URL: https://issues.apache.org/jira/browse/SPARK-21740
> Project: Spark
>  Issue Type: Bug
>  Components: Spark Core
>Affects Versions: 2.0.0, 2.2.0
>Reporter: Paul Wu
>Priority: Major
>  Labels: jdbc, phoenix
>
>   The reason for this is that Phoenix JDBC driver does not support "INSERT", 
> but "UPSERT".
> Exception for the following program:
> 17/08/15 12:18:53 ERROR Executor: Exception in task 0.0 in stage 1.0 (TID 1)
> org.apache.phoenix.exception.PhoenixParserException: ERROR 601 (42P00): 
> Syntax error. Encountered "INSERT" at line 1, column 1.
>   at 
> org.apache.phoenix.exception.PhoenixParserException.newException(PhoenixParserException.java:33)
> {code:java}
> public class HbaseJDBCSpark {
> private static final SparkSession sparkSession
> = SparkSession.builder()
> .config("spark.sql.warehouse.dir", "file:///temp")
> .config("spark.driver.memory", "5g")
> .master("local[*]").appName("Spark2JdbcDs").getOrCreate();
> static final String JDBC_URL
> = "jdbc:phoenix:somehost:2181:/hbase-unsecure";
> public static void main(String[] args) {
> final Properties connectionProperties = new Properties();
> Dataset jdbcDF
> = sparkSession.read()
> .jdbc(JDBC_URL, "javatest", connectionProperties);
> jdbcDF.show();
> String url = JDBC_URL;
> Properties p = new Properties();
> p.put("driver", "org.apache.phoenix.jdbc.PhoenixDriver");
> //p.put("batchsize", "10");
> jdbcDF.write().mode(SaveMode.Append).jdbc(url, "javatest", p);
> sparkSession.close();
> }
> // Create variables
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org